Self-Host a Portal

​

Prerequisites

​

Software Requirements

• Working Docker installation

​

Financial Requirements

• Minimum 10,000 SQD tokens
  • Tokens remain locked while your Portal instance is active
  • More tokens = more bandwidth
  • Tokens can be in your wallet or SQD-issued vesting contract
• Arbitrum ETH for gas fees

​

Hardware Requirements

• Minimum 25GB unused RAM (we’re working to reduce this requirement)
• Additional hardware depends on your use case:
  • Private single-user Portal instances can run on a laptop with sufficient RAM
  • Public or high-throughput Portal instances require more robust infrastructure

​

Portal Setup

docker run --rm subsquid/rpc-node:0.2.5 keygen > <KEY_FILE_PATH>

Your peer ID: THIS IS WHAT YOU NEED TO COPY

git clone https://github.com/subsquid/sqd-portal
cd sqd-portal

cp mainnet.env .env

echo KEY_PATH=<KEY_FILE_PATH> >> .env

docker compose up -d

docker compose logs -f

cargo run --release

apt-get install protobuf-compiler

​

Using your portal

npx --yes npm-check-updates --filter "@subsquid/*" --target "@portal-api" --upgrade

sed -i -e 's/[\^~=<>]*\([0-9\.]*-portal-api\.[0-9a-f]\{6\}\)/\1/g' package.json

rm -r node_modules package-lock.json
npm i

http://<host>:8000/datasets/<dataset-slug>

http://127.0.0.1:8000/datasets/ethereum-mainnet

​

On EVM

+  .setPortal('http://127.0.0.1:8000/datasets/ethereum-mainnet')
-  .setGateway('https://v2.archive.subsquid.io/network/ethereum-mainnet')

​

On Solana

git clone https://github.com/subsquid-labs/solana-example/
cd solana-example
git checkout portal-api
npm i
npm run build
docker compose up -d
npx squid-typeorm-migration apply
node -r dotenv/config lib/main.js

​

Migrating an Existing Squid

+  .setPortal('https://portal.tethys.sqd.dev/datasets/solana-beta')
-  .setGateway('https://v2.archive.subsquid.io/network/solana-mainnet')
-  .setRpc({
-    client: new SolanaRpcClient({
-      url: process.env.SOLANA_NODE
-    })
-  })

-import \{DataSourceBuilder, SolanaRpcClient\} from '@subsquid/solana-stream'
+import \{DataSourceBuilder\} from '@subsquid/solana-stream'

+  .setBlockRange(\{from: 325000000\})
-  .setBlockRange(\{from: 303262650\})

SLOT_NUMBER=320276063 curl --request POST --url https: "//api.mainnet-beta.solana.com --header 'accept: application/json' --header 'content-type: application/json' --data "\{\"id\": 1, \"jsonrpc\": \"2.0\", \"method\": \"getBlock\", \"params\": [$\{SLOT_NUMBER\}, \{\"encoding\": \"jsonParsed\", \"maxSupportedTransactionVersion\": 0\}]\}" | jq | grep blockHeight"

-  slot: block.header.slot,
+  slot: block.header.number,

  .setFields({
    block: { // block header fields
      timestamp: true,
+      height: true
    },

​

Token Requirements and Compute Units

10^-5 _ range_length_blocks _ transactions_per_block

• This assumes lightweight queries - that is, queries that fetch much less data than the total dataset size. For heavyweight queries multiply the estimate by a factor of 2-5.
• This is a rough estimate. Multiply it by ten to get a somewhat safe figure. If you want to minimize your SQD lockup, start at that safe figure, then measure the actual amount of CU you spend and reduce the lockup accordingly.
• If your network has an Etherscan-style explorer, you can estimate the transactions_per_block by visiting its front page, reading the “Transactions” stat and dividing it by the “Last finalized block” height.

num*chunks_in_range = range_length * (eth*chunks / eth_height) * (chain_txs_per_block / eth_txs_per_block)

X / (Y \* boost_factor)

​

Example

• Visiting bscscan.com we find that there’s a total of 6.4B txs made over 44M blocks. We can roughly estimate the chain density at 150 txs/block.
• We can now estimate the cost of each query to be on the order of 10^-5 * 10^6 * 150 = 1500 CU.
• When ran once every 60 minutes or 3 epochs at 20 min per epoch, such queries would require roughly 1500 / 3 = 500 SQD locked up. This assumes a short lockup (under 60 days).
• Finally we apply the recommended margins:
  • For lightweight queries it is 10x and the final sum amounts to 5000 SQD - less than the global minimum lockup of 10000.
  • If you’re running extremely heavy queries, the recommended margin is 50x and the final recommended lockup is 25000 SQD. Use this value as a starting point and iterate if you’d like to minimize your lockup.
    • If you’d like to reduce the amount of SQD even further, one option is to lock the tokens up for longer: with a two-year long lockup heavy queries you’ll get three times the CUs and the final recommended lockup will be 10000 even for heavy queries.

​

High throughput portals

1. Use a single wallet to register as many peer IDs as you need Portal instances.
2. Make one SQD lockup for all your Portal instances. See Token requirements and compute units to get an idea of how large your stake should be.
3. Run Portal instances in parallel, balancing traffic between the instances.

​

Troubleshooting

curl --compressed http://127.0.0.1:8000/metrics | grep throttled