Skip to main content

​
Build an Indexer from Scratch

This guide demonstrates how SDK packages can be combined into a working indexer (called a squid). This page goes through all the technical details to make the squid architecture easier to understand. If you want to get to a working indexer ASAP, bootstrap from a template.

​
Prerequisites

Before you begin, ensure you have:

​
Project: Polkadot Balance Transfers API

In this tutorial, you’ll build an indexer that tracks balance transfers on Polkadot, saves the data to PostgreSQL, and serves it as a GraphQL API.

​
Required packages

From the requirements above, you’ll need these packages:
  • @subsquid/substrate-processor - for retrieving Substrate chain data
  • @subsquid/typeorm-store, @subsquid/typeorm-codegen, and @subsquid/typeorm-migration - for saving data to PostgreSQL

​
Optional packages

  • @subsquid/substrate-typegen - for generating type-safe wrappers for Substrate events and calls
  • @subsquid/graphql-server / OpenReader - for serving GraphQL API

​
Build the Indexer

1

Initialize the project

Create a new folder and initialize a Node.js project:
npm init
Create a .gitignore file:
.gitignore
node_modules
lib
2

Install dependencies

Install the required packages:
npm i dotenv typeorm @subsquid/substrate-processor @subsquid/typeorm-store @subsquid/typeorm-migration @subsquid/graphql-server
Install development dependencies:
npm i typescript @subsquid/typeorm-codegen @subsquid/substrate-typegen --save-dev
3

Configure TypeScript

Create a tsconfig.json file with minimal configuration:
tsconfig.json
{
  "compilerOptions": {
    "rootDir": "src",
    "outDir": "lib",
    "module": "commonjs",
    "target": "es2020",
    "esModuleInterop": true,
    "skipLibCheck": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true
  }
}
4

Define the data schema

Create a schema.graphql file to define your data model:
schema.graphql
type Transfer @entity {
  id: ID!
  from: String! @index
  to: String! @index
  amount: BigInt!
  blockNumber: Int!
  timestamp: DateTime!
}
Generate TypeORM classes from the schema:
npx squid-typeorm-codegen
The TypeORM classes are now available at src/model/index.ts.
5

Set up the database

Create environment variables in a .env file:
.env
DB_NAME=squid
DB_PORT=23798
RPC_POLKADOT_HTTP=https://rpc.polkadot.io
Create a docker-compose.yaml file for PostgreSQL:
docker-compose.yaml
services:
  db:
    image: postgres:15
    environment:
      POSTGRES_DB: "${DB_NAME}"
      POSTGRES_PASSWORD: postgres
    ports:
      - "${DB_PORT}:5432"
Start the database:
docker compose up -d
Compile the TypeORM classes:
npx tsc
Generate and apply the database migration:
npx squid-typeorm-migration generate
npx squid-typeorm-migration apply
6

Generate type-safe wrappers

Create a typegen.json file to specify which Substrate types to generate:
typegen.json
{
  "outDir": "src/types",
  "specVersions": "https://v2.archive.subsquid.io/metadata/polkadot",
  "pallets": {
    "Balances": {
      "events": ["Transfer"]
    }
  }
}
Generate TypeScript type-safe wrappers:
npx squid-substrate-typegen typegen.json
The type-safe wrappers are now available in src/types.
7

Create the processor

Create src/main.ts with the following code:
main.ts
import { SubstrateBatchProcessor } from "@subsquid/substrate-processor";
import { TypeormDatabase } from "@subsquid/typeorm-store";
import { events } from "./types";
import { Transfer } from "./model";

const processor = new SubstrateBatchProcessor()
  .setGateway("https://v2.archive.subsquid.io/network/polkadot")
  .setRpcEndpoint({
    url: process.env.RPC_POLKADOT_HTTP,
    rateLimit: 10,
  })
  .setBlockRange({ from: 1000000 })
  .addEvent({
    name: [events.balances.transfer.name],
  })
  .setFields({
    event: {
      args: true,
    },
    block: {
      timestamp: true,
    },
  });

const db = new TypeormDatabase();

processor.run(db, async (ctx) => {
  const transfers: Transfer[] = [];
  
  for (let block of ctx.blocks) {
    for (let event of block.events) {
      if (event.name === events.balances.transfer.name) {
        let decoded = events.balances.transfer.v1050.decode(event);
        
        transfers.push(
          new Transfer({
            id: event.id,
            from: decoded.from,
            to: decoded.to,
            amount: decoded.amount,
            blockNumber: block.header.height,
            timestamp: new Date(block.header.timestamp!),
          })
        );
      }
    }
  }
  
  await ctx.store.insert(transfers);
});
The typegen tool automatically detects which spec versions are needed and generates versioned decoders. The example uses v1050 - your actual version may differ.
8

Run the processor

Compile the project:
npx tsc
Start the processor:
node -r dotenv/config lib/main.js
The processor is now indexing balance transfers from the Polkadot blockchain.
9

Start the GraphQL server

Update your .env file to include the GraphQL server port:
.env
DB_NAME=squid
DB_PORT=23798
RPC_POLKADOT_HTTP=https://rpc.polkadot.io
GRAPHQL_SERVER_PORT=4350
In a separate terminal, start the GraphQL server:
npx squid-graphql-server
The GraphQL API with GraphiQL is available at localhost:4350/graphql.

​
Next Steps

Congratulations! You’ve built a complete Substrate blockchain indexer from scratch.

Substrate Tutorial

Follow a complete Substrate indexing tutorial

Development Flow

Learn about the general development workflow
The commands in this tutorial are often abbreviated as custom sqd commands in production squids. Use the commands.json file from the Substrate template as a starter.