Skip to main content

Documentation Index

Fetch the complete documentation index at: https://beta.docs.sqd.dev/llms.txt

Use this file to discover all available pages before exploring further.

Pipes SDK ships a built-in per-batch profiler. It records how long each part of the pipeline takes. When a metricsServer() is attached to the source, the profiler output is served as JSON at http://localhost:<metrics_port>/profiler and rendered live in Pipes UI.

​
Enabling the profiler

The profiler is on by default when process.env.NODE_ENV !== 'production' and off otherwise. Override explicitly with profiler: true or profiler: false on the source. 
substratePortalStream({
  portal: 'https://portal.sqd.dev/datasets/',
  outputs: /* ... */,
  metrics: metricsServer({ port: 9090 }),
  profiler: true,   // force on — useful in production
})

​
Interpreting the output

The pipeline is represented as a tree. Each node reports how long was spent in that stage of a batch. A typical tree looks like: 
batch
├── fetch data
├── apply transformers
│   ├── track progress
│   └──  decoder
├── clickhouse
│   ├── data handler
│   ├── insert cursor
│   └── cleanup cursors
└── metrics processing

​
Custom spans

Wrap any code in your target or transformer to get it to appear as a tree node. ctx.profiler.start() is a no-op when the profiler is disabled, so the instrumentation is safe to leave in place. 
onData: async ({ data, ctx }) => {
  const span = ctx.profiler.start('my measure')
  await myDataProcessing(data)
  span.end()
},
The named span appears under its parent node in the tree: 
batch
...
├── clickhouse
│   ├── data handler
│   │   └── my measure
...