Skip to main content

​
Where to keep state of stateful transforms

The choice of state storage depends on your data volume, latency requirements, and computational resources.
  • Keep state in RAM if it can be cheaply reconstructed from limited historical data (example - 1-minute candles)
This approach offers the lowest latency.
  • Keep state in ClickHouse and use materialized views if your transforms meet these criteria:
Do not form long dependency chains in latency-critical apps: each materialized view that depends on another materialized view increases the final latency.
For very frequent changes on lightweight data, prefer plain (non-materialized) views. You must have computational resources to spare on the database machine to take this approach.
  • If your destination requires pre-transformed data, consider using stateful transformers:

​
More

TBA