Sui gRPC streaming powers low-latency checkpoint indexing and resilient data pipelines

Developers on Sui can now tap into sui grpc streaming to build faster, more reliable indexing pipelines for real-time blockchain data.

Hybrid streaming model transforms Sui data access

The Sui blockchain has introduced gRPC streaming as a primary data source for its indexing infrastructure, enabling real-time checkpoint ingestion with minimal latency. Moreover, this design targets applications that must react as soon as data is finalized.

The platform combines streaming capabilities with traditional polling methods to maintain data accuracy and system resilience. This hybrid model allows immediate access to finalized checkpoints while preserving backward compatibility with existing custom indexers already deployed on Sui.

The Custom Indexing Framework underpins this streaming-first architecture without requiring changes to checkpoint processing logic. However, teams can still rely on their existing pipelines while layering in streaming where it brings tangible benefits.

Streaming removes polling delays for checkpoint ingestion

The new gRPC streaming capability fundamentally changes how indexers receive blockchain data on Sui. Full nodes now push checkpoint data directly to indexers as soon as finalization occurs, rather than waiting for scheduled fetches.

This push-based model eliminates repeated polling cycles that previously introduced delays between checkpoint creation and downstream processing. As a result, latency-sensitive tools can react closer to real time without tuning complex polling intervals.

According to the documentation, the system delivers “real-time checkpoints as soon as they’re finalized” with “faster data, resilient pipelines, less infra work on Sui”. That said, operators can still configure safety nets to protect against connection issues and service disruptions.

The streaming mechanism operates through a straightforward configuration step where developers add a streaming-url argument that points to a full node endpoint. The indexer then receives checkpoints as event streams instead of fetching them at predetermined intervals.

This event-driven model is particularly valuable for monitoring systems, real-time analytics platforms, and other latency-sensitive applications. Moreover, it simplifies infrastructure by reducing the need for aggressive polling strategies and related operational tuning.

Mandatory polling fallbacks safeguard historical data

Sui pairs streaming with mandatory polling-based fallback sources to handle inherent limitations of long-lived connections. A streaming link only delivers data starting from the moment it is established, so historical checkpoints still require additional mechanisms.

The General-Purpose Indexer showcases this hybrid design in production. It uses streaming as its primary ingestion path while maintaining polling sources as safety mechanisms for historical data and recovery scenarios.

This configuration keeps indexed data current while enabling clean restarts and seamless recovery from failures. However, if a connection drops, the system can resume from the last known checkpoint using polling, then return to streaming once the link stabilizes.

In practice, this hybrid pattern functions similarly to a sui checkpoint streaming fallback strategy. Developers obtain the low-latency benefits of pushed updates without compromising completeness or reliability.

Framework design enables incremental streaming adoption

The Custom Indexing Framework separates checkpoint processing from data ingestion. Indexers consume and transform checkpoints through a unified interface, without coupling logic to specific sources such as gRPC streams or HTTP polling.

This abstraction allows teams to evolve ingestion strategies as requirements change, without rewriting core processing components. Moreover, it keeps codebases simpler by concentrating data-handling logic in a single layer.

The documentation notes that with gRPC streaming, “there is no need to poll, no guesswork around timing, and no artificial delay introduced by fetch intervals”. That said, operators can still choose polling where workloads do not require ultra-low latency.

Developers can enable sui grpc streaming gradually, based on individual workload characteristics. Applications that prioritize data freshness and real-time responsiveness gain the most from immediate streaming adoption.

By contrast, systems focused on batch analytics, offline processing, or simpler workflows can continue using polling-only configurations. The framework supports both approaches under the same processing model, simplifying multi-application environments.

Configuration keeps logic stable while sources evolve

Existing custom indexers built on the official framework require minimal changes to take advantage of streaming. Adding gRPC capability involves including a streaming-url parameter alongside the existing remote-store-url configuration value.

The checkpoint processing logic remains unchanged throughout this transition. Moreover, the framework automatically manages source switching during operation so that indexers keep a consistent view of the network state.

This design helps prevent common failure modes where systems either lose data or lag significantly behind the chain. The framework coordinates interactions between streaming and polling, maintaining continuity across restarts and network interruptions.

Overall, Sui’s hybrid streaming and polling architecture offers real-time checkpoints, resilient pipelines, and a clear migration path for indexers seeking low-latency ingestion without sacrificing reliability.