v1.14.0

This release brings performance improvements to the substreams engine, through the introduction of a new "QuickSave" feature, and a switch to wasmtime as the default runtime for Rust modules.

Server

• Implement "QuickSave" feature to save the state of "live running" substreams stores when shutting down, and then resume processing from that point if the cursor matches.

  • enabled if the "QuickSaveStoreURL" attribute is not empty in the tier1 config
  • requires the "CheckPendingShutdown" module to be passed to the app via NewTier1()

• Rust modules will now be executed with wasmtime by default instead of wazero.

  • Prevents the whole server from stalling in certain memory-intensive operations in wazero.
  • Speed improvement: cuts the execution time in half in some circumstances.
  • Wazero is still used for modules with wbindgen and modules compiled with tinygo.
  • Set env var SUBSTREAMS_WASM_RUNTIME=wazero to revert to previous behavior.

CLI

• Fixed --skip-package-validation to also skip sub packages being imported.
• Trim down packages when using 'imports': only the modules explicitly defined in the YAML manifest and their dependencies will end up in the final spkg.

v1.13.0

Server

Request Pool and Worker Pool

• Added GlobalRequestPool to the Tier1Modules struct in app/tier1.go and integrated it into the Run method to enhance request lifecycle management.
  When set, the GlobalRequestPool will manage the borrowing, quotas, and keep-alive mechanisms for user requests via requests to a GRPC remote server.

• Added WorkerPoolFactory to the Tier1Modules struct in app/tier1.go and integrated it into the Run method to enhance worker lifecycle management.
  When set, the WorkerPool will manage the borrowing, quotas, and keep-alive mechanisms for worker subrequests on tier2, via requests to a GRPC remote server.

Performance

• Added 'shared cache' on tier1: execution of modules near the HEAD of the chain will be done once for a given module hash and the result shared between requests.
  This will reduce CPU usage and increase performance when many requests are using the same modules (ex: foundational modules)

• Improved "time to first block" when a lot of cached files exist on dependency substreams modules by skipping reads segments that won't be used and assuming stores "full KVs" are always filled sequentially (since they are!)

• Limit parallel execution of a stage's layer.
  Previously, the engine was executing modules in a stage's layer all in parallel. We now change that behavior, development mode will from now on execute every sequentially and when in production mode will limit parallelism to 2 (hard-coded) for now.
  The auth plugin can control that value dynamically by providing a trusted header X-Sf-Substreams-Stage-Layer-Parallel-Executor-Max-Count.

• Fixed a regression since "v1.12.2" where the SkipEmptyOutput instruction was ignored in substreams mappers

CLI

• Removed enforcement of BUFBUILD_AUTH_TOKEN environment variable when using descriptor sets. It appears there is now a public free tier to query those which should work in most cases.
• When running Solana package, set base58 encoding by default in the GUI.
• Add Sei Mainnet to the ChainConfigByID map.

v1.12.4

Server

• Fix log regression on 'substreams request stats' (bad value for production_mode/tier)

v1.12.3

Server

• Improve noop-mode: will now only send one signal per bundle, without any data
• Improve logging

Client

• Add --noop-mode flag to substreams run as a simple way to force the server to generate caches in production-mode.

v1.12.2

• Add Stellar Mainnet and Testnet to the HardcodedEndpoints map.
• Fix a panic when a substreams was using an index as an input which contained empty output

v1.12.1

• Fixed tier2 app not setting itself as ready on startup

• Added extra ad-hoc prometheus labels 'tools prometheus-explorer' as query params to each endpoint.

v1.12.0

Server-side

• Fix a thread leak in cursor resolution resulting in a bad value for active_connections metric
• Fix detection of accepted gzip compression when multiple values are sent in the Grpc-Accept-Encoding header (ex: Python library)
• Properly accept and compress responses with gzip for browser HTTP clients using ConnectWeb with Accept-Encoding header
• Allow setting subscription channel max capacity via SOURCE_CHAN_SIZE env var (default: 100)
• Added tier1 app configuration option to limit max active requests a single instance can accept before starting to reject them with 'Unavailable' gRPC code.
• Added tier1 & tier2 app new Prometheus metric substreams_{tier1,tier2}_rejected_request_counter, to track rejected request, especially when hard limit is reached.

Client-side

• improvements to 'tools prometheus-explorer'

  • change flags lookup_interval and lookup_timeout to --interval and --timeout
  • now support relative block (default is now: -1) and does not use 'final-blocks-only' flag on request
  • add --max-freshness flag to check for block age (when using relative block)
  • add substreams_healthcheck_block_age_ms prometheus metric
  • --block-height is now a flag instead of a positional argument
  • improve logging
  • removed "3 retries" that were built in and causing more confusion

• add User-Agent headers depending on the client command

v1.11.3

Server-side

• Fixed: detection of gzip compression on 'connect' protocol (js/ts clients)
• Added: tier1.Config EnforceCompression to refuse incoming connections that do not support GZIP compression (default: false)

v1.11.2

Server-side

• Fix too many memory allocations impacting performance when stores are used

CLI

• Force topological ordering of protobuf descriptors when 'packing' an spkg (affecting current substreams-js clients)
• Allow substreams pack to be able to do a "re-packing" of an existing spkg file. Useful to apply the protobuf descriptor ordering fix.

Docker image

• Rebuilt of v1.11.1 to generate Docker latest tag with revamp Docker image building.
• Substreams CLI is now built with using Ubuntu 22, previous releases were built using Ubuntu 20.
• Substreams Docker image is now using ubuntu:22 as its base, previous releases were built using ubuntu:20.04.

v1.11.1

• Fix the gui breaking when the network field is not set in the spkg
• Fixed SUBSTREAMS_REGISTRY_TOKEN environment variable not taking precedence over the registry-token file.