Skip to main content

Ingest SDK

What is the Ingest SDK?​

The SDK is composed of several published Golang packages under github.com/stellar/go for acquiring and parsing data from the Stellar network. It provides language level bindings which convert the binary XDR encoded streams emitted from the network into fluent programmatic data model bindings.

Why use the Ingest SDK?​

Applications can leverage the SDK to rapidly develop ingestion pipelines capable of acquiring real-time or historical Stellar network data and deriving custom data models. The SDK enables applications to traverse the hierarchal data structures of the network: history archives, ledgers, transactions, operations, ledger state changes, and events.

Use the SDK for an intuitive, compile-time, type-safe developer experience to work with the main types of network data:

Ledger Entries​

Obtain the final state of ledger entries on the network at close of any recent or historically aged checkpoint ledger sequence. A Checkpoint ledger occurs once every 64 ledgers, during which the network will publish this data to history archives in the format of compressed files which contain lists of BucketEntry, wherein each contains one LedgerEntry and the LedgerKey.

Ledger entries are cryptographically signed as part of each ledger and therefore represent the trusted, cumulative state at a point in time for assets related to an account or contract. Examples of asset types:

  • trustlines which hold token balances
  • offers which hold bid and asks on the stellar network DEX
  • contract data which holds key/value stores for contracts

Ledger Metadata​

Gain access to all transactions and their nested operations which were included in each closed ledger. It also contains the changes(pre/post) for each ledger entry when it is changed due to transaction activity and all Soroban contract events emitted as a result of contract invocations during transaction execution. It is effectively a commit log for ledger entries.

Use the metadata to detect incremental changes in ledger entries that occur as a result of each transaction. These changes and the events emitted after each transaction represent valuabe data models specific to a point in time(expressed as a ledger sequence on the network), some real world examples:

  • a payment in the form of a token transfer between accounts
  • an offer is placed to trade one token for another token at a given price on the dex
  • token transfers between accounts and contracts

The SDK provides packages which clients can use to acquire and parse the metadata in streaming data format, with readers and callback functions.

The streams of ledger metadata can be sourced in unbounded fashion as ongoing real-time ledgers close on the Stellar network, and also in a historical replay mode with a bounded range of past ledger sequences.