Overview
This tutorial walks through how an application can leverage common streaming data patterns to ingest Stellar network transaction data using a few select packages from the Stellar Go Repo github.com/stellar/go collectively known as the 'Ingestion' SDK:
The Ingestion SDK packages​
github.com/stellar/go/amount
utility package to convert prices from network transaction operations to stringgithub.com/stellar/go/historyarchive
github.com/stellar/go/support/datastore
github.com/stellar/go/support/storage
utility package with convenient wrappers for accessing history archives, and avoid low-level http aspectsgithub.com/stellar/go/ingest
provides parsing functionality over the network ledger data, converts to more developer-centricLedgerTransaction
modelgithub.com/stellar/go/ingest/ledgerbackend
provides the captive core backend implementationgithub.com/stellar/go/network
provides convenient pre-configured settings for Testnet and Mainnet networksgithub.com/stellar/go/xdr
a complete Golang binding to the Stellar network data model
Ingestion project setup​
Project requirements​
To build an example streaming network ingestion pipeline from live Stellar network transaction data, you'll need:
- A developer workstation with Go programming language runtime installed
- An IDE to edit Go code, VSCode is good if one is needed
- A newly initialized, empty Go project folder.
mkdir pipeline; cd pipeline; go mod init example/pipeline
stellar-core
must be installed on your workstation and available on your o/s PATH
The Stellar network data model is defined in an IDL format expressed in XDR encoding. Our example application is only interested in a small subset of the overall transaction data model related to buying and selling of assets, i.e. a payment, and defines its own data model internally:
- Example
::AppPayment
Timestamp: uint
BuyerAccountId: string
SellerAccountId: string
AssetCode: string
Amount: string
}
The example application will run a network ingestion pipeline to derive a smaller ApplicationPayment
model from the Stellar network transaction data model as 'source of origin' and thus enable the application to avoid large compute resources that would have been required for maintaining storage of the full Stellar network data model.
The ingestion pipeline will perform three distinct stream processor roles:
Inbound Adapter​
Acts as the 'source of origin' for the pipeline. Retrieves LedgerCloseMeta generated from a Stellar network using captive core. LedgerCloseMeta
is the top-level aggregate in the Stellar data model of which all Stellar network transaction data is nested within. Publishes the LedgerCloseMeta
onto the pipeline.
Transformer​
Subscribes to receive LedgerCloseMeta
from the pipeline. Uses the Go SDK package github.com/stellar/go/xdr to parse the nested network data model for payment operations and convert those into a new instance of application data model ApplicationPayment
instances. Publishes ApplicationPayment
to the pipeline.
Outbound Adapter​
Acts as the termination of the pipeline, it subscribes to receive ApplicationPayment
and publishes the data off the pipeline and to an external data store, a ZeroMQ Publisher Socket, which is essentially a message broker.
Summary​
Refer to Ingestion Pipeline Sample Application for complete code demonstrating usage of the 'ingestion' SDK packages to create these adapters and transformers and run a live pipeline against the Stellar network.