Complete code for a small sample of a consumer pipeline of Stellar network ledger metadata using the Stellar Go [Ingest SDK](https://developers.stellar.org/docs/build/apps/ingest-sdk/overview.md#the-ingestion-sdk-packages) to demonstrate data pipeline from ledger metadata to derived data model with event-driven, distributed processing to sample microservice (Python script) as subscriber.

This example uses the ZeroMQ [goczmq](https://github.com/zeromq/goczmq) Go wrapper SDK, which requires a few o/s [dependent libraries to also be installed on the host machine](https://github.com/zeromq/goczmq?tab=readme-ov-file#dependencies).

This example requires having access to a public ledger metadata lake that is actively populated with latest ledgers from Stellar mainnet. For purposes of the example it uses a reference data lake hosted on [AWS Open Data S3](../../../data/apis/rpc/admin-guide/data-lake-integration#1-accessing-a-data-lake)

### Step# 1 - Create example directory and copy following files to your workstation.



### `main.go`



### `config.toml`

The CDP configuration settings, this file defines the data storage which contains the pre-generated Ledger Metadata files. The [S3 Public Blockchain](https://registry.opendata.aws/aws-public-blockchain) bucket for [Stellar Pubnet](https://aws-public-blockchain.s3.us-east-2.amazonaws.com/index.html#v1.1/stellar/ledgers/pubnet) is used in this example.



### `distributed_payment_subsciber.py`

A Python script demonstrating how we now have distributed processing and event driven architecture by leveraging the MQ Broker to push derived application payment data model out to other microservices. Make sure to `pip install pyzmq`



### Step# 2 - Compile and run the ingestion pipeline example.



### Step# 3 - Run the distributed pipeline consumer

In separate terminal, run `python distributed_payment_subsciber.py`, this will perform distributed pipeline topology, as it receives messages with payment info from the pipeline process and does additional processing(printing it to console).
```