Data Lake Storage Format

Galexie stores ledger metadata in an object store as compressed XDR files.

Unless you are building for a highly specialized case, use the ingest SDK or Stellar RPC rather than reading these files directly. Both tools hide the layout described here, so your application does not have to manage it.

Read this page if one of these applies to you:

• You read ledgers in bulk, for example for a backfill or an analytics job, and want to fetch objects from the bucket in parallel without an RPC server in the path.
• Your application is not written in Go, so it cannot use the ingest SDK, and you do not want to run Stellar RPC for history.
• You build a data store that other Stellar tools must be able to read.

SEP-54: Ledger Metadata Storage is the specification for this format. This page summarizes the specification and shows the layout of the public Stellar data lake.

Object keys

Galexie writes all ledger files under one configurable path in the bucket. Inside that path, each file sits in a partition directory:

<ledgers-path>/<partition>/<batch>.xdr.zst

Galexie omits the partition directory when files_per_partition is 1.

Both the partition name and the batch name start with 8 hexadecimal digits. That value is 0xFFFFFFFF minus the first ledger sequence of the range. The subtraction reverses the order, so a listing returns the newest ledgers first. A client finds the latest ledger with one listing request and no index.

The partition name then repeats the first and last ledger sequence of the partition:

FFFFFFFF--0-15

The batch name repeats the first and last ledger sequence of the batch. When a batch holds one ledger, the name carries that one sequence only:

FFFFFFFD--2-3.xdr.zst
FFFFFFFD--2.xdr.zst

Put together, the full key of the first batch above is:

<ledgers-path>/FFFFFFFF--0-15/FFFFFFFD--2-3.xdr.zst

note

A data store does not necessarily hold the whole ledger history. List the oldest partition to find where its history starts.

File contents

Each ledger file is a Zstandard compressed LedgerCloseMetaBatch XDR value:

// Batch of ledgers along with their transaction metadata
struct LedgerCloseMetaBatch
{
    // starting ledger sequence number in the batch
    uint32 startSequence;

    // ending ledger sequence number in the batch
    uint32 endSequence;

    // Ledger close meta for each ledger within the batch
    LedgerCloseMeta ledgerCloseMetas<>;
};

The ledgersPerBatch value in the configuration file sets the number of ledgers in each batch. Batch ranges start at sequence 0, but Galexie exports from ledger 2, so the first batch of a data store can hold fewer ledgers than the rest. Refer to the XDR documentation for more information on the encoding.

Configuration file

SEP-54 requires a JSON configuration object at <ledgers-path>/.config.json. A client reads it to learn the batch and partition sizes before it builds any object key.

Property	Type	Description
networkPassphrase	string	The passphrase of the Stellar network that produced the ledgers.
version	string	The version of the configuration file schema.
compression	string	The compression algorithm. Only zstd is supported.
ledgersPerBatch	integer	The number of ledgers in each batch file.
batchesPerPartition	integer	The number of batch files in each partition directory.

The Galexie configuration file names the last two values ledgers_per_file and files_per_partition.

Data stores written by Galexie versions before v23.0.0 do not hold this file, and their object keys end in .xdr.zstd instead of .xdr.zst. A client that reads one must get the batch and partition sizes from its own configuration. The Go datastore package does that when the file is absent.

For example, the Pubnet data store in the public Stellar data lake holds one ledger per batch and 64,000 batches per partition:

v1.1/stellar/ledgers/pubnet/.config.json

{
  "networkPassphrase": "Public Global Stellar Network ; September 2015",
  "version": "1.0",
  "compression": "zstd",
  "ledgersPerBatch": 1,
  "batchesPerPartition": 64000
}

With those sizes, ledger 50000000 falls in the partition that starts at ledger 49984000, so its object key is:

v1.1/stellar/ledgers/pubnet/FD054DFF--49984000-50047999/FD050F7F--50000000.xdr.zst

The public Stellar data lake

A Stellar ledger data lake is available through the AWS Open Data program. The bucket is public and needs no credentials. The Pubnet ledgers sit in one data store:

s3://aws-public-blockchain/v1.1/stellar/ledgers/pubnet

Download ledger 50000000 from the example above and print it as JSON with stellar xdr decode:

curl -s https://aws-public-blockchain.s3.amazonaws.com/v1.1/stellar/ledgers/pubnet/FD054DFF--49984000-50047999/FD050F7F--50000000.xdr.zst \
  | zstd -dc \
  | stellar xdr decode --type LedgerCloseMetaBatch --input single --output json-formatted

Testnet is reset regularly, and each reset starts a new data store. The data store for each reset sits in a directory named after the reset date:

s3://aws-public-blockchain/v1.1/stellar/ledgers/testnet/<reset-date>

List v1.1/stellar/ledgers/testnet/ to find the current reset directory, and use that directory as the <ledgers-path>. Some older reset directories have no .config.json. Read those as described in Configuration file.

Object metadata

Galexie also attaches metadata to each ledger object. The metadata lets a client filter objects without downloading them. Data stores that cannot hold user-defined object metadata, such as a plain file system, omit it.

Key	Description
start-ledger	The first ledger sequence in the batch.
end-ledger	The last ledger sequence in the batch.
start-ledger-close-time	The Unix timestamp of the close time of the first ledger.
end-ledger-close-time	The Unix timestamp of the close time of the last ledger.
protocol-version	The protocol version of the last ledger.
core-version	The version of Stellar Core that produced the ledgers.
network-passphrase	The passphrase of the Stellar network that produced the ledgers.
compression-type	The compression algorithm of the object.
version	The version of Galexie that wrote the object.

On Amazon S3 these arrive as x-amz-meta- response headers:

curl -sI https://aws-public-blockchain.s3.amazonaws.com/v1.1/stellar/ledgers/pubnet/FD054DFF--49984000-50047999/FD050F7F--50000000.xdr.zst

Data integrity

The format carries no proof that the ledgers in a data store are the ledgers the network closed. Verifying that is outside the scope of SEP-54. Read from a data store you run, or from an operator you trust.