Skip to content

These new docs are in beta. Please submit bugs to GitHub issues.


Running

Once your Horizon database and Captive Core configuration is set up properly, you’re ready to run Horizon. Simply run stellar-horizon with the appropriate parameters set (or stellar-horizon-cmd serve if you installed via the package manager, which will automatically import your configuration), which starts the HTTP server and starts logging to standard out. When run, you should see output similar to:

INFO[...] Starting horizon on :8000                     pid=29013

Note: the numbers shown above may be different for your installation.

The log line above announces that Horizon is ready to serve client requests. Next, you can confirm that Horizon is responding correctly by loading the root resource. In the example above, that URL would be http://127.0.0.1:8000/, and simply running curl http://127.0.0.1:8000/ shows you that the root resource can be loaded correctly:

JSON
{
  "_links": {
    "account": {
      "href": "http://127.0.0.1:8000/accounts/{account_id}",
      "templated": true
    },
    "accounts": {
      "href": "http://127.0.0.1:8000/accounts{?signer,sponsor,asset,cursor,limit,order}",
      "templated": true
    }
  }
}

Ingesting Transactions

Horizon provides most of its utility through ingested data, and your Horizon server can be configured to listen for and ingest transaction results from the Stellar network. However, this option is disabled by default.

We highly recommend having more than one ingesting instance, as this makes it easier to avoid downtime during upgrades and adds resilience to your infrastructure, ensuring you always have the latest network data.

Ingesting Live Data

To enable ingestion, you must either pass --ingest=true on the command line or set the INGEST environment variable to “true”. Note that you can have multiple ingesting machines in your cluster.

Ingesting Historical Data

To ingest historical data from Stellar Core, you need to run horizon db reingest range <start> <end>. If you’re running a full validator with published history archive(s), for example, you might want to ingest the network’s entire history. You can run this process in the background while your Horizon server is up. This continuously decrements the history.elder_ledger in your /metrics endpoint until NUM_LEDGERS is reached and the backfill is complete.

Reingesting Ledgers

To reingest older ledgers—occasionally necessary after a version upgrade—or to ingest ledgers closed by the network before you started Horizon use the horizon db reingest range <start> <end> command:

horizon1> horizon db reingest range 1 10000
horizon2> horizon db reingest range 10001 20000
horizon3> horizon db reingest range 20001 30000
# ... etc.

This allows reingestion to be split up and done in parallel by multiple Horizon processes.

Managing Storage for Historical Data

Over time, the recorded network history will grow unbounded, increasing storage used by the database. Horizon needs sufficient disk space to expand the data ingested from Stellar Core. Unless you need to maintain a history archive, you may configure Horizon to only retain a certain number of ledgers in the database. This is done using the --history-retention-count flag or the HISTORY_RETENTION_COUNT environment variable. Set the value to the number of recent ledgers you wish to keep around, and every hour the Horizon subsystem will reap expired data. Alternatively, you may execute the command horizon db reap to force a collection.

If you configure this value, we also recommend configuring the CATCHUP_RECENT value for your Captive Core instance.

Common Issues

Some endpoints are not available during state ingestion

Endpoints that display state information are not available during initial state ingestion and will return a 503 Service Unavailable/Still Ingesting error. An example is the /paths endpoint (built using offers). Such endpoints will become available after state ingestion is done (usually within a couple of minutes).

State ingestion is taking a lot of time

State ingestion shouldn’t take more than a couple of minutes on an AWS c5.xlarge instance or equivalent.

It’s possible that the progress logs (see below) will not show anything new for a longer period of time or print a lot of progress entries every few seconds. This happens because of the way history archives are designed. The ingestion is still working but it’s processing entries of type DEADENTRY. If there is a lot of them in the bucket, there are no active entries to process. We plan to improve the progress logs to display actual percentage progress so it’s easier to estimate an ETA.

If you see that ingestion is not proceeding for a very long period of time:

  1. Check the RAM usage on the machine. It’s possible that system ran out of RAM and it using swap memory that is extremely slow.
  2. If above is not the case, file a new issue in the Horizon repository.

CPU usage goes high every few minutes

This is by design. Horizon runs a state verifier routine that compares state in local storage to history archives every 64 ledgers to ensure data changes are applied correctly. If data corruption is detected, Horizon will block access to endpoints serving invalid data.

We recommend keeping this security feature turned on; however, if it’s causing problems (due to CPU usage) this can be disabled via the --ingest-disable-state-verification CLI flag or INGEST_DISABLE_STATE_VERIFICATION env variable.

Reading the logs

In order to check the progress and status of ingestion you should check the logs; all logs related to ingestion are tagged with service=ingest.

It starts with informing you about state ingestion:

INFO[...] Starting ingestion system from empty state...  pid=5965 service=ingest temp_set="*io.MemoryTempSet"
INFO[...] Reading from History Archive Snapshot         ledger=25565887 pid=5965 service=ingest

During state ingestion, Horizon will log number of processed entries every 100,000 entries (there are currently around 7M entries in the public network):

INFO[...] Processing entries from History Archive Snapshot  ledger=25565887 numEntries=100000 pid=5965 service=ingest
INFO[...] Processing entries from History Archive Snapshot  ledger=25565887 numEntries=200000 pid=5965 service=ingest
INFO[...] Processing entries from History Archive Snapshot  ledger=25565887 numEntries=300000 pid=5965 service=ingest
INFO[...] Processing entries from History Archive Snapshot  ledger=25565887 numEntries=400000 pid=5965 service=ingest
INFO[...] Processing entries from History Archive Snapshot  ledger=25565887 numEntries=500000 pid=5965 service=ingest

When state ingestion is finished, it will proceed to ledger ingestion starting from the next ledger after checkpoint ledger (25565887+1 in this example) to update the state using transaction metadata:

INFO[...] Processing entries from History Archive Snapshot  ledger=25565887 numEntries=5300000 pid=5965 service=ingest
INFO[...] Processing entries from History Archive Snapshot  ledger=25565887 numEntries=5400000 pid=5965 service=ingest
INFO[...] Processing entries from History Archive Snapshot  ledger=25565887 numEntries=5500000 pid=5965 service=ingest
INFO[...] Processed ledger                              ledger=25565887 pid=5965 service=ingest type=state_pipeline
INFO[...] Finished processing History Archive Snapshot  duration=2145.337575904 ledger=25565887 numEntries=5529931 pid=5965 service=ingest shutdown=false
INFO[...] Reading new ledger                            ledger=25565888 pid=5965 service=ingest
INFO[...] Processing ledger                             ledger=25565888 pid=5965 service=ingest type=ledger_pipeline updating_database=true
INFO[...] Processed ledger                              ledger=25565888 pid=5965 service=ingest type=ledger_pipeline
INFO[...] Finished processing ledger                    duration=0.086024492 ledger=25565888 pid=5965 service=ingest shutdown=false transactions=14
INFO[...] Reading new ledger                            ledger=25565889 pid=5965 service=ingest
INFO[...] Processing ledger                             ledger=25565889 pid=5965 service=ingest type=ledger_pipeline updating_database=true
INFO[...] Processed ledger                              ledger=25565889 pid=5965 service=ingest type=ledger_pipeline
INFO[...] Finished processing ledger                    duration=0.06619956 ledger=25565889 pid=5965 service=ingest shutdown=false transactions=29
INFO[...] Reading new ledger                            ledger=25565890 pid=5965 service=ingest
INFO[...] Processing ledger                             ledger=25565890 pid=5965 service=ingest type=ledger_pipeline updating_database=true
INFO[...] Processed ledger                              ledger=25565890 pid=5965 service=ingest type=ledger_pipeline
INFO[...] Finished processing ledger                    duration=0.071039012 ledger=25565890 pid=5965 service=ingest shutdown=false transactions=20

Managing Stale Historical Data

Horizon ingests ledger data from an managed, possibly-remote, pared-down Captive Stellar Core instance. In the event that Captive Core crashes, lags, or if Horizon stops ingesting data for any other reason, the view provided by Horizon will start to lag behind reality. For simpler applications, this may be fine, but in many cases this lag is unacceptable and the application should not continue operating until the lag is resolved.

To help applications that cannot tolerate lag, Horizon provides a configurable “staleness” threshold. Given that enough lag has accumulated to surpass this threshold (expressed in number of ledgers), Horizon will only respond with an error: stale_history. To configure this option, use either the --history-stale-threshold command line flag or the HISTORY_STALE_THRESHOLD environment variable.

Note: Non-historical requests (such as submitting transactions or finding payment paths) will not error out when the staleness threshold is surpassed.

Last updated Apr. 12, 2021

Next Up: Monitoring
Page Outline