> ## Documentation Index > Fetch the complete documentation index at: https://developers.stellar.org/llms.txt > Use this file to discover all available pages before exploring further. # Ingestion > Horizon API provides most of its utility through ingested data, and your Horizon server can be configured to listen for and ingest transaction results from... Horizon API provides most of its utility through ingested data, and your Horizon server can be configured to listen for and ingest transaction results from the Stellar network. Ingestion enables API access to both current state (e.g. someone's balance) and historical state (e.g. someone's transaction history). ## Ingestion Types There are two primary ingestion use cases for Horizon operations: * Ingesting **live** data to stay up to date with the latest ledgers from the network, accumulating a sliding window of aged ledgers; * Ingesting **historical** data to retroactively add network data from a time range in the past to the database. ## Determine Storage Space You should think carefully about the historical timeframe of ingested data you'd like to retain in Horizon's database. The storage requirements for transactions on the Stellar network are substantial and are growing unbounded over time. This is something that you may need to continually monitor and reevaluate as the network continues to grow. We have found that most organizations need only a small fraction of recent historical data to satisfy their use cases. Through analyzing traffic patterns on SDF's Horizon instance, we see that most requests are for very recent data. To keep your storage footprint small, we recommend the following: * Use **live** ingestion, use **historical** ingestion only in limited exceptional cases. * If your application requires access to all network data, no filtering can be done, we recommend limiting historical retention of ingested data to a sliding window of 1 month (`HISTORY_RETENTION_COUNT=518400`). Note that you must set this explicitly: Horizon's default is `HISTORY_RETENTION_COUNT=0`, which retains all history and never purges ingested data. * If your application can work on a [filtered network dataset](https://developers.stellar.org/docs/data/apis/horizon/admin-guide/ingestion-filtering.md) based on specific accounts and assets, then we recommend applying ingestion filter rules. When using filter rules, it provides benefit of choice in longer historical retention timeframe since the filtering is reducing the overall database size to such a degree, historical retention(`HISTORY_RETENTION_COUNT`) can be set in terms of years rather than months or even disabled(`HISTORY_RETENTION_COUNT=0`). * If you cannot limit your history retention window to 30 days and cannot use filter rules, we recommend considering [Stellar Hubble Data Warehouse](https://developers.stellar.org/docs/data/analytics/hubble) for any historical data. ### Ingesting Live Data This option is enabled by default and is the recommended mode of ingestion to run. It is controlled with environment configuration flag `INGEST`. Refer to [Configuration](https://developers.stellar.org/docs/data/apis/horizon/admin-guide/configuring.md) for how an instance of Horizon performs the ingestion role. For high availability requirements, **we recommend deploying more than one live ingesting instance**, as this makes it easier to avoid downtime during upgrades and adds resilience, ensuring you always have the latest network data (refer to [Ingestion Role Instance](https://developers.stellar.org/docs/data/apis/horizon/admin-guide/configuring.md#multiple-instance-deployment)). ### Ingesting Historical Data Import network data from a past date range into the database: ```text Example theme={"theme":{"light":"github-light","dark":"github-dark"}} stellar-horizon db reingest range ``` Running any historical range of ingestion requires coordination with the data retention configuration chosen. When setting a temporal limit on history with `HISTORY_RETENTION_COUNT=`, the temporal limit takes precedence, and any data ingested beyond that limit will be automatically purged. Typically the only time you need to run historical ingestion is once when boot-strapping a system after first deployment, from that point forward **live** ingestion will keep the database populated with the expected sliding window of trailing historical data. Maybe one exception is if you think you have a gap in the database caused by the **live** ingestion being down, in which case you can run historical ingestion range to essentially gap fill. You can run historical ingestion in parallel in background while your main Horizon server separately performs **live** ingestion. If the range specified overlaps with data already in the database, it is ok and will simply be overwritten, effectively idempotent. #### Parallel Ingestion Workers You can parallelize the ingestion of target historical ledger range by dividing it into sequential slices of smaller ranges and run the db reingest range command for each sub-range in parallel as a separate process on the same or a different machine. The shorthand rule for best performance is to identify the number of CPU cores available per target machine, if multi-core, then add `--parallel-workers ` to the command, this will enable the command to further parallelize internally within a single process using multiple threads and sub-divided smaller ranges. ```text Example theme={"theme":{"light":"github-light","dark":"github-dark"}} # target range 1 30000, on single machine with 1 CPU core horizon1> stellar-horizon db reingest range 1 30000 # target range 1 30000, on single machine with 4 CPU cores horizon1> stellar-horizon db reingest range 1 30000 --parallel-workers 4 # target range 1 30000, on two machines, each has 2 CPU cores horizon1> stellar-horizon db reingest range 1 15000 --parallel-workers 2 horizon2> stellar-horizon db reingest range 15001 30000 --parallel-workers 2 ``` ### Notes #### Some endpoints may report not available during **live** ingestion * Endpoints that display current state information from **live** ingestion may return `503 Service Unavailable`/`Still Ingesting` error. An example is the `/paths` endpoint (built using offers). Such endpoints will become available after **live** ingestion has finished network synchronization and catch up (usually within a couple of minutes). #### If more than five minutes has elapsed with no new ingested data: * Verify the host machine meets recommended [Prerequisites](https://developers.stellar.org/docs/data/apis/horizon/admin-guide/prerequisites.md). * Check Horizon log output. * If there are many `level=error` messages, it may point to an environmental issue, inability to access the database. * **Live** ingestion will emit two key log lines about once every 5 seconds based on latest ledger emitted from network. Tail the Horizon log output and grep for presence of these lines with a filter: ``` tail -f horizon.log | | grep -E 'Processed ledger|Closed ledger' ``` If you don't see output from this pipeline every couple of seconds for a new ledger then ingestion is not proceeding, look at full logs and see if any alternative messages are printing reasons to the contrary. May see lines mentioning 'catching up' When connecting to pubnet, as it can take up to 5 minutes for the captive core process started by Horizon to catch up to pubnet network. * Check RAM usage on the machine, it's possible that system ran low on RAM and is using swap memory which will result in slow performance. Verify host machine meets minimum RAM [prerequisites](https://developers.stellar.org/docs/data/apis/horizon/admin-guide/prerequisites.md). * Verify the read/write throughput speeds on the volume that current working directory for horizon process is using. Based on [prerequisites](https://developers.stellar.org/docs/data/apis/horizon/admin-guide/prerequisites.md), volume should have at least 10mb/s, one way to roughly verify this on host machine(linux/mac) command line: ``` sudo dd if=/dev/zero of=/tmp/test_speed.img bs=1G count=1 ``` #### Monitoring Ingestion Process For high-availability deployments, it is recommended to implement monitoring of ingestion process for visibility on performance/health. Refer to [Monitoring](https://developers.stellar.org/docs/data/apis/horizon/admin-guide/monitoring.md) for accessing logs and metrics from Horizon. Stellar publishes the example [Horizon Grafana Dashboard](https://grafana.com/grafana/dashboards/13793-stellar-horizon), which demonstrates queries against key horizon ingestion metrics, specifically look at the `Local Ingestion Delay [Ledgers]` and `Last ledger age` in the `Health Summary` panel.