Skip to main content

Scaling

Horizon enables different logical tiers that can be scaled independently for increasing throughput, isolation, and availability. The following components can be independently scaled:

  • Web service API (serving)
  • Captive Core (ingestion and transaction submission)
  • Database (storage)

Single Instance Deployment​

It is recommend to start with a single instance deployment, and scale up based on the needs of your particular use-case.

This deployment is intended for use with minimal history retention (<= 30 days) and minimal request volume.

In this setup, a single instance of Horizon performs all three roles; ingestion, transaction submission, and end-user API requests.

Scaling to Multiple Instances​

There are a few reasons you may choose to scale to multiple instances of Horizon.

  • Horizontally scaling enables you to serve more API requests and at a faster rate
  • Redundancy enables zero downtime in the cases where Horizon requires downtime on upgrade (migrations, state rebuilds, etc)
  • Protection against potential ingestion lag, which could result in downtime for end-users

Multiple instances of Horizon can be configured to point to the same database, and the ingestion process will not perform redundant work in these cases.

When scaling Horizon, it is worth it to note that Horizon's rate limiting should be disabled and rate limiting should be managed external to Horizon within infrastructure. Horizon's rate limiting implementation is managed in-memory, so does not work with multiple instances.

Logically Isolating Ingestion​

Ingestion is the process by which new ledgers are propagated into Horizon's database. It's health is critical, as degredations in performance can result in falling behind the last closed ledger, leaving your end-users unaware of the current state of the network, and unable to successfully submit new transactions. Any lag in ingestion would likely be considered downtime for your service

Horizon allows you to independently configure the different roles that it performs, including ingestion. The below diagram illustrates how you could logically separate the instances serving API requests from the instances performing ingestion, and introduce a read-only replica database in order to further isolate these components. This setup has quite a few advantages:

  • Each "role" Horizon plays can be independently scaled
  • API instances are significantly ligher weight from a hardware requirements perspective, since they do not need to run captive core
  • API instances can be horizontally scaled or dynamically scaled, based on your specific end-user needs
  • Ingestion and it's performance is isolated from API activity, so bursts in user activity cannot degrade it and cause ingestion lag. Ingestion health is critical, as degredations in performance can result in falling behind the last closed ledger, leaving your end-users unaware of the current state of the network, and unable to successfully submit new transactions

The Horizon API role requires only read-only permissions to a database for all actions it performs. However, the API instances will need to delegate all transaction submission requests to an instance which runs captive core. Further database replicas could be added if necessary to support more requests.

Logically Isolating Transaction Submission​

In the above example, ingestion is safely isolated from most API traffic, which has historically been the large majority of traffic. However, transaction submission still needs to be served by a core instance, and so API instances must passthrough their transaction submission requests to an ingesting instance.

The below diagram illustrates how we could further isolate (and scale) transaction submission, by way of using core watcher instances, rather than Horizon instances running captive core. This allows us to further protect ingestion, preventing downtime and ingestion lag. It also makes it possible to horizontally scale transaction submission itself, independent of the rest of the API traffic.