Upgrading

Here we'll describe the recommended steps for upgrading a Horizon 2.x installation.

Pre-requisites

• An existing Horizon deployment consisting of one or more instances of Horizon.
• All instances are on same 2.x version to begin.
• If bare-metal install, you have shell, or command line access to each host having a Horizon installation.
• If deployed direct on Docker daemon, you have command line access to the host that is running the Docker daemon.
• If deployed on Kubernetes with Helm chart, you have kubectl and helm command line tools on your workstation and a user login with appropriate access levels to change resources in target namespace of Horizon deployment on the cluster.

Assess current installation

• Identify the list of all instances of Horizon that need to be upgraded.

  • Bare-metal installations: the list of hosts is managed by you.

  • Docker daemon deployments: the list of hosts and running containers is managed by you.

  • Kubernetes deployments: get the list of pods that are deployed from your prior Helm installation, they will have an annotation for release=your_helm_horizon_installation_name:

    bash

    kubectl get pods -l release=your_helm_horizon_installation_name -n <your_horizon_namespace>

• Identify your current Horizon software version:

  • Obtain command line access to the operating system of each Horizon instance:
    • Bare-metal installations, this is typically ssh on Linux or powershell on Windows.
    • Docker daemon deployments, use docker exec -it <containerid> /bin/bash
    • For Kubernetes deployments, use kubectl exec -it <pod_name> -n <horizon_namespace> -- /bin/bash
  • On command line of each instance, run stellar-horizon version

• All instances should report the same version, if not, the system may be inconsistent, use this upgrade as opportunity to establish consistency and get them all on same version.

Determine the target version for upgrade

Now that you know your current Horizon version, visit Horizon Releases and choose the next greater version above your current version to upgrade. Follow steps recommended by GitHub to compare releases, click on the Compare dropdown of the chosen release, and then select your current release and GH will display the differences between versions, select the Files changed tab, and go to the services/horizon/CHANGELOG.md, it will highlight the new release notes for changes that have occurred between your current version and the new version you selected. Review this and look for any Breaking Changes, State Rebuild and DB Schema Migration sections for consideration, as the latter two will also mention expected time for the state rebuild or db migration to apply respectively.

Install the new version

Now that you have indentified the new version and are aware of the potential impacts from upgrading to new version based on release notes, such as state rebuilds and db migrations, you are informed and ready to proceed with upgrade.

Upgrading production deployents should leverage a secondary, hot-backup deployment, also known as a blue/green model and perform the upgrade on the inactive deployment first. This will avoid downtime of system to your external users, as the upgrade takes place on the inactive deployment.

A good strategy for upgrading Horizon and applicable to single or multi-instance deployments - shut all instances down, install new Horizon version on one of the ingesting instances first. The reason being Horizon software will only initate State Rebuild and DB Schema Migration actions related to an upgrade on an instance that it detects ingestion has been enabled with configuration parameter, INGEST=true. This lowers complexity for you during the upgrade as you only need to focus on one instance and it avoids potential concurrent Horizon ingestion processes attempting the same upgrade on the database.

• Bare-metal installations, stop the Horizon process on all instances first, then shell into one instance that is configured for ingestion, and use apt package manager on linux.

  bash

  sudo apt update
  sudo apt install stellar-horizon=new_horizon_debian_pkg_version

  Restart Horizon using the configuration already in place, but include APPLY_MIGRATIONS=true environment variable, this will trigger Horizon to automatically run any db migrations that it detects are needed.

• Docker daemon deployments, stop all docker containers first, then choose one container that has ingestion enabled, set the new tag for the image based on release published on dockerhub - stellar/stellar-horizon, and restart the container in docker daemon, include APPLY_MIGRATIONS=true environment variable to the container envrionment, this will trigger Horizon to automatically run any db migrations that it detects are needed.

• For Helm installations on Kubernetes, first use your helm cli tool to stop all Horizon instances by scaling all your Horizon installations(for ingest and web) down to 0 replicas, which you've created prior on run steps.

  bash

  helm upgrade all-my-horizon-installations \
  --namespace my-horizon-namespace-on-cluster \
  --set ingest.replicaCount=0 \
  --set web.replicaCount=0

  Now, use helm to start just a single Horizon instance from a helm installation that has ingestion enabled on kubernetes cluster, you will set the global.image.horizon.tag to the release tag published on stellar/stellar-horizon

  bash

  helm upgrade my-horizon \
  --namespace my-horizon-namespace-on-cluster  \
  --set global.image.horizon.tag=new_horizon_release_number \
  --set ingest.horizonConfig.applyMigrations=True \
  --set ingest.replicaCount=1

Confirming the upgrade on single ingestion instance first

If you have monitoring infrastructure in place, then you have two options for assessing the upgrade status:

• View metrics outputs using grafana dashboards that leverage queries on the Horizon metrics data model to check key stats like ingestion and network ledgers are advancing and in step.

• View the Horizon web server 'status' url path on the upgraded instance:

  bash

  curl http://localhost:8000/

  The response will be HTTP status code 200 and body of response will be a text based json data structure with diagnostic info on current Horizon software version, and ledger numbers for ingestion and network, refresh the url every 5 seconds or so, and should see the ingestion and network ledger numbers advancing and in step, indicating good connection to the network and ingestion.

If metrics and/or the Horizon 'status' url respones don't indicate healthy status based on advancing ledger ingestion, two steps to triage further:

• A delay in Horizon achieving healthy status after an upgrade is expected and legitmate for any upgrade cases where State Rebuild or DB Migration was noted in the release delta as part of prior Determine the target version for upgrade step. Typically the notes will also mention relative timeframe expectations for those to complete which can be factored in to how long to wait on delay.
• Check the logs from the upgraded instance to confirm what's going on. Any State Rebuild or DB Migration initiated will be mentioned. For example, a db migration will be noted in logs with following lines for start and finish:

  2023/09/22 18:27:01 Applying DB migrations...
  2023/09/22 18:27:01 successfully applied 5 Horizon migrations

Upgrade all remaining instances

At this point, you have upgraded one ingesting instance to the new Horizon version, it has automatically updated the database if required and the instance is running with healthy status. Now, install the same Horizon software version on the remainder of instances, restarting each after the upgrade. For bare-metal and docker daemon installations that will likely be self explanatory on how to accomplish that for remainder of instances, on helm chart installations, run the helm upgrade again, setting the image tag and also restoring original replicaCounts:

bash

helm upgrade all-my-horizon-installations \
--namespace my-horizon-namespace-on-cluster \
--set ingest.replicaCount=1 \
--set web.replicaCount=1 \
--set global.image.horizon.tag=new_horizon_release_number \
--set ingest.horizonConfig.applyMigrations=False

For production deployments following the hot backup or blue/green model, this is the opportunity to confirm the inactive deployment has taken the upgrade correctly and stable, at which point, switch the load balancers now to forward traffic to the inactive deployment, making it the active deployment. Now, can take time to perform same upgrade on the other deployment which is now inactive.