Skip to main content

Monitoring

To ensure that your instance of Horizon is performing correctly, we encourage you to monitor it and provide both logs and metrics to do so.

Metrics

Metrics are collected while a Horizon process is running and they are exposed privately via the /metrics path, accessible only through the Horizon admin port. You need to configure this via --admin-port or ADMIN_PORT, since it's disabled by default. If you're running such an instance locally, you can access this endpoint:

$ stellar-horizon --admin-port=4200 &
$ curl localhost:4200/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.665e-05
go_gc_duration_seconds{quantile="0.25"} 2.1889e-05
go_gc_duration_seconds{quantile="0.5"} 2.4062e-05
go_gc_duration_seconds{quantile="0.75"} 3.4226e-05
go_gc_duration_seconds{quantile="1"} 0.001294239
go_gc_duration_seconds_sum 0.002469679
go_gc_duration_seconds_count 25
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 23
and so on...

Logs

Horizon will output logs to standard out. Information about what requests are coming in will be reported, but more importantly, warnings or errors will also be emitted by default. A correctly running Horizon instance will not output any warning or error log entries.

Below we present a few standard log entries with associated fields. You can use them to build metrics and alerts. Please note that these represent Horizon app metrics only. You should also monitor your hardware metrics like CPU or RAM Utilization.

Starting HTTP request

KeyValue
msgStarting request
client_nameValue of X-Client-Name HTTP header representing client name
client_versionValue of X-Client-Version HTTP header representing client version
app_nameValue of X-App-Name HTTP header representing app name
app_versionValue of X-App-Version HTTP header representing app version
forwarded_ipFirst value of X-Forwarded-For header
hostValue of Host header
ipIP of a client sending HTTP request
ip_portIP and port of a client sending HTTP request
methodHTTP method (GET, POST, ...)
pathFull request path, including query string (ex. /transactions?order=desc)
streamingBoolean, true if request is a streaming request
refererValue of Referer header
reqRandom value that uniquely identifies a request, attached to all logs within this HTTP request

Finished HTTP request

KeyValue
msgFinished request
bytesNumber of response bytes sent
client_nameValue of X-Client-Name HTTP header representing client name
client_versionValue of X-Client-Version HTTP header representing client version
app_nameValue of X-App-Name HTTP header representing app name
app_versionValue of X-App-Version HTTP header representing app version
durationDuration of request in seconds
forwarded_ipFirst value of X-Forwarded-For header
hostValue of Host header
ipIP of a client sending HTTP request
ip_portIP and port of a client sending HTTP request
methodHTTP method (GET, POST, ...)
pathFull request path, including query string (ex. /transactions?order=desc)
routeRoute pattern without query string (ex. /accounts/{id})
statusHTTP status code (ex. 200)
streamingBoolean, true if request is a streaming request
refererValue of Referer header
reqRandom value that uniquely identifies a request, attached to all logs within this HTTP request

Metrics

Using the entries above you can build metrics that will help understand performance of a given Horizon node. For example:

  • Number of requests per minute.
  • Number of requests per route (the most popular routes).
  • Average response time per route.
  • Maximum response time for non-streaming requests.
  • Number of streaming vs. non-streaming requests.
  • Number of rate-limited requests.
  • List of rate-limited IPs.
  • Unique IPs.
  • The most popular SDKs/apps sending requests to a given Horizon node.
  • Average ingestion time of a ledger.
  • Average ingestion time of a transaction.

Alerts

Below are example alerts with potential causes and solutions. Feel free to add more alerts using your metrics:

AlertCauseSolution
Spike in number of requestsPotential DoS attackLower rate-limiting threshold
Large number of rate-limited requestsRate-limiting threshold too lowIncrease rate-limiting threshold
Ingestion is slowHorizon server spec too lowIncrease hardware spec
Spike in average response time of a single routePossible bug in a code responsible for rendering a routeReport an issue in Horizon repository.

I'm Stuck! Help!

If any of the above steps don't work or you are otherwise prevented from correctly setting up Horizon, please join our community and let us know. Either post a question at our Stack Exchange or chat with us on Keybase in #dev_discussion to ask for help.