System Health & Prometheus Metrics

The Athena server counts what it is doing: agent connections, heartbeat and database writes, REST API latency, commands, deployments, agent updates, CPU and memory. It shows those numbers in two places that read the same counters: the live Settings → System Health tab in the web console, and a Prometheus /metrics endpoint that Prometheus, Grafana or any other tool that reads the Prometheus text format can scrape. Because both read the same numbers, the console and your dashboards always agree.

This page covers the server's own performance counters. For the up/down health checks (/health, database, PKI, audit and disk status), see Server Health & Monitoring. For CPU, memory and disk on managed endpoints, see Resource Metrics & Performance.

Who can see it#

Operators hold settings.health.view but cannot open Settings, so they read System Health over REST. See Permission Model & Role Bundles.

Opening System Health#

1. Open Settings.
2. Click the System Health tab, or go straight to https://<server>:8443/settings/system-health.

While the tab is open, the server sends it new numbers every 10 seconds. The page does not poll. A badge at the top right reads Live with the time of the last update. If the browser loses its live connection, the badge turns amber and reads Live updates paused. The tab keeps the last numbers and reconnects by itself.

Use the 15m / 1h range selector to choose how much history the charts draw. The server keeps history in memory for the last 60 minutes, at one point every 10 seconds. Restarting the server clears that history and resets every counter. For long-term trends, scrape /metrics with Prometheus.

Key indicators#

The tiles at the top summarise the last 5 minutes. A tile turns amber when it crosses the threshold below. p95 means 95 % of the operations finished faster than the value shown.

Below the tiles, four charts plot Agents connected (against registered), REST latency, Heartbeat batch duration and CPU & memory over the selected range.

Database, Work and Process panels#

• Database: heartbeat queue, batch p95 and the last batch, heartbeats dropped, write-lock wait p95, inventory writes in flight, inventory write p95, the Agents list query p95, and database size. On the built-in SQLite database every write goes through a single writer, so a growing Heartbeat queue or Write-lock wait means writes are waiting for each other. A Writer behind marker appears in that case. This is most likely when a whole fleet reconnects at once. For fleets like that, use PostgreSQL.
• Work: commands, deployments, agent updates (automatic and pushed, listed separately) and monitor evaluations since the server started, each with OK / Failed / Active counts. It also shows alerts fired and firing now, agent-update failures by reason, and each scheduled job with its last run.
• Process: uptime, GC heap and pause time, thread pool, open console sessions and live-update connections, logins since start (with failures and rate-limit bans), and the days left on each certificate.
• Prometheus endpoint: the listener's address and exposure (public - no auth, loopback only or disabled), whether a scrape token is required, the last scrape time and the number of series. If the listener is not running, the box shows why. This box also has the Copy and token buttons.

Scraping /metrics with Prometheus#

By default the server serves Prometheus text on its own port:

http://<server>:9464/metrics

This port serves /metrics and nothing else. It is not the web console port. Do not point Prometheus at 8443/metrics, which only returns the web page. A minimal scrape configuration:

scrape_configs:
  - job_name: athena
    scrape_interval: 15s
    static_configs:
      - targets: ['athena.example.local:9464']

The Copy Prometheus scrape config button in the Prometheus endpoint box on the System Health tab copies a configuration like this for your server.

Counters reset when the server restarts. In Grafana, use rate() or increase() on any metric whose name ends in _total. Two useful queries:

# REST p95 over 5 minutes, excluding the long-lived agent stream
histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{http_route!~".*Connect"}[5m])))

# Heartbeat batch p95 - the database write bottleneck
histogram_quantile(0.95, sum by (le) (rate(athena_heartbeat_batch_duration_seconds_bucket[5m])))

Useful metrics#

Athena's own metrics start with athena_. Durations end in _seconds, sizes in _bytes, and counters in _total. Durations are histograms, so they also have _bucket, _sum and _count series. The endpoint also carries the standard ASP.NET Core HTTP, Kestrel and SignalR metrics (for example http_server_request_duration_seconds). On PostgreSQL it adds the Npgsql connection-pool metrics. A selection:

For alerting, athena_certificate_expiry_days and athena_licence_days_remaining let you warn well before an expiry, and a rising athena_heartbeat_queue_length or athena_heartbeats_dropped_total warns that the database cannot keep up.

Restricting who can read /metrics#

/metrics exposes only counts, durations, versions, days until expiry and licence limits. It never contains host names, user names, agent IDs, IP addresses, the licence customer name or ID, certificate subjects or any secret. Every label comes from a fixed list, and any other label is dropped before it is published.

To lock it down further, use any combination of the following. All of them are off by default.

When both an allow-list and a token are set, the address is checked first (403), then the token (401).

Setting the Metrics options#

The Metrics:* options go in appsettings.local.json in the server's data directory (see Data Directories & Backup), or in environment variables with a double underscore, such as Metrics__ListenAddress. They take effect after a server restart.

{
  "Metrics": {
    "Enabled": true,
    "ListenAddress": "0.0.0.0",
    "Port": 9464,
    "UseHttps": true,
    "AllowedIps": [ "10.20.0.15", "10.30.0.0/24" ]
  }
}

# The same as environment variables (double underscore for the colon)
Metrics__UseHttps=true
Metrics__AllowedIps__0=10.20.0.15
Metrics__AllowedIps__1=10.30.0.0/24

Scrape token#

1. Open Settings → System Health and find the Prometheus endpoint box.
2. Click the Generate / rotate scrape token button.
3. Copy the token right away. It is shown once and cannot be displayed again. The server keeps only a hash of it.
4. Save the token in a file on the Prometheus host and reference it from the scrape configuration. Every scrape must now send it as a Bearer token.

scrape_configs:
  - job_name: athena
    scrape_interval: 15s
    scheme: https                      # only if Metrics:UseHttps is true
    tls_config:
      ca_file: /etc/prometheus/athena-ca.crt   # CA that issued the server's web certificate
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/athena-scrape-token
    static_configs:
      - targets: ['athena.example.local:9464']

• Takes effect immediately, with no restart. Rotating a token makes the old one stop working at once, so update the Prometheus credentials file and reload Prometheus right after.
• Revoke: click Revoke scrape token next to it. /metrics is then open again. Metrics:AllowedIps still applies.
• Per server: the token is not copied to other servers by Settings Sync. Each server has its own token.

REST API#

Invoke-RestMethod "https://athena.contoso.com:8443/api/metrics/summary?historyMinutes=15" `
  -Headers @{ Authorization = "Bearer $token" } -SkipCertificateCheck |
  Select-Object -ExpandProperty data

See API Reference for signing in and getting a token.

Audit trail#

Individual scrapes are counted (athena_metrics_scrapes_total), not audited. See Audit & SIEM.

Troubleshooting#