System Health Tile Reference

Settings → System Health shows the Athena server's own health as a grid of tiles. This page has one section per tile: what it measures, the exact rule that turns it amber (warning) or red (critical), the Prometheus metric behind it, what a healthy server shows, and what to check first when it isn't healthy. Each tile's ? button in the console opens the matching section of the built-in help; the section anchors on this page use the same names (for example #tile-heartbeat).

For how to open the tab, who can see it, the charts and panels, and how to scrape /metrics, see System Health & Prometheus Metrics.

How to read the tiles#

• Window. Unless a tile says otherwise, it is judged over the last 5 minutes (30 samples, one every 10 seconds). A tag in the tile's corner such as 5 MIN or 24 H names a different window. Because the window rolls, a tile goes green again by itself once a problem is over.
• p95 means 95 % of the operations were faster than the value shown. A value shown as > 300 s is beyond the largest range the server measures.
• Amber is a warning: look at it soon. Red is critical: act now. A tile with no rule for a level never shows that colour.
• Unknown is never green. If the server can't read a tile's data (a failed database query or an unreadable file), the tile reads data unavailable with the reason, in amber, and a Server internals tile reports the status unknown. This applies to Scheduled task failures, Database size and Certificate expiry, the tiles that read the database or files. The others read the server's in-memory counters.
• Counters restart with the server. In PromQL always use rate() or increase() on a _total metric, never its raw value.

Red tiles reach /health

While any Server internals tile, the Heartbeat batch p95 tile or Database health is red, the server's /health endpoint answers {"status":"degraded"}. It is still HTTP 200, so a load balancer keeps the server in rotation. /health/details lists the red tiles under critical in its internals entry. The check reads the same in-memory numbers as the tab and runs no database query, so frequent probes cost nothing. The User experience tile is left out on purpose: it measures browsers, not the server. See Server Health & Monitoring.

Tile layout

All thresholds at a glance

Top-row tiles#

Agents online

Agents with a live connection to this server right now, out of the agents registered. The bar fills against the licence agent limit (or the registered count when the licence has no limit), and the line below says how many agents the licence still allows. It counts; it does not judge. Prometheus: athena_agents_connected, athena_agents_registered, athena_licence_agents_limit.

Healthy: a flat line close to the number of machines you expect to be switched on, dipping only around a server restart and recovering within a few minutes. Keep at least 10 % headroom under the licence limit so new enrolments are never refused.

When it drops or doesn't recover:

1. Look at Agent disconnects: transport_error means the network dropped them, server_shutdown means this server restarted.
2. Look at Agent TLS handshake p95: a slow handshake keeps agents from getting back in.
3. Check that the agent port (8444) is reachable from the agents. See Network Ports & Firewall.
4. If the bar is near the licence limit, new agents are refused. See Licensing.

Agent connect p95

How long the slowest 5 % of agent connections took from the connection being accepted to its start-up work being done (configuration and assignments sent). The line below shows the worst 10-second interval of the last hour and when it was. Prometheus: athena_agent_connect_duration_seconds.

Amber: p95 > 2 s. Red: none. Healthy: well under a second in steady state (0.05 s at 500 agents in testing). It rises only while many agents connect at once.

1. Check for a reconnect storm: Agents online climbing fast and Agent disconnects showing server_shutdown or transport_error. A storm settles by itself within a few minutes.
2. Look at Heartbeat batch p95 and the Database panel. Connect work writes to the database, so a busy writer slows every connect.
3. Look at CPU. Pinned near 100 % during a storm means the server needs more cores.
4. If it stays high with no storm, check the server log for slow database calls, and consider PostgreSQL above about 3,000 agents.

REST p95

Response time of the slowest 5 % of API and console requests, measured on the server (no network time). Agent connections, the console's live connection, the live-update hub and the /metrics scrape are left out, so this is what people and scripts feel. Prometheus: http_server_request_duration_seconds (without rpc_system="grpc").

Amber: p95 > 500 ms. Red: none. Healthy: tens of milliseconds (20–25 ms at 500–2,000 agents in testing). What a user feels is this plus the network round trip.

1. Open the REST latency chart. A spike that lines up with Agents online jumping is a reconnect storm and passes.
2. Look at CPU and the Process panel's thread-pool queue. A queue above 100 means the server is starved of threads.
3. Look at the Database panel: a long write-lock wait slows every page that writes.
4. In Grafana, split the query by http_route to find the one slow endpoint, then look for it in the server log.

Error rate

The share of REST requests in the window that failed with a server error (HTTP 5xx). The line below adds the count of 5xx responses and the 4xx (client error) share. Prometheus: http_server_request_duration_seconds_count by http_response_status_code.

Amber: 5xx > 1 % of REST requests. Red: none. Healthy: 0 %. A high 4xx share is usually a script with a wrong URL or an expired token, not a server fault.

1. Open the server log for the same minutes and find the exception behind the 5xx responses.
2. Look at Database health: a damaged or full database makes requests fail.
3. Look at Scheduler and CPU: an overloaded server times requests out.
4. If one route fails, find it by splitting the PromQL by (http_route).

Heartbeat batch p95

How long the database took to write one batch of agent heartbeats (up to 500 agents at a time). Every agent has one heartbeat slot in memory, which is what every screen shows as "last seen". A newer heartbeat overwrites the slot, so nothing is ever dropped. The database is written when something changes (online/offline, IP address, agent version, boot time) and otherwise at least every Agent:HeartbeatPersistSeconds (default 300 s). The line below shows how many slots are pending (due to be written now) and the persist lag: how far past its due time the oldest pending slot is. A rising lag means the writer is falling behind.

A batch that never finishes records no duration, so the tile also watches the batch running now and the time since the last batch finished. When the writer is stuck, the line below starts with writer stuck and for how long.

The tile also checks intake: heartbeats actually read in the last 5 minutes against what the connected agents should have sent (connected agents × time ÷ the heartbeat interval in Settings → Agents). A heartbeat the server never reads is neither queued nor dropped, so only intake can see it. Intake is judged once at least 100 heartbeats are expected in the window (about 10 agents at a 30 s interval). The heartbeat each agent sends right after it connects isn't counted, so intake doesn't overshoot while agents connect.

Prometheus: athena_heartbeat_batch_duration_seconds, athena_heartbeat_batch_running_seconds, athena_heartbeat_batch_since_last_seconds, athena_heartbeat_slots, athena_heartbeat_slots_pending, athena_heartbeat_persist_lag_seconds, athena_heartbeat_persisted_total, athena_heartbeat_coalesced_total, athena_heartbeats_received_total, athena_heartbeat_intake_ratio.

# stuck writer: a long batch, or nothing finished while heartbeats wait
max(athena_heartbeat_batch_running_seconds) > 60 or (athena_heartbeat_batch_since_last_seconds > 60 and athena_heartbeat_slots_pending > 0)
# writer behind: due slots waiting past due
athena_heartbeat_persist_lag_seconds > 30
# intake: heartbeats read vs expected (1.0 = every heartbeat arrived)
athena_heartbeat_intake_ratio < 0.9

Amber / red: see the summary table. Healthy: under 0.1 s in steady state, persist lag near 0. On PostgreSQL, Database:HeartbeatWriters (default 4) writers run in parallel. SQLite has one.

1. If intake is low, look at Inventory backlog and the Process panel's thread pool. Also check that the agents use the heartbeat interval set in Settings → Agents: an agent on a longer interval reads as low intake.
2. Check for a reconnect storm (see Agent connect p95). On SQLite a storm slows batches for a few minutes and recovers.
3. Look at the Database panel's write-lock wait. On SQLite every write goes through one writer, so heartbeats wait behind inventory and other writes.
4. Look at Database size: a WAL bigger than the database slows writes. Restarting the server forces a checkpoint.
5. If it stays above 1 s in steady state, give the server more and faster cores, and move to PostgreSQL above about 3,000 agents (or earlier if the whole fleet restarts together).

CPU

CPU used by the Athena server process as a share of all cores (100 % = every core busy). The line below shows the core count and the busiest 10-second interval of the last hour. Prometheus: athena_process_cpu_seconds_total (rate() divided by the core count).

Amber: > 85 % of all cores. Red: none. Healthy: 10 % at 500 agents and 35 % at 2,000 on the 2-core test server. Near 100 % only during a reconnect storm or bulk enrolment, for a few minutes.

1. Check what happened at the peak time: a storm, an enrolment burst (Agent registrations) or a large deployment.
2. Look at the Process panel: a high GC pause share means memory pressure, a long thread-pool queue means starvation.
3. If it stays above 85 % in steady state, add cores. See Server Sizing.

Memory

The server process working set (memory the operating system has given it) and the .NET managed heap inside it. No thresholds. Prometheus: athena_process_working_set_bytes, athena_dotnet_gc_heap_bytes.

Healthy: rises with the fleet and then levels off (about 0.7 GB at 500 agents and 1.4 GB at 2,000 in testing). It should not climb without end.

1. Compare with Agents online: memory that follows the fleet is normal.
2. Look at the Process panel: a heap that keeps growing with no fleet growth is worth reporting to support, with the times and the server version.
3. Restarting the server frees the memory but clears the 60-minute chart history.

Agent registrations

New agent enrolments per minute in the window, the p95 time of one enrolment over the last hour, and how many enrolments happened since the server started. Enrolment is the most expensive call per agent, because the server generates a key for each one. No thresholds. Prometheus: athena_agent_registrations_total{result}, athena_agent_registration_duration_seconds.

1. If enrolments fail, look at result in Prometheus: bad_token means an expired or wrong registration token, licence_limit means the licence is full.
2. If CPU is pinned during a rollout, enrol in smaller waves.

Certificate expiry

The days left on whichever server certificate expires first: the web certificate (port 8443), the agent certificate (port 8444) or the certificate authority. The Process panel lists all of them. Prometheus: athena_certificate_expiry_days{cert} (server_web, server_grpc, ca).

Amber: < 14 days left. Red: none. Healthy: more than 30 days at all times. An expired agent certificate or CA stops every agent connection. The tile names the certificate. Renew it as described in Certificates & PKI (and Appliance Web Certificate on the appliance).

Licence

Days until the installed licence expires, with the expiry date and licence type. Prometheus: athena_licence_days_remaining (absent when no licence is installed).

Amber: < 14 days left. Red: none. Install the renewed licence under Settings → License. See Licensing.

Database health

The worst result of the database health checks (integrity, foreign keys, journal mode, WAL size, free pages, disk space and, on PostgreSQL, connections, long transactions and more), how many checks have a problem, and when they last ran. The checks run at startup, once a day, and when you click Check now. Click the tile to open Settings → Database. Prometheus: athena_db_health_check_status{check} (0 ok, 1 warning, 2 critical).

Amber: any check at warning. Red: any check critical. A critical check also raises the red bar at the top of every page and a server alert.

1. Open Settings → Database and read the failing check. Each one says what it means and what to do. See Database Backup & Maintenance.
2. Integrity or foreign-key failures: stop and restore from the latest good backup before more data is written. See Database Restore & Recovery.
3. Disk space: free space or grow the disk now. SQLite can't write without it.
4. Click Check now after the fix to confirm.

Scheduler

Whether the job scheduler started: green with the start time, red with the reason when it failed. While it is down no scheduled task runs. Prometheus: athena_scheduler_up, athena_scheduler_startup_failures_total{stage}.

Amber: none. Red: the scheduler did not start. It normally starts a few seconds after the server.

1. Read the reason on the tile. Job recovery failed almost always means a damaged database: check Database health.
2. Fix or restore the database, then restart the server. The scheduler only starts with the server.
3. Afterwards, check Scheduled task failures for tasks that missed their runs.

Server internals row#

These tiles watch the server's plumbing. A red tile here makes /health report degraded.

Event bus queue

Agent events (connected, disconnected, inventory received, scan results, module messages) waiting for the parts of the server that handle them. The core and every module (for example the policy modules) have their own queue with a capacity of 10,000, so the tile shows the fullest queue by name and how full it is (for example bitlocker 62%), with the total across all queues and the events dropped in the last 5 minutes underneath. A filling queue means that module's handler is slow, usually because the database is busy, while the others keep flowing. A dropped event is an update a module silently missed. Prometheus: athena_event_bus_queue_length{module}, athena_event_bus_queue_capacity, athena_event_bus_dropped_total{event}.

The rule is per queue, never the total: four modules at 2,000 events each are four queues at 20 %, not one at 80 %. A short burst above half that drains within the minute is not a warning. Healthy: near 0 and nothing dropped.

1. Check the database first: Heartbeat batch p95 and the write-lock wait in the Database panel. Event handlers wait on the same writer.
2. Check for a reconnect storm. Short drops during a storm recover; sustained drops don't.
3. Look in the server log for a slow or failing module handler.
4. If it happens in steady state, the server is under-sized for the fleet: more cores, or PostgreSQL above about 3,000 agents.

Agent disconnects

Agent disconnects per minute by reason over the last 5 minutes. The big number is transport_error per minute: the network or TLS broke the connection. The line below shows client_closed per minute (the agent hung up, which is normal for restarts and laptops going to sleep) and the counts of server_shutdown (this server stopped) and replaced (the same agent connected again before its old connection was closed). Only transport_error is judged. Prometheus: athena_agent_disconnects_total{reason}.

Amber: transport_error > 10/min. Red: transport_error > 100/min.

1. A spike of transport_error with Agent TLS handshake p95 rising is a reconnect storm, usually after a network blip or a load-balancer timeout.
2. Check the network path to port 8444: load-balancer idle timeouts (agents keep a long-lived connection), firewalls resetting long connections, VPN drops.
3. Check that the certificates and CA are valid (Certificate expiry).
4. If replaced keeps climbing in steady state, two machines may share one identity (a cloned machine). Re-enrol the clone. See Agent Enrollment.

For how a silent connection is detected and marked offline, see Agent Status & Health.

Agent TLS handshake p95

The slowest 5 % of completed TLS handshakes on the agent port (8444 by default, Server:ApiPort) over the last 5 minutes, plus connections the server rejected or queued and handshakes that failed. Agents use mutual TLS, so every reconnect costs a full handshake. Failed handshakes are counted separately and kept out of the p95. Most are timeouts at about 10 s: a client that opened the connection and never finished TLS. Prometheus: kestrel_tls_handshake_duration_seconds, kestrel_rejected_connections_total, kestrel_queued_connections (all with server_port="8444").

# completed handshakes (what the tile shows)
histogram_quantile(0.95, sum by (le) (rate(kestrel_tls_handshake_duration_seconds_bucket{server_port="8444", error_type=""}[5m])))

# failed or timed-out handshakes in 5 minutes, by reason
sum by (error_type) (increase(kestrel_tls_handshake_duration_seconds_count{server_port="8444", error_type!=""}[5m]))

Amber: completed-handshake p95 > 1 s, or any rejected or queued connection, or any failed or timed-out handshake. Red: completed-handshake p95 > 5 s. Healthy: tens of milliseconds. Connections are only rejected or queued when a connection limit is configured; with no limit both stay 0.

1. A high handshake time is almost always a reconnect storm: check Agent disconnects for the reason and CPU for saturation.
2. If agents reconnect in a loop, find why they drop. Fixing the drop fixes the handshake load.
3. Rejected or queued connections mean a connection limit was reached: raise or remove it.
4. Failed handshakes: a timeout of about 10 s is a client that connected and stalled (an agent restarting mid-connect, a scanner, or a load-balancer health check that only opens TCP). A burst right after a server restart is reconnecting agents and clears within 5 minutes. For a steady stream, list the connections to the agent port (Get-NetTCPConnection -LocalPort 8444 on Windows, ss -tn sport = :8444 on Linux) and match the addresses against Agents. An AuthenticationException is a client certificate the server refused: re-enrol that agent.

Scheduled task failures

Failed runs of scheduled tasks (backups, cleanups, scheduled commands and deployments) in the last 24 hours, read from the task history in the database, so it survives a server restart. The line below names the active recurring or cron task that is furthest past its schedule: when it last succeeded and how often it should run. A task that has never succeeded counts from when it was created. Prometheus: athena_scheduled_job_runs_total{job_type,result}, athena_scheduled_job_last_success_timestamp_seconds{job_type}. Prometheus can't tell individual user tasks apart; the tile can.

The tile also asks the Scheduler: when it failed to start, the tile is red whatever the history says and reads scheduler down. A clean history only means nothing failed, not that anything ran. In the first seconds after boot the tile is unknown. Healthy: 0 failures, and every task within one interval of its last success.

1. Open Scheduler in the menu and open the task's history. Each failed run's error message is there.
2. A backup task: check disk space and that the backup folder is writable.
3. A task far past its interval with no failures hasn't been running at all: check the Scheduler tile, and that the task is still active.
4. An expired licence stops scheduled tasks: check Licence.

Database size

The size of the database: on SQLite the main file plus its write-ahead log (WAL), on PostgreSQL the database size. The line below shows the WAL on its own and how fast the total grew per hour (after 10 minutes of samples). The size is re-read every 5 minutes. The rule is the same as the WAL size database health check, so this tile and Settings → Database always agree. Prometheus: athena_db_size_bytes{file} (main, wal).

Amber: WAL > 1 GB, or WAL > main file once WAL ≥ 64 MB. Red: none. Healthy: a WAL that stays in megabytes, and steady growth that matches your inventory and retention.

1. WAL bigger than the main database: something is holding a long read open, so SQLite can't checkpoint. Look for a database browser with the file open, a copy or backup tool reading it, or a stuck query in the server log.
2. Restart the server to force a checkpoint. The WAL shrinks back into the main file.
3. Fast growth: check data retention, and run Optimize Database on Settings → Database in a maintenance window.
4. Watch the disk-space check under Database health while it grows.

Agent refresh queue

Agents waiting for their refresh: when an agent connects or reports, the server re-evaluates its tags, then its monitors, then its collections. There is at most one entry per agent. The line below shows the lowest value in the last 2 minutes, so the tile only turns amber when the queue stays high, not on a short burst. Prometheus: athena_agent_refresh_pending.

Amber: > 500 pending for 2 min. Red: > 2000 pending for 2 min. Healthy: near 0, with bursts up to the number of agents that just connected that drain within a minute or two. A queue near the fleet size means tags, monitors and collections are going stale.

1. Check the database: refreshes read and write it, so a slow Heartbeat batch p95 or a long write-lock wait slows them.
2. Check for a reconnect storm (every connecting agent queues one refresh).
3. Look for very large tag scripts or collection rules: each refresh evaluates all of them.
4. If it stays high in steady state, add cores or move to PostgreSQL above about 3,000 agents.

Inventory backlog

Inventory messages waiting to be saved. The server reads an agent's inventory off its connection and saves it in the background, at most 16 saves at a time (Database:InventoryWriteConcurrency). Each agent's messages are saved in the order they arrived. A newer snapshot of a module replaces an older one still waiting; this is normal and is counted as superseded. The line below shows how long the oldest message has waited, the memory the queue holds against its budget, and messages dropped in 5 minutes.

The queue has a memory budget: by default a tenth of the memory the server may use (the container limit when there is one), at least 256 MB and at most 2 GB. Inventory:IntakeBudgetMB sets it exactly. Over the budget, the agent that has waited longest loses its waiting messages (counted as dropped). Nothing stays missing: once the backlog has eased, the server asks each of those agents for exactly the modules it lost, a few agents at a time, and an agent that is offline by then sends everything when it reconnects. Heartbeats, command results and everything else keep arriving while inventory waits.

Prometheus: athena_inventory_queue_length, athena_inventory_queue_bytes, athena_inventory_queue_budget_bytes, athena_inventory_queue_oldest_age_seconds, athena_inventory_dropped_total{reason="superseded|budget"}, athena_inventory_recollect_requests_total{result="sent|offline"}, athena_inventory_write_duration_seconds.

Amber: oldest queued inventory waited > 60 s, or any inventory dropped over the budget in 5 min. Red: oldest waited > 300 s, or the queue holds ≥ 90 % of its budget. Healthy: near 0 in steady state. When many agents connect, each sends a full snapshot, so the queue rises into the thousands and drains within a few minutes.

1. Check for a reconnect storm: it drains once the storm settles.
2. Look at the Database panel's inventory write p95 and write-lock wait. On SQLite every save shares one writer with heartbeats.
3. Raise Inventory:IntakeBudgetMB above the default only if the server has memory to spare. The real fix is faster saves.
4. If it stays high in steady state, move to PostgreSQL above about 3,000 agents, or give the server faster disks.

Telemetry export

Whether the optional OpenTelemetry (OTLP) exporter set up in Settings → Telemetry is delivering. Each signal (metrics, traces, logs) is tracked on its own: the last successful export and the failures since that success. Saving the Telemetry settings starts the count again, so an old endpoint's failures never colour a new one. The tile reads Off (export not enabled), Waiting (enabled, nothing sent yet) or Healthy. Prometheus: athena_otlp_export_failures_total{signal}, athena_otlp_export_last_success_timestamp_seconds.

Amber: an export failed after the last successful one, or none has succeeded yet and one failed. Red: an export has been failing continuously for more than 5 min, or every enabled signal is failing.

Read the error on the tile or in Settings → Telemetry. "Timed out" to localhost usually means localhost resolved to IPv6 (::1) while the collector listens on 127.0.0.1; use 127.0.0.1. "HTTP 401" or "gRPC status 16" means the headers (API key) are wrong. "Unreachable" means nothing listens at that address and port. See Telemetry Export (OpenTelemetry).

User experience#

What the people using the console actually felt, measured in their browsers and collected over each browser's live connection to the server (the browser pushes nothing on its own). One wide tile with four measurements; the worst of the four sets its colour:

The ux section of GET /api/metrics/summary carries the same numbers since server start; the tile uses only the last 5 minutes. Prometheus: athena_ui_navigation_duration_seconds, athena_ui_page_load_seconds{phase="first_interactive"}, athena_ui_circuit_latency_seconds, athena_ui_errors_total, athena_ui_slow_navigations_total.

# navigation p95 per page (the tile shows the worst one)
histogram_quantile(0.95, sum by (le, http_route) (rate(athena_ui_navigation_duration_seconds_bucket[5m])))

# UI errors and slow navigations in 5 minutes, by page
sum by (http_route) (increase(athena_ui_errors_total[5m]))
sum by (http_route) (increase(athena_ui_slow_navigations_total[5m]))

Healthy: navigation p95 of a few hundred milliseconds and no UI errors at all. Over a real network, add the round trip to every number. One error is worth reading in the log.

1. A slow page: the tile names it. A page waits for the REST calls and database reads behind it, so check REST p95 and the database tiles for the same minutes.
2. High circuit latency: either the network between users and the server (compare with a user on the server's own LAN) or a busy server (check CPU and the thread-pool queue under Process).
3. Slow page load with normal navigation: the first load downloads the console and opens its live connection. Look at the network path, and at a proxy that buffers or blocks WebSockets.
4. UI errors: find them in the server log by page. A js_error is also in the user's browser console.

Related#

• System Health & Prometheus Metrics — opening the tab, panels, /metrics, scrape token and REST API.
• Server Health & Monitoring — /health and /health/details.
• Telemetry Export (OpenTelemetry) — the exporter behind the Telemetry export tile.
• Server Sizing & Hardware Requirements — when the tiles say the server is too small.
• PostgreSQL Database — for large fleets.
• Database Backup & Maintenance — the checks behind Database health.