Agent Status & Health

Every machine in your fleet carries a status that tells you, at a glance, whether Athena can reach it and whether it needs attention. This page explains exactly how that status is worked out — the difference between the raw status stored for an agent and the derived Online / Offline / Warning / Maintenance buckets shown on the Dashboard and the fleet view — how the agent's live connection and the heartbeat-timeout window decide online versus offline, how a machine that goes silent (a laptop closing its lid, a pulled cable) is detected, and a practical runbook for the most common question of all: "why does my agent show offline?"

The raw agent status#

Each agent record stores a single status value that reflects its lifecycle and its last reported condition. You will see these on an agent's row and on its details view:

The raw status is only half the story. When Athena summarises your fleet — the Dashboard cards, the status filter, and the agent statistics — it combines this stored status with two extra signals: how recently the agent last checked in and whether the agent is in maintenance mode. Those combined results are the buckets described next.

Derived status buckets#

The summary buckets are what you filter and count by. They are computed on the server every time you load the fleet, so they always reflect the live picture rather than a value written once and left to go stale.

The heartbeat window#

Agents hold a long-lived connection to the server and prove they are alive by sending a lightweight heartbeat over it on a fixed cadence. While that connection is up, the agent is Online. When the agent is not connected, the server falls back to the heartbeat age: the agent counts as online only if a heartbeat has arrived recently enough — that "recently enough" span is the heartbeat-timeout window. The same two settings also decide how long a silent connection is tolerated:

With the shipped configuration, agents check in roughly once a minute (every 59 seconds) and the server tolerates a gap of 600 seconds (10 minutes) — about ten missed heartbeats — before treating a machine as gone. That headroom absorbs dropped check-ins and brief network blips without the fleet flickering between online and offline. Both values are edited under Settings → Agents → Heartbeat Settings, where the interval accepts 5–300 seconds and the timeout must be at least three times the interval (up to 900 seconds); raising the interval automatically lifts the timeout to that minimum.

When a connection goes silent#

An agent that shuts down cleanly, or loses its network in a way the server can see, ends its connection and is marked Offline at once. Some machines vanish without saying goodbye — a laptop that goes to sleep or a VM that is suspended can leave its connection half-open. Athena handles that in two layers, so a silent machine cannot stay Online indefinitely:

The silent-connection limit is read when an agent connects, so after you change the heartbeat settings it applies to each agent from its next connection. When the machine wakes up, the agent reconnects and its next heartbeat brings it back to Online.

Each time the limit ends a connection, the server log records a warning of the form Agent <hostname>: no message for 600 s - ending the stream as offline (see Server Logging), and the Prometheus counter athena_agent_disconnects_total is incremented with reason="heartbeat_timeout" (see System Health & Metrics). A steady stream of these is normal for a fleet of laptops; on always-on servers it points to network or power problems.

Live connection cross-check#

Heartbeats are recorded to the database in small batches, so for a moment or two after a burst of check-ins the stored figures can lag reality. To avoid under-reporting, the server also knows which agents hold a live connection to it right now, and cross-checks the two: if more agents are actively connected than the database has yet recorded as online, the live connection count wins. The online figure is never allowed to exceed the total number of registered agents, which keeps the offline count from ever going negative during that brief write delay.

The practical upshot: the online/offline numbers you see are the more optimistic of "what the database last recorded" and "who is connected this instant," so a healthy agent that just reconnected is reflected immediately rather than after the next batch write.

For an agent whose state has not changed, the server saves its last-heartbeat time to the database at most every 5 minutes (Agent:HeartbeatPersistSeconds, default 300), while each agent's own row and details view always show the latest heartbeat. To compensate, the fleet counts and the Online/Offline status filter accept a stored heartbeat up to the timeout plus that save interval old. A real disconnect is saved immediately, so this allowance never delays an agent going offline.

How the fleet stays current#

The Dashboard and fleet view update in real time. They subscribe to agent heartbeat events over a live server connection, so a status pill flips the moment an agent comes online or drops off — no manual reload and no polling timer. If the console loses that live connection, the page shows a Live updates paused – reconnecting badge until it is restored; see Dashboard.

Checking status#

Beyond the console, you can read status programmatically. The fleet listing accepts a status filter, and there is a dedicated statistics endpoint that returns the derived counts.

REST API#

List agents filtered by a derived bucket, or fetch the fleet counts in one call:

# Only the machines that are currently offline
GET /api/agents?status=Offline

# Fleet totals: total, online, offline, maintenance, warning
GET /api/agents/stats

The status filter understands Online, Offline, Warning, Maintenance, Pending, and the raw statuses. Combine it with the os filter and a search term to narrow further; see the API Reference.

PowerShell#

The Athena module exposes the same filters and the fleet stats:

# All offline agents
Get-AthenaAgent -Status Offline

# Offline Linux machines only (server-side OS filter)
Get-AthenaAgent -Status Offline -OS Linux

# Fleet health at a glance
Get-AthenaAgentStats

Get-AthenaAgent -Status accepts Online, Offline, Pending, and Maintenance. To find machines whose certificate is close to expiry, use the certificate tooling in Certificates & PKI (for example the expiring-certificates report), which is what the Warning bucket is built from.

Runbook: an agent shows offline#

"Offline" simply means the server has not heard a recent heartbeat. Work through these checks in order — most offline agents are resolved by the first two.

1. Confirm the machine and the agent service are actually running. A powered-off or sleeping host, or a stopped agent service, is the most common cause. On Windows the agent runs as a service; see Windows Agent (and Linux Agent) for how to check and restart it.
2. Check the last-heartbeat time. On the agent details view, a last heartbeat about one timeout window (10 minutes with the shipped settings) before the agent went offline points to a machine that went to sleep or lost its network silently — see When a connection goes silent; a heartbeat of Never means the agent never completed a successful check-in after enrollment.
3. Verify network reachability to the server. The agent must be able to reach the server's agent-connection port. Confirm DNS, routing, and firewall rules along the path; see Network Ports & Firewall.
4. Rule out a revoked or expired certificate. An agent whose certificate was revoked can no longer connect and will settle as offline. A certificate nearing expiry shows up under Warning first; agents auto-renew ahead of expiry, but a machine that was offline through its renewal window may need attention. See Certificates & PKI.
5. Read the agent's own logs. The agent records why it cannot connect — certificate, TLS, or network errors are all logged locally and can be streamed to the console. See Agent Logs.
6. Consider whether the timeout is simply too tight. Laptops that sleep overnight are expected to go offline — that is correct behaviour, and they return to Online when they wake. But if always-on machines flicker offline, check the server log for “no message for … s” warnings; if their real check-in gaps are longer than the window, raise the Heartbeat Timeout as described above.