Staged Agent Rollouts

With automatic updates on, a newly activated agent version doesn't have to reach the whole fleet at once. Turn on staged rollout and Athena sends the version to a canary group first, then to everyone else in waves. If too many agents in a wave fail, the rollout halts on its own, so a bad build stops at a few machines instead of reaching all of them. This is the pilot-then-production client upgrade pattern from System Center, built into the server.

Turning staging on#

Go to Settings → Agents → Updates. The Staged rollout section is below the auto-update options. Tick Stage agent updates (canary first, then waves), fill in the fields, and click Save in the section header. The other fields are greyed out while the box is cleared. With staging off, an activated version is pushed to every older online agent at once.

An out-of-range value isn't saved. You get an error that names the field and its allowed range, for example RolloutWaveDelayMinutes must be 0-10080.

How a rollout moves#

A rollout starts when you activate an agent binary while both auto-update and staging are on. Each operating system has its own rollout.

1. Plan. The server lists every registered agent of that OS on an older version. Offline agents are included. If none are older, no rollout is created. Matching agents in the canary collection or with the canary tag form the canary wave. The rest are split into wave 1, wave 2 and so on, by the wave size you set.
2. First wave. The first wave starts immediately. Its online agents get the update right away. Its offline agents get it when they reconnect.
3. Track. About every 30 seconds the server checks each agent in the waves that have started, and records an outcome for it (see Wave outcomes).
4. Next wave. The next wave starts only when (a) every agent in the current wave has an outcome and (b) Minimum time between waves has passed since the current wave started. If a wave takes longer than the delay to finish, the next one starts as soon as it finishes.
5. Complete. When the last wave has finished, the rollout is Completed. From then on, auto-update handles any remaining older agent of that OS normally when it connects, including agents enrolled after the rollout was planned.

While a rollout is running, only agents in a wave that has started can get the version automatically. An agent in a later wave, or one enrolled after the plan was made, waits.

Wave outcomes#

An agent that goes offline partway through an update is only marked failed when the wait time runs out, because no failure has been recorded for it yet.

The automatic halt#

The server checks the current wave's failure rate on every check, without waiting for the wave to finish. The rate is the number of failed agents divided by the number of agents in the wave that were attempted (pushed to, or failed). When at least one agent has failed and the rate is above Stop when a wave's failures exceed, the rollout changes to Halted. The reason is shown, for example canary: 2 of 5 agent(s) failed, over the 20 % threshold.

• At 0 %, one failure halts the rollout.
• At 100 %, the rollout never halts.
• While a rollout is halted, no agent gets that version automatically. That includes agents in a wave that already started and are just coming back online.

Watching and controlling a rollout#

Under the fields, Rollouts lists the latest rollout for each operating system, shown as the OS, the version and a status badge. It updates live, so you don't need to refresh the page. If there is no rollout yet, it shows No rollout yet. One starts when a new agent version is activated with auto update and staging on. Each rollout has a table with one row per wave: Wave (its name and agent count), Done, Failed, Skipped, Pending and Started. The current wave's row is marked. While a rollout is running, a line shows when the next wave can start.

• Pause works only on a running rollout.
• Resume works on a paused or halted rollout. On a halted rollout it overrides the halt: more failures in that wave won't halt it again, and the next wave starts under the normal rules. A later wave can still halt the rollout.
• Abort ends the rollout permanently. Agents that already updated keep the new version. The aborted version isn't pushed automatically again. Only activating a version for that OS starts a new rollout, and a manual push still works.

If you activate another version for an OS that already has an active rollout, the old rollout is aborted with the reason replaced by the rollout of <version>, and the new rollout takes its place. The server keeps only the latest rollout for each OS.

Server restarts#

Rollout progress is saved in the server's data directory as agent-rollouts.json. A restart in the middle of a rollout doesn't re-push to the fleet, and a halted rollout stays halted. Include this file in your data-directory backups. If the file can't be read when the server starts, the error goes to the server log and the server starts with no rollouts. A halt that was in place is lost, and that version is again pushed by normal auto-update. Rollout state belongs to one server, so this feature assumes a single Athena server node.

Audit events#

Every change of state is written to the audit log with resource type AgentRollout:

Events the server raises by itself (wave started, halted, completed) show the user System (staged rollout). Wave pushes appear in the update history with the same name in Initiated By.

Configuration#

The Settings fields are stored under AgentUpdate:Rollout in the server configuration. You can also set them in appsettings.json or as environment variables, for example AgentUpdate__Rollout__Enabled=true.

Roles#

See Permission Model.

REST API#

The control calls return the updated rollout. An unknown ID returns 404 (Rollout not found). An action the current status doesn't allow returns 409, for example Cannot pause a rollout that is halted. Each rollout has id, os, version, status (running, paused, halted, aborted or completed), currentWave, startedAt, startedBy, finishedAt, reason, nextWaveAt and waves. Each wave has index, name, startedAt, settledAt, total, done, failed, skipped, pending and failedAgentIds (up to 20 failed agents to look at first).

PowerShell#

Module version 2.14.0 and later includes four cmdlets. All of them support -WhatIf:

# See every rollout and its per-wave progress
Get-AthenaAgentRollout | Select-Object -ExpandProperty Waves

# Pause whatever is running
Get-AthenaAgentRollout | Where-Object Status -eq running | Suspend-AthenaAgentRollout

# Look at a halt, then override it and continue
$r = Get-AthenaAgentRollout | Where-Object Status -eq halted
$r.Reason
Resume-AthenaAgentRollout -Id $r.Id

See the PowerShell Module reference for full cmdlet details.