Settings Sync (multi-server)

Settings Sync lets one Athena server push its configuration to other Athena servers over TLS, so you can keep a whole estate of on-premises servers configured the same way from a single place. Nominate a hub, list the servers it should push to, choose which settings groups travel, and every change flows outward — automatically after each save, or on demand. Downstream servers can pass what they receive on to their peers, so a change made at headquarters reaches branch and sub-branch servers in one motion. There is no polling and no cloud broker: it is server-to-server, on your network.

Who can configure it

The Settings Sync tab and every peer/token action require the Admin role (the settings.manage permission). See Roles & Permissions. Configuration lives under Settings → Sync in the web console, and the same operations are available through the REST API and the PowerShell module.

How it works#

Settings Sync is a push model. A parent server builds a bundle of selected settings and posts it to each peer (a downstream server) over HTTPS. The peer applies the bundle to its own configuration and — if cascading is enabled — pushes it on to its own peers. Nothing is ever pulled; a server only ever receives a bundle another server chose to send it.

Term	Meaning
Parent	A server that pushes its settings out. Any server can be a parent simply by adding peers.
Peer (downstream)	A server this one pushes to. It is identified by a name and base URL, and authenticated with a sync token the peer issued.
Sync token	A long-lived bearer credential a server issues so a parent may push to it. Only its hash is stored; the cleartext is shown once at creation.
Bundle	The payload a parent pushes: a set of settings groups plus a small header used for loop detection and change tracking.
Instance ID	Each server's stable identity in the fleet. It is used to detect and break cascade loops, and must be unique per server.
Cascade	When a peer re-pushes a bundle it received on to its own peers, enabling A → B → C topologies.

Direction of the token matters

A sync token is issued on the server that will receive the settings, and pasted into the parent that pushes them. In other words: the child hands a token to the parent. Getting this backwards is the most common setup mistake — the token you create on your own Sync tab is for a server upstream of you, not for the peers you list below it.

What is synced — and what is not#

You choose which settings groups each peer receives. A bundle only carries the groups you tick; everything else is left exactly as the peer had it. The available groups are:

Group	Covers
General	Application name and other general/display settings.
Security	Security settings such as password policy and session behaviour.
Agents	Agent-wide configuration defaults.
Scheduler	Scheduler settings.
Logging	Server logging configuration.
Syslog	Audit forwarding / syslog settings.
LDAP	LDAP / Active Directory connection and role mapping.
OpenID Connect	Keycloak / OIDC single sign-on settings.

Some things are deliberately never synced

A bundle never carries the peer list itself, the database configuration, this server's own URLs and file paths, or its certificates. That is by design: syncing the topology would let one server rewrite the whole fleet, and machine-specific values (like where a server lives on disk) must never be overwritten from elsewhere. Each server keeps its own identity while sharing policy.

Secret values inside a synced group — the LDAP bind password and the OIDC client secret — are included when a bundle is actually pushed to a peer, so the peer ends up with a working configuration. They travel only over the direct, authenticated TLS connection between the two servers. A read-only preview of the bundle (see the API) always masks those secrets instead.

The Sync tab#

Settings → Sync is laid out so the two directions of sync can never be confused:

Status strip (top) — shows at a glance whether sync is on (Sync enabled / Sync disabled), how many destination servers are configured, whether pushes are automatic (Automatic push on changes) or Manual push only, and — under THIS SERVER — this server's Instance ID.
Outgoing peers sub-tab — the servers this one pushes its settings to, shown as cards with each peer's last-push outcome, plus the Push now and Add peer buttons.
Incoming access sub-tab — the sync tokens this server has issued so that an upstream server can push settings to it.
Server behavior panel (beside the list) — the switches that control how this server participates, described below.

Adding or editing a peer and issuing a token both open in a dialog. Outcomes of Save, Delete, Test, Push now, Issue token and Revoke appear as short notifications; a failed push or test stays on screen longer (about 12 seconds) because it carries a reason you will usually need to act on.

Setting (Server behavior)	What it does
Enable settings sync	Master switch. When off, this server neither pushes nor accepts a pushed bundle.
Push when settings change	Automatically push shortly after a synced settings group is saved on this server. Changes made in quick succession are collapsed into a single push.
Cascade to my own peers	After applying a bundle received from a parent, push it on to this server's own peers. This is what makes A → B → C work.
Debounce (seconds) (under Advanced settings)	How long to wait after the last change before an on-change push fires, so a burst of edits becomes one push. Defaults to 5 seconds.
Instance ID	This server's identity in the sync path, shown in the status strip as plain text. It cannot be edited in the console: two servers sharing an ID would defeat the loop guard that stops a bundle circling the fleet.

Pushes happen on change, on demand, or on a schedule — never by polling

With Push when settings change on, a save triggers a single debounced push a few seconds later. You can also push explicitly with the Push now button (or the API / PowerShell), or on a timetable with a scheduled job (see Scheduled pushes). In every case the receiver is a passive endpoint — it never reaches out to or polls a parent.

Scheduled pushes

To re-send your settings on a regular timetable — for example a nightly push that brings any branch back into line — create a job under Scheduled Jobs and choose the Settings Sync action (Push this server's settings to its downstream peers). Creating the job requires the scheduler.manage permission. Each run pushes to every enabled peer, sending each peer only the groups ticked for it. The run counts as successful only if every peer accepted the push; when any peer fails, the run is recorded as failed and its error lists each failing peer by name with the reason, so a partially delivered policy is visible in the job's run history.

Setting up a link#

To make Server A push to Server B, you configure both ends once. All of this is on the Settings → Sync tab of each server.

On Server B (the receiver): open the Incoming access sub-tab and click Issue token. Give it a description (e.g. push from HQ) and an optional validity in days (blank = never expires). When the token is issued the dialog closes and the token is shown on the page — copy it immediately. Only its hash is stored, so it is shown exactly once.
On Server A (the sender): open the Outgoing peers sub-tab and click Add peer. Enter a name (a label only — it is not sent anywhere), Server B's base URL (including scheme and port, e.g. https://branch-01:8443), and paste the token you copied from Server B. Tick the settings groups you want Server A to push to Server B.
Test it: click Test on the peer card. Reachable proves Server B answered and accepted the stored token; Not reachable comes with the reason (for example a rejected token, an HTTP error, or a connection/certificate failure).
Enable sync on both servers (master switch on), then either turn on Push when settings change on Server A or click Push now, which immediately pushes to every enabled peer.

When you later edit a peer, its token box shows A token is stored - leave blank to keep it: leave it empty to keep the existing token, or paste a new one to replace it. A peer you switch to disabled is skipped by every push — manual, automatic and scheduled.

Network path

The parent connects to each peer's normal HTTPS address. Make sure the peer's base URL is reachable from the parent and that any firewall allows it — see Network Ports & Firewall. Because the token rides an Authorization header over TLS, the peer must present a certificate the parent trusts. That is either a CA the parent's operating system trusts, or a receiver CA uploaded on the parent under Settings → Sync → Outgoing peers → Trusted CAs — see Trusting a peer server.

Topologies#

Pattern	How to build it
Hub and spoke	One central server lists every branch as a peer and pushes to them all. Leave cascade off on the branches. Best when every branch is reachable from the hub.
Cascade (A → B → C)	The hub pushes to regional servers; each regional server has Cascade to my own peers on and lists its own downstream branches. A single push at the hub flows to the whole tree. Useful when a sub-branch is only reachable from its region, not from HQ.
Scoped fan-out	Push different groups to different peers — e.g. LDAP/OIDC to every server for consistent sign-in, but Scheduler and Agent settings only to a subset.

Loops are broken automatically

Every bundle records the list of servers it has already passed through (by Instance ID). A server that sees its own ID in that list refuses the bundle, so even a mis-wired topology that forms a ring cannot send a change round and round forever. This is why each server's Instance ID must be unique and stable across restarts.

What happens on the receiver#

When a peer accepts a bundle it applies each included group to its own configuration and reports, group by group, what it applied. A whole bundle can be refused for a few clear reasons — and a refusal is reported back to the pusher rather than silently dropped:

Outcome	Meaning
Applied	The bundle was accepted; each group's result says whether it was written or skipped.
Refused — loop detected	The receiver's own Instance ID was already in the bundle's path, so it declined to re-apply it.
Refused — sync disabled	The receiver's master switch is off, so it does not accept pushes.
Refused — unsupported version	The bundle's wire format is one the receiver does not understand.
Refused — revision mismatch	The bundle's contents do not match the fingerprint it was sent with, so it is treated as corrupted or tampered with and nothing in it is applied.
Refused — not licensed	The receiver's license does not include settings sync (or has lapsed). The receiver answers HTTP 403 with its own explanation, which the sender shows as-is. See Licensing.
Rejected — no valid token	No valid sync token was presented, so the push was never authenticated.

What the receiver records

Every push that reaches a receiver leaves one row in the receiver's audit log, recorded under the user name settings-sync:

An applied push (event SettingsSyncApplied) names the sending server by its organisation name, its URL and its Instance ID, gives the bundle revision and how many groups were applied, and then spells out what actually changed, field by field — for example Agent: HeartbeatIntervalSeconds: 151 -> 173. If nothing differed, it says No values differed from this server's current settings. The row is a warning rather than informational when some groups failed to apply.
A refused push (event SettingsSyncRejected) names the sender and its Instance ID together with the refusal reason from the table above.

Secret values are never written to the audit log. When a synced password, secret or key genuinely changes, the row records only that it changed — Ldap: BindPassword: changed (value withheld). A push that re-sends the same secret is not reported as a change, so a value withheld entry always means a real rotation. The sending server records its own side of every push too (event SettingsSyncPushed), including the failure reason when a push does not succeed. Changes to the Sync settings themselves are audited like any other settings change.

If someone has the Settings page open on the receiver when a push is applied, the page reloads its values by itself and shows the banner These settings were just updated by a sync push from another server. The page has been refreshed. — so nobody keeps editing stale values.

Troubleshooting a failed push#

A sync failure always involves two servers, so every failed-push message names the peer, states the cause, and says which server to fix. The same message appears in the notification, on the peer card's last-push line (next to its Push failed badge), in Invoke-AthenaSyncPush / API results, in the sender's audit log, and in a scheduled job's run history. Where the peer returned a response body or the connection raised an error, the original text is appended so it can be quoted in a support ticket.

What the message says	Cause	Where to fix it
Peer 'name' rejected the sync token (HTTP 401/403)	The token was revoked, expired, or issued by a different server.	On the peer: issue a new token under Settings → Sync (Incoming access), then edit the peer on the sender and paste it in.
Peer 'name' refused the push (HTTP 403): reason	The peer refused for its own stated reason — most often that its license does not cover settings sync.	On the peer, as the quoted reason says. Re-issuing the token will not help.
The receiving server 'host' is not accepting settings pushes	Settings Sync is turned off on the receiver.	On the receiver: Settings → Sync → turn Enable settings sync on, then push again.
Peer 'name' answered HTTP 404 at URL	Something answered at that address, but it has no settings-sync endpoint.	On the sender: check the peer's base URL points at an Athena server running a version that supports sync.
Peer 'name' refused the bundle as a loop (HTTP 409) / Loop detected	The bundle already passed through that server — the topology forms a circle, or two servers share an Instance ID.	Remove the peer entry that closes the circle, or give the cloned server its own Instance ID.
Peer 'name' refused the bundle as too large (HTTP 413)	The bundle exceeded what the peer accepts.	On the sender: push fewer settings groups to that peer.
Peer 'name' failed while applying the bundle (HTTP 5xx)	An error on the peer while applying.	On the peer: check that server's log for the matching error — the sender is not at fault.
Could not establish TLS with peer 'name' … its certificate is not trusted	The sender does not trust the peer's certificate authority.	On the sender: download the peer's CA from `https://<peer>/api/pki/ca/download?format=pem` and upload it under Settings → Sync → Outgoing peers → Trusted CAs. No OS trust store change or restart is needed. If it is already listed, check that the peer URL's host name matches the peer's certificate. Full steps: Trusting a peer's certificate.
Could not resolve 'host' for peer 'name'	The sender cannot turn the peer's name into an address. This is a DNS problem, not a certificate problem.	On the sender: add a DNS record or a hosts-file entry. Do not switch to the IP address — the peer's certificate is issued for its name.
Peer 'name' refused the connection on port N	The name resolved and the host answered, but nothing is listening on that port.	On the peer: confirm Athena is running and the port is open in its firewall.
Could not reach peer 'name' … the connection timed out	Usually a firewall silently dropping the traffic.	Check the network path and firewalls between the two servers — see Network Ports & Firewall.
No usable sync token is stored for this peer	The sender has no token for the peer, or can no longer read the one it stored (for example after its data-protection keys changed).	On the sender: edit the peer and paste the token again (issue a fresh one on the peer if you no longer have it).

Test gives a shorter answer than a push

The peer card's Test button reports a briefer result — The peer rejected the sync token, The peer answered HTTP N, or Could not reach the peer followed by the underlying error. If a test fails and the cause is not obvious, run Push now once: the push result carries the full explanation from the table above.

Managing sync tokens#

The Incoming access sub-tab (Sync Tokens) lists the tokens this server has issued for parents to push to it. For each token you can see its description, who created it, when, its expiry (blank means it never expires), how many times it has been used and when last used — but never the cleartext, which exists only at the moment of creation.

Revoke a token to cut a parent off immediately: a revoked token is refused on the very next push, even if it had not yet expired. The record is retained (marked revoked) so the audit trail survives. Issue a fresh token and update the parent's peer entry to restore the link. Rotating tokens periodically is good practice; setting a validity in days makes a token self-expire.

PowerShell#

The Athena PowerShell module exposes the whole feature, which makes it easy to script fleet-wide setup. All of these require the Admin role. Connect to each server with Connect-Athena first.

Cmdlet	Purpose
`Get-AthenaSyncSettings`	Read this server's sync configuration.
`Set-AthenaSyncSettings`	Update it (e.g. `-Enabled`, `-PushOnChange`, `-CascadeEnabled`, `-DebounceSeconds`).
`New-AthenaSyncToken`	Issue a token on a receiver (`-Description`, optional `-ValidityDays`). Prints the cleartext once.
`Get-AthenaSyncToken`	List issued tokens (hash-only).
`Revoke-AthenaSyncToken`	Revoke a token by ID.
`New-AthenaSyncPeer`	Register a downstream peer (`-Name`, `-BaseUrl`, `-Token`, optional `-Groups`).
`Get-AthenaSyncPeer`	List peers and their last-push status.
`Set-AthenaSyncPeer`	Update a peer.
`Remove-AthenaSyncPeer`	Remove a peer (this server stops pushing to it).
`Test-AthenaSyncPeer`	Probe reachability and token validity for one peer.
`Invoke-AthenaSyncPush`	Push now — to one peer with `-PeerId`, or to every enabled peer.

Set up a one-way HQ → branch link end to end:

# On the BRANCH (receiver): issue a token for HQ to use
Connect-Athena -BaseUrl https://branch-01:8443
$t = New-AthenaSyncToken -Description "push from HQ" -ValidityDays 365
$t.Token   # copy this — it is shown only once

# On HQ (sender): register the branch as a peer, pushing only sign-in settings
Connect-Athena -BaseUrl https://hq:8443
Set-AthenaSyncSettings -Enabled $true -PushOnChange $true
New-AthenaSyncPeer -Name "branch-01" -BaseUrl "https://branch-01:8443" `
    -Token $t.Token -Groups Ldap,Oidc

# Send the current settings now, and report any failures
Invoke-AthenaSyncPush | Where-Object { -not $_.Success } |
    Format-Table PeerName, Message

REST API#

Every action is also available under /api/settings/sync and requires the settings.manage permission. See the API Reference for authentication.

Method & path	Purpose
`GET /api/settings/sync`	Read the sync settings.
`PUT /api/settings/sync`	Update the sync settings.
`GET /api/settings/sync/peers`	List downstream peers (tokens never returned).
`POST /api/settings/sync/peers`	Add a peer.
`PUT /api/settings/sync/peers/{id}`	Update a peer. Omit the `token` field (or send it masked) to keep the stored token; send a new value to replace it; send an empty string to clear it.
`DELETE /api/settings/sync/peers/{id}`	Remove a peer.
`POST /api/settings/sync/peers/{id}/test`	Test one peer (returns a structured pass/fail result).
`POST /api/settings/sync/push`	Push to all enabled peers, or one with `?peerId=`.
`GET /api/settings/sync/tokens`	List issued tokens (hash-only).
`POST /api/settings/sync/tokens`	Issue a token (cleartext returned once).
`DELETE /api/settings/sync/tokens/{id}`	Revoke a token.
`GET /api/settings/sync/bundle`	Preview the bundle this server would push (secrets masked).

A peer's test and a failed push return HTTP 200 with a structured result carrying success: false and a reason, so scripts get the failure detail rather than a generic error. A bundle the receiver refuses comes back as HTTP 409 with the rejection reason.

Configuration file#

The Sync settings also live in the server's appsettings.json under a SettingsSync section, and can be seeded there before first run. The peer list and issued tokens are stored in the database, not the file, so they are never part of a pushed bundle.

"SettingsSync": {
  "Enabled": false,
  "PushOnChange": false,
  "CascadeEnabled": true,
  "DebounceSeconds": 5,
  "InstanceId": ""
}

Changes made in the console take effect without restarting the server. If you set values in the file, use the web console or the cmdlets above once the server is running to add peers and issue tokens.

Air-gapped friendly

Settings Sync is server-to-server on your own network with no external dependency, so it works the same in a fully air-gapped deployment as it does on a connected LAN. It is a good fit for multi-site organisations that run several isolated Athena servers but want one consistent policy across them.