Telemetry Export (OpenTelemetry / OTLP)

The Athena server can send its own metrics, traces and logs over OTLP, the OpenTelemetry protocol, to an OpenTelemetry Collector or any other backend that accepts OTLP. You set it up in Settings → Telemetry. It is an extra output only: the Prometheus /metrics endpoint and the System Health tab keep working as before. Changes apply when you save. No restart is needed.

For the numbers the server shows in its own console and on /metrics, see System Health & Prometheus Metrics. For the server log files, see Server Logging.

Who can see and change it#

A custom role with settings.health.view but without settings.manage sees the form as read-only, with the note "Read-only: changing these settings needs settings.manage." See Permission Model & Role Bundles.

Turning it on#

1. Open Settings and click the Telemetry tab (right after System Health), or go to https://<server>:8443/settings/telemetry.
2. Tick Export telemetry over OTLP.
3. Enter the collector's Endpoint and pick the Protocol.
4. If your collector needs an API key or other header, enter it in Headers.
5. Choose which Signals to send: Metrics, Traces and Logs. All three are on by default.
6. Click Test connection. This sends one test span and one metric point using the values in the form, before they are saved.
7. Click Save. The toast reads "Telemetry settings saved and applied." and the exporter starts at once.

Settings#

Every export also carries service.version (the server version) and service.instance.id (the server's machine name in lower case). Each export request times out after 10 seconds.

If a value is not valid, Save shows one of these messages and nothing is changed:

• The endpoint must be an http:// or https:// URL.
• The endpoint must not have a query string.
• The trace sampling ratio must be between 0 and 1.
• The metric export interval must be 1 to 3600 seconds.
• Headers must be key=value pairs separated by commas.

Headers are write-only#

• Headers usually hold an API key, so they are stored encrypted and are never shown again. Once saved, the field is empty and its placeholder reads "Stored - leave blank to keep".
• Leave the field blank when you save to keep the stored headers. Type a new value to replace them.
• To delete them, tick Remove the stored headers and save.
• Over REST, GET returns ******** when headers are stored. Sending ******** or a blank value keeps them. "clearHeaders": true removes them.
• If the server can no longer decrypt the stored headers (for example after its key ring was replaced), it logs a warning and exports without them. Enter the headers again and save.

What is sent#

Only what the server is doing is sent, not the data it manages. Each signal is filtered before it leaves the server.

Export status#

The Status row on the Telemetry tab updates live after every export attempt. It shows the overall state, the endpoint, the time of the last successful export, and the failures since that success. Below it, one chip per enabled signal shows that signal's state and its failure count. While an export is failing, Last error shows the most recent error.

• A toast appears once when export starts failing ("Telemetry export is failing: …") and once when it recovers ("Telemetry export recovered."). You do not get a toast for every retry.
• Saving the settings resets the counts, so an old endpoint's failures never show against a new one.
• The Telemetry export tile on Settings → System Health shows the same state. It reads "OTLP export is off" when the exporter is off.
• For alerting from Prometheus, /metrics publishes athena_otlp_export_failures_total{signal} (a lifetime total, not reset on save) and athena_otlp_export_last_success_timestamp_seconds.

# Any OTLP export failures in the last 5 minutes
increase(athena_otlp_export_failures_total[5m])

Environment variables and appsettings.json#

The standard OpenTelemetry OTEL_* environment variables are honoured. For each field, an environment variable wins over the Settings tab, and the tab wins over the default. A field set by a variable is read-only in the tab and shows "Set by environment variable …", so a container's configuration cannot be overridden by accident from the console.

Log minimum level has no environment variable. It always comes from the tab or from configuration.

You can also configure the exporter in the Telemetry:Otlp section of appsettings.json, or with Telemetry__Otlp__* environment variables in a container. The keys are Enabled, Endpoint, Protocol, Headers, Metrics, Traces, Logs, TraceSampleRatio, LogMinimumLevel, ServiceName, DeploymentEnvironment and MetricExportIntervalSeconds. A plain-text Headers value set this way works. The tab always stores headers encrypted.

{
  "Telemetry": {
    "Otlp": {
      "Enabled": true,
      "Endpoint": "http://otel-collector.corp.local:4317",
      "Protocol": "grpc",
      "ServiceName": "athena-server",
      "DeploymentEnvironment": "production"
    }
  }
}

# docker-compose.yml: the same, with the standard OpenTelemetry variables
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
      - OTEL_SERVICE_NAME=athena-server
      - OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production

• When you save in the tab, the settings are written to the server's appsettings.local.json in its data directory.
• Per server. Settings Sync does not copy telemetry settings, so each server keeps its own endpoint and its own API key.
• If the exporter cannot start at server start-up (for example a malformed endpoint), the server logs the error and keeps running without the exporter. When it starts, the server log shows Telemetry: OTLP export ON to … with the endpoint, protocol and signals.

REST API#

$headers = @{ Authorization = "Bearer $token" }
$t = (Invoke-RestMethod "https://athena.contoso.com:8443/api/settings/telemetry" `
  -Headers $headers -SkipCertificateCheck).data
$t.status | Select-Object state, endpoint, lastSuccessAt, failures, lastError
$t.status.signals | Format-Table signal, state, failures, lastError

See API Reference for signing in and getting a token.

Audit trail#

Each save writes one SettingsChanged event per changed field, with the old and new value and the user who saved. A change to the headers is recorded only as [Hidden] → [Changed]. The header value itself is never recorded. See Audit Event Types Reference.

Troubleshooting#

Related#

• System Health & Prometheus Metrics: the same counters in the console and on /metrics.
• Server Health & Monitoring: up/down health checks.
• Server Logging: the server's own log files and retention.
• Audit Forwarding to SIEM: send the audit trail to syslog.
• Data Directories & Backup: where saved settings live.