Monitoring and Operations

Metrics collection

Prometheus, running on the coordinator, collects metrics from every component of the system at intervals of 2–15 s. Services that export metrics register with Consul, and Prometheus discovers them through the Consul catalogue. Newly scheduled imagers and exporters are therefore monitored without changes to its configuration (Reddy et al. 2024).

Source	Interval	Metrics
epic++ imagers	2 s	Execution time of each pipeline stage, GPU kernel time, packet-loss statistics, current tuning, and the full runtime configuration as labels
Nomad agents	2 s	Scheduler state, job and allocation status, resource use
NVIDIA GPU exporter	2 s	Utilization, memory, temperature, fan speed, power, clocks
systemd exporter	2 s	State of system services
Node exporter	15 s	CPU, memory, disk, file-system and network use
PostgreSQL exporter	15 s	Connections, transaction rates, database and table sizes

The exporters run on every node as a Nomad system job. The imager metrics include the configuration under which each instance is running and the identifier of its build, so any change in performance can be traced to the configuration in force at the time.

Alerting

Prometheus evaluates a set of alert rules continuously and forwards active alerts to Alertmanager. Alertmanager groups them and posts them to the operators' Slack channel.

Alert	Condition	Hold time
NodeDown	A node stops reporting metrics	2 min
SystemServiceDown	Nomad, Consul, Vault, Docker, the reverse proxy or the NFS server is not running	5 min
EpicImagerDown	An imager's metrics endpoint is unreachable	5 min
EpicNoFEngineData	An imager receives no F-engine packets	2 min
PostgresDown	A database instance is unreachable	—
DiskAlmostFull	Less than 10% free space on a file system	30 min
ZfsPoolUnhealthy	The storage pool is degraded	—
MonitoringDiscoveryFailing	Service discovery returns no targets	15 min

Alertmanager applies two inhibition rules to limit redundant notifications:

• an unreachable node suppresses all other alerts from that node;
• a critical alert suppresses warnings for the same condition.

Notifications repeat every six hours until the condition clears.

Dashboards

Grafana presents the Prometheus data as dashboards covering the computing, storage and network resources of every node, together with the performance of each imaging pipeline (Reddy et al. 2024). The GPU dashboard tracks utilization, memory, power and fan speed on the nodes that run the imagers. These are the principal indicators of whether an imager is keeping pace with real time.

Operating procedures

Routine operations are carried out through Nomad, Consul and Ansible rather than by direct intervention on the nodes:

• Changing imaging parameters. The imager configuration of a node is edited in the Consul key–value store. Nomad restarts that imager with the new configuration, and the other nodes are unaffected.
• Deploying a new imager build. The nightly build job publishes a package. The build hash in the imager job is then updated and the job resubmitted. Nomad replaces the instances one node at a time and requires each new instance to pass its health checks.
• Recovering from a power or network outage. A dedicated playbook restarts the HashiCorp agents in order, unseals Vault if automatic unsealing is unavailable, verifies the database workers, and resubmits all jobs.
• Controlled shutdown. A kill-switch playbook stops the scheduler, the service mesh and the secrets store on all nodes.
• Data retention. Partition creation and retention in the pixel database run automatically. A manual dry-run mode allows the retention policy to be inspected before it is applied.

Most failures of individual processes are handled automatically. Nomad restarts failed tasks according to each job's restart policy, and the imager exits deliberately when its input ceases so that it is restarted cleanly. The alert rules above report the conditions that require operator action.