Orchestration

All long-running processes of the system, and the periodic tasks, are deployed as jobs of the HashiCorp Nomad cluster manager. Two further HashiCorp tools support it (Reddy et al. 2024):

• Consul provides service discovery, health checking and a key–value configuration store.
• Vault stores credentials and other secrets.

Each tool runs as an agent on every node. The coordinator runs the Nomad and Consul servers, which hold the cluster state, together with Vault. The compute nodes run Nomad and Consul clients, and the coordinator also runs a Nomad client so that shared services can be scheduled on it. Agents locate one another through Consul, so nodes can join the cluster without changes to existing configuration. Access to all three tools is controlled by access-control lists, with a default-deny policy in Consul.

Nomad jobs

Each job declares its placement constraints, resources, health checks and restart policy. It also declares the configuration and secrets it consumes. Three Nomad drivers are used:

• raw_exec for GPU and network-intensive processes, which require direct access to devices and to raw sockets;
• docker for off-the-shelf services;
• the periodic scheduler for recurring tasks.

Job	Type	Placement	Function
epic-imager	Service	One per compute node	Real-time imager (epic++)
epic-image-updater	Periodic (daily)	One compute node	Builds the imager from the phoenix branch and publishes a versioned package
watchdog	Service	Coordinator	gRPC watch-list service
media-server	System	All nodes	RTMP servers receiving the imager video streams
epictv-streaming	Periodic (daily)	Compute nodes	Forwards each sub-band's stream to PeerTube
epictv-grid	Periodic (daily)	Coordinator	Composes the six streams into a single mosaic stream
overlay	Service	Coordinator	Renders the coordinate-grid overlay for the video
peertube_runner	System	Compute nodes	Video transcoding for PeerTube
waterfalls	Service	Coordinator	Web application (Nuxt front end, FastAPI back end)
prometheus, alertmanager	Service	Coordinator	Metrics collection and alert routing
cluster-monitoring	System	All nodes	GPU, node, systemd and PostgreSQL exporters
vouch_oauth_backend	Service	Coordinator	Single sign-on for administrative interfaces
wikijs	Service	Coordinator	Internal operations documentation
ensuring-mounts	Periodic (hourly)	Compute nodes	Verifies the network file-system mount

The imager job

The epic-imager job runs one instance per compute node, with a distinct-host constraint. Each instance proceeds through the following lifecycle:

1. Retrieval. A pre-start task retrieves the versioned imager package from the coordinator's internal file server.
2. Preparation. Pre-start tasks set the GPU to its maximum-performance clock state and grant the executable the raw-network capability required for ibverbs packet capture.
3. Configuration. The command-line configuration of the instance is read from the Consul key–value store. Each compute node has its own key, so imagers on different sub-bands may be configured independently. The watchdog address is resolved from the Consul service catalogue. The database credentials are read from Vault.
4. Execution. The imager runs under a launcher that activates the Python environment required by the embedded interpreter.
5. Registration. The imager registers its Prometheus endpoint as a Consul service, with an HTTP health check every 10 s.
6. Teardown. A post-stop task restores the default GPU clocks.

The configuration templates use a restart change mode. Any change to an imager's configuration key, or to its secrets, therefore restarts that imager alone, which allows imaging parameters to be changed without redeploying the job (Reddy et al. 2024). The imager also terminates deliberately when the input stream ceases (see Imager Pipeline), and Nomad then restarts it.

Configuration and secrets

Configuration. Runtime configuration is held in a single Consul key–value namespace, which records:

• the imager options for each node;
• the video endpoints and durations;
• the identifier of the most recent imager build;
• the parameters of the video overlay;
• the configuration of the web application.

Secrets. Database credentials, streaming keys, the alerting webhook and OAuth credentials are held in Vault's key–value engine. They are injected into jobs only through Nomad's Vault integration: each job receives a short-lived token restricted to the policy it requires.

Vault starts in a sealed state after a restart. Its master key is split into five shares, any three of which unseal it. To keep unplanned reboots from leaving Vault sealed, the primary Vault is configured for automatic unsealing through the transit engine of a second, independent Vault instance on a compute node (Reddy et al. 2024).

Software delivery

The imager is delivered through a build-and-publish cycle:

1. Nightly build. The epic-image-updater job pulls the phoenix branch of LWA_EPIC. If the commit differs from the last recorded build, it compiles epic++ and packages the executable with its Python modules, gRPC stubs and shared libraries. The package is named by commit hash.
2. Publication. The package is written to the coordinator's internal file server, and the new hash is recorded in Consul.
3. Deployment. The imager job is pinned to a specific build hash. A build is put into service by updating that hash, which gives an explicit, reversible deployment step.

Provisioning

The entire cluster is provisioned with Ansible, through a numbered sequence of playbooks that can be replayed on new hardware. The deployment scripts are intended to make the system portable to other dense, compact arrays (Reddy et al. 2024).

Stage	Playbooks	Function
Base system	01–03	Inventory, operating-system packages, container runtime, HashiCorp packages, internal certificate authority and host certificates
Control plane	04–07	Consul (TLS, gossip encryption, ACLs); primary and auto-unseal Vault; Nomad (ACLs); Nomad–Vault integration and policies
Runtime environments	08, 15	Python environment of the imager; build dependencies (libpq, FFmpeg libraries)
Database	09, 09b, 20, 21	PostgreSQL 16, Citus coordinator and workers, extensions, partition management and retention, mount-ordered start-up of workers
Web and access	10, 10b, 12	Reverse proxy and TLS; RTMP; single sign-on
Monitoring	11, 13	Prometheus, exporters, Alertmanager, database monitoring roles
Video	14, 14b, 17	PeerTube, transcoding runners, media servers, streaming jobs
Applications	16, 17, 18, 19	Watchdog, imager and build jobs, internal wiki, Waterfalls
Operations	95–99	Controlled shutdown, full restart (including unsealing and job resubmission), unsealing, teardown