System Architecture
Requirements, cluster organization and principal services of the EPIC real-time sky-monitoring system at LWA-SV.
The imager described in The LWA-SV Imager is one component of a larger system. That system, designated "Alchemist" (Reddy et al. 2024), turns six independent imaging pipelines into a continuously operating sky monitor. It is responsible for:
- deploying and configuring the imagers;
- maintaining the list of monitored sources;
- storing and serving the extracted data;
- distributing the live video;
- monitoring the health of every component.
This section describes the deployed system. The configuration is maintained as infrastructure code in the epic-sv-devops repository.
Requirements
The design follows from the data rates of the imager and from the scientific use of its products (Reddy et al. 2024).
| Requirement | Consequence for the design |
|---|---|
| Six imagers, each processing one 3.2 MHz sub-band at 80 ms cadence | One GPU node per F-engine sub-band; nodes operate independently |
| Full images would amount to ≈110 TB per day | Only pixel patches around watched sources are retained at full cadence; full images are archived as longer accumulations |
| ≈7 × 10⁶ pixel rows per hour per source; ≳10⁹ rows per day in total | A distributed, time-partitioned database with a fixed retention period |
| Pixel data retained for six months; accumulations and video retained permanently | Automated partition management; network storage for files |
| Follow-up of external alerts (CHIME/FRB, DSA-110, realfast, LIGO/Virgo/KAGRA) | A watch-list service that accepts time-limited entries |
| Unattended, continuous operation | Declarative scheduling, automatic restarts, service discovery, metrics and alerting |
| Portability to other compact arrays | All provisioning expressed as reproducible configuration (Ansible) |
Cluster organization
The system follows a client–server model. A coordinator node schedules work and hosts the shared services, and six compute nodes execute the imaging pipelines. Compute nodes are added by provisioning them and joining them to the cluster; services discover one another through Consul, so no static configuration needs to change.
| Node | Count | Role | Principal services |
|---|---|---|---|
Coordinator (epic-sv) | 1 | Control plane, shared services, storage | Nomad, Consul and Vault servers; PostgreSQL/Citus coordinator; watchdog; Prometheus and Alertmanager; PeerTube; Waterfalls; reverse proxy; NFS export |
Compute (epic1–epic6) | 6 | Imaging | Nomad and Consul clients; one epic++ imager with one GPU; Citus worker; RTMP media server; video transcoding runner; metrics exporters |
The nodes are connected by two networks:
- Station data network. Carries the F-engine multicast at 100 GbE. Each compute node subscribes to one sub-band.
- Cluster network. Carries scheduling, database, metrics and video traffic.
All nodes mount a shared file system exported by the coordinator over NFS. This file system holds the database storage, the build artifacts, the monitoring data and the archived data products.
Principal services
The services shown in Figure 3 fall into five groups. Each is described on the page indicated.
| Group | Services | Description |
|---|---|---|
| Orchestration | Nomad, Consul, Vault | Scheduling of all jobs, service discovery, configuration store, secrets (Orchestration) |
| Imaging | epic++ imager, build job | One imager per compute node; nightly build from the phoenix branch (Orchestration) |
| Data | Citus database, watchdog, network storage | Pixel spectrograms, session metadata, watch list, archived images (Data Management) |
| Distribution | Media servers, PeerTube, Waterfalls | Live video and web access to spectrograms (Video and Web Services) |
| Operations | Prometheus, Alertmanager, Grafana, Ansible | Metrics, alerting, dashboards and provisioning (Monitoring and Operations) |
Flow of data through the system
The path from the telescope to a stored product is as follows (Figure 3):
- The station F-engine multicasts channelized voltages for six sub-bands.
- On each compute node, the imager subscribes to its sub-band and forms images (see Imager Pipeline).
- The imager obtains the watch list from the watchdog over gRPC. It extracts the pixel patches of visible sources and inserts them into the distributed database.
- Accumulated images are written as FITS files to network storage.
- A video stream of each sub-band is published to a local media server. Scheduled streaming jobs forward it to the PeerTube instance (EPIC TV).
- The Waterfalls application queries the database for spectrograms and accepts alerts from external facilities. Those alerts are recorded as time-limited entries, which the watchdog adds to the watch list.
Throughout, every service exports metrics to Prometheus, which evaluates alert rules and notifies the operators.