Data Management

Distributed pixel database

At 80 ms cadence, each watched source produces a $5\times 5$ patch of pixels per image, which amounts to approximately $7\times 10^6$ rows per hour per source (Reddy et al. 2024). With six sub-bands and several sources visible at any time, the system ingests of order $10^9$ rows, about 1 TB, per day. Two requirements follow:

• insertion must keep pace with six concurrent imagers;
• queries for a source over an interval of minutes must remain interactive.

Both are met by distributing the database across the cluster with the Citus extension to PostgreSQL 16:

• Coordinator. The coordinator node runs the Citus coordinator, which holds the metadata and routes queries.
• Workers. Each of the six compute nodes runs a Citus worker that stores a share of the data.
• Storage. The data directories of all seven instances reside on the shared network file system. A start-up dependency ensures that no worker starts before the file system is mounted.

The pixel table

The table sevilleta.epic_pixels (schema in Data Products) is a distributed table. It is also range-partitioned on the image timestamp, img_time. Partitioning serves three purposes:

1. Insertion. New rows are written to the most recent partition only, which keeps index maintenance local.
2. Query. A query for an interval touches only the partitions that overlap it.
3. Retention. Expired data are removed by dropping whole partitions rather than by deleting rows.

Partitions span 6 hours and are maintained by two scheduled jobs inside the database (pg_cron):

Job	Schedule	Action
Pre-creation	Hourly, at minute 15	Creates partitions two days ahead of the present time
Retention	Hourly, at minute 30	Drops partitions older than six months, at most one day of partitions per run

Limiting each retention run to one day of partitions bounds the duration of the locks it takes, so ingestion by the imagers is not stalled.

An earlier design created hourly partitions with the pg_partman extension (Reddy et al. 2024). It was superseded by native Citus time partitioning, which creates and drops partitions consistently across the coordinator and all workers.

Metadata tables

The pixel table is accompanied by smaller tables, kept on the coordinator:

Table	Content	Writer
epic_img_metadata2	Imaging sessions: tuning, channel width, integration time, image and patch size, imager version, watched sources	Imager
epic_files_metadata	Archived FITS files and their imaging parameters	Imager
epic_daily_digest	Per-source Stokes I and V spectra of each accumulated image	Imager
epic_watchdog	Watch-list entries with their validity windows	Watchdog, Waterfalls

Watch list and external triggers

The set of sources for which pixels are recorded is defined by the epic_watchdog table. Each entry contains:

• a source name and J2000 position;
• a patch size;
• a validity window (t_start, t_end);
• a watch mode, a reason and an author;
• for triggered entries, the originating VOEvent.

The watchdog service runs on the coordinator. It periodically reloads the table and treats entries whose validity window contains the present time as the active watch list. Entries whose window has elapsed are marked as watched. The service exposes the active list through the gRPC method fetch_watchlist, which each imager calls at intervals of no less than 10 s (see Imager Pipeline). Two watch modes are supported:

• Continuous entries, such as the Sun, bright pulsars and the A-team sources, have effectively unlimited windows.
• Timed entries default to a window of seven days.

CHIME/FRB alerts are received by the back end of the Waterfalls application, which authenticates each alert with a shared secret. An alert creates or extends a watch-list entry if it satisfies three conditions:

• signal-to-noise ratio of at least 10;
• importance of at least 0.98;
• dispersion measure of at least 10 pc cm⁻³.

The validity window begins at the burst's arrival time at infinite frequency. It extends for twice the dispersion delay between 400 MHz and 28.1 MHz:

$$\Delta t_{\mathrm{watch}}=2\times 4.15\times 10^3\mathrm{s}\mathrm{DM}\left[{\left(\frac{28.1}{\mathrm{MHz}}\right)}^{-2}-{\left(\frac{400}{\mathrm{MHz}}\right)}^{-2}\right],$$

where DM is in pc cm⁻³. This guarantees that the dispersed burst crosses the entire LWA band while the source is being recorded.

For repeating sources, the existing entry is extended rather than duplicated. Triggered entries use a $5\times 5$ patch, and each accepted alert is reported to the operators.

File storage

The coordinator exports a ZFS-backed file system to all nodes over NFS. It holds:

Content	Retention
Accumulated FITS image cubes and PNG quick-look images	Permanent
Live-stream recordings and daily replays from PeerTube	Permanent; short recordings are pruned
Database storage of the coordinator and workers	Governed by partition retention
Imager build packages and video overlays	Retained by version
Prometheus time-series data	Prometheus retention

A periodic job on each compute node verifies the network mount every hour. An alert is raised if a disk falls below 10% free space or if a ZFS pool becomes unhealthy.