Imager Pipeline

The real-time imager, epic++, is a C++17/CUDA application. Its source is in epic-astronomy/LWA_EPIC, branch phoenix.

The processing is expressed as a dataflow graph in RaftLib:

• Each stage executes on a dedicated, pinned CPU thread.
• Stages exchange reference-counted handles to pre-allocated, page-locked buffers, so data are not copied between stages.
• An embedded Python interpreter (pybind11) performs the astrometric and file-handling operations, using LSL and Astropy: antenna geometry, cable models, coordinate transformations, FITS output and database helpers. These operations are invoked only on retuning, per output file, or per watch-list update.

Dataflow graph

Figure 1 shows the graph for a single GPU. It is constructed in src/raft_kernels/epic_executor.hpp. A single process can serve up to three GPUs, each with an independent instance of the graph and its own multicast stream. In the LWA-SV deployment, each node runs one GPU on one sub-band.

Packet capture and gulp assembly

Packets are received through raw ibverbs queue pairs, with flow steering on the multicast address and port. The implementation is adapted from Bifrost and bypasses the kernel network stack. The receive path proceeds as follows:

1. The Ethernet, IP and UDP headers (42 bytes) are discarded.
2. The CHIPS header is byte-swapped and validated.
3. Payloads are copied into place with 256-bit SIMD operations (Google Highway, AVX2). One vector corresponds to one board's 16 stands × 2 polarizations for a single channel.

Packets are assembled into gulps of --nts spectra. The default of 1000 spectra corresponds to 40 ms. The first gulp is aligned to an integer UTC second. Gulp memory is ordered as [channel][time][stand][polarization], so that each GPU thread block reads a contiguous region for its channel.

Packet loss is handled at two levels:

• Board. A board that delivers fewer than 50% of its expected packets within a gulp has its data set to zero.
• Gulp. A gulp with fewer than 50% of its packets is discarded.

After 1000 consecutive empty gulps the process terminates, and the cluster scheduler restarts it.

Each gulp carries metadata:

• chan0 and the number of channels
• the sequence range
• a time tag in units of the 196 MHz sample clock
• the fraction of valid data

Correlator

The correlator stage first compares the gulp's tuning against the previous gulp:

• Health-check tuning. The stage emits an empty, flagged output in place of an image.
• Changed chan0 or number of channels. The imaging configuration is recomputed for each channel: antenna positions in grid units, phase corrections, gridding-kernel weights and the grid-correction image. A new imaging session is opened with a new identifier.
• Otherwise. The fused imaging kernel is launched.

The kernel is described in GPU Kernel. Images are accumulated over --seq_accum milliseconds. At LWA-SV this is 80 ms, or two gulps. The work is distributed across eight CUDA streams by channel, and only the final gulp of each integration is synchronized with the host.

The output contains one float4 per pixel per channel, holding $(X{X}^{\ast },Y{Y}^{\ast },\mathrm{Re}X{Y}^{\ast },\mathrm{Im}X{Y}^{\ast })$, with shape [channel][x][y][4].

Channel reduction

The channel reducer sums groups of --chan_nbin adjacent channels to reduce the downstream data volume. The LWA-SV configuration retains full 25 kHz resolution (chan_nbin = 1). A full-resolution copy of each image is passed separately to the video streamer.

Pixel extraction

The index fetcher retrieves the watch list from the watchdog service over gRPC, at intervals of no less than 10 s. Each entry specifies a name, a J2000 position and a patch size. Pixel indices are then computed for each source in four steps:

1. The positions of Solar-system bodies are computed for the image epoch at the LWA-SV site.
2. A world coordinate system in SIN projection is constructed. It is centered on the zenith (local sidereal time and site latitude) and precessed to J2000.
3. The pixel nearest the source is identified and expanded to a $k\times k$ patch, typically $5\times 5$.
4. Patches containing pixels below the elevation limit (--elev_limit_deg, default 10°) are excluded.

The pixel extractor then copies these patches from every image and forwards them to the database ingester. Each pixel becomes one row, containing all channels and four polarization products. The full image is passed to the accumulator.

Database ingestion

The database ingester writes pixel rows to PostgreSQL through libpqxx, with one multi-row INSERT per source per image. It also maintains a table of imaging sessions. The schemas are given in Data Products.

Accumulation and archiving

The accumulator sums --nimg_accum consecutive images; at LWA-SV this is 16 images, or 1.28 s. Accumulation restarts after a change of tuning or a gap in time.

The disk saver then performs three operations:

• It writes each accumulated cube to a FITS file with a SIN-projection world coordinate system, together with a PNG quick-look image.
• It registers the file in the database.
• It appends Stokes I and V spectra of the watched sources to the daily digest.

Video stream

The video streamer converts the Stokes I image ($X{X}^{\ast }+Y{Y}^{\ast }$) of the first channel into a 10-bit video frame every 80 ms. It uses the FFmpeg libraries in-process, and applies the following operations in order:

1. per-frame normalization
2. Lanczos rescaling
3. gamma correction
4. a horizon mask
5. a perceptual color map
6. a frequency and time annotation
7. temporal denoising

The frames are encoded as H.264 at 12.5 frames per second and transmitted over RTMP to a relay. The relay distributes the stream as EPIC TV.

Resources and real-time constraint

Resource	Allocation
CPU	One pinned core per pipeline stage, approximately nine per GPU
Host memory	Lock-free pools of page-locked buffers; 100 gulps for packet assembly
GPU	Fused imaging kernel on eight CUDA streams; approximately 35–39 ms per 40 ms gulp on an RTX 4090
Python	Invoked on retuning, per FITS file and per watch-list update; the interpreter lock is otherwise released

The system operates in real time provided that the GPU stage processes each 40 ms gulp in less than 40 ms. All other stages run asynchronously downstream. Execution times for each stage, packet-loss statistics and the active tuning are exported as Prometheus metrics.