Building and Running
Software dependencies, build procedure, configuration parameters and deployment of the epic++ imager at LWA-SV.
Build procedure
epic++ requires three things:
- Linux;
- a CPU supporting AVX2;
- an NVIDIA GPU of the Ada generation. The CUDA architecture is set to
89inCMakeLists.txt.
git clone --recurse-submodules -b phoenix https://github.com/epic-astronomy/LWA_EPIC.git
cd LWA_EPIC
conda env create -f conda_env.yml # Python 3.10 environment "epic310"
conda activate epic310
# Obtain NVIDIA MathDx (cuFFTDx) and place it under src/extern/ (see README)
mkdir build && cd build
cmake .. && make -j
The dependencies supplied as git submodules are:
- RaftLib (dataflow runtime)
- Google Highway (portable SIMD)
- pybind11 (embedded Python)
- glog (logging)
- cxxopts (command-line parsing)
- libpqxx (PostgreSQL client)
- prometheus-cpp (metrics)
The executable additionally links against:
- the CUDA runtime
- NVML
- libibverbs
- the FFmpeg libraries
The Python environment provides:
- LSL
- Astropy
- SciPy
- NumPy
- psycopg2
- SQLAlchemy
- gRPC
The build places a link to epic_utils.py beside the executable, so that the embedded interpreter can import it.
Configuration parameters
The principal parameters are listed below. The complete set is printed by epic++ --help.
| Parameter | Default | LWA-SV | Description |
|---|---|---|---|
--addr, --port | 239.168.40.11, 4015 | one sub-band per node | F-engine multicast group(s) and port(s) |
--printendpoints | — | — | Report station endpoints and their tunings, then exit |
--imagesize | 128 | 128 | Image dimension (64 or 128) |
--imageres | 1.0 | 1.056 | Pixel size at zenith (degrees) |
--nts | 1000 | 1000 | Spectra per gulp (1000 × 40 µs = 40 ms) |
--seq_accum | 40 | 80 | Integration time per image (ms) |
--nimg_accum | 1 | 16 | Images per archived cube |
--channels | 128 | 128 | Number of channels imaged (at most 128 on an RTX 4090) |
--support | 2 | 5 | Gridding-kernel support (cells; 1–9) |
--kernel_oversample | 2 | 4 | Oversampling factor for the grid-correction computation |
--accum_16bit | off | on | Accumulate in 16-bit floating point |
--chan_nbin | 4 | 1 | Channels summed before extraction and archiving |
--elev_limit_deg | 10 | 10 | Minimum elevation for pixel extraction (degrees) |
--watchdog_addr | localhost:2023 | — | Address of the watch-list service (gRPC) |
--epic_data_schema | public | sevilleta | PostgreSQL schema for output tables |
--out_dir | ./ | — | Output directory for FITS and PNG files |
--stream_url, --video_size, --stream_cmap | — | — | RTMP destination, frame width and color map |
--metrics_bind_addr | 127.0.0.1:8080 | — | Prometheus endpoint |
--runtime | 5 | — | Duration of the run (s) |
--ngpus, --gpu_ids, --nstreams | 1, —, 8 | 1, 0, 8 | GPU selection and number of CUDA streams |
--offline | — | — | Replace packet capture with a synthetic source |
Database connection parameters are taken from the standard PostgreSQL environment variables (PGHOST, PGDATABASE, PGUSER, and so on).
Deployment at LWA-SV
At LWA-SV each imager runs as a Nomad job on one of six GPU nodes. Its configuration is held in Consul and its credentials in Vault. The deployment, including scheduling, data management, video distribution and monitoring, is described in System Design.
Diagnostics
| Aim | Method |
|---|---|
| Identify the frequency carried by each sub-band | Run --printendpoints |
| Run the dataflow graph without network input | Use --offline |
| Monitor real-time operation | Check the per-stage execution times exported to Prometheus. The criterion is that the BlockEpicImager time remain below the gulp duration (40 ms) |
| Increase diagnostic output | Set the glog verbosity variable GLOG_v (levels 1–4) |