Daily image
Monitoring and Control, a year of progress
© Henk Mulder
Since the last daily image there has been substantial development in monitoring and control. Live data every few seconds, live spectrum plots in the same web based system as all other information, live RTSM data. Everything is integrated, including network information, CEP6, Cobalt flagging, observation info, serial numbers, firmware versions, disk usage, Raspberry Pi data, antenna statuses, logging, calibration information, database syncing and Nomad cluster job information.
In the LOFAR1 world we relied on terminals, log files and individual observation pages to piece together what happened after the fact. Now data from any source can be combined and used for alerting and events appear on the homepage as they occur. Shutting down a station, spotting an oscillating antenna, checking the spectrum and marking the antenna as broken is now a matter of seconds.
The centralized system also exposes issues we did not even know existed. One example is discovering that the Sun can cause false polarity swap indications during the daytime. The fix came from tracking the Sun, now visible in Grafana, and filtering those events. The same applies to maintenance planning, seeing which LBA or HBA require attention first, where major events occur, when they happen and why. Any information can be combined to build a plot or comparison. Tango version, calibration file date, 48 V input voltage and frequency band in one graph is possible when needed to reach the root cause of a problem. Different users can shape the system to their needs.
LOFAR2 will be a different operational environment. The gap between what we had and what is coming cannot be captured in a single daily image. Drop by, explore it and start using it.