Cycle 20 observing projects

At the end of the 8 March proposal submission deadline, 42 proposals have been received. These represent oversubscription factors of 1.54 in telescope observing hours, 1.27 in processing hours and 2.00 in LTA storage space. The technical assessment of the proposals has been completed, and the PC meeting is planned to take place on May 11, 2023 after which proposers will be informed about the outcome of their proposals. The Cycle 20 observations will run from 1 June 2023 to 31 May 2024.

Juelich and SURFsara LTA sites issues

Intermittent issues with data staging and downloads from the Juelich LTA site have been reported in the past few weeks. The problem has been resolved now. There is also extended maintenance activities ongoing at the SURFsara LTA site currently; please take note of the notifications shown on the LOFAR LTA portal page when attempting to access the archive. Contact SDC-Helpdesk if you experience any issues with staging and downloads.

Issue with determining the sensitivity of beamformed observations

To determine the sensitivity of beamformed observations a factor to take into account is the number of antennas that are off because of system issues. For this we have provided access to the timestamps of the disabling and enabling of antennas through this file:
https://proxy.lofar.eu/array_status/STATIONS/DATA/hardware_states_latest.txt . Unfortunately, the file does not contain updates after June 2019 (except for a few antennas in FR606). We are investigating the issue. For data taken after June 2019, the correction will currently not work. Please contact the SDC helpdesk in case of further questions.

LOFAR password self-service

The password Self-service is active although the integration with MoM/NorthStar/LTA is not yet available. In the meantime, users can go to the link: https://webportal.astron.nl/pwm, to reset their password when having problem with LTA access. Users may also contact SDC-Helpdesk, if further support is needed.

LOFAR family meeting 2023

This year’s LOFAR family meeting will be held in Olsztyn, Poland, from 12 to 16 June. It will cover all the main areas of LOFAR science and use. The conference will feature five days of contributed and invited talks as well as posters. LOFAR Users sessions, Extra Room meetings and other interesting events are planned. Here are the links to registration, abstract submission, and hotel reservations. The deadline for abstract submission is 30 April 2023.

ASTRON SDC

ASTRON SDC will cease Python 2.x support for the lofar_stager_api (https://www.astron.nl/lofarwiki/doku.php?id=public:lta_tricks#python_module_for_staging) effective October 1st, 2023. Please adjust your systems/workflows accordingly.

• 38 stations operational in the Netherlands: 24 core and 14 remote stations. 14 international stations in operations: DE601, DE602, DE603, DE604, DE605, FR606, SE607, UK608, DE609, PL610, PL611, PL612, IE613, LV614. New international stations will be built in Italy and Bulgaria in the coming years. Both stations will be equipped with LOFAR2.0 hardware, whose installation/rollout is expected in 2025.
• Antenna elements of all stations are performing nominally, except CS005 HBA that has about 19% non-operational elements. Those elements are either broken to be fixed or broken beyond repairs. The stations CS030, RS406b and DE605 also have 10 – 13 % non-operational HBA tiles. The overview of non-operational antenna elements for LBA and HBA is available here. At the station level, all (available) stations are online.
• No major failures occurred on CEP hardware/software over the past couple of months.

• Cycle 19 observations started on 1 December 2022 and will run till 31 May 2023. The observing schedule can be found here.

• At the end of observing week 17 ending March 26, 2023, we have done 2030 hours of production observations, with an average observing efficiency of 74%.

LOFAR Development program (W. van Cappellen)

• Progress highlights are also presented in the LOFAR Development Newsletter.
• The lead time of electronic components is currently long and uncertain. Our strategy is to buy as soon as possible to avoid further delays and price increases. At the time of writing, 90% of the LOFAR2.0 hardware has been ordered. The hardware for the 2 production test stations is expected to arrive in Q1 2024. The integration and rollout of all other stations will start at the end of 2024.
• The verification of the first phase of the LOFAR2.0 Test Station (L2TS) has completed and only revealed small issues that can be easily solved. A long-term stability test was successfully executed over the Christmas period.
• The hardware to extend the L2TS to a full station (i.e. all antennas are connected) is arriving. The hardware will be installed in the field in April 2023.
• The upgrade of CEP4 to CEP6, COBALT2 to COBALT3, and the central network infrastructure in Groningen is scheduled in the second half of 2024. That is, after the completion of cycle 20 and before the LOFAR2.0 roll-out.

SDC program (J. Swinbank)

• Over the last few months, the major goal of SDC development has continued to be supporting the LDV project in its operational activities. In particular, this has focused on the development of “LDVSpec”, a web based interface the LDV team can use to define processing tasks before they are sent on to other systems for execution. Although it's initial development was driven by the LDV project, this tool will ultimately form an important part of the way data processing is managed throughout the SDC. See the Fig. 3 for an example of LDVSpec in action.
• Development of ADEX, the SDC's next-generation data discovery and access tool, has been reinvigorated. Current work focuses on the so-called “back end” part of the application, which will run on SDC servers and acts as the glue between the web content the user sees (“the front end”) and the Virtual Observatory services which provide access to the data archives themselves. While we focus on the back-end, there's not much progress on the “real” user interface yet, but we did put together a test interface — ADEX-Labs — that we use for testing and benchmarking the backend; see Fig. 4!
• We are continuing to work hard on a first release of the Rapthor direction-dependent calibration and imaging pipeline for LOFAR. Building on and working with the expertise and tooling developed across the LOFAR community, Rapthor — together with LINC — will provide a standardized and supported imaging solution within the context of the SDC. We hope to have more news soon, and, in particular, we are working hard to have results available in time for the LOFAR Family Meeting later this year.
• The ESCAPE H2020 Project (https://projectescape.eu/) reached a successful conclusion at the end of January. ESCAPE has been a major part of our work in the SDC over the last several years, as ASTRON and the SDC team have had major roles in work packages 2 (developing “data lake” systems for managing and distributing massive datasets) and 4 (developing virtual observatory standards) and leading work package 5 (the development of the ESAP Science Analysis Platform). Although ESCAPE has ended, these ideas will live on: we are already putting VO standards to work in the ADEX back-end, while the data lake and ESAP are being further worked on in the context of the SKA Regional Centre network — and, we hope, will ultimately form the basis of further LOFAR-facing services.

LOFAR Data Valorization (LDV; C. Baldovin - on behalf of the LDV project team)

• The main objectives of the LOFAR Data Valorisation (LDV) project are:
  • To give added value to the LOFAR data in the LTA, generating higher-level data products and make these available to the community;
  • To reduce the volume of the data in the archive, which will directly reduce the operational costs for LOFAR.
• There has been a considerable effort into designing, developing, and implementing the workflows to be deployed in the SURF infrastructure first - in Juelich and Poznan later - to allow the compression of imaging data, the removal of duplicates and re-quantization of beamformed data (BF).
• The processing of early BF data started in December 2022, however due to technical issues (see previous newsletter) the processing until recently did not reach the end state: ingesting of newly generated data in the LTA. We can now report that this has been solved and that LDV is running routine operations on BF data. This is a particularly important milestone since it marks the beginning of the end-to-end operations phase in LDV; the pace of operations will steadily increase as we expand the processing to re-quantization of BF and compression of imaging data later this year.
• Other important news is that the LDV grant from NWO has been renewed for another 2-years (January 2023-December 2024). The grant provides computing and storage resources, plus technical support from SURF. This support is essential to the completion of LDV objectives.

Telescope Manager Specification System (TMSS; A. Schoenmakers, S. ter Veen)

• TMSS was moved to a new and faster server and we see a big improvement in the responsiveness of the system.
• Commissioning of the observing modes is progressing and we expect to switch all production observations to TMSS during April.
• Further commissioning is ongoing for expected observing setups for cycle 20, getting dynamic scheduling ready for cycle 20 (currently testing with ~60 pulsars in for pulsar timing) and to test the responsive telescope functionality.
• Cycle 20 should run fully with TMSS. Some administrative functionality may only be commissioned after the cycle start, as well as final observing strategies that can run later in the cycle, but well in time for their campaigns to run.

The dates of LOFAR Status Meetings, roll-outs and stop days are listed in an online calendar that is available here. In particular, we emphasize:

• Next LOFAR bulletin: July 2023
• LOFAR family meeting 2023 in Poland: June 12 – 16, 2023