LOFAR is preparing the upgrade to the LOFAR2.0 system, which will offer simultaneous low- and high-band observing, increased field-of-view, and various other improvements to the sensitivity and operation of the telescope. The new system will generate considerably more data than the current one. The need for processing and storage of this data represents a significant challenge, as it outstrips current affordable solutions. To realize the goals of the new Observatory, the ILT board has approved a data life cycle that optimises science data processing capacity and efficiency with respect to affordable data storage. This includes data retirement. The ILT anticipates that (final) science data products, generated through standard pipelines (such as image cubes) can be kept available indefinitely (under open access policies), but that raw/intermediate products may only be retained for further processing within a restricted dwell time.

To prepare the ILT infrastructure for the LOFAR2.0 data challenges, a first step needs to be taken now that involves the retirement of early Cycle data from the LOFAR Long-Term Archive (LTA). After careful analysis, it has been concluded that data from Cycle 0 up to and including Cycle 6 is either low in metadata quality or already valorized and published (https://science.astron.nl/telescopes/lofar/science-with-lofar/lofar-papers/) or, based on the download statistics, of little demonstrable interest to the current community. The ILT has therefore approved that mid-2024 this data will be deleted from all LTA sites.

Through this notification, we give the opportunity to any users (i.e. not only the projects’ PI’s) who may need to retrieve data from these early LOFAR Cycles, to signal this to the Observatory to coordinate the data download. The requests will be evaluated by the Observatory for technical feasibility and reviewed by the ILT-board for approval. We emphasize that all data concerned is public data.

This excel spreadsheet overview lists all the data that will be retired mid-2024.

The spreadsheet contains two tabs:

1. ‘Projects’: in this tab you will find the general details of the projects whose data will be retired: Project code, title, abstract and size in TB of the project at each of the LTA sites (SURF – column D, Juelich – column E, and Poznan – column F) along with the total size of the project across all sites (column G);
2. ‘Observation and Pipeline ID’s’: in here, you will find two tables. In the table on the left you will find the list of all the projects whose data are selected for retirement. In this table, you can select the projects for which you want to display more information in the table on the right. A ‘multi-select’ option is available in the top right corner of the table. The table on the right provides the detailed list of observation and pipeline products to be retired (column E) and their size (in TB) at each LTA site (SURF – column F, Juelich – column G, Poznan – column H). The total is given in column I.

Note that data sizes in the spreadsheet have byte level accuracy but are shown with one decimal accuracy.

More information about the data products can be obtained by querying the LOFAR LTA. The LTA portal provides documentation on how to use the query service.

If you encounter issues in opening the spreadsheets on your system, please get in contact with us through the SDC Helpdesk.

If you need to download some of the data targeted for retirement, we request you to adhere to the guidelines given below.

To avoid excessive load of the data access services at the LTA sites, the Observatory needs to be informed of your data download plans and needs to evaluate them in relation to data access requirements for other users and purposes. For this, we ask you to submit a formal request for data download by using the Google form linked below. You will be asked to answer questions about the data you wish to retrieve, their size at the various LTA sites, and your data management plan.

https://forms.gle/n8i25WfrDXAxq7U19

The deadline for submitting your request is Friday, 1 December, 12 UT (noon).

All the requests will be evaluated by the Observatory for technical feasibility and will be presented to the ILT-board for their approval and to evaluate whether specific actions are needed. Observatory personnel will notify you of the outcome of your request as soon as possible after this, along with instructions on when and how to initiate the data transfers. Instructions may include recommendations on how efficient data transfers can be achieved given the size and location of the data in relation to the destination(s) indicated in the provided data access plan.

If you have any questions about the above, do not hesitate to contact us through the SDC Helpdesk.