This web page lists the issues found in the system that might have affected the analysis of LOFAR data during the production Cycles.

Wrong information in antenna tables of LOFAR Measurement Sets (13/02/13 - 10/02/14)

During a software roll out few years ago, on 13th February 2013, a bug was inadvertently introduced in a procedure used to store information in the LOFAR internal database (LOFAR_4 OTDB). This bug was related to a function called ‘getbrokenhardware', used by the system to store in the Measurement Sets (MS's) information about malfunctioning antenna elements. It initiated a cumulative effect whereby the information stored in the MS's after the aforementioned date consisted of the addition over time of all newly diagnosed malfunctioning elements to the original broken elements information, while not removing from the list elements which subsequently became operational after repair.

The issue was discovered by the EoR KSP while performing deep wide-field calibration and imaging. It affects all reduction software that uses the antenna element information in the MS's to simulate the beam response of the LOFAR array.

Please be aware that the visibilities recorded in your MS's are correct; only the information stored in the Antenna Table is wrong.

Therefore, the quality of the data is still preserved.

Depending on the processing procedures you applied to achieve your final results, these might be affected in different fundamental ways.

In particular, please note the following.

This issue does not affect:

• Products of the ASTRON pre-processing, calibrator, and target pipelines;

• Solution transfer to the target field from calibrators observed with dedicated bandwidth;

• Calibration results at the field center obtained adopting an external model of the sky, unless your science case requires thermal noise limited maps where the positions and fluxes of off-axis sources become important.

This issue affects only interferometric observations that have been processed using:

• Products of the ASTRON imaging pipeline;

• The ‘smart demix' (currently not available in the Standard Imaging Pipeline);

• BBS (used for wide-field self calibration);

• AWImager.

Since the specific effects depend on the state of the list on the observing date, as well as on the characteristics of the observed fields, it is hard to quantify for all interferometric observations performed during the last year.

Luckily it is possible to extract a history of broken and repaired antenna elements since the moment that the bug was introduced into the system.

This problem was solved during a software roll out on 10th February 2014. From that date onward all MS's antenna tables report the correct information again.

The script to correct for this issue can be downloaded from the link below:

fixinfo

We advise Users to compare the results obtained after correcting the MS's antenna tables with those with wrong information in order to estimate the impact of this issue on their scientific goals.

The software requirements to run this script on your systems are:

1. TaQL (so casacore)

2. pyrap

3. LofIm (in particular, it uses makebeamtables and addfailedtileinfo)

Usage:

tar xvf fixinfo.tar

cd fixinfo

#running the program with full path is needed at some systems, you can use `pwd` to get the path name #on most systems.

full/path/to/fixinfo/fixbeaminfo path/to/my_ms/my_ms.MS

The updated number of flagged elements can be listed, e.g.,

taql 'select ANTENNA_ID, ntrue(ELEMENT_FLAG), [select NAME from my_ms.MS::ANTENNA][ANTENNA_ID]  from my_ms.MS::LOFAR_ANTENNA_FIELD'

This will give something like:

0 48 CS001HBA0

1 48 CS001HBA1

2 48 CS002HBA0

3 56 CS002HBA1

...

55 0 RS407HBA

56 0 RS409HBA

57 6 RS503HBA

58 0 RS508HBA

Note that in this case, it concerns the HBA dual mode, so half the tiles (24=48 polarizations) are flagged for HBA0 etc. CS002HBA1 in the above example, has HBA0 flagged (48 pols) + 4 additional tiles (8 pols) because they were set to broken.

The plots of the history of operational antenna elements for all LOFAR stations since February 2013 can be found here.

In case you experience issues when running this script, please contact SDC operations through the SDC-helpdesk which is hosted at sdchelpdesk

Incorrect values in the WEIGHT_SPECTRUM column of interferometric COBALT data (19/03/14 - 31/10/14)

This is an error associated with the COBALT correlator that affects data recorded between 19-03-2014 and 31-10-2014. A solution to this problem is described below. The error has been corrected at the correlator level so all data recorded after 31-10-2014 is not affected by this problem

An error occurred in the code of the COBALT correlator which resulted in incorrect values being written to the WEIGHT_SPECTRUM column of interferometric COBALT data. This error arose whenever a station was flagged for either part of, or for the entire observation. COBALT incorrectly wrote -1 in the WEIGHT_SPECTRUM column of the Measurement Set (MS), which eventually results in a weight of around 21.447 (it should be between 0 and 1)

Due to the wrong weights, baselines with flagged stations are not actually flagged later on. Their contribution prevents the solver from converging, and can lead to non-optimal demixing and BBS results. This means that there is a chance that some A-team signal is left in the visibilities, so we ask that you inspect your data carefully.

The issue was discovered by ASTRON while analyzing suspicious demixing results and investigating the reason of long processing time for LOFAR pipelines. Once identified, the problem has been promptly fixed at the correlator level. In particular, all observations performed after 31-10-2014  do not suffer from this issue

Depending on the data type, the issue can be addressed as follows:

Raw data: in order to solve the issue the data need to be processed using a new version of the software, no older than 2014-10-30. To check which version of LOFAR software is installed in your own computing facility, run the command:

use <Your_Lofar_enviromnent>

versionlofarstman | grep "package revision"

returns (for example): package revision  = 30484 (last change in package)

The install is ok if package revision >= 30324.

Processing the data with the new version of the software will automatically solve the problem (with a fix in the LofarStMan storage manager).

If needed, ask your software or system administration to update your LOFAR software. The latest version is always available at:  https://svn.astron.nl/LOFAR/trunk

Build instructions are available at:  http://www.lofar.org/operations/doku.php?id=public:user_software:documentation:lofar-cmake

Processed data:

- If autoweight=TRUE was not used in the pre-processing:

If the data was not averaged, flag the data for which WEIGHT_SPECTRUM > 1, for example by using the following TaQL command:

taql ‘update <your_MS> set FLAG=True where any(WEIGHT_SPECTRUM>1) and ANTENNA1!=ANTENNA2

If the data was averaged, flag the data for which WEIGHT_SPECTRUM > 1, for example by using the following TaQL command:

taql ‘update <your_MS> set FLAG=True where any(WEIGHT_SPECTRUM > <averaging_frequency_step_used> * <averaging_time_step_used>) and ANTENNA1!=ANTENNA2

- If autoweight=TRUE was used in the pre-processing (default setting in the ASTRON pre-processing pipelines), it is advised to flag data for which there are outliers in the WEIGHT_SPECTRUM column of the MS.

In case you experience issues when running these commands or you need help in any of these steps, please do not hesitate to contact SDC Operations through the SDC-helpdesk which is hosted at https://support.astron.nl/sdchelpdesk.

Incorrect information about broken tiles and antenna elements in LOFAR Measurement Sets (March 2015)

During a software roll out on Jan 26th 2015, the information on broken tiles and antenna elements, which is stored in a LOFAR internal database, was inadvertently erased. Consequently, from that moment onward the list of malfunctioning antenna elements stored in the LOFAR Measurement Sets (MS's) only contained newly diagnosed malfunctioning elements. This problem was solved on February 25th 2015. Since then, the MS's contain again the proper broken tile and element information.

Please be aware that the visibilities recorded in the MS's during the aforementioned dates are correct; only the FLAG information in the LOFAR-ANTENNA-FIELD table is wrong. Therefore, the quality of the data is still preserved.

Depending on the processing procedures applied to your data to achieve your final results, these might be affected in different fundamental ways. In particular, please note the following.

This issue does not affect:

• Products of ASTRON pre-processing, calibrator, and target pipelines;

• Solution transfer to the target field from calibrators observed with dedicated bandwidth;

• Calibration results at the field center obtained adopting an external model of the sky, unless your science case requires thermal noise limited maps where the positions and fluxes of off-axis sources become important.

This issue affects only interferometric observations that have been processed using:

• Products of the ASTRON imaging pipeline;

• The ‘smart demix' (currently not available in the Standard Imaging Pipeline);

• BBS (used for wide-field self calibration);

• AWImager.

Since the specific effects depend on the state of the list on the observing date, as well as on the characteristics of the observed fields, it is hard to quantify for each interferometric observation performed during the last month.

The Radio Observatory could extract a history of broken and repaired antenna elements relative to the aforementioned period and consequently repair the database.

The script to correct your data for this issue can be downloaded from the link below:

fixbeaminfo_March2015

We advise Users to compare the results obtained after correcting the MS's with those with wrong broken antenna elements information in order to estimate the impact of this issue on their scientific goals.

The software requirements to run this script on your systems are:

1. TaQL (so casacore)

2. pyrap

3. LofIm (in particular, it uses makebeamtables and addfailedtileinfo)

Usage:

tar xvf fixbeaminfo_March2015.tar

cd fixbeaminfo

./fixbeaminfo path/to/my_ms/my_ms.MS

The updated number of flagged elements can be listed, e.g.,

taql 'select ANTENNA_ID, ntrue(ELEMENT_FLAG), [select NAME from my_ms.MS::ANTENNA][ANTENNA_ID]  from my_ms.MS::LOFAR_ANTENNA_FIELD'

This will give something like:

0 48 CS001HBA0

1 48 CS001HBA1

2 48 CS002HBA0

3 56 CS002HBA1

...

55 0 RS407HBA

56 0 RS409HBA

57 6 RS503HBA

58 0 RS508HBA

Note that in this case, it concerns the HBA dual mode, so half the tiles (24=48 polarizations) are flagged for HBA0 etc. CS002HBA1 in the above example, has HBA0 flagged (48 pols) + 4 additional tiles (8 pols) because they were set to broken.

In case you experience issues when running this script, please contact SDC Operations through the SDC-helpdesk which is hosted at https://support.astron.nl/sdchelpdesk.

Polarization leakage in beam formed data (12/07/14 - 21/01/15)

The report below concerns beam-formed data recorded with COBALT between 2014-07-12 and 2015-01-21.

The new clock correction for coherent beam-forming of the core stations, introduced at 2014-07-12, influences the data in two negative ways:

• Polarization studies: The different stokes parameters are mixed, most notably stokes Q and V, but this is direction dependent.

• Flux measurements may also be affected because the addition of the signal is not fully coherent.

At 2015-01-21 the previous situation has been restored. Please note that at this point LOFAR does not have any polarization calibration and a smaller incoherency effect may still be present in the data. The effect is described in more detail below.

The single clock that is used for the core stations delivers the same clock signal to these stations. This allows for coherent beam-forming.  However, there is a different delay between the clock and each state.  This delay gives a frequency dependent phase shift, a phase gradient, but there is also a phase zero offset. The values can be different for the X and Y polarization. If the phase zero offset is not applied correctly, there is leakage between the stokes parameters. With the previous correlator, BG/P, only the phase gradient could be applied, but no phase zero offset. With COBALT, it is possible to apply the phase zero offset. This was introduced at 2014-07-12. However, the values that were applied did not decrease spectral leakage, but instead increased the values, creating a stronger Stokes V intensity, probably from the Stokes Q parameter, but this can be direction dependent. This was reported by a user at 2015-01-07. Following this report, tests were performed to see what the current optimum values are by trying three different settings. The best result was obtained by setting the phases back to 0 at 2015-01-21. Please note that there may still be polarisation leakage, but less than before this date. There is currently no polarisation calibration method available to correct this. We are still attempting to find optimum calibration values, also to add the core coherently and will also investigate the polarization aspect. We are also investigating a conversion method for data taking during this period. A similar issue is known at international stations, but no fix will be available on the short term.

Imaging data will have had different delays during this period as well, but this is corrected in the standard calibration procedures.

Issue in Skymodels created with PyBDSM (03/03/15 - 13/05/15)

From 3rd March until the 13th May 2015 a bug was inadvertently introduced in the source finder software PyBDSM.

This was caused by a change to the astropy WCS library in astropy version 1.0 causing PyBDSM to write NaNs (not a number) to output sky models written in BBS format.

The NaNs occurred in the "Orientation" column of sources of type "Gaussian" (sources of other types were not affected).

This bug produces flat antenna gain tables (parmdb) when solving with BBS using a model extracted with PyBDSM in the period between the 3rd March and the 13th May 2015

The LOFAR pipeline products are not affected by this issue since the models used both in the Calibrator and Imaging pipelines have a different format.

We advise users to check their self-calibration results if these were obtained by using models extracted with PyBDMS during the above mentioned period.

Station Adder correction - LB pipeline ( up to 28/07/15)

The Long Baseline (LB) pipeline increases the signal-to-noise ratio for baselines containing international stations by combining all of the core stations into a single "Super" station. This is done by the "Station Adder" in NDPPP.

The LB working group found that the Station Adder provides images and/or calibration solutions that are noisier than expected. The cause of this higher noise was identified in an incorrect calculation of the visibilities for the super station: while a weighted sum of the visibilities should be calculated for the output combined visibility, only a simple sum of visibilities is actually calculated by NDPPP. The use of a simple sum instead of a weighted one results in an increase in image noise of about 30%.

A corrected version of NDPPP will be in production (LofIm) on 28 Jul 2015 and will be implemented in the ASTRON pipelines from the next software roll-out, which is currently scheduled in early September 2015. For data sets that have already been processed through the ASTRON pipelines, a script (fix_weightedsum_uvw.txt) is provided to fix this issue.

LOFAR Measurement Sets structure temporarily not conform with CASA (29/06/15 - 24/08/15)

During the period June 29th - August 24th 2015 the ASTRON pipelines wrote LOFAR Measurement Sets (MSs) in a manner inconsistent with the MS definition v2 because:

• a change in the coordinates by NRAO since release 2.0 introduced a type Undefined and an MS produced with a casacore newer than release 1.7 is not readable by old CASA.

Also, all LOFAR MS have been generated with two tables (STATE,PROCESSOR) missing. However, since release 4.1 a check for tables is done by CASA, thus causing a failure.

Users working with CASA 4.1 or higher on MS generated before September 2nd 2015 need to fix their MSes.

A script to add missing tables in the MS to conform with CASA standards and to make a fix in the MeasInfo in the spectral window table is available here. The software requirements to run this script on your systems are:

1. TaQL (so casacore)

2. pyrap

Usage:

> tar -xvf fixLofarMSAug2015.py.tar

> python fixLofarMSAug2015.py <name MS>

In case you experience issues when running this script, please contact SDC Operations through the SDC-helpdesk which is hosted at https://support.astron.nl/sdchelpdesk.

Wrong Imaging pipeline settings (up to 26/06/15)

An error was found in the settings of the phase-only calibration parset of the ASTRON Imaging Pipeline. The error was fixed on the 26th of June 2015.

The settings used applied the default parameters present in the solution tables (parmdb) to the phases, which is zero in the case of the phase. The effect is that no correction is applied to the phase and therefore the phase values in the CORRECTED_DATA column are the same as those in the DATA column.
To fix the issue we performed the following changes:

- in the BBS step Solve the mode has been changed from Phase to Complex
- in the BBS step Correct the Phasors.Enable=F has been enabled to Phasors.Enable=T.

In order to obtain the proper results from your gms phase-only calibration, we advise you to repeat the initial phase-only calibration using the parset reported below.
BBS.Step.Solve.Baselines=

BBS.Step.Solve.Correlations=[]

BBS.Step.Solve.Model.Bandass.Enable=F

BBS.Step.Solve.Model.Beam.ConjugateAF=F

BBS.Step.Solve.Model.Beam.Element.Path=${LOFARROOT}/share

BBS.Step.Solve.Model.Beam.Enable=true

BBS.Step.Solve.Model.Beam.Mode=ARRAY_FACTOR

BBS.Step.Solve.Model.Cache.Enable=T

BBS.Step.Solve.Model.DirectionalGain.Enable=F

BBS.Step.Solve.Model.Flagger.Enable=F

BBS.Step.Solve.Model.Flagger.Threshold=0

BBS.Step.Solve.Model.Gain.Enable=T

BBS.Step.Solve.Model.Ionosphere.Degree=0

BBS.Step.Solve.Model.Ionosphere.Enable=F

BBS.Step.Solve.Model.Ionosphere.Type=

BBS.Step.Solve.Model.Phasors.Enable=T

BBS.Step.Solve.Model.Sources=[]

BBS.Step.Solve.Operation=SOLVE

BBS.Step.Solve.Output.Column=

BBS.Step.Solve.Output.WriteCovariance=F

BBS.Step.Solve.Output.WriteFlags=F

BBS.Step.Solve.Solve.Algorithm=L2

BBS.Step.Solve.Solve.CalibrationGroups=[]

BBS.Step.Solve.Solve.CellChunkSize=10

BBS.Step.Solve.Solve.CellSize.Freq=0

BBS.Step.Solve.Solve.CellSize.Time=1

BBS.Step.Solve.Solve.EpsilonL1=[1e-4,1e-5,1e-6]

BBS.Step.Solve.Solve.ExclParms=[]

BBS.Step.Solve.Solve.Log.Level=NONE

BBS.Step.Solve.Solve.Log.Name=solver_log

BBS.Step.Solve.Solve.Mode=COMPLEX

BBS.Step.Solve.Solve.Options.BalancedEqs=F

BBS.Step.Solve.Solve.Options.ColFactor=1e-9

BBS.Step.Solve.Solve.Options.EpsDerivative=1e-9

BBS.Step.Solve.Solve.Options.EpsValue=1e-9

BBS.Step.Solve.Solve.Options.LMFactor=1.0

BBS.Step.Solve.Solve.Options.MaxIter=100

BBS.Step.Solve.Solve.Options.UseSVD=T

BBS.Step.Solve.Solve.OutlierRejection.Enable=F

BBS.Step.Solve.Solve.OutlierRejection.Threshold=[7.0,5.0,4.0,3.5,3.0,2.8,2.6,2.4,2.2,2.5]

BBS.Step.Solve.Solve.Parms=["Gain:0:0:Phase:*","Gain:1:1:Phase:*"]

BBS.Step.Solve.Solve.PhaseShift.Direction=[]

BBS.Step.Solve.Solve.PhaseShift.Enable=F

BBS.Step.Solve.Solve.PropagateSolutions=F

BBS.Step.Solve.Solve.Resample.CellSize.Freq=0

BBS.Step.Solve.Solve.Resample.CellSize.Time=0

BBS.Step.Solve.Solve.Resample.DensityThreshold=1.0

BBS.Step.Solve.Solve.Resample.Enable=F

BBS.Step.Solve.Solve.UVRange=[]

BBS.Step.Solve.Model.TEC.Enable =F

BBS.Step.Solve.Model.Beam.UseChannelFreq=T

BBS.Step.Correct.Baselines=

BBS.Step.Correct.Correlations=[]

BBS.Step.Correct.Model.Bandpass.Enable=F

BBS.Step.Correct.Model.Beam.ConjugateAF=F

BBS.Step.Correct.Model.Beam.Element.Path=${LOFARROOT}/share

BBS.Step.Correct.Model.Beam.Enable=true

BBS.Step.Correct.Model.Beam.Mode=ARRAY_FACTOR

BBS.Step.Correct.Model.Cache.Enable=true

BBS.Step.Correct.Model.DirectionalGain.Enable=F

BBS.Step.Correct.Model.Flagger.Enable=F

BBS.Step.Correct.Model.Flagger.Threshold=0

BBS.Step.Correct.Model.Gain.Enable=T

BBS.Step.Correct.Model.Ionosphere.Degree=0

BBS.Step.Correct.Model.Ionosphere.Enable=F

BBS.Step.Correct.Model.Ionosphere.Type=

BBS.Step.Correct.Model.Phasors.Enable=T

BBS.Step.Correct.Model.Sources=[]

BBS.Step.Correct.Operation=CORRECT

BBS.Step.Correct.Output.Column=CORRECTED_DATA

BBS.Step.Correct.Output.WriteCovariance=T

BBS.Step.Correct.Output.WriteFlags=F

BBS.Step.Correct.Model.TEC.Enable =F

BBS.Step.Correct.Model.Beam.UseChannelFreq=T

BBS.Step.nrOfDefaultBBSStep=2

BBS.Strategy.Baselines=*&

BBS.Strategy.ChunkSize=100

BBS.Strategy.Correlations=[]

BBS.Strategy.InputColumn=DATA

BBS.Strategy.Steps=[Solve, Correct]

BBS.Strategy.TimeRange=[]

BBS.Strategy.UseSolver=F

Inaccurate flagging of LBA data (19/10/12 - 25/01/16)

Low band (LBA) data recorded and (pre-)processed between 19-10-2012 and 25-01-2016 were not properly flagged by the Radio Observatory pipelines because of an error in the definition of the flagging strategy used for LBA data (LBAdefault) by AOflagger via NDPPP. The level of accuracy of the flagging step differs in different data, depending on both pointing and processing specifications such as, e.g., the target elevation and frequency/time averaging factors. A solution to this problem is described below. The error has been corrected at the parset level so all data recorded and processed after 25-01-2016 are not affected by this problem.

An error occurred in the code (specifically a “write flags to file” action being incorrectly placed) of the LBA flagging strategy which resulted in a crash of the AO-flagger and a non-optimal RFI detection / flagging results.
This implies that some RFI could be left in the visibilities of your observations, so we ask that you inspect your data carefully during the data reduction.

NOTE: The visibilities were properly recorded and can be fixed in various ways, according to the degree of processing applied by ASTRON in the pre-processing phase before the visibilities were sent to the LTA.

Below we describe the steps to perform to fix the issue, depending on the stage of pre-processing applied to your visibilities.

1. Flagged data that did not undergo any frequency compression: to solve the issue the data need to be flagged again. A new version of the LBAdefault strategy can be used, but the developer’s advise is to use the standard AOFlagger strategy (see below).
2.  Flagged data that have been averaged in frequency: in order to mitigate the issue the data need to be flagged again using a new version of the LBAdefault strategy. The developer’s advise is to use the HBAdefault strategy, instead of the LBAdefault strategy. The effectiveness of this step will depend on the level of frequency and time averaging.

USAGE

To select the flagging strategies, users should use the following syntax in the NDPPP parset:

aoflagger.type=aoflagger

aoflagger.strategy=           # to select the default AOFlagger strategy

or

aoflagger.type=aoflagger

aoflagger.strategy=LBAdefault # to select the LBA (HBA) strategies

To inform future users of this error, a pop-up warning will be implemented in the LTA when users retrieve data observed during the aforementioned period.

In case you experience issues when applying these instructions or you need help in any of these steps, please do not hesitate to contact SDC Operations through the SDC-helpdesk which is hosted at https://support.astron.nl/sdchelpdesk

Station sensitivity issue (31/08/15 - 07/06/16)

Between 31^st August 2015 and 7^th June 2016 an issue existed where the sensitivity of some stations was reduced when observing using the HBA. The observed effect:

• A number of stations may have had no sensitivity (i.e., not even RFI was visible in inspection plots) or obviously reduced sensitivity compared to others;

• A number of stations may have suffered a sudden gain or loss in sensitivity during the observation;

• The number of stations affected and the exact stations affected varied with observation and was not predictable. However:

  • It predominantly, but not exclusively, affected HBA1 core stations;

  • It was more apparent when larger numbers of stations were used: e.g., HBA_DUAL with the full array was strongly affected whereas it was only occasionally noticed in core-only observations.

When it occurred:

• The effect appeared for any HBA observation following on from a previous HBA observation if the gap between observations was less than three minutes.

• In a run of HBA observations, the third observation onwards would have a greater number of stations affected (possibly 10 or more).

The issue was caused by the HBA analogue tile beam being incorrectly set under some circumstances. There were two problems:

1. The analogue beam was sometimes set incorrectly at the HBA tiles due to a communication failure. In the past the HBA analogue pointing was sent to the tiles at fixed intervals of 180s. An RSP Driver update issued on 31^st August 2015 updated the analogue pointing only if had changed from that stored in cache. Communication failures could mean that the analogue beam was not updated when it should have been.

2. The analogue tile beam is re-calculated every 180s (the so-called “analogue heartbeat”). If a new beam were set at a time which was not in sync with the heartbeat of a currently-set beam, setting of the new beam was delayed until the next heartbeat and the duration of the new beam was not changed to account for this difference in time. Since beam duration is currently equal to the observation duration, this could lead to the beam from an observation remaining set at the station for up to 180s after the formal end of the observation. When the gap between successive observations was one minute, this could lead to a situation where the beam of the previous observation remained set for up to two minutes at the start of the new observation. This has been the situation since 22^nd March 2011, but only became noticeable when the gap between observations was reduced to one minute and only became an issue for the whole observation in combination with the other bug detailed above.

Both of these problems have now been resolved: The HBA analogue pointing is now updated every 10s, the duration of any beam will not exceed the duration of the observation and updates to the pointing have been made more robust against communication failures.

In the majority of cases, the effect on observations should be minimal:

Most interferometric observing runs are structured such that a long observation of the target is bookended by short observations of the calibrator of, typically, 10 minutes in length. In this case, the first calibrator will usually be unaffected and only the first few minutes of the target observation and the same for the second calibrator may be affected. The effect will be seen as low amplitudes for cross-correlation intensities on baselines where one or both stations suffer from reduced sensitivity.

The majority of beam-formed observations using the core only are unaffected. Unfortunately, it will not be possible to tell without studying the station dynamic spectra (which are only available for two months currently) or unless the beam intensity(s) are obviously reduced compared to expected.

We are very sorry, but there is no way in which data taken can be corrected for this issue.

Delay compensation issue (14/04/14 - 13/09/16)

Between April 14-th 2014 (LOFAR Release 2.2) and September 13-th 2016 (LOFAR Release 2.16.7) there was an issue with the LOFAR array, specifically with the antenna cable delay compensation not being properly set which has potentially affected some observations.

The issue was caused by RCUs keeping the previous cable delay settings (and ignoring new ones), if the RCUs were still powered on after the end of a previous observation. This happens whenever there is a gap smaller than about 3 minutes between observations.

The issue occurs if and only if RCUs switch modes while the intra-observation time gap is not larger than 3 minutes. It also occurs if the switch is from HBA to any LBA mode, or between any two LBA modes. If a chain of observations all have less than 3 minute gaps, the cable delay setting used for the entire sequence is that of the first observation. Within this scenario, if the splitter in the core stations switches state (a change between HBA and HBA_DUAL mode) the RCUs belonging to ring 1 will be reset and receive proper delays at the time of the ring splitter switching state, meaning that for HBA1 the issue ceases to exist, but is still present for HBA0.

The issue manifests itself as a multi-MHz-scale ripple over the pass-band of the affected stations causing a "ripple effect" in sensitivity with an amplitude of up to 50% of the peak gain (as shown in the figure below); some SBs having higher sensitivity than others, with lower frequencies being more affected than higher ones. Unfortunately, there is no way to recover the loss of sensitivity and distorted beam shapes in affected data. A list of project / observations which may be affected by this issue can be downloaded here.

Figure 1: An example of bad amplitude solutions (reflecting the station sensitivity). with a clear modulation in frequency, here indicated as SB number.

Faulty LBA tables (21/09/15 - 22/11/16)

Some of the LBA calibration tables installed in the stations in the period between 21st September 2015 and 22nd November 2016 have suffered from format issues that prevented them to be correctly read by the system during observations. This means that at the stations where such faulty tables were installed, data were collected without the application of a calibration table and the delay corrections that it provides. Instead of the table installed, a "default" (empty) table was loaded. It is estimated that on average in LBA one can expect that the lack of a calibration table would turn into a noise up to about 20% larger than with proper calibration. Unfortunately, this cannot be corrected in the data reduction process.

Since the end of November 2016, a new alert system has been implemented to avoid that such a problem may affect again LOFAR data in the future.

Between 19 May 2016 and 22 November 2016, the following stations had unreadable calibration tables in LBA OUTER:

Core Stations: 001, 002, 003, 004, 005, 006, 007, 011, 017, 024, 026, 028, 030, 032, 101, 103, 201, 302, 501, and 401

Remote stations: 210, 306, 307, 310, 406, 503, and 508.

This affected the following projects:
IPS_Commissioning
LC5_001
LC6_001
LC6_002
LC6_007
LC6_008
LC6_010
LC6_019
LC6_023
LC6_027
LC7_009
LT5_009

Between 21 September 2015 and 22 November 2016, station CS013 had an unreadable LBA INNER calibration table.

This affected the following projects:
Commissioning2016 (Total power spectroscopy)
DDT5_002
Hexacopter
LC3_022
LC6_001
MSSS_LBA

The associated observation IDs are given in the following spreadsheets:

LBA_Inner_missing_tables

LBA_Outer_missing_tables

Cobalt geographical delays offset (19/12/13 - 29/06/15)

It has been discovered that the geographical delays applied to the data stream by the Cobalt correlator were off by 1 second. A fix was implemented, but observations performed between December 19, 2013 and June 29, 2015 are affected. Due to the magnitude of the offset, only long baseline data reduction can potentially have issues. For data reduction strategies using the standard long baseline pipeline, calibration solution transfer for up to 2 degrees in HBA (using appropriate calibrators) shows no adverse effects due to this issue. Also, standard long baseline processing in LBA shows not to be affected by this issue.

In beam-formed observations, the issue resulted in a slight mis-pointing of tied-array beams:  The pointing direction of the tied-array beam was calculated for one second later than the actual time, meaning that the beam would have been pointing slightly west of the specified position.  This means that the actual pointing direction would have a slightly lower right ascension than specified.  Hence a value of 15arcsec cos(declination) should be subtracted from the specified pointing to correct for this issue.

For all relevant timing issues, please consult the timing delays system history page.

Delay compensation DE601 (09/12/16 - 06/06/17)

Data taken between December 9, 2016 and June 6, 2017 have an extra delay of 1 micro second on baselines involving DE601. There was an extra delay manually introduced into the system due to the speciffics of its GPS unit. The unit was replaced in December, but the extra system delay for that station was not removed until June.

This can be accounted for during data reduction by fringe-fitting assuming that there was no large frequency averaging applied. For example, a frequency resolution of 4 ch/ SB is sufficient for reduction, but data averaged to 1 ch/SB will have issues.

A possible fringe fitting strategy is to subtract 1 microsecond delay from all the baselines containing DE601 and fringe fit using a delay window of around 600 nanoseconds.

The issue affects only interferometric observations using the Cobalt correlator.

Wrong position DE605 ( up to 20/04/17)

Data taken before April 20, 2017 had wrong position coordinates for station DE605 reported in the metadata. An estimate of the effect on relevant observing modes is given below.

• Interferometric mode:
  Based on the LCBS survey catalog (Jackson et al. 2016, A&A, 595, A86), source detection rates on baselines between the phased LOFAR core and DE601 as well as DE605 are found to be consistent with each other within 5%. This conclusion is robust against ionospheric effects, since DE601 is located close to DE605 (55 km distance). Thus, the station position difference error does not have an effect on interferometric observations performed prior to the correction at the cited significance level.

• Pulsar timing in stand-alone mode:
  The data before the fix are beam-formed correctly to the off-set phase center, as the error is less than one phase wrap. This ANTENNA position should be used in the analysis for those dates. After the correction, the data is beam-formed correctly to the new phase center. For combining the data between these two parts, treat it as two telescopes, or make a conversion between the two systems. If this is not done, there is an error of order 100 ns between the two positions, which is AZ/EL dependent.

• The current up-to-date DE605 coordinates are:
  # ITRF2005 target_date = 2017.5
  # Created: 2017-04-13 08:23:22
  LBA
  4005681.284790000 450968.703003000 4926457.990
  HBA
  4005717.989780000 451028.465007000 4926424.423