The timing of LOFAR is complicated because it is more than just one central clock and the system has changed over time. Therefore this overview has been created to explain how the system works and to track fundamental changes in the system.
The clocks of LOFAR are synchronized by GPS receivers to time the second, with the finer time resolution provided by a rubidium clock. This clock gives the timestamp for the different systems. The actual time of the received signal for each antenna depends on a number of delays in various parts of the LOFAR system. The best values are measured regularly and applied to the system. At certain points there was a fundamental change in the setup of LOFAR, and thus in the delays that need to be applied. The purpose of this page is to communicate what changes have been made, what calibration values have been measured and how these values can be obtained.
There are different delays in the system between the arrival of the signals at the antenna and the final data product that has been produced. A schematic is presented in figure 1. The signal first arrives at the antenna. It then goes through a cable that is connected to the Receiver Unit (RCU). These cables can have different lenghts for the different antenna’s in the station. At the RCU the timestamp is given to the data. This timestamp is corrected for the cable delay, up to the sample time. A further correction is applied as part of the station calibration. The timestamp is provided by a clock per station (remote and international stations) or by the single clock for the core (current situation, see history). The core station timestamps are therefore delivered with a delay. The timestamp given to the data is thus too early. A correction for this is made at the correlator.
Figure 1: Schematic overview of applied delays in the LOFAR system.
An overview of all software changes is available at the LOFAR wiki http://www.lofar.org/wiki/doku.php?id=maintenance:lofar_software_changes . Here only the issues relevant to timing are presented.
Fundamental system changes
The table below displays the dates of fundamental system changes. They are described below in detail.
| 2011-05-11 | Installation single clock for the superterp |
| 2012-10-31 | Single clock expanded to all stations. SyncOptics installed. |
| 2013-08-28 | 1.38 us offset of superterp stations introduced |
| 2013-11-14 | 1.38 us offset of superterp stations fixed |
| 2014-05-12 | COBALT replaces BlueGene/P as correlator |
| 2014-06-06 | New delay tables for Cobalt |
| 2015-01-26 | Reverted back to old tables because of polarisation issue in pulsar pipelines |
| 2015-06-01 | Longer cable in single clock station, 19 ns offset added for all core stations |
| 2015-06-29 | COBALT pointing calculation corrected from one second off-set |
Inter-Station timing
Installation of a single clock for the superterp. (2011-05-11)
At 11 May 2011 a single clock was installed for the superterp stations ( CS002 CS003 CS004 CS005 CS006 CS007 ) to allow coherent beam forming. The GPS and Rubidium module are located in the Concentrator Node and from there on the reference signals are distributed via fibre connections to six superterp stations. The timestamp is generated at the concentrator node, and this arrives with a delay to the station. The timestamp of the data is thus too early, and a correction needs to be applied. This is both a delay and a phase-offset. At first only the delay is applied at the correlator and before data is written to disk. The values are calculated with respect to the median of the other stations. They are given in the parset from a specific observation. The parset may also be part of the metadata of a measurement set and beam formed data
The keys are:
PIC.Core.CS002HBA.clockCorrectionTime = 8.291395e-06
PIC.Core.CS002HBA.phaseCenter = [3826583.3205,460955.708,5064894.168]
PIC.Core.CS002HBA0.clockCorrectionTime = 8.291555e-06
PIC.Core.CS002HBA0.phaseCenter = [3826601.004,460953.354,5064881.107]
PIC.Core.CS002HBA1.clockCorrectionTime = 8.291235e-06
PIC.Core.CS002HBA1.phaseCenter = [3826565.637,460958.062,5064907.229]
PIC.Core.CS002LBA.clockCorrectionTime = 8.291395e-06
where CS002 needs to be replaced with the actual station name.
Where can the user find this data?
Beamformed data: PROCESS_HISTORY / OSERVATION_PARSET
import h5py
f=h5py.File(<filename>,r)
parset=f[u'SUB_ARRAY_POINTING_XXX'][u'PROCESS_HISTORY'].attrs['OBSERVATION_PARSET']
with XXX the pointing (usually 000).
Correlated data:
The parset is stored as an array of key=value in table:
HISTORY / APP_PARAMS
Installation of a single clock for all core stations. (2012-10-31)
At 2012-10-31 the single clock was expanded to all core stations. Also syncoptics fibers replaced the copper wires, increasing the stability.
1.38 us offset of superterp stations (introduced 2013-08-28, fixed 2013-11-14)
A 1.38 us offset was discovered from single clock stations of the superterp compared to other core stations and adjusted since 2013-11-14 5:18:00 UT. The clockCorrectionTime was reduced by this number. This error was introduced on 2013-08-28.
Installation of COBALT
At 2014-05-12 the BlueGene/P correlator got replaced by the COBALT correlator. This means that issues with the old correlator are no longer there, but issues with the new correlator start at this date. Also, the clock corrections changed, as now also a phase could be applied. The values are no longer available in the parset on CEP2 (TBB data), but are still available in the pulsar and correlated data. The values and changes can also be found in the LOFAR svn repository. The location is:
https://svn.astron.nl/LOFAR/trunk/RTCP/Cobalt/GPUProc/etc/parset-additions.d/default/StationCalibration.parset
The trunk version may not always be in sync with the different release branches and sometimes are only updated later.
The structure of the keys is now:
PIC.Core.<stationname><antennatype>.<antennaset>.<filter>.phase0.X = <value>
PIC.Core.<stationname><antennatype>.<antennaset>.<filter>.phase0.Y = <value>
PIC.Core.<stationname><antennatype>.<antennaset>.<filter>.delay.X = <value>
PIC.Core.<stationname><antennatype>.<antennaset>.<filter>.delay.Y = <value>
antennatype in [LBA,HBA]
antennaset in [LBA_INNER, LBA_OUTER,HBA_ZERO,HBA_ONE,HBA_DUAL,HBA_DUAL_INNER,HBA_JOINED,HBA_JOINED_INNER]
filter in [LBA_10_70,LBA_30_70,LBA_10_90,LBA_30_90,HBA_110_190,HBA_170_230,HBA_210_250]
examples:
PIC.Core.CS001LBA.LBA_INNER.LBA_10_90.phase0.X = 0.000000e+00
PIC.Core.CS001LBA.LBA_INNER.LBA_10_90.phase0.Y = 0.000000e+00
PIC.Core.CS001LBA.LBA_INNER.LBA_10_90.delay.X = 4.675295e-06
PIC.Core.CS001LBA.LBA_INNER.LBA_10_90.delay.Y = 4.674852e-06
PIC.Core.CS001HBA0.HBA_ZERO.HBA_110_190.phase0.X = 0.000000e+00
PIC.Core.CS001HBA0.HBA_ZERO.HBA_110_190.phase0.Y = 0.000000e+00
PIC.Core.CS001HBA0.HBA_ZERO.HBA_110_190.delay.X = 4.675321e-06
PIC.Core.CS001HBA0.HBA_ZERO.HBA_110_190.delay.Y = 4.674847e-06
Changes between revisions, example values for CS001 and CS002
Revisions 29188 29151 (initialisation of COBALT tables
Date: 2014-05-09 13:58:51.505842+00:00
dDelayX: [ 1.11440386e-08 1.09030225e-08 2.23078814e-06 2.23074676e-06] std: 6.65046e-06
dDelayY: [ 1.06701918e-08 1.04910214e-08 2.23114193e-06 2.23147799e-06] std: 6.65038e-06
dPhaseX: [ 0.49561861 0.53390652 0.49561861 0.53390652]
dPhaseY: [-0.08541571 0.1352897 -0.08541571 0.1352897 ]
Revisions 29417 29188
Date: 2014-06-04 14:53:39.488859+00:00
dDelayX: [ 2.31921149e-10 2.31011654e-10 4.49745130e-10 -2.38742359e-10] std: 5.64933e-10
dDelayY: [ 5.19776222e-10 6.68933353e-10 -4.86579665e-11 1.74168235e-10] std: 5.96959e-10
dPhaseX: [-0.47510129 -0.74432021 0.08626603 0.4866339 ]
dPhaseY: [-0.44889069 -0.37937009 0.0705175 -0.04821974]
Revisions 29455 29417 (Revert to 29188 values. Not sure if these were actually applied, no entries in software changes)
Date: 2014-06-06 12:37:29.638227+00:00
dDelayX: [ -2.31921149e-10 -2.31011654e-10 -4.49745130e-10 2.38742359e-10] std: 5.64933e-10
dDelayY: [ -5.19776222e-10 -6.68933353e-10 4.86579665e-11 -1.74168235e-10] std: 5.96959e-10
dPhaseX: [ 0.47510129 0.74432021 -0.08626603 -0.4866339 ]
dPhaseY: [ 0.44889069 0.37937009 -0.0705175 0.04821974]
Revisions 29650 29455
Date: 2014-06-25 13:22:46.962918+00:00
dDelayX: [ 2.87855073e-10 -5.93900040e-10 4.57930582e-10 -5.74800652e-10] std: 1.31625e-09
dDelayY: [ 9.45874490e-10 1.90084393e-10 -4.36557457e-11 -2.16914486e-10] std: 1.26369e-09
dPhaseX: [-0.30759931 -0.72099388 -0.4138284 -0.56180561]
dPhaseY: [-0.4375329 -0.21744861 0.1209249 -0.21931329]
Revisions 30737 29650 (See Polarisation issue below, revert to revision 29455 values.)
Date: 2015-01-21 10:57:34.757096+00:00
dDelayX: [ -2.87855073e-10 5.93900040e-10 -4.57930582e-10 5.74800652e-10] std: 1.31625e-09
dDelayY: [ -9.45874490e-10 -1.90084393e-10 4.36557457e-11 2.16914486e-10] std: 1.26369e-09
dPhaseX: [ 0.30759931 0.72099388 0.4138284 0.56180561]
dPhaseY: [ 0.4375329 0.21744861 -0.1209249 0.21931329]
Revisions 31945 30737 (See Network reconfiguration below)
Date: 2015-06-29 13:04:34.976108+00:00; Rolled out already on June 1st, release
dDelayX: [ 1.90002538e-08 1.89997991e-08 1.89997991e-08 1.90002538e-08] std: 5.12555e-13
dDelayY: [ 1.89997991e-08 1.89997991e-08 1.90002538e-08 1.89997991e-08] std: 5.39181e-13
dPhaseX: [ 0. 0. 0. 0.]
dPhaseY: [ 0. 0. 0. 0.]
Polarisation issue
With COBALT it was possible to add a phase as well as a delay for each station, and this was done at 2014-07-12. Unfortunately the applied values appeared to give polarisation leakage. This is because the ionospheric effects were not averaged out over the observation. It was decided that the optimum solution was to go back to the previously applied values at 2015-01-21. See also http://www.astron.nl/radio-observatory/observing-capabilities/depth-technical-information/system-notes/polarization-leaka
Network reconfiguration
During the network reconfiguration at 18 to 28 may 2015 a longer cable was needed to connect the single clock that is used for all core stations. This caused an extra delay between the stations compared to the remote and international stations. An initial estimated value of 19 ns was added to the delay correction of the stations at 2015-06-01. This value should be accurate within 2 ns. The value will be measured more precisely in the next calibration run (Status 2015-11-18).
Pointing calculation off by one second in COBALT (2014-05-12 - 2015-06-29 )
In June 2015 a bug was discovered in the geographical delays applied at Cobalt. The delay correction for each second was calculated for the next second. This means the pointing has been off by 15 arcsec cos(delta), causing larger delays on longer baselines, since the start of the use of Cobalt. This was corrected in LOFAR Release 2.11, and rolled out at 2015-06-29. This influences the absolute pointing and timing between 2014-05-12 and 2015-06-29. [Redmine issue 8077]
A roll-back to the previous situation has been seen in pulsar timing data since December 2015. This issue is currently under investigation.
Timing issue April 2018 for 8 core stations.
From 2018-04-24 till 2018-04-30 there was a clock offset of 1.4 microseconds with respect to the other stations for CS002, CS003, CS004, CS005, CS006, CS007, CS302 and CS401. The different boards can have an offset in the order of 10s of ns. This caused the sensitivity of these stations to drop significantly.
Intrastation timing
Cable delays
The different antennas generally have a different cable length to connect them to the receiver units. A first estimate of this delay is applied in a sample shift up to a 5 nanosecond (6.25 nanosecond for 160MHz clock). The remaining delay is corrected for in the station calibration.
Station calibration
A further correction is applied as part of the station calibration. Station calibration is performed with respect to the same antenna. Therefore, an update of the station calibration table will potentially introduce a small (<= 5 ns ) absolute timing residuals / phase gradients. An overview of when station calibration was changed is under construction.
In very early commissioning data (2009 or 2010) the correction for the cable delay in each station was not applied. This caused an interference pattern for the LBA arrays, and the HBA arrays from remote and international stations because of a different cable length for the different antenna's. Also this means an absolute offset of 200-465 nanosecond, compared to observations where the sample shift was applied.
International stations in stand-alone operation
The timing of international stations is only influenced by the calibration table. More information about when these are installed will be available soon.
Polarization
There is currently no system in place to measure the offset between the two polarizations. It is therefore possible that there is an off-set between the X and Y polarisation and that this is visible in the data, depending on how much these two contribute.
TBB data
The only correction on the TBB data is the integer correction on the cable delay for the antennas. The other corrections need to be applied by the user. This may already be part of the software (PyCRtools).