The BackendDATA table provides the data dumps collected during the subscan integration. In particular, it shows the dump MJD, its integration time INTEGTIM, its phase number through cycles ( ISWITCH) and the data itself (channels) in the DATA column.
It may happen that a dump is flagged out by the control system. In this case, its ISWITCH value is 0. Such a dump should be discarded during the calibration process. This is achieved with the command MSET CALIBRATION BAD YES|NO (default NO, i.e. discard bad time dumps).
In practice, this is not just a matter of skipping undesired rows when
dealing with the DATA column. The first issue is that the values
in the associated columns are unreliable. This is particularly true
with the MJD value which is not set to the actual MJD of the
dump. This introduces inconsistent values in this column, resulting in
an unsorted column, breaking the dichotomic search engine. In order to
solve this issue, MRTCAL compresses at read time the MJD, INTEGTIM, and ISWITCH columns, discarding the rows
where ISWITCH is null. The size of those columns is decreased
from to
(with
).
MRTCAL keeps also track of the discarded dumps thanks to two new
columns added. One column named FOREPOIN (size )
provides a forward pointer, i.e. FOREPOIN(i) is the position of
the
dump (as found in the original columns) in the compressed
columns. If the dump is bad, the corresponding position is set to 0
(i.e. dump not available in the compressed columns). It also adds a
column named BACKPOIN (size
) which provides a
backward pointer: BACKPOIN(j) is the position of the
dump (as found in the compressed columns) in the original columns.
At this stage, MRTCAL has patched the BackendDATA columns so that the bad dumps are removed, but this is not fully transparent for later use.