next up previous contents
Next: Dump cycle identification Up: Reading the DATA column Previous: Reading the DATA column   Contents


Discarding bad time dumps

The BackendDATA table provides the data dumps collected during the subscan integration. In particular, it shows the dump MJD, its integration time INTEGTIM, its phase number through cycles ( ISWITCH) and the data itself (channels) in the DATA column.

It may happen that a dump is flagged out by the control system. In this case, its ISWITCH value is 0. Such a dump should be discarded during the calibration process. This is achieved with the command MSET CALIBRATION BAD YES|NO (default NO, i.e. discard bad time dumps).

In practice, this is not just a matter of skipping undesired rows when dealing with the DATA column. The first issue is that the values in the associated columns are unreliable. This is particularly true with the MJD value which is not set to the actual MJD of the dump. This introduces inconsistent values in this column, resulting in an unsorted column, breaking the dichotomic search engine. In order to solve this issue, MRTCAL compresses at read time the MJD, INTEGTIM, and ISWITCH columns, discarding the rows where ISWITCH is null. The size of those columns is decreased from $N_{tot}$ to $N_{good}$ (with $N_{tot} = N_{good} + N_{bad}$).

MRTCAL keeps also track of the discarded dumps thanks to two new columns added. One column named FOREPOIN (size $N_{tot}$) provides a forward pointer, i.e. FOREPOIN(i) is the position of the $i^{th}$ dump (as found in the original columns) in the compressed columns. If the dump is bad, the corresponding position is set to 0 (i.e. dump not available in the compressed columns). It also adds a column named BACKPOIN (size $N_{good}$) which provides a backward pointer: BACKPOIN(j) is the position of the $j^{th}$ dump (as found in the compressed columns) in the original columns.

At this stage, MRTCAL has patched the BackendDATA columns so that the bad dumps are removed, but this is not fully transparent for later use.


next up previous contents
Next: Dump cycle identification Up: Reading the DATA column Previous: Reading the DATA column   Contents
Gildas manager 2023-06-01