next up previous contents
Next: About this document ... Up: Implementation Previous: Memory management   Contents

In practice

One can check how the memory management described above applies to his dataset by enabling the debugging messages before invoking the command XY_MAP. In practice, reusing the same dataset as in the Table [*], this gives:

sic message class s+d             ! Enable Debug messages
sic logical VOLATILE_MEMORY 6GiB  ! Custom memory buffer
xy_map myfile.tab
sic message class s-d             ! Disable Debug messages

which shows (some outputs have been hidden for clarity):

I-XY_MAP,  Reading columns X Y W
[...]
Table size: 61263 positions x 20740 values
[...]
Cube size: 148 by 142 pixels x 20737 channels
[...]
I-XY_MAP,  Creating file: myfile.lmv
I-XY_MAP,  Creating file: myfile.wei
D-XY_MAP,  Using at most VOLATILE_MEMORY = 6.00 GB
D-XY_MAP,  Mblocks: 3, MC: 9787, Size: 29361 20737
D-XY_MAP,  --------------------
D-XY_MAP,  Doing block 1/3, channels 1:9787, Nchan = 9787
D-XY_MAP>REALLOCATE>2D,  Allocated workspace of size: 9787 x 61263 = 2.23 GB
D-XY_MAP,  Reading table...
D-XY_MAP,  Sorting table...
D-XY_MAP>REALLOCATE>2D,  Allocated workspace of size: 9787 x 61263 = 2.23 GB
D-XY_MAP>REALLOCATE>3D,  Allocated workspace of size: 9787 x 148 x 142 = 784.6 MB
D-XY_MAP,  Convolving
D-XY_MAP,  Transposing to LMV...
D-XY_MAP>REALLOCATE>3D,  Allocated workspace of size: 148 x 142 x 9787 = 784.6 MB
D-XY_MAP,  Writing LMV cube...
D-XY_MAP,  --------------------
D-XY_MAP,  Doing block 2/3, channels 9788:19574, Nchan = 9787
D-XY_MAP>REALLOCATE>2D,  Workspace already allocated at an appropriate size: 9787 x 61263 = 2.23 GB
D-XY_MAP,  Reading table...
D-XY_MAP,  Sorting table...
D-XY_MAP>REALLOCATE>2D,  Workspace already allocated at an appropriate size: 9787 x 61263 = 2.23 GB
D-XY_MAP>REALLOCATE>3D,  Workspace already allocated at an appropriate size: 9787 x 148 x 142 = 784.6 MB
D-XY_MAP,  Convolving
D-XY_MAP,  Transposing to LMV...
D-XY_MAP>REALLOCATE>3D,  Workspace already allocated at an appropriate size: 148 x 142 x 9787 = 784.6 MB
D-XY_MAP,  Extending LMV cube...
D-XY_MAP,  --------------------
D-XY_MAP,  Doing block 3/3, channels 19575:20737, Nchan = 1163
D-XY_MAP>REALLOCATE>2D,  Allocated workspace of size: 1163 x 61263 = 271.8 MB
D-XY_MAP,  Reading table...
D-XY_MAP,  Sorting table...
D-XY_MAP>REALLOCATE>2D,  Allocated workspace of size: 1163 x 61263 = 271.8 MB
D-XY_MAP>REALLOCATE>3D,  Allocated workspace of size: 1163 x 148 x 142 = 93.2 MB
D-XY_MAP,  Convolving
D-XY_MAP,  Transposing to LMV...
D-XY_MAP>REALLOCATE>3D,  Allocated workspace of size: 148 x 142 x 1163 = 93.2 MB
D-XY_MAP,  Extending LMV cube...
D-XY_MAP,  --------------------
D-XY_MAP,  Writing weight image...

I-XY_MAP,  Time elapsed in command (total):            54.53 sec
I-XY_MAP,  Time elapsed reading XYW:                   36.94 sec
I-XY_MAP,  Time elapsed reading the table:              5.26 sec
I-XY_MAP,  Time elapsed sorting the table:              2.01 sec
I-XY_MAP,  Time elapsed transposing table or cube:      1.80 sec
I-XY_MAP,  Time elapsed convolving:                     5.63 sec
I-XY_MAP,  Time elapsed writing the cube:               2.80 sec

We find here some interesting informations. The input table provides $N_{posi} = 61263$ positions times $N_c = 20737$ channels (+ 3 values for X,Y,W). The output cube has $N_x = 148$ times $N_y = 142$ pixels per plane. Using VOLATILE_MEMORY = 6 GiB, XY_MAP chooses to split the processing in $n = 3$ blocks, using $n_c = 9787$ channels per block (1163 in the last one).

In each block, we can see the allocation of buffers for each task, namely $M_{td}$, $M_{tt}$, $M_{cd}$, and $M_{ct}$. The total is 5.99 GB, which fits exactly in the VOLATILE_MEMORY limit. We see that because the .tab is ordered VP, the table does not need to be transposed. Note also that the first slice of the cube is written at the end of the first block. The others slices are then written by extending the cube.

Finally, the command shows a summary of the processing time per task3. The key point here is that reading the X,Y,W columns takes more than 50% of the total processing time. Again, this is because the table is ordered VP, so the columns are not contiguous in the file (spread all other the file), which must be traversed entirely for such a small information in comparison of the 20737 channels. It would have been almost instantaneous with a PV table. Note also that reading the table data in only 5.26 seconds here, because the system has saved the file in the system cache when we read the X,Y,W columns. This is possible because this example is small and fits in memory. For larger files (greater than system memory), reading (a part of) the table would cost again and again the same time in each block as it took reading X,Y,W. Finally, the main goal of the command which is to add and convolve the data at each pixels takes only about 10% of the overall processing time.


next up previous contents
Next: About this document ... Up: Implementation Previous: Memory management   Contents
Gildas manager 2023-06-01