One can check how the memory management described above applies to his
dataset by enabling the debugging messages before invoking the command
XY_MAP. In practice, reusing the same dataset as in the
Table , this gives:
sic message class s+d ! Enable Debug messages sic logical VOLATILE_MEMORY 6GiB ! Custom memory buffer xy_map myfile.tab sic message class s-d ! Disable Debug messages
which shows (some outputs have been hidden for clarity):
I-XY_MAP, Reading columns X Y W [...] Table size: 61263 positions x 20740 values [...] Cube size: 148 by 142 pixels x 20737 channels [...] I-XY_MAP, Creating file: myfile.lmv I-XY_MAP, Creating file: myfile.wei D-XY_MAP, Using at most VOLATILE_MEMORY = 6.00 GB D-XY_MAP, Mblocks: 3, MC: 9787, Size: 29361 20737 D-XY_MAP, -------------------- D-XY_MAP, Doing block 1/3, channels 1:9787, Nchan = 9787 D-XY_MAP>REALLOCATE>2D, Allocated workspace of size: 9787 x 61263 = 2.23 GB D-XY_MAP, Reading table... D-XY_MAP, Sorting table... D-XY_MAP>REALLOCATE>2D, Allocated workspace of size: 9787 x 61263 = 2.23 GB D-XY_MAP>REALLOCATE>3D, Allocated workspace of size: 9787 x 148 x 142 = 784.6 MB D-XY_MAP, Convolving D-XY_MAP, Transposing to LMV... D-XY_MAP>REALLOCATE>3D, Allocated workspace of size: 148 x 142 x 9787 = 784.6 MB D-XY_MAP, Writing LMV cube... D-XY_MAP, -------------------- D-XY_MAP, Doing block 2/3, channels 9788:19574, Nchan = 9787 D-XY_MAP>REALLOCATE>2D, Workspace already allocated at an appropriate size: 9787 x 61263 = 2.23 GB D-XY_MAP, Reading table... D-XY_MAP, Sorting table... D-XY_MAP>REALLOCATE>2D, Workspace already allocated at an appropriate size: 9787 x 61263 = 2.23 GB D-XY_MAP>REALLOCATE>3D, Workspace already allocated at an appropriate size: 9787 x 148 x 142 = 784.6 MB D-XY_MAP, Convolving D-XY_MAP, Transposing to LMV... D-XY_MAP>REALLOCATE>3D, Workspace already allocated at an appropriate size: 148 x 142 x 9787 = 784.6 MB D-XY_MAP, Extending LMV cube... D-XY_MAP, -------------------- D-XY_MAP, Doing block 3/3, channels 19575:20737, Nchan = 1163 D-XY_MAP>REALLOCATE>2D, Allocated workspace of size: 1163 x 61263 = 271.8 MB D-XY_MAP, Reading table... D-XY_MAP, Sorting table... D-XY_MAP>REALLOCATE>2D, Allocated workspace of size: 1163 x 61263 = 271.8 MB D-XY_MAP>REALLOCATE>3D, Allocated workspace of size: 1163 x 148 x 142 = 93.2 MB D-XY_MAP, Convolving D-XY_MAP, Transposing to LMV... D-XY_MAP>REALLOCATE>3D, Allocated workspace of size: 148 x 142 x 1163 = 93.2 MB D-XY_MAP, Extending LMV cube... D-XY_MAP, -------------------- D-XY_MAP, Writing weight image... I-XY_MAP, Time elapsed in command (total): 54.53 sec I-XY_MAP, Time elapsed reading XYW: 36.94 sec I-XY_MAP, Time elapsed reading the table: 5.26 sec I-XY_MAP, Time elapsed sorting the table: 2.01 sec I-XY_MAP, Time elapsed transposing table or cube: 1.80 sec I-XY_MAP, Time elapsed convolving: 5.63 sec I-XY_MAP, Time elapsed writing the cube: 2.80 sec
We find here some interesting informations. The input table provides
positions times
channels (+ 3 values
for X,Y,W). The output cube has
times
pixels
per plane. Using VOLATILE_MEMORY = 6 GiB, XY_MAP chooses
to split the processing in
blocks, using
channels
per block (1163 in the last one).
In each block, we can see the allocation of buffers for each task,
namely ,
,
, and
. The total is 5.99
GB, which fits exactly in the VOLATILE_MEMORY limit. We see
that because the .tab is ordered VP, the table does not need to be
transposed. Note also that the first slice of the cube is written at
the end of the first block. The others slices are then
written by extending the cube.
Finally, the command shows a summary of the processing time per task3. The key point here is that reading the X,Y,W columns takes more than 50% of the total processing time. Again, this is because the table is ordered VP, so the columns are not contiguous in the file (spread all other the file), which must be traversed entirely for such a small information in comparison of the 20737 channels. It would have been almost instantaneous with a PV table. Note also that reading the table data in only 5.26 seconds here, because the system has saved the file in the system cache when we read the X,Y,W columns. This is possible because this example is small and fits in memory. For larger files (greater than system memory), reading (a part of) the table would cost again and again the same time in each block as it took reading X,Y,W. Finally, the main goal of the command which is to add and convolve the data at each pixels takes only about 10% of the overall processing time.