next up previous contents
Next: In practice Up: Implementation Previous: Implementation   Contents

Memory management

Several data buffers are needed in this problem. In particular, transposition buffers are needed to go to/from the internal gridding engine which works on velocity-first arrays. The buffers are:

  1. the table data buffer (size $M_{td} = N_{posi} \times N_c$),
  2. the table transposition/sorting buffer (size $M_{tt} = N_{posi}
\times N_c$) if (and only if) the transposition of the table (.bat) and/or sorting (positions not sorted by Y offsets) are needed. The same buffer is used for both tasks.
  3. the cube data buffer (size $M_{cd} = N_x \times N_y \times
N_c$),
  4. the cube transposition/sorting buffer (size $M_{ct} = N_x \times
N_y \times N_c$) if and only transposition of the cube is needed (.lmv).
We call $M_T = M_{td}+M_{tt}+M_{cd}+M_{ct}$ the total amount of memory allocated to the buffers. Note that they all scale as $N_c$. Some other allocatable buffers are involved (e.g. X, Y, W columns read from the table) but they are neglictible in front of the data size, and they do not scale as $N_c$.

The command XY_MAP needs memory space to allocate these buffers. However, it does not allocate freely the memory: it ensures they fit in a limited memory space (VOLATILE_MEMORY, we do not consider here the way it is defined).

If $M_T < $ VOLATILE_MEMORY, the 4 buffers above fit in the dedicated memory space. We are in the best case where all the problem fits in memory. In particular, the output cube fits in memory and will be written at once, whatever its order.

If $M_T > $ VOLATILE_MEMORY, the 4 buffers do not fit in the dedicated memory space. The problem will be iterated by blocks of channels along its velocity dimension. There will be $n$ iterations, the buffers being divided by the same and correct amount. But, in this case, there are 2 possibilities:

At this stage, $n$ is the minimum (floating point) number of divisions needed. The number of channels processed per division is then:

\begin{displaymath}
n_c = \mathrm{floor}(N_c / n)
\end{displaymath} (4)

Rounding to an integer number of channels is obviously needed as the internal engine can not process fraction of channels. Down-rounding is used because up-rounding could mean using more than VOLATILE_MEMORY per division. Because of the integer rounding, the actual number of divisions which will be performed is finally:
\begin{displaymath}
n = \mathrm{ceiling}(N_c / n_c)
\end{displaymath} (5)

Note that the last division may process less channels than the other ones, because the total number of channels $N_c$ may not be a multiple of $n_c$.


next up previous contents
Next: In practice Up: Implementation Previous: Implementation   Contents
Gildas manager 2023-06-01