Re: [Boost-users] multi_array with large dimensions crashes
Hi Pierre, Sorry for the late reply. The code work now, thanks very much. But the programme crashes with a run-time error when the dimensions of the array are huge 500x500x500. This is probably due to the limitation of the 32 bit addrress space. Is there any other work around. I heard of creating arrays with strides and allocating them in different blocks of memory rather than in contiguos memory? Thanks again Jothy On Wed, Aug 24, 2011 at 6:25 PM, Pierre-Andre Noel < noel.pierre.andre@gmail.com> wrote:
The code work now, thanks very much. But the programme crashes with a run-time error when the dimensions of the array are huge 500x500x500.
In your examples, you were using "float" variable, which requires 4 bytes to store. If you want a vector of 500*500*500=125000000 floats, that requires 500000000 bytes, i.e., 0.5 GB (or 0.466 GiB), of contiguous space in memory.
This is probably due to the limitation of the 32 bit addrress space.
For 32 bits, the total amount of addressable space is 4 GB. So no, that is not directly the problem.
I fail to see how strides may help. However, requesting memory in a non-contiguous way may certainly help. The easiest way to do so would be to use a vector at the first level. typedef std::vector< boost::multi_array< data_type, 2 > > vector_of_2D_multi_array; When using this kind of object, remember that this is not a 3 dimensional multi_array but instead a vector of two dimensional ones. If you cannot reach 500x500x500 with this setting, try to see if your OS imposes limitations to the amount of memory that can be allocated to a single process. Finally, if you aim for even larger sizes, the amount of system memory will eventually become a problem. In such cases, you may want to consider distributed computing... Hope it helps, Pierre-André On Thu, Aug 25, 2011 at 5:04 AM, Jothy <jothybasu@gmail.com> wrote:
Just for curiosity,... can a custom allocator help in this case? As you know the third template parameter of a multi_array is an allocator. Thanks, Alfredo
I would like to point out something (that may be known), but may be confusing to others that read this post and will affect the data written to the arrays as it appears linearly. This is regarding the fastest increasing axis: in C/C++ float c_array[z_dim][y_dim][x_dim] ; defines a row major array with the x_dim axis being the axis of the fastest increasing index. You can specify any axis you want to be the fastest increasing dimension just so long is that is how you expect the data to appear in linear memory and it is accessed accordingly. I think: boost::extents[xDim][yDim][zDim] should be: boost::extents[zDim][yDim][xDim] as: #include <boost/multi_array.hpp> int main( void ) { int x_dim = 4; int y_dim = 3; int z_dim = 2; float* mat_1_data = new float[x_dim * y_dim * z_dim]; float* mat_2_data = new float[x_dim * y_dim * z_dim]; boost::multi_array_ref<float, 3> mat_1( mat_1_data, boost::extents[z_dim][y_dim][x_dim] ); boost::multi_array_ref<float, 3> mat_2( mat_2_data, boost::extents[x_dim][y_dim][z_dim] ); for( int z = 0; z < z_dim; z++ ){ for( int y = 0; y < y_dim; y++ ){ for( int x = 0; x < x_dim; x++ ){ mat_1[z][y][x] = z * x_dim * y_dim + y * x_dim + x; } } } for( int x = 0; x < x_dim; x++ ){ for( int y = 0; y < y_dim; y++ ){ for( int z = 0; z < z_dim; z++ ){ mat_2[x][y][z] = x * y_dim * z_dim + y * z_dim + z; } } } for( int i = 0; i < 12; i++ ) { printf( "%f : %f\n", mat_1_data[i], mat_2_data[i] ); } delete [] mat_1_data; delete [] mat_2_data; } yields: 0.000000 : 0.000000 1.000000 : 1.000000 2.000000 : 2.000000 3.000000 : 3.000000 4.000000 : 4.000000 5.000000 : 5.000000 6.000000 : 6.000000 7.000000 : 7.000000 8.000000 : 8.000000 9.000000 : 9.000000 10.000000 : 10.000000 11.000000 : 11.000000 with the first matrix mat_1 being "typical usage" where x is the axis of fastest increase as: mat_2[x][y][z] = x * y_dim * z_dim + y * z_dim + z; indexing is a bit odd, but perfectly acceptable so long as this is what is desired if the order of the for loops is changed for mat_2 so it loops z then y then x the array will not be written linearly and can be seen if the array is printed while it is being filled in. though what was originally posted may very well be what is desired by the original author.
participants (4)
-
alfC -
Brian Davis -
Jothy -
Pierre-Andre Noel