Re: [Boost-users] multi_array with large dimensions crashes
Hi Pierre, Sorry for the late reply. The code work now, thanks very much. But the programme crashes with a run-time error when the dimensions of the array are huge 500x500x500. This is probably due to the limitation of the 32 bit addrress space. Is there any other work around. I heard of creating arrays with strides and allocating them in different blocks of memory rather than in contiguos memory? Thanks again Jothy On Wed, Aug 24, 2011 at 6:25 PM, Pierre-Andre Noel < noel.pierre.andre@gmail.com> wrote:
Yes, 500, 300 and 400 were the example dimensions you used in your earlier message.
Did you got it to work?
Pierre-André
PS: This email was not sent to the whole list.
On Wed, Aug 24, 2011 at 11:55 AM, Jothy <jothybasu@gmail.com> wrote:
You mean 500,300,400 are the required dimesnions and xDim,yDim,zDim are site_type parameters.
So the code should be like below I guess for an array of [x][y][z]
array_ref_type::size_type xDim(x), yDim(y), zDim(z);
array_storage_type Storage(xDim*yDim*zDim); array_ref_type Array(&Storage.front(),boost::extents[xDim][yDim][zDim]);
Thanks
Jothy
On Wed, Aug 24, 2011 at 3:51 PM, Pierre-Andre Noel < noel.pierre.andre@gmail.com> wrote:
I am not familiar with multi_array, but it seems that you are creating a >1GB object on the stack. This will almost always crash. Try creating it on the heap instead. Be prepared for std::bad_alloc.
But how to do this? I am new to c++.
I suggest you to either 1. read on the operators "new" and "delete" or 2. use a std::vector to allocate the memory then use a boost::multi_array_ref to refer to it. Here's an example:
#include <vector> #include <boost/multi_array.hpp>
int main() { typedef float data_type; typedef std::vector<data_type> array_storage_type; typedef boost::multi_array_ref<data_type,3> array_ref_type;
array_ref_type::size_type xDim(500), yDim(300), zDim(400); array_storage_type Storage(xDim*yDim*zDim); array_ref_type Array(&Storage.front(),boost::extents[xDim][yDim][zDim]); // You may now use Array as usual. }
Hope it helps,
Pierre-André Noël
On Wed, Aug 24, 2011 at 5:20 AM, Jothy <jothybasu@gmail.com> wrote:
Thanks for pointing this out.
But how to do this? I am new to c++.
Jothy
On Tue, Aug 23, 2011 at 10:42 PM, Juraj Ivančić < juraj.ivancic@gmail.com> wrote:
On 23.8.2011. 12:18, Jothy wrote:
Hi all,
I have just started to use boost libs. I am trying to create a large float array (mandatory for my purpose). it works fine till certain limit, but crashes above that. I think it about allocating memory in the stack.
My code is like this
typedefboost::multi_array<**float,3>array_type; typedefarray_type::indexindex; array_typeArray(boost::**extents[xDim][yDim][zDim]);//**500,300,400 for instance
Can some one help me to resolve this?
I am not familiar with multi_array, but it seems that you are creating a >1GB object on the stack. This will almost always crash. Try creating it on the heap instead. Be prepared for std::bad_alloc.
HTH
______________________________**_________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/**mailman/listinfo.cgi/boost-**users<http://lists.boost.org/mailman/listinfo.cgi/boost-users>
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
The code work now, thanks very much. But the programme crashes with a run-time error when the dimensions of the array are huge 500x500x500.
In your examples, you were using "float" variable, which requires 4 bytes to store. If you want a vector of 500*500*500=125000000 floats, that requires 500000000 bytes, i.e., 0.5 GB (or 0.466 GiB), of contiguous space in memory.
This is probably due to the limitation of the 32 bit addrress space.
For 32 bits, the total amount of addressable space is 4 GB. So no, that is not directly the problem.
Is there any other work around. I heard of creating arrays with strides and allocating them in different blocks of memory rather than in contiguos memory?
I fail to see how strides may help. However, requesting memory in a non-contiguous way may certainly help. The easiest way to do so would be to use a vector at the first level. typedef std::vector< boost::multi_array< data_type, 2 > > vector_of_2D_multi_array; When using this kind of object, remember that this is not a 3 dimensional multi_array but instead a vector of two dimensional ones. If you cannot reach 500x500x500 with this setting, try to see if your OS imposes limitations to the amount of memory that can be allocated to a single process. Finally, if you aim for even larger sizes, the amount of system memory will eventually become a problem. In such cases, you may want to consider distributed computing... Hope it helps, Pierre-André On Thu, Aug 25, 2011 at 5:04 AM, Jothy <jothybasu@gmail.com> wrote:
Hi Pierre,
Sorry for the late reply.
The code work now, thanks very much. But the programme crashes with a run-time error when the dimensions of the array are huge 500x500x500. This is probably due to the limitation of the 32 bit addrress space. Is there any other work around. I heard of creating arrays with strides and allocating them in different blocks of memory rather than in contiguos memory?
Thanks again
Jothy
On Wed, Aug 24, 2011 at 6:25 PM, Pierre-Andre Noel < noel.pierre.andre@gmail.com> wrote:
Yes, 500, 300 and 400 were the example dimensions you used in your earlier message.
Did you got it to work?
Pierre-André
PS: This email was not sent to the whole list.
On Wed, Aug 24, 2011 at 11:55 AM, Jothy <jothybasu@gmail.com> wrote:
You mean 500,300,400 are the required dimesnions and xDim,yDim,zDim are site_type parameters.
So the code should be like below I guess for an array of [x][y][z]
array_ref_type::size_type xDim(x), yDim(y), zDim(z);
array_storage_type Storage(xDim*yDim*zDim); array_ref_type Array(&Storage.front(),boost::extents[xDim][yDim][zDim]);
Thanks
Jothy
On Wed, Aug 24, 2011 at 3:51 PM, Pierre-Andre Noel < noel.pierre.andre@gmail.com> wrote:
I am not familiar with multi_array, but it seems that you are creating a >1GB object on the stack. This will almost always crash. Try creating it on the heap instead. Be prepared for std::bad_alloc.
But how to do this? I am new to c++.
I suggest you to either 1. read on the operators "new" and "delete" or 2. use a std::vector to allocate the memory then use a boost::multi_array_ref to refer to it. Here's an example:
#include <vector> #include <boost/multi_array.hpp>
int main() { typedef float data_type; typedef std::vector<data_type> array_storage_type; typedef boost::multi_array_ref<data_type,3> array_ref_type;
array_ref_type::size_type xDim(500), yDim(300), zDim(400); array_storage_type Storage(xDim*yDim*zDim); array_ref_type Array(&Storage.front(),boost::extents[xDim][yDim][zDim]); // You may now use Array as usual. }
Hope it helps,
Pierre-André Noël
On Wed, Aug 24, 2011 at 5:20 AM, Jothy <jothybasu@gmail.com> wrote:
Thanks for pointing this out.
But how to do this? I am new to c++.
Jothy
On Tue, Aug 23, 2011 at 10:42 PM, Juraj Ivančić < juraj.ivancic@gmail.com> wrote:
On 23.8.2011. 12:18, Jothy wrote:
> Hi all, > > I have just started to use boost libs. I am trying to create a large > float array (mandatory for my purpose). it works fine till certain > limit, but crashes above that. I think it about allocating memory in > the > stack. > > My code is like this > > typedefboost::multi_array<**float,3>array_type; > typedefarray_type::indexindex; > array_typeArray(boost::**extents[xDim][yDim][zDim]);//**500,300,400 > for instance > > > Can some one help me to resolve this? >
I am not familiar with multi_array, but it seems that you are creating a >1GB object on the stack. This will almost always crash. Try creating it on the heap instead. Be prepared for std::bad_alloc.
HTH
______________________________**_________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/**mailman/listinfo.cgi/boost-**users<http://lists.boost.org/mailman/listinfo.cgi/boost-users>
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Just for curiosity,... can a custom allocator help in this case? As you know the third template parameter of a multi_array is an allocator. Thanks, Alfredo
I would like to point out something (that may be known), but may be confusing to others that read this post and will affect the data written to the arrays as it appears linearly. This is regarding the fastest increasing axis: in C/C++ float c_array[z_dim][y_dim][x_dim] ; defines a row major array with the x_dim axis being the axis of the fastest increasing index. You can specify any axis you want to be the fastest increasing dimension just so long is that is how you expect the data to appear in linear memory and it is accessed accordingly. I think: boost::extents[xDim][yDim][zDim] should be: boost::extents[zDim][yDim][xDim] as: #include <boost/multi_array.hpp> int main( void ) { int x_dim = 4; int y_dim = 3; int z_dim = 2; float* mat_1_data = new float[x_dim * y_dim * z_dim]; float* mat_2_data = new float[x_dim * y_dim * z_dim]; boost::multi_array_ref<float, 3> mat_1( mat_1_data, boost::extents[z_dim][y_dim][x_dim] ); boost::multi_array_ref<float, 3> mat_2( mat_2_data, boost::extents[x_dim][y_dim][z_dim] ); for( int z = 0; z < z_dim; z++ ){ for( int y = 0; y < y_dim; y++ ){ for( int x = 0; x < x_dim; x++ ){ mat_1[z][y][x] = z * x_dim * y_dim + y * x_dim + x; } } } for( int x = 0; x < x_dim; x++ ){ for( int y = 0; y < y_dim; y++ ){ for( int z = 0; z < z_dim; z++ ){ mat_2[x][y][z] = x * y_dim * z_dim + y * z_dim + z; } } } for( int i = 0; i < 12; i++ ) { printf( "%f : %f\n", mat_1_data[i], mat_2_data[i] ); } delete [] mat_1_data; delete [] mat_2_data; } yields: 0.000000 : 0.000000 1.000000 : 1.000000 2.000000 : 2.000000 3.000000 : 3.000000 4.000000 : 4.000000 5.000000 : 5.000000 6.000000 : 6.000000 7.000000 : 7.000000 8.000000 : 8.000000 9.000000 : 9.000000 10.000000 : 10.000000 11.000000 : 11.000000 with the first matrix mat_1 being "typical usage" where x is the axis of fastest increase as: mat_2[x][y][z] = x * y_dim * z_dim + y * z_dim + z; indexing is a bit odd, but perfectly acceptable so long as this is what is desired if the order of the for loops is changed for mat_2 so it loops z then y then x the array will not be written linearly and can be seen if the array is printed while it is being filled in. though what was originally posted may very well be what is desired by the original author.
participants (4)
-
alfC -
Brian Davis -
Jothy -
Pierre-Andre Noel