Re: [Boost-users] multi_array with large dimensions crashes

Hi Pierre, Sorry for the late reply. The code work now, thanks very much. But the programme crashes with a run-time error when the dimensions of the array are huge 500x500x500. This is probably due to the limitation of the 32 bit addrress space. Is there any other work around. I heard of creating arrays with strides and allocating them in different blocks of memory rather than in contiguos memory? Thanks again Jothy On Wed, Aug 24, 2011 at 6:25 PM, Pierre-Andre Noel < noel.pierre.andre@gmail.com> wrote:
Yes, 500, 300 and 400 were the example dimensions you used in your earlier message.
Did you got it to work?
Pierre-André
PS: This email was not sent to the whole list.
On Wed, Aug 24, 2011 at 11:55 AM, Jothy
wrote: You mean 500,300,400 are the required dimesnions and xDim,yDim,zDim are site_type parameters.
So the code should be like below I guess for an array of [x][y][z]
array_ref_type::size_type xDim(x), yDim(y), zDim(z);
array_storage_type Storage(xDim*yDim*zDim); array_ref_type Array(&Storage.front(),boost::extents[xDim][yDim][zDim]);
Thanks
Jothy
On Wed, Aug 24, 2011 at 3:51 PM, Pierre-Andre Noel < noel.pierre.andre@gmail.com> wrote:
I am not familiar with multi_array, but it seems that you are creating a >1GB object on the stack. This will almost always crash. Try creating it on the heap instead. Be prepared for std::bad_alloc.
But how to do this? I am new to c++.
I suggest you to either 1. read on the operators "new" and "delete" or 2. use a std::vector to allocate the memory then use a boost::multi_array_ref to refer to it. Here's an example:
#include <vector> #include
int main() { typedef float data_type; typedef std::vector
array_storage_type; typedef boost::multi_array_ref array_ref_type; array_ref_type::size_type xDim(500), yDim(300), zDim(400); array_storage_type Storage(xDim*yDim*zDim); array_ref_type Array(&Storage.front(),boost::extents[xDim][yDim][zDim]); // You may now use Array as usual. }
Hope it helps,
Pierre-André Noël
On Wed, Aug 24, 2011 at 5:20 AM, Jothy
wrote: Thanks for pointing this out.
But how to do this? I am new to c++.
Jothy
On Tue, Aug 23, 2011 at 10:42 PM, Juraj Ivančić < juraj.ivancic@gmail.com> wrote:
On 23.8.2011. 12:18, Jothy wrote:
Hi all,
I have just started to use boost libs. I am trying to create a large float array (mandatory for my purpose). it works fine till certain limit, but crashes above that. I think it about allocating memory in the stack.
My code is like this
typedefboost::multi_array<**float,3>array_type; typedefarray_type::indexindex; array_typeArray(boost::**extents[xDim][yDim][zDim]);//**500,300,400 for instance
Can some one help me to resolve this?
I am not familiar with multi_array, but it seems that you are creating a >1GB object on the stack. This will almost always crash. Try creating it on the heap instead. Be prepared for std::bad_alloc.
HTH
______________________________**_________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/**mailman/listinfo.cgi/boost-**usershttp://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

The code work now, thanks very much. But the programme crashes with a run-time error when the dimensions of the array are huge 500x500x500.
In your examples, you were using "float" variable, which requires 4 bytes to store. If you want a vector of 500*500*500=125000000 floats, that requires 500000000 bytes, i.e., 0.5 GB (or 0.466 GiB), of contiguous space in memory.
This is probably due to the limitation of the 32 bit addrress space.
For 32 bits, the total amount of addressable space is 4 GB. So no, that is not directly the problem.
Is there any other work around. I heard of creating arrays with strides and allocating them in different blocks of memory rather than in contiguos memory?
I fail to see how strides may help. However, requesting memory in a
non-contiguous way may certainly help. The easiest way to do so would be to
use a vector at the first level.
typedef std::vector< boost::multi_array< data_type, 2 > >
vector_of_2D_multi_array;
When using this kind of object, remember that this is not a 3 dimensional
multi_array but instead a vector of two dimensional ones.
If you cannot reach 500x500x500 with this setting, try to see if your OS
imposes limitations to the amount of memory that can be allocated to a
single process.
Finally, if you aim for even larger sizes, the amount of system memory will
eventually become a problem. In such cases, you may want to consider
distributed computing...
Hope it helps,
Pierre-André
On Thu, Aug 25, 2011 at 5:04 AM, Jothy
Hi Pierre,
Sorry for the late reply.
The code work now, thanks very much. But the programme crashes with a run-time error when the dimensions of the array are huge 500x500x500. This is probably due to the limitation of the 32 bit addrress space. Is there any other work around. I heard of creating arrays with strides and allocating them in different blocks of memory rather than in contiguos memory?
Thanks again
Jothy
On Wed, Aug 24, 2011 at 6:25 PM, Pierre-Andre Noel < noel.pierre.andre@gmail.com> wrote:
Yes, 500, 300 and 400 were the example dimensions you used in your earlier message.
Did you got it to work?
Pierre-André
PS: This email was not sent to the whole list.
On Wed, Aug 24, 2011 at 11:55 AM, Jothy
wrote: You mean 500,300,400 are the required dimesnions and xDim,yDim,zDim are site_type parameters.
So the code should be like below I guess for an array of [x][y][z]
array_ref_type::size_type xDim(x), yDim(y), zDim(z);
array_storage_type Storage(xDim*yDim*zDim); array_ref_type Array(&Storage.front(),boost::extents[xDim][yDim][zDim]);
Thanks
Jothy
On Wed, Aug 24, 2011 at 3:51 PM, Pierre-Andre Noel < noel.pierre.andre@gmail.com> wrote:
I am not familiar with multi_array, but it seems that you are creating a >1GB object on the stack. This will almost always crash. Try creating it on the heap instead. Be prepared for std::bad_alloc.
But how to do this? I am new to c++.
I suggest you to either 1. read on the operators "new" and "delete" or 2. use a std::vector to allocate the memory then use a boost::multi_array_ref to refer to it. Here's an example:
#include <vector> #include
int main() { typedef float data_type; typedef std::vector
array_storage_type; typedef boost::multi_array_ref array_ref_type; array_ref_type::size_type xDim(500), yDim(300), zDim(400); array_storage_type Storage(xDim*yDim*zDim); array_ref_type Array(&Storage.front(),boost::extents[xDim][yDim][zDim]); // You may now use Array as usual. }
Hope it helps,
Pierre-André Noël
On Wed, Aug 24, 2011 at 5:20 AM, Jothy
wrote: Thanks for pointing this out.
But how to do this? I am new to c++.
Jothy
On Tue, Aug 23, 2011 at 10:42 PM, Juraj Ivančić < juraj.ivancic@gmail.com> wrote:
On 23.8.2011. 12:18, Jothy wrote:
> Hi all, > > I have just started to use boost libs. I am trying to create a large > float array (mandatory for my purpose). it works fine till certain > limit, but crashes above that. I think it about allocating memory in > the > stack. > > My code is like this > > typedefboost::multi_array<**float,3>array_type; > typedefarray_type::indexindex; > array_typeArray(boost::**extents[xDim][yDim][zDim]);//**500,300,400 > for instance > > > Can some one help me to resolve this? >
I am not familiar with multi_array, but it seems that you are creating a >1GB object on the stack. This will almost always crash. Try creating it on the heap instead. Be prepared for std::bad_alloc.
HTH
______________________________**_________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/**mailman/listinfo.cgi/boost-**usershttp://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Just for curiosity,... can a custom allocator help in this case? As you know the third template parameter of a multi_array is an allocator. Thanks, Alfredo

I would like to point out something (that may be known), but may be
confusing to others that read this post and will affect the data written to
the arrays as it appears linearly. This is regarding the fastest
increasing axis:
in C/C++ float c_array[z_dim][y_dim][x_dim] ;
defines a row major array with the x_dim axis being the axis of the fastest
increasing index.
You can specify any axis you want to be the fastest increasing dimension
just so long is that is how you expect the data to appear in linear memory
and it is accessed accordingly.
I think:
boost::extents[xDim][yDim][zDim]
should be:
boost::extents[zDim][yDim][xDim]
as:
#include
participants (4)
-
alfC
-
Brian Davis
-
Jothy
-
Pierre-Andre Noel