Interprocess : vectors of vectors

newer
[Graph] Problem with large vertex...

older
Interprocess : offset_ptr used as...

Oodini

13 May 2013 13 May '13

3:05 p.m.

Hello, I'd like to share a vector of vectors (jagged 2D array). The data come from a deserialization. I know the size of the "external" vector. The sizes of the "internal" vectors are discovered during deserialization. I read : http://www.boost.org/doc/libs/1_52_0/doc/html/interprocess/quick_guide.html#... and regarding this line : managed_shared_memory segment(create_only, "MySharedMemory", //segment name 65536); how to set the size (3rd argument), when considering my use case ? Even with a simple vector, and if the number and the types of the elements are know? i woldn't know what size to set for truncation. On VS 2010, a sizeof(vector<int>) is 40; I suppose I have to add it sizeof(int)*nbElements to get the full size. 1. I don't know if this assumption is valid 2. If so, I don't know if it's still valid when using Boost.Interprocess (I suppose it isn't) 3. I don't know how to generalize that to a vector of vectors in order what should be the truncation size. Thanks for help. PS : I also read http://www.boost.org/doc/libs/1_52_0/doc/html/interprocess/managed_memory_se..., but I havn't been enlightened about these points.

Show replies by date

Nat Linden

13 May 13 May

7:07 p.m.

On Mon, May 13, 2013 at 11:05 AM, Oodini wrote:

...

I'd like to share a vector of vectors (jagged 2D array). The data come from a deserialization. I know the size of the "external" vector. The sizes of the "internal" vectors are discovered during deserialization. Even with a simple vector, and if the number and the types of the elements are know? i woldn't know what size to set for truncation. On VS 2010, a sizeof(vector<int>) is 40; I suppose I have to add it sizeof(int)*nbElements to get the full size.

3. I don't know how to generalize that to a vector of vectors in order what should be the truncation size.

Who serializes the vector of vectors? Where were they created? Would they have been Boost.Interprocess vectors in that original process? If so, is there a way to determine the memory consumption in the *original* instance, and serialize that total size requirement before serializing the vector of vectors?

Oodini

14 May 14 May

7:54 a.m.

...

Who serializes the vector of vectors? Where were they created? Would they have been Boost.Interprocess vectors in that original process?

They are not originally vectors of vectors. The data come from a database, and are serialized in Python. So no C++ and Boost.Inteprocess. This work is done by one of my coworker.

Nat Linden

12:41 p.m.

On Tue, May 14, 2013 at 3:54 AM, Oodini wrote:

...

They are not originally vectors of vectors. The data come from a database, and are serialized in Python. So no C++ and Boost.Inteprocess.

Graphical user interface toolkits typically have methods that ask, "How much screen space do you require?" Container panels implement this by recursively querying their contents and then adding their own overhead. But I've never heard of a C++ container library that predictively does the same for memory consumption: "Given such-and-such content, how many bytes of memory *would* you require?" I can see that for certain use cases, that could be very handy, especially in embedded or realtime systems. I suspect, though, that the eventual solution to this particular problem will be a higher-level interface to shared memory that can grow or shrink the resource as needed. For right now, I can only suggest that you overallocate and hope for the best. I will be delighted to hear better suggestions from others!

Oodini

1:02 p.m.

...

But I've never heard of a C++ container library that predictively does the same for memory consumption: "Given such-and-such content how many bytes of memory *would* you require?" I can see that for certain use cases, that could be very handy, especially in embedded or realtime systems.

I am studybing this point. If you construct an empty vector in a managed shared memory, you have already 152 bytes consumed. I'd like to be sure that this value is always same.

...

I suspect, though, that the eventual solution to this particular problem will be a higher-level interface to shared memory that can grow or shrink the resource as needed. For right now, I can only suggest that you overallocate and hope for the best. I will be delighted to hear better suggestions from others!

It seem you can already do that : http://www.boost.org/doc/libs/1_52_0/doc/html/interprocess/managed_memory_se... But as I already know the number of bytes I have to deserialize, I'd like to avoid reallocation.

Oodini

1:09 p.m.

...

I am studybing this point. If you construct an empty vector in a managed shared memory, you have already 152 bytes consumed. I'd like to be sure that this value is always same.

Correction. By applying the function get_free_memory(), I can see that as soon as you create the managed shared memory, you lose some bytes : typedef allocator ShMemIntAllocator; typedef vector MyVector;managed_shared_memory shmem(create_only, "name", 65536); // shmem.get_free_memory() = 65384 -=> 152 bytes have been used MyVector* vv = shmem.construct<MyVector>("Vector")(shmem.get_segment_manager()); // shmem.get_free_memory() = 65296-=> 88 bytes have been used On Visual Studio 2010, an empty vector consumes 40 bytes.

Jan Herrmann

1:48 p.m.

On 14.05.2013 15:09, Oodini wrote:

...

...
I am studybing this point. If you construct an empty vector in a managed shared memory, you have already 152 bytes consumed. I'd like to be sure that this value is always same.

Correction. By applying the function get_free_memory(), I can see that as soon as you create the managed shared memory, you lose some bytes :

typedef allocator ShMemIntAllocator; typedef vector MyVector;managed_shared_memory shmem(create_only, "name", 65536);

// shmem.get_free_memory() = 65384 -=> 152 bytes have been used MyVector* vv = shmem.construct<MyVector>("Vector")(shmem.get_segment_manager()); // shmem.get_free_memory() = 65296-=> 88 bytes have been used

On Visual Studio 2010, an empty vector consumes 40 bytes.

Why not simply try different matrices with some larger memory segments, measure needed size and create a heuristic like: size=serialized_size*factor+extra_memory which full fills your need. Additionally I would avoid reallocations (no push_back) of vector inside shared memory to avoid fragmentation. Jan Herrmann

Nat Linden

2:49 p.m.

On Tue, May 14, 2013 at 9:02 AM, Oodini wrote:

...

...
I suspect, though, that the eventual solution to this particular problem will be a higher-level interface to shared memory that can grow or shrink the resource as needed.

...

It seem you can already do that :

...

http://www.boost.org/doc/libs/1_52_0/doc/html/interprocess/managed_memory_se...

But as I already know the number of bytes I have to deserialize, I'd like to avoid reallocation.

? Isn't that the point of your original question: you do not, in fact, know in advance the number of bytes you will need? If you have a way to move forward without getting stuck on this issue, finish your code, evaluate its performance and then -- if necessary -- run a profiler to figure out what actually needs improving. http://en.wikipedia.org/wiki/Program_optimization#Quotes

Oodini

3:05 p.m.

...

Isn't that the point of your original question: you do not, in fact, know in advance the number of bytes you will need ?

I do know the full size, but I don't know the size of each vector. The data are organized as : <nbElements> elt1 elt2 ... <nbElements> elt1 elt2 elt3 ..., So, if I have 2 inner vectors, I'll have, ie : 5 1.2 3.3 2.5 4.6 0.2 2 0.5 5.6 nbElments are integers, but the other values have a known other type. Also, I know the number of "nbElements tags". So I know the number of inner vectors I'll need. I know also the number of values. But I don't know in advance how the values are spread in the vectors. I have to read the file.

...

If you have a way to move forward without getting stuck on this issue, finish your code, evaluate its performance and then -- if necessary run a profiler to figure out what actually needs improving.

Unfortunately, I don't have access to a profiler. But anyway, these classes seem to consume a lot of RAM compared to a solution based on simple pointers (I havn't compared to the offset_ptr<>, though...).

Nat Linden

3:22 p.m.

On Tue, May 14, 2013 at 11:05 AM, Oodini wrote:

...

...
Isn't that the point of your original question: you do not, in fact, know in advance the number of bytes you will need ?

...

I do know the full size, but I don't know the size of each vector.

You know the number of data items. You don't know the number of bytes. Hence your original question.

...

...
If you have a way to move forward without getting stuck on this issue, finish your code, evaluate its performance and then -- if necessary run a profiler to figure out what actually needs improving.

...

Unfortunately, I don't have access to a profiler.

I would look for one only if the program's performance is observed to be inadequate. I would spend effort on computing the correct initial size for the shared memory only if an observed performance problem is proven to be due to resizing shared memory. You're trying to solve a problem you don't even know you have. You are of course free to spend your time doing that; I'll stop trying to dissuade you.

Oodini

3:28 p.m.

...

You know the number of data items. You don't know the number of bytes. Hence your original question.

I do know the number of bytes of my data. I don't know the number of bytes to allocate through Interprocess, because I don't know the bytes that Interprocess consumes for its data structures.

Anthony Foiani

4:18 p.m.

Oodini -- Oodini writes:

...

I'd like to share a vector of vectors (jagged 2D array). The data come from a deserialization. I know the size of the "external" vector.

The sizes of the "internal" vectors are discovered during deserialization.

managed_shared_memory segment(create_only, "MySharedMemory", //segment name 65536);

how to set the size (3rd argument), when considering my use case ?

If you have plenty of memory (especially virtual memory), then just make it huge, and don't worry about using it all. Alternately, get some guarantee from the previous step about the largest possible sizes. You probably need some maximum anyway; otherwise, one day someone will try to pump gigabytes of data into your routine, and things will go pear-shaped. If you can make multiple passes over the serialized data, then you can calculate exactly how many elements in each vector, and size your segment based on that, but as you ask...

...

Even with a simple vector, and if the number and the types of the elements are know? i woldn't know what size to set for truncation. On VS 2010, a sizeof(vector<int>) is 40; I suppose I have to add it sizeof(int)*nbElements to get the full size.

1. I don't know if this assumption is valid

Probably not perfectly valid, but it should get you to the right order of magnitude.

...

2. If so, I don't know if it's still valid when using Boost.Interprocess (I suppose it isn't)

It might be; the main difference with most of the interprocess containers is that they work with offset pointers, not with raw pointers. Offset pointers should be the same size as regular pointers, although there might be a bit of extra bookkeeping in the vector "header" itself.

...

3. I don't know how to generalize that to a vector of vectors in order what should be the truncation size.

It's easier to talk about this if we have some concrete types: namespace bi = boost::interprocess; typedef bi::managed_shared_memory managed_shared_memory_t; typedef bi::allocator< int, managed_shared_memory_t::segment_manager > shared_int_allocator_t; typedef bi::vector< int, shared_int_allocator_t > inner_vec_t; typedef bi::allocator< inner_vec_t, managed_shared_memory_t::segment_manager > shared_inner_vec_allocator_t; typedef bi::vector< inner_vec_t, shared_inner_vec_allocator_t > outer_vec_t; Further, assume we can do two passes over the data, and end up with a vector of the length of each serialized sub-vector: std::vector< size_t > subs; We can compute the amount of user-visible space like so: // first, the size of the outermost containing struct size_t total_memory = sizeof( outer_vec_t ); // this is too tight of a constraint, since the outer_vec_t will // really allocate more than the requested number of inner_vec_t // structs, but as a minimum: total_memory += subs.size() * sizeof( inner_vec_t ); // and then we have to add in the memory for each inner vector: for ( const size_t inner_len : subs ) total_memory += inner_len * sizeof( int ); (There are probably some int vs. size_t type conversion issues in that code; season with casts or larger types to taste.) However, realize that this is a bare minimum: the actual amount of memory needed is guaranteed to be larger for at least the following reasons: a. vectors tend to allocate extra space (think "reserve()" and "capacity()" vs. "resize()" and "size()"). b. chunks of memory are often larger than actually requested (to improve performance, or to stash housekeeping information with each chunk). c. the "managed" part of "managed memory segment" includes storage for object name to location+size mapping, which reduces the total amount of memory available in the segment. So whatever value you get for "total_memory", remember that you need to increase it by some amount. If you have the memory to spare, go for double, and you should be fine; if you're tight, try maybe 20% more, and pay extra attention to your exception paths.

...

Thanks for help.

Hope this helps. I've written a few other replies on related interprocess issues: http://preview.tinyurl.com/a877vlf http://preview.tinyurl.com/c5l6mtw

...

PS : I also read http://www.boost.org/doc/libs/1_52_0/doc/html/interprocess/managed_memory_se..., but I havn't been enlightened about these points.

In a sense, that should be at a lower level than the questions you're working on. The segments just provide raw memory; it's up to the containers to use it. Hope this helps, Tony

Oodini

16 May 16 May

11:25 a.m.

...

If you have plenty of memory (especially virtual memory), then just make it huge, and don't worry about using it all.

Actualy, my data will take about 26 Gb... Tha's why I want to control the memory.

...

there might be a bit of extra bookkeeping in the vector "header" itself.

That's one of the problem.

...

a. vectors tend to allocate extra space (think "reserve()" and "capacity()" vs. "resize()" and "size()").

reserve(1) or reserve(5) take the same amount of memory.

...

b. chunks of memory are often larger than actually requested (to improve performance, or to stash housekeeping information with each chunk).

Yes. I'd like to control this behaviour.

...

So whatever value you get for "total_memory", remember that you need to increase it by some amount.

As we are supposed to provide an exact value for the shared memory, there should be a way to know exactly how much memory consume the data structures.

...

If you have the memory to spare, go for double, and you should be fine; if you're tight, try maybe 20% more, and pay extra attention to your exception paths.

I can't... Thanks a lot for your contribution. I switck back on an implementation based on pointers instead of vectors.

Anthony Foiani

12:52 p.m.

Oodini writes:

...

...
If you have plenty of memory (especially virtual memory), then just make it huge, and don't worry about using it all.

Actualy, my data will take about 26 Gb...

Ouch!

...

Tha's why I want to control the memory.

Yeah, if you can't fit it all in core, then Boost.Interprocess isn't going to be enough. (On the other hand, depending on how expensive your programmers are, remember that fast computers with 64GiB RAM are no longer outrageously expensive, even as rented hosts. A quick look found me a 32GiB system for about 1500USD, 64GiB for 2500USD. Here in the USA, it's not unusual to figure 3k-5k USD/wk for developer time, and if it takes you more than a week to work around the memory limit, then you're better off just getting the new machine. But anyway...)

...

...
there might be a bit of extra bookkeeping in the vector "header" itself.

That's one of the problem.

I'd be surprised if the interprocess vector cost more than another pointer or two over the regular std::vector. They both need the extra space for capacity vs. size, pointer to start of memory, etc.

...

reserve(1) or reserve(5) take the same amount of memory.

Right, because at the lowest level, they either have internal optimizations that limit the options, or the level "under" the vector will only give out a certain smallest size of memory block.

...

...
b. chunks of memory are often larger than actually requested (to improve performance, or to stash housekeeping information with each chunk).

Yes. I'd like to control this behaviour.

You can't really control it without reworking your entire memory allocation infrastructure from scratch. Even malloc/free tends to use this trick: if you requested, 0x200 bytes, it would actually allocate 0x204 bytes, put the value 0x200 in the first 4-byte word, and then hand you the pointer to allocated_range+4). That's how "free" knew how big of a chunk it was using, without relying on an external lookup table.

...

...
So whatever value you get for "total_memory", remember that you need to increase it by some amount.

As we are supposed to provide an exact value for the shared memory, there should be a way to know exactly how much memory consume the data structures.

Using shm from C++ through Boost.Interprocess, I've always just given it extra space. On a machine with virtual memory, it should only consume as much RAM as you're actually using in the segment, not the entire segment. I've also had the luxury of working on problems that fit easily into their hosts, e.g., a few megabytes on a multi-gigabyte machine. So the question has never really come up for me.

...

...
If you have the memory to spare, go for double, and you should be fine; if you're tight, try maybe 20% more, and pay extra attention to your exception paths.

I can't... Thanks a lot for your contribution.

You're welcome. Sorry I didn't have better answers for you.

...

I switck back on an implementation based on pointers instead of vectors.

So you do have enough RAM to fit it all in core, but not with the overhead of the vector structures? Interesting. You might see if there are ragged / sparse matrix classes that could be adapted to Boost.Interprocess; those might be closer to your use case than the general-purpose vector-of-vectors. Good luck! Best regards, Anthony Foiani

Oodini

2:05 p.m.

...

(On the other hand, depending on how expensive your programmers are remember that fast computers with 64GiB RAM are no longer outrageously expensive, even as rented hosts.)

Actually, it isn't to avoid t buy PCs with a lto of RAM machines, but to save time when launching the application. I let you imagine how long it takes to load the data in RAM when I debug (even if I use then a light dataset of 5 GB).

...

I'd be surprised if the interprocess vector cost more than another pointer or two over the regular std::vector. They both need the extra space for capacity vs. size, pointer to start of memory, etc.

I didn't use vectors, but arrays based on pointers. I started a new thread about porting that with offset_ptr.

4301

Age (days ago)

4304

Last active (days ago)

List overview

Download

14 comments

4 participants

participants (4)

Anthony Foiani
Jan Herrmann
Nat Linden
Oodini