[serialization] [1.35.0] Potential Memory Buster, Deserializing std::vector<T, boost::pool_allocator<T> >
Hi Everyone, In the 1.35.0 release, I see that boost/archive/array/iarchive.hpp has a commented out 'load_optimized' function which is not commented out in the trunk. When deserializing to an std::vector<uint32_t, boost::pool_allocator<uint32_t> >, I see that the optimized implementation will first get the correct size, set the receiving container's size correctly, and then put the contents of the array into the container. The question I have is why is this function commented out in the 1.35.0 release? I had to find out the hard way that this causes insane memory usage when deserializing to an std::vector<uint32_t, boost::pool_allocator<uint32_t> >, mainly because if it uses an implementation that grows the vector one element at a time, can (and usually would) cause multiple calls to resize the container and make a lot of calls to the pool allocator. The way the pool allocator grows causes an abrupt usage in memory unnecessarily. Should this be considered a blocker issue that merits a 1.35.1 release? Insights would be most appreciated. My solution for the meantime is to stick to the Boost svn trunk version that I know the current application I'm working on works fine with -- eventually though I hope to be able to move to a "stable" Boost release. -- Dean Michael C. Berris Software Engineer, Friendster, Inc.
On Jun 13, 2008, at 9:55 AM, Dean Michael Berris wrote:
Hi Everyone,
In the 1.35.0 release, I see that boost/archive/array/iarchive.hpp has a commented out 'load_optimized' function which is not commented out in the trunk.
When deserializing to an std::vector<uint32_t, boost::pool_allocator<uint32_t> >, I see that the optimized implementation will first get the correct size, set the receiving container's size correctly, and then put the contents of the array into the container.
The question I have is why is this function commented out in the 1.35.0 release?
We commented this out since we found out just before the release that this optimization prevents tracking of vectors. 1.36 will implement this in a slightly different way, which will enable some more optimizations. Matthias
On Sat, Jun 14, 2008 at 5:27 PM, Matthias Troyer <troyer@phys.ethz.ch> wrote:
On Jun 13, 2008, at 9:55 AM, Dean Michael Berris wrote:
The question I have is why is this function commented out in the 1.35.0 release?
We commented this out since we found out just before the release that this optimization prevents tracking of vectors. 1.36 will implement this in a slightly different way, which will enable some more optimizations.
Okay, sounds good to me. Thanks Matthias! -- Dean Michael C. Berris Software Engineer, Friendster, Inc.
On Jun 14, 2008, at 5:35 PM, Dean Michael Berris wrote:
On Sat, Jun 14, 2008 at 5:27 PM, Matthias Troyer <troyer@phys.ethz.ch> wrote:
On Jun 13, 2008, at 9:55 AM, Dean Michael Berris wrote:
The question I have is why is this function commented out in the 1.35.0 release?
We commented this out since we found out just before the release that this optimization prevents tracking of vectors. 1.36 will implement this in a slightly different way, which will enable some more optimizations.
Okay, sounds good to me.
It is on the trunk now Matthias
On Sun, Jun 15, 2008 at 2:43 AM, Matthias Troyer <troyer@phys.ethz.ch> wrote:
On Jun 14, 2008, at 5:35 PM, Dean Michael Berris wrote:
On Sat, Jun 14, 2008 at 5:27 PM, Matthias Troyer <troyer@phys.ethz.ch> wrote:
On Jun 13, 2008, at 9:55 AM, Dean Michael Berris wrote:
The question I have is why is this function commented out in the 1.35.0 release?
We commented this out since we found out just before the release that this optimization prevents tracking of vectors. 1.36 will implement this in a slightly different way, which will enable some more optimizations.
Okay, sounds good to me.
It is on the trunk now
Wow, that was quick. Thanks again Matthias! Let me try and convince people over here to use the latest in the trunk. -- Dean Michael C. Berris Software Engineer, Friendster, Inc.
On Jun 15, 2008, at 7:16 AM, Dean Michael Berris wrote:
On Sun, Jun 15, 2008 at 2:43 AM, Matthias Troyer <troyer@phys.ethz.ch> wrote:
On Jun 14, 2008, at 5:35 PM, Dean Michael Berris wrote:
On Sat, Jun 14, 2008 at 5:27 PM, Matthias Troyer <troyer@phys.ethz.ch
wrote:
On Jun 13, 2008, at 9:55 AM, Dean Michael Berris wrote:
The question I have is why is this function commented out in the 1.35.0 release?
We commented this out since we found out just before the release that this optimization prevents tracking of vectors. 1.36 will implement this in a slightly different way, which will enable some more optimizations.
Okay, sounds good to me.
It is on the trunk now
Wow, that was quick. Thanks again Matthias!
Let me try and convince people over here to use the latest in the trunk.
If you don't need tracking of vectors you can just uncomment the commented out parts in 1.35.0 Matthias
On Sun, Jun 15, 2008 at 2:53 PM, Matthias Troyer <troyer@phys.ethz.ch> wrote:
On Jun 15, 2008, at 7:16 AM, Dean Michael Berris wrote:
On Sun, Jun 15, 2008 at 2:43 AM, Matthias Troyer <troyer@phys.ethz.ch> wrote:
On Jun 14, 2008, at 5:35 PM, Dean Michael Berris wrote:
On Sat, Jun 14, 2008 at 5:27 PM, Matthias Troyer <troyer@phys.ethz.ch> wrote:
On Jun 13, 2008, at 9:55 AM, Dean Michael Berris wrote:
The question I have is why is this function commented out in the 1.35.0 release?
We commented this out since we found out just before the release that this optimization prevents tracking of vectors. 1.36 will implement this in a slightly different way, which will enable some more optimizations.
Okay, sounds good to me.
It is on the trunk now
Wow, that was quick. Thanks again Matthias!
Let me try and convince people over here to use the latest in the trunk.
If you don't need tracking of vectors you can just uncomment the commented out parts in 1.35.0
This sounds like a good plan too. When you say the version in 1.35.0 cannot track vectors, does that mean we cannot merely tell the difference between a serialized array and a serialized vector? I'm a little lost as to why there should be a difference between a normally serialized/deserialized vector and a static array (T[]). More precisely, a serialized array should be deserializable to a vector and a serialized vector should be deserializable to a statically (appropriately) sized array as well. Or is there a deeper reason why this behavior isn't acceptable? -- Dean Michael C. Berris Software Engineer, Friendster, Inc.
participants (3)
-
Dean Michael Berris -
Dean Michael Berris -
Matthias Troyer