According to documentation to "Pool" library, it is intended to be used "when there is a lot of allocation and deallocation of small objects". In our product we also use such pool for allocation of "normal" - heavier objects. In such case we can't benefit from significant improvement in memory consumption per object (overhead of malloc's internal structure relative to used size of the object), but the allocations themselves are much more efficient. Additional advantage is memory fragmentation - all objects in the pool reside in continuous memory blocks. The main problem with such usage is, in our case, that the pool never "returns" memory to the system, unless it's released. For example, if I use the pool for allocation of Event objects, and at some point of time I had 10000 concurrent events, the pool will occupy all that memory while the process is alive, even if for most of the time I use a small portion of it. As a solution I was thinking about adding a "defragment" function to memory pool, which will compact it by moving live objects to free cells at the "beginning" of the pool. Such feature can be implemented with help of additional "index" table of pointers. Of cause, instead of returning pointer to allocated object, the pool will return a wrapper object, actually pointing to that index table (for which separate "classical" pool can be used). So, the first question is: do you think such feature will be helpful? If yes - please share your thoughts regarding suggested and/or other implementations? Michael
Moving live objects is very dangerous, and I think it would cause a lot of problem on threads' synchronization. 2007/5/8, Michael Gopshtein <mgopshtein@gmail.com>:
According to documentation to "Pool" library, it is intended to be used "when there is a lot of allocation and deallocation of small objects".
In our product we also use such pool for allocation of "normal" - heavier objects. In such case we can't benefit from significant improvement in memory consumption per object (overhead of malloc's internal structure relative to used size of the object), but the allocations themselves are much more efficient.
Additional advantage is memory fragmentation - all objects in the pool reside in continuous memory blocks.
The main problem with such usage is, in our case, that the pool never "returns" memory to the system, unless it's released. For example, if I use the pool for allocation of Event objects, and at some point of time I had 10000 concurrent events, the pool will occupy all that memory while the process is alive, even if for most of the time I use a small portion of it.
As a solution I was thinking about adding a "defragment" function to memory pool, which will compact it by moving live objects to free cells at the "beginning" of the pool. Such feature can be implemented with help of additional "index" table of pointers. Of cause, instead of returning pointer to allocated object, the pool will return a wrapper object, actually pointing to that index table (for which separate "classical" pool can be used).
So, the first question is: do you think such feature will be helpful?
If yes - please share your thoughts regarding suggested and/or other implementations?
Michael
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Yes, "defragment" or other automation memory management is useful, but the existing code depends on native pointer would be very hard to work together, almost all C++ code need to recode if they want to use these futures. 2007/5/8, Michael Gopshtein <mgopshtein@gmail.com>:
According to documentation to "Pool" library, it is intended to be used "when there is a lot of allocation and deallocation of small objects".
In our product we also use such pool for allocation of "normal" - heavier objects. In such case we can't benefit from significant improvement in memory consumption per object (overhead of malloc's internal structure relative to used size of the object), but the allocations themselves are much more efficient.
Additional advantage is memory fragmentation - all objects in the pool reside in continuous memory blocks.
The main problem with such usage is, in our case, that the pool never "returns" memory to the system, unless it's released. For example, if I use the pool for allocation of Event objects, and at some point of time I had 10000 concurrent events, the pool will occupy all that memory while the process is alive, even if for most of the time I use a small portion of it.
As a solution I was thinking about adding a "defragment" function to memory pool, which will compact it by moving live objects to free cells at the "beginning" of the pool. Such feature can be implemented with help of additional "index" table of pointers. Of cause, instead of returning pointer to allocated object, the pool will return a wrapper object, actually pointing to that index table (for which separate "classical" pool can be used).
So, the first question is: do you think such feature will be helpful?
If yes - please share your thoughts regarding suggested and/or other implementations?
Michael
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
"Atry" <pop.atry@gmail.com> wrote in message news:fd8b80680705080147t5c561dd8l9fc0ee8ede2120c7@mail.gmail.com...
Yes, "defragment" or other automation memory management is useful, but the existing code depends on native pointer would be very hard to work together, almost all C++ code need to recode if they want to use these futures.
I do "feel" that just moving object's memory is not safe, but can't find explanation why it will cause problems. Can you give an example of a class for which this will not work?
on Tue May 08 2007, "Michael Gopshtein" <mgopshtein-AT-gmail.com> wrote:
"Atry" <pop.atry@gmail.com> wrote in message news:fd8b80680705080147t5c561dd8l9fc0ee8ede2120c7@mail.gmail.com...
Yes, "defragment" or other automation memory management is useful, but the existing code depends on native pointer would be very hard to work together, almost all C++ code need to recode if they want to use these futures.
I do "feel" that just moving object's memory is not safe, but can't find explanation why it will cause problems. Can you give an example of a class for which this will not work?
struct B; struct A { A(B*p) : p(p) {} B* p; }; struct B : noncopyable { B() : px(new A(this)) {} auto_ptr<A> px; // invariant: px->p == this }; -- Dave Abrahams Boost Consulting http://www.boost-consulting.com Don't Miss BoostCon 2007! ==> http://www.boostcon.com
"David Abrahams" <dave@boost-consulting.com> wrote in message news:87lkfyklpu.fsf@grogan.peloton...
on Tue May 08 2007, "Michael Gopshtein" <mgopshtein-AT-gmail.com> wrote:
"Atry" <pop.atry@gmail.com> wrote in message news:fd8b80680705080147t5c561dd8l9fc0ee8ede2120c7@mail.gmail.com...
Yes, "defragment" or other automation memory management is useful, but the existing code depends on native pointer would be very hard to work together, almost all C++ code need to recode if they want to use these futures.
I do "feel" that just moving object's memory is not safe, but can't find explanation why it will cause problems. Can you give an example of a class for which this will not work?
struct B;
struct A { A(B*p) : p(p) {} B* p; };
struct B : noncopyable { B() : px(new A(this)) {} auto_ptr<A> px; // invariant: px->p == this };
So, basically, it's any case where the object has pointers to it's internal members, right? Are there other cases that the object can't be "moved" (it's more of a theoretical interest)?
-- Dave Abrahams Boost Consulting http://www.boost-consulting.com
Don't Miss BoostCon 2007! ==> http://www.boostcon.com
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Hello Michael, Thursday, May 10, 2007, 12:13:18 AM, you wrote:
"David Abrahams" <dave@boost-consulting.com> wrote in message news:87lkfyklpu.fsf@grogan.peloton...
on Tue May 08 2007, "Michael Gopshtein" <mgopshtein-AT-gmail.com> wrote:
"Atry" <pop.atry@gmail.com> wrote in message news:fd8b80680705080147t5c561dd8l9fc0ee8ede2120c7@mail.gmail.com...
Yes, "defragment" or other automation memory management is useful, but the existing code depends on native pointer would be very hard to work together, almost all C++ code need to recode if they want to use these futures.
I do "feel" that just moving object's memory is not safe, but can't find explanation why it will cause problems. Can you give an example of a class for which this will not work?
struct B;
struct A { A(B*p) : p(p) {} B* p; };
struct B : noncopyable { B() : px(new A(this)) {} auto_ptr<A> px; // invariant: px->p == this };
So, basically, it's any case where the object has pointers to it's internal members, right?
Are there other cases that the object can't be "moved" (it's more of a theoretical interest)?
In general, if there is anywhere in the Universe a pointer or reference to the object or its part, the object cannot be moved. -- Best regards, Andrey mailto:andysem@mail.ru
"Andrey Semashev" <andysem@mail.ru> wrote in message news:135894700.20070510002632@mail.ru...
Hello Michael,
Thursday, May 10, 2007, 12:13:18 AM, you wrote:
"David Abrahams" <dave@boost-consulting.com> wrote in message news:87lkfyklpu.fsf@grogan.peloton...
on Tue May 08 2007, "Michael Gopshtein" <mgopshtein-AT-gmail.com> wrote:
"Atry" <pop.atry@gmail.com> wrote in message news:fd8b80680705080147t5c561dd8l9fc0ee8ede2120c7@mail.gmail.com...
Yes, "defragment" or other automation memory management is useful, but the existing code depends on native pointer would be very hard to work together, almost all C++ code need to recode if they want to use these futures.
I do "feel" that just moving object's memory is not safe, but can't find explanation why it will cause problems. Can you give an example of a class for which this will not work?
struct B;
struct A { A(B*p) : p(p) {} B* p; };
struct B : noncopyable { B() : px(new A(this)) {} auto_ptr<A> px; // invariant: px->p == this };
So, basically, it's any case where the object has pointers to it's internal members, right?
Are there other cases that the object can't be "moved" (it's more of a theoretical interest)?
In general, if there is anywhere in the Universe a pointer or reference to the object or its part, the object cannot be moved.
That I understand, but in current context I'm talking about internal members only - assuming that all external classes access the "moving" class though special wrapper (and not directly by pointer), which is also updated.
Best regards, Andrey mailto:andysem@mail.ru
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 5/7/07, Michael Gopshtein <mgopshtein@gmail.com> wrote:
As a solution I was thinking about adding a "defragment" function to memory pool, which will compact it by moving live objects to free cells at the "beginning" of the pool. Such feature can be implemented with help of additional "index" table of pointers. Of cause, instead of returning pointer to allocated object, the pool will return a wrapper object, actually pointing to that index table (for which separate "classical" pool can be used).
Maybe you're looking for the "release_memory" method in the pool interface. If you're using the pool_allocator as a replacement to the standard allocators, it might be interesting to note that the documentation mentions how to use "release_memory": http://boost.org/libs/pool/doc/interfaces.html " pool_alloc The pool_alloc interface is a Singleton Usage interface with Exceptions. It is built on the singleton_pool interface, and provides a Standard Allocator-compliant class (for use in containers, etc.). Example: void func() { std::vector<int, boost::pool_allocator<int> > v; for (int i = 0; i < 10000; ++i) v.push_back(13); } // Exiting the function does NOT free the system memory allocated by the pool allocator // You must call // boost::singleton_pool<boost::pool_allocator_tag, sizeof(int)>::release_memory() // in order to force that " Hope this helps. -- Dean Michael C. Berris http://cplusplus-soup.blogspot.com/ mikhailberis AT gmail DOT com +63 928 7291459
"Dean Michael Berris" <mikhailberis@gmail.com> wrote in message news:6adba9f0705080520h27fcf20coc6b353141e18e8f8@mail.gmail.com...
On 5/7/07, Michael Gopshtein <mgopshtein@gmail.com> wrote:
As a solution I was thinking about adding a "defragment" function to memory pool, which will compact it by moving live objects to free cells at the "beginning" of the pool. Such feature can be implemented with help of additional "index" table of pointers. Of cause, instead of returning pointer to allocated object, the pool will return a wrapper object, actually pointing to that index table (for which separate "classical" pool can be used).
Maybe you're looking for the "release_memory" method in the pool interface. If you're using the pool_allocator as a replacement to the standard allocators, it might be interesting to note that the documentation mentions how to use "release_memory":
http://boost.org/libs/pool/doc/interfaces.html
...
release_memory: "Frees every memory block that doesn't have any allocated chunks" This might be useful, I agree. And the chances to get such free blocks can be improved by using smaller block size, although it's hard for me to predict what can be the ratio of "free" blocks, even if total number of objects is relatively small.
Hope this helps.
-- Dean Michael C. Berris http://cplusplus-soup.blogspot.com/ mikhailberis AT gmail DOT com +63 928 7291459 _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
participants (5)
-
Andrey Semashev -
Atry -
David Abrahams -
Dean Michael Berris -
Michael Gopshtein