Stupid ostringstream question
I have been searching the world in vain for a stringstream that works with a static buffer, and does not use the heap for memory allocation. We have an application that uses lots (12-16Gb) of memory. This puts a strain on the memory allocation routines, and when we want to create a small string we find that stringstream is very, very expensive, and sprintf into a static buffer much, much faster. Does boost provide anything like this? This actually came up when we realized that boost lexical_cast was using stringstream. When we replaced it with sprintf, our runtimes (when memory usage was very high) went down significantly. I should also point out that ANYTHING we did with dynamic memory at that point in the application (vector::push_back, for example) was hideously slow. I'd like to have the best of both worlds: the relatively clear syntax of the ostringstream inserters, working on a static buffer. Any help (code samples especially) appreciated. Bill
Have you tried investing in a replacement memory manager? It might give you a bigger overall payback than just tinkering with one aspect. Best-of-luck, - Richard -----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Bill Lear Sent: 27 July 2007 20:36 To: boost-users@lists.boost.org Subject: [Boost-users] Stupid ostringstream question I have been searching the world in vain for a stringstream that works with a static buffer, and does not use the heap for memory allocation. We have an application that uses lots (12-16Gb) of memory. This puts a strain on the memory allocation routines, and when we want to create a small string we find that stringstream is very, very expensive, and sprintf into a static buffer much, much faster. Does boost provide anything like this? This actually came up when we realized that boost lexical_cast was using stringstream. When we replaced it with sprintf, our runtimes (when memory usage was very high) went down significantly. I should also point out that ANYTHING we did with dynamic memory at that point in the application (vector::push_back, for example) was hideously slow. I'd like to have the best of both worlds: the relatively clear syntax of the ostringstream inserters, working on a static buffer. Any help (code samples especially) appreciated. Bill _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On Friday, July 27, 2007 at 20:47:54 (+0100) Richard writes:
Have you tried investing in a replacement memory manager?
Such as?
It might give you a bigger overall payback than just tinkering with one aspect.
Certainly possible, however, it is also possible that the sheer size and number of objects we have in our application makes it inherently difficult for dynamically-allocated memory for small objects to work very well at very high memory usages. Thus the hope for a hybrid static store and ostringstream. Bill
Well - I have zero experience here. I remember that they were frequently advertised in the back of computer mags a few years back. Try Google? I saw... http://www.freedownloadscenter.com/Programming/C_and_C++_Tools_and_Component s/SwiftAlloc_Memory_manager_replacement.html and http://www.newcodeinc.com/ Another possibility might be to look at how your app uses memory. If you have vast numbers of objects all the same size you might benefit from a home written replacement for operator new. If you special case them all into a special pool it might get them out of the way of everything else. The general purpose manager that ships with the compiler is just that - general! Sounds like you have a specific problem. HTH, - Richard -----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Bill Lear Sent: 27 July 2007 21:01 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Stupid ostringstream question On Friday, July 27, 2007 at 20:47:54 (+0100) Richard writes:
Have you tried investing in a replacement memory manager?
Such as?
It might give you a bigger overall payback than just tinkering with one aspect.
Certainly possible, however, it is also possible that the sheer size and number of objects we have in our application makes it inherently difficult for dynamically-allocated memory for small objects to work very well at very high memory usages. Thus the hope for a hybrid static store and ostringstream. Bill _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Most of std classes support custom allocators. basic_stringstream and vector do this as well. You can preallocate the memory needed and create objects in this memory locations. For more help see the placement new operator. But your question here is very general. It is for example possible use some COW implementation of your objects together with reference counted content. You objects are not duplicated on every copy, but only when one tries to write to the content. To give you better advices you should provide more specific examples. With Kind Regards, Ovanes On Fri, July 27, 2007 22:17, Richard wrote:
Well - I have zero experience here.
I remember that they were frequently advertised in the back of computer mags a few years back. Try Google? I saw...
http://www.freedownloadscenter.com/Programming/C_and_C++_Tools_and_Component s/SwiftAlloc_Memory_manager_replacement.html
and
Another possibility might be to look at how your app uses memory.
If you have vast numbers of objects all the same size you might benefit from a home written replacement for operator new. If you special case them all into a special pool it might get them out of the way of everything else. The general purpose manager that ships with the compiler is just that - general! Sounds like you have a specific problem.
HTH,
- Richard
-----Original Message----- From: boost-users-bounces@lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Bill Lear Sent: 27 July 2007 21:01 To: boost-users@lists.boost.org Subject: Re: [Boost-users] Stupid ostringstream question
On Friday, July 27, 2007 at 20:47:54 (+0100) Richard writes:
Have you tried investing in a replacement memory manager?
Such as?
It might give you a bigger overall payback than just tinkering with one aspect.
Certainly possible, however, it is also possible that the sheer size and number of objects we have in our application makes it inherently difficult for dynamically-allocated memory for small objects to work very well at very high memory usages. Thus the hope for a hybrid static store and ostringstream.
Bill _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
With Kind Regards, Ovanes Markarian
On Friday, July 27, 2007 at 22:49:22 (+0200) Ovanes Markarian writes:
Most of std classes support custom allocators. basic_stringstream and vector do this as well. You can preallocate the memory needed and create objects in this memory locations. For more help see the placement new operator.
Hmm, yes, they do, but they also have a pitfall. If I create a new templated ostringstream with a custom allocator, it then means that the string that is returned also must use this custom allocator, and is incompatible with standard strings. I guess that's not surprising, however, now that I think about it... Bill
On Fri, July 27, 2007 23:11, Bill Lear wrote:
On Friday, July 27, 2007 at 22:49:22 (+0200) Ovanes Markarian writes:
Most of std classes support custom allocators. basic_stringstream and vector do this as well. You can preallocate the memory needed and create objects in this memory locations. For more help see the placement new operator.
Hmm, yes, they do, but they also have a pitfall. If I create a new templated ostringstream with a custom allocator, it then means that the string that is returned also must use this custom allocator, and is incompatible with standard strings. I guess that's not surprising, however, now that I think about it...
I don't think it is so. std::string and a stream class are containers of elements. How they contain them is their own beer. They exchange them for example via operators << and >> where it is defined how the strings are serialized or deserialized to the stream. In these operators you provide as template params Elems stored in stream and traits about these Elems. And the rest is how you handle it. With Kind Regards, Ovanes Markarian
On Friday, July 27, 2007 at 23:33:21 (+0200) Ovanes Markarian writes:
On Fri, July 27, 2007 23:11, Bill Lear wrote:
On Friday, July 27, 2007 at 22:49:22 (+0200) Ovanes Markarian writes:
Most of std classes support custom allocators. basic_stringstream and vector do this as well. You can preallocate the memory needed and create objects in this memory locations. For more help see the placement new operator.
Hmm, yes, they do, but they also have a pitfall. If I create a new templated ostringstream with a custom allocator, it then means that the string that is returned also must use this custom allocator, and is incompatible with standard strings. I guess that's not surprising, however, now that I think about it...
I don't think it is so. std::string and a stream class are containers of elements. How they contain them is their own beer. They exchange them for example via operators << and >> where it is defined how the strings are serialized or deserialized to the stream. In these operators you provide as template params Elems stored in stream and traits about these Elems. And the rest is how you handle it.
Well, stringstream defines a type, from g++ header: typedef basic_string<char_type, _Traits, _Alloc> __string_type; and str() is defined to return this: __string_type str() { ...} So, if I have a std::string, and an ostringstream with a different allocator, say my_ostringstream, I cannot do this: my_ostringstream os; os << ...; std::string s = os.str(); Bill
On Sat, July 28, 2007 00:07, Bill Lear wrote: [...]
Well, stringstream defines a type, from g++ header:
typedef basic_string<char_type, _Traits, _Alloc> __string_type;
and str() is defined to return this:
__string_type str() { ...}
So, if I have a std::string, and an ostringstream with a different allocator, say my_ostringstream, I cannot do this:
my_ostringstream os; os << ...; std::string s = os.str();
Ok, but you understand that this approach, will probably consume twice as much memory, since str() leaves stream as it is and does not remove characters read. A was meaning this: my_ostringstream os; os << ...; std::string s; os >> s; And this should work with custom allocators in stringstream and string. With Kind Regards, Ovanes Markarian
--- Bill Lear <rael@zopyra.com> wrote:
On Friday, July 27, 2007 at 20:47:54 (+0100) Richard writes:
Have you tried investing in a replacement memory manager?
Such as?
Maybe you can try to use libhoard. It is very easy to try and the results can be stunning! http://www.cs.umass.edu/~emery/hoard/hoard-documentation.html Greets, Andrej
It might give you a bigger overall payback than just tinkering with one aspect.
Certainly possible, however, it is also possible that the sheer size and number of objects we have in our application makes it inherently difficult for dynamically-allocated memory for small objects to work very well at very high memory usages. Thus the hope for a hybrid static store and ostringstream.
Bill _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users
___________________________________________________________ Yahoo! Mail is the world's favourite email. Don't settle for less, sign up for your free account today http://uk.rd.yahoo.com/evt=44106/*http://uk.docs.yahoo.com/mail/winter07.htm...
On 27/07/07, Bill Lear <rael@zopyra.com> wrote:
Certainly possible, however, it is also possible that the sheer size and number of objects we have in our application makes it inherently difficult for dynamically-allocated memory for small objects to work very well at very high memory usages. Thus the hope for a hybrid static store and ostringstream.
Not sure if it is exactly what you're looking for, but Boost.Pool might be of some use here. Regards, Darren
Bill Lear wrote:
I have been searching the world in vain for a stringstream that works with a static buffer, and does not use the heap for memory allocation. We have an application that uses lots (12-16Gb) of memory. This puts a strain on the memory allocation routines, and when we want to create
Which platform?
a small string we find that stringstream is very, very expensive, and sprintf into a static buffer much, much faster.
Does boost provide anything like this?
Have you looked at the boost::iostream library which abstracts out sources/sinks that you can use with buffers that you control? Also have you looked at the boost::format library as a typesafe replacement for sprintf? Jeff Flinn
On Friday, July 27, 2007 at 16:40:05 (-0400) Jeff Flinn writes:
Bill Lear wrote:
I have been searching the world in vain for a stringstream that works with a static buffer, and does not use the heap for memory allocation. We have an application that uses lots (12-16Gb) of memory. This puts a strain on the memory allocation routines, and when we want to create
Which platform?
Linux, with Intel and gcc compilers.
Have you looked at the boost::iostream library which abstracts out sources/sinks that you can use with buffers that you control?
Also have you looked at the boost::format library as a typesafe replacement for sprintf?
Hmm, not yet, but thanks for the pointers. Bill
Bill Lear wrote:
I have been searching the world in vain for a stringstream that works with a static buffer, and does not use the heap for memory allocation. We have an application that uses lots (12-16Gb) of memory. This puts a strain on the memory allocation routines, and when we want to create a small string we find that stringstream is very, very expensive, and sprintf into a static buffer much, much faster.
Does boost provide anything like this?
No, but there is std::(i/o)strstream. It's deprecated but it should work.
Bill Lear <rael <at> zopyra.com> writes:
I have been searching the world in vain for a stringstream that works with a static buffer, and does not use the heap for memory allocation. We have an application that uses lots (12-16Gb) of memory. This puts a strain on the memory allocation routines, and when we want to create a small string we find that stringstream is very, very expensive, and sprintf into a static buffer much, much faster.
Does boost provide anything like this?
This actually came up when we realized that boost lexical_cast was using stringstream. When we replaced it with sprintf, our runtimes (when memory usage was very high) went down significantly. I should also point out that ANYTHING we did with dynamic memory at that point in the application (vector::push_back, for example) was hideously slow.
I'd like to have the best of both worlds: the relatively clear syntax of the ostringstream inserters, working on a static buffer.
Any help (code samples especially) appreciated.
Bill
I wrote some while ago a wrapper which writes to a static buffer and flushes then to the debugger output window. One can easily write things like: dbg_cout << 2 << "Hello" << std::endl It originates from the book 'Standard C++ IOStreams and Locales' 3.4.1.1.2 and is basically an adapter of a 'basic_streambuf'. The iostream library is not easy to plug in, and I think that Boost Iostreams helps in this area. Unfortunately I haven't worked with it yet. If you are interested I can mail you the code, but since it is not a Boost topic I better not show it here... wkr, me
On Monday, July 30, 2007 at 11:40:07 (+0000) gast128 writes:
... I wrote some while ago a wrapper which writes to a static buffer and flushes then to the debugger output window. One can easily write things like:
dbg_cout << 2 << "Hello" << std::endl
It originates from the book 'Standard C++ IOStreams and Locales' 3.4.1.1.2 and is basically an adapter of a 'basic_streambuf'. The iostream library is not easy to plug in, and I think that Boost Iostreams helps in this area. Unfortunately I haven't worked with it yet.
If you are interested I can mail you the code, but since it is not a Boost topic I better not show it here...
Well, I'd be grateful if you sent it to me. If it's good enough, perhaps one day it could be accepted into boost ... who knows? Bill
participants (8)
-
Andrej van der Zee -
Bill Lear -
Darren Garvey -
gast128 -
Jeff Flinn -
Ovanes Markarian -
Peter Dimov -
Richard