[date_time] [MSVC8] Memory leak in operator<<(ptime)
The following test app illustrates a memory leak that I'v been able to produce so far with MSVC 8.0 (earlier versions not tested). GCC 4.0 seems to be unaffected. The leakage in case of this test app is around 12kb/s. #include <iostream> #include <boost/date_time/posix_time/posix_time.hpp> using namespace boost::posix_time; int main(int argc, char *argv[]) { ptime t(second_clock::local_time()); while (true) { std::cout << t << "\r"; } return 0; } Tested against Boost 1.33.1 release. Compiled with cl test.cpp /I c:\boost_1_33_1 /D _CRT_SECURE_NO_DEPRECATE /EHsc /D BOOST_ALL_NO_LIB -- Alo Sarv
Could you please provide more information about your testing methods. Did you try this test with other objects besides ptime? What about other non-date_time objects? Thanks, Bart "Alo Sarv" <alo.sarv@gmail.com> wrote in message news:14a0620d0602161156s55640094xabd31ed80ebde1cb@mail.gmail.com...
The following test app illustrates a memory leak that I'v been able to produce so far with MSVC 8.0 (earlier versions not tested). GCC 4.0 seems to be unaffected. The leakage in case of this test app is around 12kb/s.
#include <iostream> #include <boost/date_time/posix_time/posix_time.hpp> using namespace boost::posix_time; int main(int argc, char *argv[]) { ptime t(second_clock::local_time()); while (true) { std::cout << t << "\r"; } return 0; }
Tested against Boost 1.33.1 release.
Compiled with cl test.cpp /I c:\boost_1_33_1 /D _CRT_SECURE_NO_DEPRECATE /EHsc /D BOOST_ALL_NO_LIB
-- Alo Sarv
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On 2/17/06, Bart <bartmann.nsd@gmail.com> wrote:
Could you please provide more information about your testing methods.
Did you try this test with other objects besides ptime? What about other non-date_time objects?
Yes. Tested time_duration, date, date_duration and date_period operator<<, and they did not show any memory leakage. For testing, I compiled/ran the test application, and watched the memory usage in Windows Task Manager. No other user-defined or Boost types show this leakage, so as far as I can tell, this leakage is limited to ptime operator<<. -- Alo Sarv
On Fri, 17 Feb 2006 15:44:14 +0200, Alo Sarv wrote
On 2/17/06, Bart <bartmann.nsd@gmail.com> wrote:
Could you please provide more information about your testing methods.
Did you try this test with other objects besides ptime? What about other non-date_time objects?
Yes. Tested time_duration, date, date_duration and date_period operator<<, and they did not show any memory leakage. For testing, I compiled/ran the test application, and watched the memory usage in Windows Task Manager. No other user-defined or Boost types show this leakage, so as far as I can tell, this leakage is limited to ptime operator<<.
Well, that's odd. The only place that date-time does an allocation is in operator<< and operator>>. So there 'could' be a problem with leaking in that code -- either a bug in date-time code or standard library code. However, time_duration and posix_time use the exact same allocation techniques (and facet code) so should exhibit the exact same leaking behavior. Basically, what gets allocated is a custom date-time facet that gets imbued into the stream. This gets done on the first call to operator>> or operator<< on a particular stream. After that it just uses the facet avoiding the allocation. The facet is written to work with C++ I/O streams which has a reference counting and should destroy the facet when the stream is destroyed. In the case of std::cout this may be after main has exited. If you want to trace it thru you could explicitly create and delete a stringstream and check it's behavior. The date-time code that does the allocations is in date_time/posix_time/posix_time_io.hpp. Jeff
On 2/17/06, Jeff Garland <jeff@crystalclearsoftware.com> wrote:
Well, that's odd. The only place that date-time does an allocation is in operator<< and operator>>. So there 'could' be a problem with leaking in that code -- either a bug in date-time code or standard library code. However, time_duration and posix_time use the exact same allocation techniques (and facet code) so should exhibit the exact same leaking behavior. Basically, what gets allocated is a custom date-time facet that gets imbued into the stream. This gets done on the first call to operator>> or operator<< on a particular stream. After that it just uses the facet avoiding the allocation. The facet is written to work with C++ I/O streams which has a reference counting and should destroy the facet when the stream is destroyed. In the case of std::cout this may be after main has exited.
Yes, the first place I looked was that code as well, and thought perhaps the custom facet doesn't get destroyed properly. However, that facet is only created once, so that doesn't seem to be the cause, especially considering other operator<<'s also use basically the same code.
If you want to trace it thru you could explicitly create and delete a stringstream and check it's behavior. The date-time code that does the allocations is in date_time/posix_time/posix_time_io.hpp.
These two snippets also exhibit the leakage (around 50-80kb/s): using namespace boost::posix_time; ptime p(second_clock::local_time()); while (true) { std::ostringstream *o = new std::ostringstream; *o << p; delete o; } while (true) { std::ostringstream() << p; } So clearly it's not some kind of hidden std::cout buffering by the library that could've caused this. I also stepped through the entire call hiearchy of the operator<< in debugger but couldn't find anything of use / suspicious. I attempted to use Purify to trace the leakage further, but apparently (at least my version) can't understand the debug format of MSVC8, and I don't have older version of the compiler around right now. But it did display some 4000 bytes leaked, just didn't say from where. -- Alo Sarv
I sent my first reply through a newsgroup reader but it seems it got lost somewhere... There is a bug in std::basic_iostream in VC8 (http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=e...) that affects all classes derived from it. During the output of a ptime, a basic_stringstream object is created/destroyed and that is causing the memory leak. I personally think Microsoft should release a patch for this but it looks like they won't until the next release. You might be able to obtain a hotfix by contacting Microsoft Technical support. On the other hand, std::basic_ostringstream/std::basic_istringstream do not suffer the same memory leak. And maybe data_time library can change to use them instead of std::basic_stringstream (line 304 in time_facet.hpp as of release 1.33.0). I haven't really looked into data_time library so I can not say this for sure. HTH, Sean ----- Original Message ----- From: "Alo Sarv" <alo.sarv@gmail.com> To: <boost@lists.boost.org> Sent: Friday, February 17, 2006 12:46 PM Subject: Re: [boost] [date_time] [MSVC8] Memory leak in operator<<(ptime)
On 2/17/06, Jeff Garland <jeff@crystalclearsoftware.com> wrote:
Well, that's odd. The only place that date-time does an allocation is in operator<< and operator>>. So there 'could' be a problem with leaking in that code -- either a bug in date-time code or standard library code. However, time_duration and posix_time use the exact same allocation techniques (and facet code) so should exhibit the exact same leaking behavior. Basically, what gets allocated is a custom date-time facet that gets imbued into the stream. This gets done on the first call to operator>> or operator<< on a particular stream. After that it just uses the facet avoiding the allocation. The facet is written to work with C++ I/O streams which has a reference counting and should destroy the facet when the stream is destroyed. In the case of std::cout this may be after main has exited.
Yes, the first place I looked was that code as well, and thought perhaps the custom facet doesn't get destroyed properly. However, that facet is only created once, so that doesn't seem to be the cause, especially considering other operator<<'s also use basically the same code.
If you want to trace it thru you could explicitly create and delete a stringstream and check it's behavior. The date-time code that does the allocations is in date_time/posix_time/posix_time_io.hpp.
These two snippets also exhibit the leakage (around 50-80kb/s):
using namespace boost::posix_time; ptime p(second_clock::local_time());
while (true) { std::ostringstream *o = new std::ostringstream; *o << p; delete o; }
while (true) { std::ostringstream() << p; }
So clearly it's not some kind of hidden std::cout buffering by the library that could've caused this. I also stepped through the entire call hiearchy of the operator<< in debugger but couldn't find anything of use / suspicious.
I attempted to use Purify to trace the leakage further, but apparently (at least my version) can't understand the debug format of MSVC8, and I don't have older version of the compiler around right now. But it did display some 4000 bytes leaked, just didn't say from where.
-- Alo Sarv
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
There is a bug in std::basic_iostream in VC8 (http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=e...) that affects all classes derived from it. During the output of a ptime, a basic_stringstream object is created/destroyed and that is causing the memory leak. I personally think Microsoft should release a patch for this but it looks like they won't until the next release. You might be able to obtain a hotfix by contacting Microsoft Technical support. On the other hand, std::basic_ostringstream/std::basic_istringstream do not suffer the same memory leak. And maybe data_time library can change to use them instead of std::basic_stringstream (line 304 in time_facet.hpp as of release 1.33.0). I haven't really looked into data_time library so I can not say this for sure. HTH, Sean "Alo Sarv" <alo.sarv@gmail.com> wrote in message news:14a0620d0602161156s55640094xabd31ed80ebde1cb@mail.gmail.com...
The following test app illustrates a memory leak that I'v been able to produce so far with MSVC 8.0 (earlier versions not tested). GCC 4.0 seems to be unaffected. The leakage in case of this test app is around 12kb/s.
#include <iostream> #include <boost/date_time/posix_time/posix_time.hpp> using namespace boost::posix_time; int main(int argc, char *argv[]) { ptime t(second_clock::local_time()); while (true) { std::cout << t << "\r"; } return 0; }
Tested against Boost 1.33.1 release.
Compiled with cl test.cpp /I c:\boost_1_33_1 /D _CRT_SECURE_NO_DEPRECATE /EHsc /D BOOST_ALL_NO_LIB
-- Alo Sarv
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Fri, 17 Feb 2006 13:47:06 -0500, Sean Huang wrote
There is a bug in std::basic_iostream in VC8
Thx for tracking this down.
(http://lab.msdn.microsoft.com/productfeedback/viewfeedback.aspx?feedbackid=e...) that affects all classes derived from it. During the output of a ptime, a basic_stringstream object is created/destroyed and that is causing the memory leak. I personally think Microsoft should release a patch for this but it looks like they won't until the next release. You might be able to obtain a hotfix by contacting Microsoft Technical support.
On the other hand, std::basic_ostringstream/std::basic_istringstream do not suffer the same memory leak. And maybe data_time library can change to use them instead of std::basic_stringstream (line 304 in time_facet.hpp as of release 1.33.0). I haven't really looked into data_time library so I can not say this for sure.
Perhaps it would work, but I have to say I feel highly unmotivated to spend my time working trying to work around this vendor bug. This bug is bad enough that anyone seriously using C++ will have to use the workaround patch available on the MS website. Jeff
From: "Jeff Garland" <jeff@crystalclearsoftware.com>
On the other hand, std::basic_ostringstream/std::basic_istringstream do not suffer the same memory leak. And maybe data_time library can change to use them instead of std::basic_stringstream (line 304 in time_facet.hpp as of release 1.33.0). I haven't really looked into data_time library so I can not say this for sure.
Perhaps it would work, but I have to say I feel highly unmotivated to spend my time working trying to work around this vendor bug. This bug is bad enough that anyone seriously using C++ will have to use the workaround patch available on the MS website.
Jeff
I can totally understand your reluctance. This is really a show-stopper bug for many of us. But your efforts will certainly be appreciated by many boost users. You could also consider the work-around as a minor improvement if I undertsand it correctly. Best regards, Sean
On Sat, 18 Feb 2006 01:37:16 -0500, Sean Huang wrote
From: "Jeff Garland" <jeff@crystalclearsoftware.com>
On the other hand, std::basic_ostringstream/std::basic_istringstream do not suffer the same memory leak. And maybe data_time library can change to use them instead of std::basic_stringstream (line 304 in time_facet.hpp as of release 1.33.0). I haven't really looked into data_time library so I can not say this for sure.
Perhaps it would work, but I have to say I feel highly unmotivated to spend my time working trying to work around this vendor bug. This bug is bad enough that anyone seriously using C++ will have to use the workaround patch available on the MS website.
Jeff
I can totally understand your reluctance. This is really a show- stopper bug for many of us. But your efforts will certainly be appreciated by many boost users. You could also consider the work- around as a minor improvement if I undertsand it correctly.
Since I don't normally test with msvc8 it's alot of work for me to setup an environment to see if making the chang would be effective -- I didn't see anything on the MS site that made it clear that moving to basic_ostringstream would actually resolve the problem. However, if you want to try out some replacements and verify that using ostringstream will work, I'm willing to test and put in the changes. So specifically if you replace std::basic_stringstream with std::basic_ostringstream in date_time/time_facet.hpp that should be a reasonable test to see if there is an effect. I've attached a changed version of the file which compiles and runs on Linux. Let me know. Jeff
Since I don't normally test with msvc8 it's alot of work for me to setup an environment to see if making the chang would be effective -- I didn't see anything on the MS site that made it clear that moving to basic_ostringstream would actually resolve the problem. However, if you want to try out some replacements and verify that using ostringstream will work, I'm willing to test and put in the changes. So specifically if you replace std::basic_stringstream with std::basic_ostringstream in date_time/time_facet.hpp that should be a reasonable test to see if there is an effect. I've attached a changed version of the file which compiles and runs on Linux. Let me know.
Jeff
Jeff, Thanks! I've tried the new version and the memory leak stopped (using the program posted by OP). I also did a search in data_time and found there are other places where basic_stringstream could be replaced with basic_ostringstream. I understand this is a lot of hassles for you so we decided we would apply the changes ourselves (is there anyway we could push Microsoft to release a patch?). As for the bug, I'll try to explain a bit more below. If you have a copy of VC8, you can take a look at <ostream> and <istream>. Both basic_ostream and basic_istream derive from basic_ios and basic_iostream derives from basic_ostream and basic_istream. The basic_iostream constructor used by basic_stringstream is coded as: explicit __CLR_OR_THIS_CALL basic_iostream(basic_streambuf<_Elem, _Traits> *_Strbuf) : basic_istream<_Elem, _Traits>(_Strbuf, false), basic_ostream<_Elem, _Traits>(_Strbuf) { // construct from stream buffer pointer } basic_ios::init is called from the constructors of both basic_istream and basic_ostream in this case. basic_ios::init in turn calls ios_base::Init where a std::locale* member _Ploc (a raw pointer) is newed. This is where the memory leak occurs. If you look at Mr. Plauger's patch, he changed one of the basic_iostream constructors from the above to: explicit __CLR_OR_THIS_CALL basic_iostream(basic_streambuf<_Elem,_Traits> *_Strbuf) : basic_istream<_Elem, _Traits>(_Strbuf, false), basic_ostream<_Elem, _Traits>(_Noinit, false) { // construct from stream buffer pointer } basic_ios::init is not called in this constructor of basic_ostream so _Ploc is only newed once. HTH, Sean
On Sat, 18 Feb 2006 14:12:16 -0500, Sean Huang wrote
Jeff,
Thanks! I've tried the new version and the memory leak stopped (using the program posted by OP).
Thx -- that's helpful.
I also did a search in data_time and found there are other places where basic_stringstream could be replaced with basic_ostringstream. I understand this is a lot of hassles for you so we decided we would apply the changes ourselves (is there anyway we could push Microsoft to release a patch?).
Don't know, but having this show up on the boost list can't hurt.
As for the bug, I'll try to explain a bit more below.
If you have a copy of VC8, you can take a look at <ostream> and <istream>. ....snip details...
I don't have a copy of VC8, but your test and explanation is enough. I've checked changes into CVS that remove all the use of basic_stringstream in date_time that should impact VC8 (there's some usage for legacy compilers that isn't easy to remove that's still there). Those interested in patching should be able to pick up these files their boost tree: date_time/format_date_parser.hpp 1.10 date_time/strings_from_facet.hpp 1.4 date_time/time_zone_base.hpp 1.6 date_time/time_facet.hpp 1.19 Thx for the help Sean. Jeff
Is this the only place this will cause trouble? A quick search shows basic_stringstream being in use in lexical case and serialization. - and in my own code :-(( Is there a problem here too? Paul PS I find the 'fixed' from Microsoft not really good enough - it involves rebuilding a dll. (Considering the amount of trouble it is likely to cause). Why can't they provide a new ready-built dll? I may get round to asking for this. -- Paul A Bristow Prizet Farmhouse, Kendal, Cumbria UK LA8 8AB Phone and SMS text +44 1539 561830, Mobile and SMS text +44 7714 330204 mailto: pbristow@hetp.u-net.com http://www.hetp.u-net.com/index.html http://www.hetp.u-net.com/Paul%20A%20Bristow%20info.html | -----Original Message----- | From: boost-bounces@lists.boost.org | [mailto:boost-bounces@lists.boost.org] On Behalf Of Jeff Garland | Sent: 18 February 2006 21:13 | To: boost@lists.boost.org | Subject: Re: [boost] [date_time] [MSVC8] Memory leak in | operator<<(ptime) | | On Sat, 18 Feb 2006 14:12:16 -0500, Sean Huang wrote | | > Jeff, | > | > Thanks! I've tried the new version and the memory leak stopped | > I also did a search in data_time | > and found there are other places where basic_stringstream could be | > replaced with basic_ostringstream.
From: "Paul A Bristow" <pbristow@hetp.u-net.com>
Is this the only place this will cause trouble?
No. Any place that uses std::basic_iostream (stringstream and fstream etc.)
A quick search shows basic_stringstream being in use in lexical case and serialization.
- and in my own code :-((
Is there a problem here too?
Yes. I think this issue should be made aware to all boost library authors/maintainers.
Paul
PS I find the 'fixed' from Microsoft not really good enough - it involves rebuilding a dll. (Considering the amount of trouble it is likely to cause).
Why can't they provide a new ready-built dll?
I may get round to asking for this.
On 2/20/06, Paul A Bristow <pbristow@hetp.u-net.com> wrote:
Why can't they provide a new ready-built dll? I may get round to asking for this.
Please do! Their work-around is preposterous. This bug is IMHO bad enough to necessitate SP1 of Visual Studio 2005. -- Caleb Epstein caleb dot epstein at gmail dot com
| -----Original Message----- | From: boost-bounces@lists.boost.org | [mailto:boost-bounces@lists.boost.org] On Behalf Of Caleb Epstein | Sent: 22 February 2006 19:30 | To: boost@lists.boost.org | Subject: Re: [boost] [date_time] [MSVC8] Memory leak in | operator<<(ptime) | | On 2/20/06, Paul A Bristow <pbristow@hetp.u-net.com> wrote: | | > Why can't they provide a new ready-built dll? | > I may get round to asking for this. I have suggested this. | Please do! Their work-around is preposterous. This bug is | IMHO bad enough to necessitate SP1 of Visual Studio 2005. Agree strongly. I can sympathize with the bug's creation, but it really needs correction. Paul -- Paul A Bristow Prizet Farmhouse, Kendal, Cumbria UK LA8 8AB Phone and SMS text +44 1539 561830, Mobile and SMS text +44 7714 330204 mailto: pbristow@hetp.u-net.com http://www.hetp.u-net.com/index.html http://www.hetp.u-net.com/Paul%20A%20Bristow%20info.html
participants (6)
-
Alo Sarv -
Bart -
Caleb Epstein -
Jeff Garland -
Paul A Bristow -
Sean Huang