Bug/Incompatibility: Visual C++ 2005 checked iterators lead to major crashes
Hi, Currently with Boost 1.33.1 and Visual C++ 2005, the libraries are compiled with the default _SECURE_SCL=1 value, which means Boost *.lib and *.dll files are compiled with checked iterator enabled. When an application using Boost is compiled with _SECURE_SCL=1 too, everything is fine. However, all hell breaks loose when the application is compiled with _SECURE_SCL=0 (which is the case of our projects). The problem arises because apparently the iterator implementation and structure is different. Therefore any call to Boost *.lib or *.dll functions with an iterator passed as an argument will result in a weird and unexplained crash (I first got the problem when using Boost.Signals). Because Visual C++ 2005 checked iterators are not Standard C++ compliant, I think that it ought to be possible to disable them when compiling Boost (e.g. using an additional bjam variable in the vc-8_0 toolset?). I sincerely think that not having this option is a serious bug, as it makes Boost useless for anyone using VC2005 and having performance constraints. In the meantime, can anyone tell me how it's possible to globally define _SECURE_SCL=0 when compiling Boost 1.33.1? I tried modifying boost\config\compiler\visualc.hpp, but apparently some Windows/VisualC headers are already included before it (causing a double definition of _SECURE_SCL). Regards, Tanguy
Tanguy Fautre wrote:
In the meantime, can anyone tell me how it's possible to globally define _SECURE_SCL=0 when compiling Boost 1.33.1?
bjam ... "-sBUILD=<define>_SECURE_SCL=0" ... -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo
Rene Rivera wrote:
Tanguy Fautre wrote:
In the meantime, can anyone tell me how it's possible to globally define _SECURE_SCL=0 when compiling Boost 1.33.1?
bjam ... "-sBUILD=<define>_SECURE_SCL=0" ...
Thanks. I just tried it, but it's only generating the debug build (where _SECURE_SCL has no effect anyway). After some googling and trial-error, here is the command line I use: bjam.exe "-sTOOLS=vc-8_0" "-sBUILD=debug <define>_CRT_SECURE_NO_DEPRECATE <runtime-link>static/dynamic release <define>_CRT_SECURE_NO_DEPRECATE <define>_SECURE_SCL=0 <runtime-link>static/dynamic" ... Tanguy
Tanguy Fautre <tanguy.fautre@spaceapplications.com> writes:
Rene Rivera wrote:
Tanguy Fautre wrote:
In the meantime, can anyone tell me how it's possible to globally define _SECURE_SCL=0 when compiling Boost 1.33.1?
bjam ... "-sBUILD=<define>_SECURE_SCL=0" ...
Thanks.
I just tried it, but it's only generating the debug build (where _SECURE_SCL has no effect anyway).
After some googling and trial-error, here is the command line I use:
bjam.exe "-sTOOLS=vc-8_0" "-sBUILD=debug <define>_CRT_SECURE_NO_DEPRECATE <runtime-link>static/dynamic release <define>_CRT_SECURE_NO_DEPRECATE <define>_SECURE_SCL=0 <runtime-link>static/dynamic" ...
Repeating <define>_CRT_SECURE_NO_DEPRECATE has no effect, FYI. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Tanguy Fautre <tanguy.fautre@spaceapplications.com> writes:
Hi,
Currently with Boost 1.33.1 and Visual C++ 2005, the libraries are compiled with the default _SECURE_SCL=1 value, which means Boost *.lib and *.dll files are compiled with checked iterator enabled.
When an application using Boost is compiled with _SECURE_SCL=1 too, everything is fine.
However, all hell breaks loose when the application is compiled with _SECURE_SCL=0 (which is the case of our projects). The problem arises because apparently the iterator implementation and structure is different.
Therefore any call to Boost *.lib or *.dll functions with an iterator passed as an argument will result in a weird and unexplained crash (I first got the problem when using Boost.Signals).
Because Visual C++ 2005 checked iterators are not Standard C++ compliant, I think that it ought to be possible to disable them when compiling Boost (e.g. using an additional bjam variable in the vc-8_0 toolset?).
Heh, I think we ought to think very carefully about changing our default. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Heh, I think we ought to think very carefully about changing our default.
I agree: normally our default is the same as the compiler's default, if you want to do something out of the ordinary then it's up to you to compile the lib's the way you want them. I don't think there's any way we can provide every possible binary. John.
John Maddock wrote:
David Abrahams wrote:
Heh, I think we ought to think very carefully about changing our default.
I agree: normally our default is the same as the compiler's default, if you want to do something out of the ordinary then it's up to you to compile the lib's the way you want them. I don't think there's any way we can provide every possible binary.
John.
I agree too. But seeing how bad the STL performance is when using the default (i.e. checked iterator on). I think the least that could be done is to talk about it in the documentation and explain how to disable the checked iterators (http://boost.org/tools/build/v1/vc-8_0-tools.html seems like a good place for this). A better solution for the user would be to provide an additional variable in the VC-8_0 toolset (e.g. bjam "-sVC80_CHECKED_ITERATOR=0"). But I dunno whether that's easily feasible. Tanguy
"John Maddock" <john@johnmaddock.co.uk> writes:
David Abrahams wrote:
Heh, I think we ought to think very carefully about changing our default.
I agree: normally our default is the same as the compiler's default, if you want to do something out of the ordinary then it's up to you to compile the lib's the way you want them. I don't think there's any way we can provide every possible binary.
a. I don't think you're actually agreeing with me. Maybe s/carefully/seriously/? b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
a. I don't think you're actually agreeing with me. Maybe s/carefully/seriously/?
b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case.
I missed (well forgot actually) that the compiler was non-comforming in that mode, I guess it's not quite non-conforming enough to cause problems for most people, although I agree that the performance issue is a very real one. The suggestion to provide a Boost.Build option for this sounds like the best approach though? Anyone want to provide a patch? John.
"John Maddock" <john@johnmaddock.co.uk> writes:
David Abrahams wrote:
a. I don't think you're actually agreeing with me. Maybe s/carefully/seriously/?
b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case.
I missed (well forgot actually) that the compiler was non-comforming in that mode, I guess it's not quite non-conforming enough to cause problems for most people, although I agree that the performance issue is a very real one. The suggestion to provide a Boost.Build option for this sounds like the best approach though? Anyone want to provide a patch?
What interface would you consider appropriate? Right now bjam ... define=_SECURE_SCL=0 ... works with BBv2. Do you want some kind of more descriptive syntax? bjam checked_iterators=no ?? That wouldn't be appropriate if _SECURE_SCL=0 has effects other than turning off iterator checking. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
I missed (well forgot actually) that the compiler was non-comforming in that mode, I guess it's not quite non-conforming enough to cause problems for most people, although I agree that the performance issue is a very real one. The suggestion to provide a Boost.Build option for this sounds like the best approach though? Anyone want to provide a patch?
What interface would you consider appropriate? Right now
bjam ... define=_SECURE_SCL=0 ...
Oh, if it's that easy then I'm happy and I'll just shut up :-) John.
At 12:14 2006-06-15, John Maddock wrote:
David Abrahams wrote:
a. I don't think you're actually agreeing with me. Maybe s/carefully/seriously/?
b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case.
I missed (well forgot actually) that the compiler was non-comforming
"in that mode" in which "mode" ?? , I guess it's not quite non-conforming enough to cause problems for
most people, although I agree that the performance issue is a very real one. The suggestion to provide a Boost.Build option for this sounds like the best approach though? Anyone want to provide a patch?
John.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Victor A. Wagner Jr. http://rudbek.com The five most dangerous words in the English language: "There oughta be a law"
"Victor A. Wagner Jr." <vawjr@rudbek.com> writes:
At 12:14 2006-06-15, John Maddock wrote:
David Abrahams wrote:
a. I don't think you're actually agreeing with me. Maybe s/carefully/seriously/?
b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case.
I missed (well forgot actually) that the compiler was non-comforming
"in that mode"
in which "mode" ??
In the mode where std::vector<T>::iterator is "checked." The type has the right syntax and mostly the right semantics but doesn't meet the complexity requirements to be a conforming iterator. -- Dave Abrahams Boost Consulting www.boost-consulting.com
At 11:37 2006-06-17, you wrote:
"Victor A. Wagner Jr." <vawjr@rudbek.com> writes:
At 12:14 2006-06-15, John Maddock wrote:
David Abrahams wrote:
a. I don't think you're actually agreeing with me. Maybe s/carefully/seriously/?
b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case.
I missed (well forgot actually) that the compiler was non-comforming
"in that mode"
in which "mode" ??
In the mode where std::vector<T>::iterator is "checked." The type has the right syntax and mostly the right semantics but doesn't meet the complexity requirements to be a conforming iterator.
I looked at the standard and I don't see anything other than "must be constant" for the operators provided, hence the missing "complexity column" from the tables. I understand they take much longer to execute than the non checked ones, but I don't find any loops (which would indicate an O(n) or worse) in <vector> . All the conditional code seems to be in the same path and executed no more often which makes it difficult to see how it changes big-O
-- Dave Abrahams Boost Consulting www.boost-consulting.com
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Victor A. Wagner Jr. http://rudbek.com The five most dangerous words in the English language: "There oughta be a law"
"Victor A. Wagner Jr." <vawjr@rudbek.com> writes:
In the mode where std::vector<T>::iterator is "checked." The type has the right syntax and mostly the right semantics but doesn't meet the complexity requirements to be a conforming iterator.
I looked at the standard and I don't see anything other than "must be constant" for the operators provided, hence the missing "complexity column" from the tables. I understand they take much longer to execute than the non checked ones, but I don't find any loops (which would indicate an O(n) or worse) in <vector> . All the conditional code seems to be in the same path and executed no more often which makes it difficult to see how it changes big-O
You may be right. Frankly I've never looked at the code used by MS and was going based on hearsay and dimly remember past experience with STLPort. IIRC from STLPort there are some operations that have to walk the entire list of iterators into a container (hence a loop), but I'm not sure if that affects conformity. std::list<T>::erase, perhaps? That's supposed to be O(1) and maybe that becomes O(N) where N is the number of iterators into the list. -- Dave Abrahams Boost Consulting www.boost-consulting.com
On 18/06/06, David Abrahams <dave@boost-consulting.com> wrote:
You may be right. Frankly I've never looked at the code used by MS and was going based on hearsay and dimly remember past experience with STLPort.
IIRC from STLPort there are some operations that have to walk the entire list of iterators into a container (hence a loop), but I'm not sure if that affects conformity. std::list<T>::erase, perhaps? That's supposed to be O(1) and maybe that becomes O(N) where N is the number of iterators into the list.
The "checked iterators" just include a check to make sure you don't go beyond the end of a container (or do various other bad things); I don't think this breaches any of the complexity requirements. The MS library in a debug build does include some code that is non-conformant because of the performance requirements. For example, the binary search family of algorithms first checks that the range is properly ordered (so they're O(N) instead of O(log N)). But the release builds don't have those checks and so don't breach the requirements. I believe your argument was that boost should disable checked iterators by default (even though they are enabled by default by the compiler) because they were non-conforming. If that isn't true then I suggest staying with the compiler default. It would still of course be useful to be able to easily build the boost libraries without checked iterators. - Alan
"Alan Stokes" <alan@alanstokes.org.uk> writes:
On 18/06/06, David Abrahams <dave@boost-consulting.com> wrote:
You may be right. Frankly I've never looked at the code used by MS and was going based on hearsay and dimly remember past experience with STLPort.
IIRC from STLPort there are some operations that have to walk the entire list of iterators into a container (hence a loop), but I'm not sure if that affects conformity. std::list<T>::erase, perhaps? That's supposed to be O(1) and maybe that becomes O(N) where N is the number of iterators into the list.
The "checked iterators" just include a check to make sure you don't go beyond the end of a container (or do various other bad things);
Like dereferencing an invalid iterator. which requires somehow identifying the iterators that have become invalid.
I don't think this breaches any of the complexity requirements.
How do you do that without looking at all the iterators to see if they're pointing at the node being deleted?
The MS library in a debug build does include some code that is non-conformant because of the performance requirements. For example, the binary search family of algorithms first checks that the range is properly ordered (so they're O(N) instead of O(log N)). But the release builds don't have those checks and so don't breach the requirements.
Is that a distinct switch from the one that makes iterators checked?
I believe your argument was that boost should disable checked iterators by default (even though they are enabled by default by the compiler) because they were non-conforming. If that isn't true then I suggest staying with the compiler default.
Yes, but is it true? A quick inspection of the <list> header suggests that it is. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case.
What's nonconformant about these checked iterators (apart from the warnings, which are annoying, but not nonconformant)?
Yuval Ronen <ronen_yuval@yahoo.com> writes:
David Abrahams wrote:
b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case.
What's nonconformant about these checked iterators (apart from the warnings, which are annoying, but not nonconformant)?
They don't meet the complexity requirements (big-O) for iterators. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Yuval Ronen <ronen_yuval@yahoo.com> writes:
David Abrahams wrote:
b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case. What's nonconformant about these checked iterators (apart from the warnings, which are annoying, but not nonconformant)?
They don't meet the complexity requirements (big-O) for iterators.
Do you happen to have a readily available link to any paper (Microsoft's or other) describing the complexity differences between the standard and its checked iterators? I tried to search MSDN, and google a bit, but found nothing...
On 15/06/06, David Abrahams <dave@boost-consulting.com> wrote:
Yuval Ronen <ronen_yuval@yahoo.com> writes:
What's nonconformant about these checked iterators (apart from the warnings, which are annoying, but not nonconformant)?
They don't meet the complexity requirements (big-O) for iterators.
Isn't that true only in debug builds, not in release builds? - Alan
David Abrahams wrote:
Yuval Ronen <ronen_yuval@yahoo.com> writes:
David Abrahams wrote:
b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case. What's nonconformant about these checked iterators (apart from the warnings, which are annoying, but not nonconformant)?
They don't meet the complexity requirements (big-O) for iterators.
(I have already posted a similar post, but it seems it didn't show up, so I re-post. Sorry if mistaken) Does anyone have a readily available link to a paper (by Microsoft or others) describing the differences between the complexity of the checked iterators and the standard (I looked in MSDN and Google, but found nothing)?
David Abrahams wrote:
Yuval Ronen <ronen_yuval@yahoo.com> writes:
David Abrahams wrote:
b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case. What's nonconformant about these checked iterators (apart from the warnings, which are annoying, but not nonconformant)?
They don't meet the complexity requirements (big-O) for iterators.
I asked this question at the Microsoft forums and the answer was that release (checked) iterators are conformant, while debug iterators are not - http://tinyurl.com/jwgvv. I guess that if anyone thinks otherwise, MS people would be glad to here about it...
Yuval Ronen <ronen_yuval@yahoo.com> writes:
David Abrahams wrote:
Yuval Ronen <ronen_yuval@yahoo.com> writes:
David Abrahams wrote:
b. Normally our default is not the same as the compiler's default when the compiler's default is nonconformant, as it is in this case. What's nonconformant about these checked iterators (apart from the warnings, which are annoying, but not nonconformant)?
They don't meet the complexity requirements (big-O) for iterators.
I asked this question at the Microsoft forums and the answer was that release (checked) iterators are conformant, while debug iterators are not - http://tinyurl.com/jwgvv.
I guess that if anyone thinks otherwise, MS people would be glad to here about it...
AFAICT, erasing a list node ends up being linear in the number of checked iterators into the list rather than O(1), regardless of the debug/release setting. Am I missing something? -- Dave Abrahams Boost Consulting www.boost-consulting.com
On 20/06/06, David Abrahams <dave@boost-consulting.com> wrote:
AFAICT, erasing a list node ends up being linear in the number of checked iterators into the list rather than O(1), regardless of the debug/release setting. Am I missing something?
Well, I don't see that, so at least one of us must be. Are you talking about the call to _Orphan_ptr? All those calls, and indeed the function itself, are inside #if _HAS_ITERATOR_DEBUGGING blocks. And if you look at yvals.h you see that _HAS_ITERATOR_DEBUGGING is normally 0 in a release build. The release mode code looks straightforward and constant-time to me. (Interestingly the erase(Iterator, Iterator) function looks to me to be quadratic in debug mode, which it really doesn't need to be. I guess once they decided to break the performance guarantees they wanted to *really* break them.) - Alan
participants (7)
-
Alan Stokes -
David Abrahams -
John Maddock -
Rene Rivera -
Tanguy Fautre -
Victor A. Wagner Jr. -
Yuval Ronen