- Boost-users - lists.preview.boost.org

problem in multithreaded regex_replace on linux
by Ashish Varman 19 Jun '06

19 Jun '06

Hello all I am getting segmentation faults when we use our program in multithreaded mode on linux (RedHat 9). The same code runs flawlessly on windows. The crash always happens in boost::regex_replace. I had read the following posts on this topic: http://lists.boost.org/MailArchives/boost/msg59110.php and http://lists.boost.org/boost-users/2005/01/9712.php Accordingly, I've initialised all the regexes before the main working threads starts. I also came across the gcc string threading issue at http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21334 and switched to STLport but to no avail. If anything, the crash happens much more quickly. Stack trace is as follows: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1129827392 (LWP 3124)] 0x4013f843 in boost::re_detail::basic_regex_formatter<boost::re_detail::string_out_iterator<stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > >, boost::match_results<wchar_t const*, stlp_std::allocator<boost::sub_match<wchar_t const*> > >, boost::regex_traits_wrapper<boost::regex_traits<wchar_t, boost::cpp_regex_traits<wchar_t> > > >::format_all() ( this=0x4357bcfc) at regex_format.hpp:133 133regex_format.hpp: No such file or directory. in regex_format.hpp (gdb) bt Thread 3 (Thread 1129827392 (LWP 3124)): #0 0x4013f843 in boost::re_detail::basic_regex_formatter<boost::re_detail::string_out_iterator<stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > >, boost::match_results<wchar_t const*, stlp_std::allocator<boost::sub_match<wchar_t const*> > >, boost::regex_traits_wrapper<boost::regex_traits<wchar_t, boost::cpp_regex_traits<wchar_t> > > >::format_all() (this=0x4357bcfc) at regex_format.hpp:133 #1 0x4013f72c in boost::re_detail::basic_regex_formatter<boost::re_detail::string_out_iterator<stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > >, boost::match_results<wchar_t const*, stlp_std::allocator<boost::sub_match<wchar_t const*> > >, boost::regex_traits_wrapper<boost::regex_traits<wchar_t, boost::cpp_regex_traits<wchar_t> > > >::format(wchar_t const*, wchar_t const*, boost::regex_constants::_match_flags) ( this=0x4357bcfc, p1=0x4357bdac, p2=0x4357bdc4, f=50331648) at regex_format.hpp:123 #2 0x4013f5bd in boost::re_detail::string_out_iterator<stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > > boost::re_detail::regex_format_imp<boost::re_detail::string_out_iterator<stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > >, wchar_t const*, stlp_std::allocator<boost::sub_match<wchar_t const*> >, wchar_t, boost::regex_traits_wrapper<boost::regex_traits<wchar_t, boost::cpp_regex_traits<wchar_t> > > >(boost::regex_traits_wrapper<boost::regex_traits<wchar_t, bo ost::cpp_regex_traits<wchar_t> > >, boost::match_results<wchar_t const*, stlp_std::allocator<boost::sub_match<wchar_t const*> > > const&, wchar_t const*, wchar_t, boost::regex_constants::_match_flags, boost::regex_traits_wrapper<boost::regex_traits<wchar_t, boost::cpp_regex_traits<wchar_t> > > const&) (out= {<iterator<stlp_std::output_iterator_tag,wchar_t,int,wchar_t*,wchar_t&>> = {<No data fields>}, out = 0x4357becc}, m=@0x80c3d70, p1=0x4357bdac, p2=0x4357bdc4, flags=50331648, t=@0x8077d48) at regex_format.hpp:569 #3 0x4013f28c in format<boost::re_detail::string_out_iterator<stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > >, boost::basic_regex<wchar_t, boost::regex_traits<wchar_t, boost::cpp_regex_traits<wchar_t> > > > (this=0x80c3d70, out= {<iterator<stlp_std::output_iterator_tag,wchar_t,int,wchar_t*,wchar_t&>> = {<No data fields>}, out = 0x4357becc}, fmt=@0x4357bdac, flags=50331648, re=@0x810eefc) at match_results.hpp:170 #4 0x4013ed6b in boost::re_detail::string_out_iterator<stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > > boost::regex_replace<boost::re_detail::string_out_iterator<stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > >, wchar_t const*, boost::regex_traits<wchar_t, boost::cpp_regex_traits<wchar_t> >, wchar_t>(boost::re_detail::string_out_iterator<stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > >, wchar_t const*, boost::re_detail::string_out_iterator<stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > >, boost::basic_regex<wchar_t, boos t::regex_traits<wchar_t, boost::cpp_regex_traits<wchar_t> > > const&, boost::re_detail::string_out_iterator<stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > > const*, boost::regex_constants::_match_flags) (out= {<iterator<stlp_std::output_iterator_tag,wchar_t,int,wchar_t*,wchar_t&>> = {<No data fields>}, out = 0x4357becc}, first=0x4357bfdc, last=0x4357bff8, e=@0x810eefc, fmt=0x4357bf3c, flags=50331648) at regex_replace.hpp:53 #5 0x4013e123 in stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > boost::regex_replace<boost::regex_traits<wchar_t, boost::cpp_regex_traits<wchar_t> >, wchar_t>(stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > const&, boost::basic_regex<wchar_t, boost::regex_traits<wchar_t, boost::cpp_regex_traits<wchar_t> > > const&, stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> > const, boost::regex_constants::_match_flags) ( s=@0x4357bfdc, e=@0x810eefc, fmt=@0x4357bf3c, flags=50331648) at regex_replace.hpp:96 #6 0x4013d9c5 in RegExRuleSet::ApplyRules(stlp_std::basic_string<wchar_t, stlp_std::char_traits<wchar_t>, stlp_std::allocator<wchar_t> >&) (this=0x807d93c, zrTok=@0x4357c12c) at common/utils/RuleApplier.cpp:192 #7 0x4014d42a in Transform1::PhonetizeAndSyllabify(stlp_std::vector<Item, stlp_std::allocator<Item> >&) (this=0x807d8e8, zrItems=@0x4188d448) at textProcessor/Transforms/Transform1/Transform1.cpp:620 #8 0x4014c0f9 in Transform1::Process(stlp_std::vector<Item, stlp_std::allocator<Item > >&) (this=0x807d8e8, zrItems=@0x4188d448) at textProcessor/Transforms/Transform1/Transform1.cpp:326 #9 0x4012f327 in Fragment::ProcessTransforms(int) (this=0x4188d448, zLangIndex=0) at common/shared/Fragment.cpp:187 #10 0x40119ef8 in Engine::ProcessFragment() (this=0x4357c50c) at main/Engine.cpp:64 #11 0x401110b4 in Input2Stage1(wchar_t const*, Engine&, int) ( zpText=0x80b9cb0, zrEngineObj=@0x4357c50c, zLangIndx=0) at main/InterfaceFunctions.cpp:137 #12 0x401039da in GetOutput (zpText=0x80b9cb0, zSamplingRate=16000, zBitRate=16, zTimeModFact=1, zPitchModFact=0, zpOutFileName=0x4357c60c "./buffer/Thread_1_Line_3.wav", zLangIndx=0, zOutput=WAVEFILE, zppWaveBuffer=0x0) at main/Interface.cpp:185 #13 0x08056ea4 in GetOutputer(void*) (zpThreadPara=0xbfffe5b0) at testing/MultiThreadedApp.cpp:194 #14 0x404da2b6 in start_thread () from /lib/tls/libpthread.so.0 The code fragment is as follows vector <Rule>::iterator yIterInRule = this->inclusiveRuleSet.begin(); for (; yIterInRule != inclusiveRuleSet.end(); yIterInRule++) { const boost::wregex &ySearchExp = yIterInRule->getSearchExpr(); yReplaceStr.assign(yIterInRule->getReplaceExpr()); if (yIterInRule->getContinousStatus() == false) { yInputStr = regex_replace(yInputStr,ySearchExp, yReplaceStr, boost::format_all | match_default | format_sed); } else { do { yInputStr = boost::regex_replace(yInputStr,ySearchExp, yReplaceStr, boost::format_all | match_default | format_sed); ystart = yInputStr.begin(); yend = yInputStr.end(); } while (boost::regex_search(ystart, yend,ySearchExp)); } // End Else }// end of inclusiveRuleSet for loop It crashes in any of the two regex_replace calls. OS is Red Hat Linux 9 The compiler is gcc-3.2.2 STLport is STLport-5.0.2 boost is boost_1_33_1 Crash happens only on multiprocessor machines. It would overcome a great roadblock if someone helps Thanks in advance Yours Sincerely Ashish Varman

2 1

hash table
by Brendan Miller 18 Jun '06

18 Jun '06

I was wondering if there's been any progress in adding hash tables to the boost library. Past posts, as long as two years ago, indicated that there was a good implementation available with standard interface, but it just hadn't been thrown in the official distribution yet. What's the holdup? You'd think hashtables would be the first thing to go into the library and not the last.

1 0

Newbie question: installation of property_tree.
by Thierry Dumont 17 Jun '06

17 Jun '06

I am trying to use "property_tree" library; as it is not yet included in the frozen distribution of boost on my Linux-Debian machine, I need to install it. Is it enough to copy the "property_tree" tree of the archive in /usr/include/boost ? Or do I need to compile something using bjam? I only see .hpp files in the distribution. Thank you in advance t.d. -- Thierry Dumont. Institut Camille Jordan -- Mathematiques-- Univ. Lyon I,43 Bd du 11 Novembre 1918, 69622 - Villeurbanne Cedex - France. Thierry.Dumont(a)maply.univ-lyon1.fr web: http://maply.univ-lyon1.fr/~tdumont

2 1

[Wave] advanced preprocessing hooks
by Andreas Sæbjørnsen 17 Jun '06

17 Jun '06

Hi, I have a question and a suggestion regarding Wave. First I have a problem when preprocessing the following file: #if 0 #elif 1 test #endif #if 1 #elif 0 int x; #endif The first '#elif' directive is registered as an 'evaluated_conditional_expression(...)', but the second '#elif' is never evaluated since the previous '#if' directive is true, and it is therefore not registered as either a 'skipped_token(...)' or an 'evaluated_conditional_expression(...)'. The way I expected it to work was to register it as a 'skipped_token(...)'. Do you have a suggestion to how I should otherwise be handling this? If not, could it be possible to add the non-evaluated '#elif' to the skipped token stream? The suggestion is for preprocessor generated tokens where the ' macro.get_position().get_file().size() == 0'. An example of this is in the following code #ifdef SOME_MACRO //DO SOMETHING #endif Since 'SOME_MACRO' is not defined the 'expanding_object_like_macro(TokenT const &macro,....)' will be passed a TokentT& const& macro with no position object, so asking for the line or column number will give a hard to debug segmentation fault. In order to detect this to avoid an error it is necessary to always ask for the size of the file as demonstrated above before asking for the line, file or column My suggestion is that alternatively a position object could be attached to 'macro', and a Boolean flag 'is_preprocessor_generated' could be used to find out if the position object is preprocessor generated or not. What do you think? Thanks Andreas

2 5

[pool] Problem with operator new[](std::size_t) and operator delete[]()
by Boris 17 Jun '06

17 Jun '06

I created a memory manager based on boost::pool and overloaded various operators like this: template <class T> class mem_manager { protected: static boost::pool<my_allocator> mem_pool; public: void *operator new(std::size_t) { return mem_pool.malloc(); } void operator delete(void *p) { mem_pool.free(p); } void *operator new(std::size_t, void *p) { return p; } void operator delete(void*, void*) { } void *operator new[](std::size_t size) { return mem_pool.ordered_malloc(size / sizeof(T)); } void operator delete[](void *p) { mem_pool.ordered_free(p); } }; Any class derived from mem_manager uses now its own boost::pool. While new and delete work fine there is a problem with new[] and delete[]. I haven't really investigated the problem yet because I'm not sure if I implemented these two operators correctly or if I understood ordered_malloc() and ordered_free() wrong. The documentation at http://www.boost.org/libs/pool/doc/interfaces/pool.html says something about merging free lists and preserving order with ordered_malloc(). Is this related to allocating an array? Or how shall I implement new[] and delete[] correctly? Boris

1 0

Re: [Boost-users] Trap in regex library (help requested)
by Marty Eaves 16 Jun '06

16 Jun '06

John, Thanks, I'll look into the STL compatibility. We were using STL port with VC6, but I think we are using the Microsoft STL with VC7. I'll double check. As far as test cases go, the regex string looks like this: "^(((((((((19)|(20)|(21)))){0,1}((((0)|(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|( 9)))(((0)|(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|(9))))){0,1}(((01)|(02)|(03)|( 04)|(05)|(06)|(07)|(08)|(09)|(10)|(11)|(12)))(((01)|(02)|(03)|(04)|(05)| (06)|(07)|(08)|(09)|(10)|(11)|(12)|(13)|(14)|(15)|(16)|(17)|(18)|(19)|(2 0)|(21)|(22)|(23)|(24)|(25)|(26)|(27)|(28)|(29)|(30)|(31)))))))" As back ground information, this string was generated from a SRGS date DTMF grammar. I am attempting to modify the code to try to determine which parts of the regex string where used in the match. This is necessary to implement support for the <tag> element in the SRGS grammar. I am looking into potentially trying to use "named groups" whenever a <tag> is encountered in the SRGS grammar as the "regex grammar" is getting generated. Thus I am starting to play with the extended match functionality. I have not modified any code yet, other than using the BOOST_REGEX_MATCH_EXTRA flag in user.hpp. This appears to cause the exception detailed in the call stack. The test case failing is using the above grammar with the value of into the "inputValue" parameter set to "2" in the code snippet below: else { // no termchar std::string tmpString; boost::match_results<std::string::const_iterator> what; try { boost::regex e(regex_grammar.c_str()); // return false for partial match, true for full match, or throw for // impossible match based on what we have so far... if(0 == boost::regex_match(inputData, what, e, boost::match_default | boost::match_partial | boost::match_extra)) { // the input so far could not possibly be valid so reject it: return SEQUENCE_NOMATCH; } } catch(...) { VXML_LOG::CLogBuff log(LOG_REC_SIZE,IVILOG_VXML_MODULE); log.WriteRecord("CDtmfGrammar_custom::check", // pTrm->GetLine(), 0, IVILOG_CLASS_TRACE, __FILE__,__LINE__, "Exception from Boost regex caught. Grammar = (%s)",regex_grammar.c_str()); return SEQUENCE_NOMATCH; } // OK so far so good, but have we finished? if(what[0].matched) { GRAMMAR_RETM_ITER iter=m_input_map.find(inputData.c_str()); if (iter != m_input_map.end()) { if(pResultValue) *pResultValue = (*iter).second; } else { if(pResultValue) { slotData = CDtmfGrammar::addWhiteSpaces(inputData); *pResultValue = slotData.c_str(); } } // excellent, we have a result: { boost::regex e(regex_grammar.c_str()); std::string newInput; bool matchOccurred = false; // loop thru 1 2 3 4 5 6 7 8 9 0 * # A B C D std::string charArray = "1234567890*#ABCD"; for(unsigned int x=0; x < charArray.length(); x++) { newInput = inputData + charArray.substr(x,1); if(0 != boost::regex_match(newInput, what, e, boost::match_default | boost::match_partial | boost::match_extra)) { matchOccurred = true; break; } } if(!matchOccurred) return SEQUENCE_OK; // we are done. } } // what we have so far is only a partial match... return SEQUENCE_MORE; } } ________________________________ Marty Eaves STAFF SOFTWARE ENGINEER CALL PROCESSING & MEDIA SERVICES RESEARCH AND DEVELOPMENT INTERVOICE, INC. P: (972) 454-8348 marty.eaves(a)intervoice.com ________________________________ Intervoice: Connecting People and Information. -----Original Message----- From: boost-users-bounces(a)lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of John Maddock Sent: Thursday, June 15, 2006 1:22 PM To: boost-users(a)lists.boost.org Subject: Re: [Boost-users] Trap in regex library (help requested) Marty Eaves wrote: > I am having an issue using the regex library. I am developing in VC7 > and this issue occurs with both Boost 1.33.0 and 1.33.1. I have > recently tried to start retrieving capture information in the regex > matches by using the match_result and sub_match classes. In order to > use these classes, the compile flag BOOST_REGEX_MATCH_EXTRA is > required > to be turned on (enabled in user.hpp). With the library is built with > this flag on, I will get the following trap when the method calling > regex_match() returns. Just a clarification: you only need that define if you want extended capture information. You can access regular Perl style marked sub-expressions perfectly well without it. > An unhandled exception of type 'System.NullReferenceException' > occurred > in tg12.exe > > > > Additional information: Object reference not set to an instance of an > object > > > > This exception appears to occur in the cleanup logic of the > ~match_results() destructor. I have yet to make any changes to the > code > to use the match_results object. In other words, the code I am using > worked previously without the BOOST_REGEX_MATCH_EXTRA flag enabled. > Are > there any compiler options required? No none, if I'm reading your stack trace right it's std::vector that's causing the failure: there's nothing fancy going on inside sub_match just a scoped_ptr to a vector or sub_matches. Do you have a test case I can look at? If you're using managed code (looks like you are?) you might want to double check to see if there are any known issues using that with the STL. John. _______________________________________________ Boost-users mailing list Boost-users(a)lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users This e-mail transmission may contain information that is proprietary, privileged and/or confidential and is intended exclusively for the person(s) to whom it is addressed. Any use, copying, retention or disclosure by any person other than the intended recipient or the intended recipient's designees is strictly prohibited. If you are the intended recipient, you must treat the information in confidence and in accordance with all laws related to the privacy and confidentiality of such information. If you are not the intended recipient or their designee, please notify the sender immediately by return e-mail and delete all copies of this email, including all attachments.

2 1

[serialization] mac/windows difference
by Hugh Hoover 16 Jun '06

16 Jun '06

I have a test program (boost::test) for a set of my application classes. When run on darwin (10.4.6, gcc4.01) the tests work correctly - that is, I can serialize the structure to an XML archive and read it back (actually, all 3 archive types). When I run the same test on windows (XP sp2, vc8), the read back of the XML archive fails very early because it apparently can't find an appropriate class to instantiate. When I look at the XML output, the windows version has a class name at the point where it fails, while the darwin version does not (there are NO class names written in the darwin archive). The code between darwin and windows is essentially the same - there aren't any differences in MY code in the serialization code. Several points: a) The topmost level is a pointer (plain) to an abstract class - the concrete subclass being serialized is very simple at this point. b) The specific class that's failing is from an STL map with pointers to an abstract base class. 1) That particular failing class is successfully tested prior to where it fails in the same test application (output to all archive types and read-back). 2) although it's using a different archive to test, initialization of all archives is identical (register_type and register_void_pointer called as appropriate) c) There is one other class that also gets serialized with a class name in addition to it's ID (I presume this means that something's up with class registration?) Ok - any suggestions on how to tackle this? I can't find anything wrong in the registration (i've walked through my part of the registration code in the debugger). I'm using boost 1.33.1 Thanks Hugh Hoover Enumclaw Software

3 3

Re: [Boost-users] [serialization-1.32.0] de-serializing newer versions that the application can deal with
by Tarjei Knapstad 16 Jun '06

16 Jun '06

[sorry for breaking the threading, just re-joined the list] > We have system data written out to an XML archive and being > de-serialized at application startup. > > One object type in the archive we written at version 1 by a newer > version of the application. We then ran an older version of the > software which only understood version 0 of the object. Hi Russel, To handle different versions of data you should split serialize() into separate save()/load() functions as described in the tutorial: http://boost.org/libs/serialization/doc/tutorial.html#splitting The implementation will look something like: void save(...) { // Serialize all data here } void load(Archive & ar, unsigned int version) { if (version > 0) { // Load data added in version 1 } if (version > 1) { // Load data added in version 2 } // Load version 0 data } Regards, -- Tarjei

4 6

Boost.property_tree help
by Demetrius Cassidy 16 Jun '06

16 Jun '06

Reading the tutorial for property_tree, using the functions ptree.find, ptree.get and ptree.get_child, they will only return the topmost node, so if I had two different children nodes under my root named '<Logistics>' with different data within the children, I would only be able to access the first one. So my question is that how would I go about navigating to each top node under my root when they are all named the same? I basicly will have a file that looks something like this: XML file contents: <Technologies> <Logistics ID="Damage Diagnostics"> <DisplayName>Damage Diagnostics</DisplayName> <Cost>1500</Cost> <Description>Increases repair ability by 10%</Description> <RepairAbility>10</RepairAbility> <Requires>Advanced Starship Repair</Requires> <Category>Logistics</Category> <Model>lrgbuilding0</Model> <AIValue>30</AIValue> </Logistics> <Logistics ID="Basic Self-Repair"> /* I need to access this node */ <DisplayName>Basic Self-Repair</DisplayName> <Cost>3000</Cost> <Description>Increases repair ability by 10%</Description> <RepairAbility>10</RepairAbility> <Requires>Damage Diagnostics</Requires> <Category>Logistics</Category> <Model>controlgrav0</Model> <AIValue>40</AIValue> </Logistics> </Technologies> I will have a very large number of nodes with the same name under my root, but each will have different data in their children that I need to access. A full DOM parser is too much overhead when I do not need any validation, but rather I need to read a large number of configuration data across XML files. I really like the ease of use of this library, but I just can't figure this out. Any help would be greatly appreciated.

2 3

templatized constructor, destructor
by Gottlob Frege 16 Jun '06

16 Jun '06

Is there really no way to templatize a) a zero argument constructor b) a destructor ? I'm trying to delay the compile of the functions until everything is known... Tony

3 2