November 2007 - Boost - lists.preview.boost.org

seek issue with boost::iostreams::stream
by Steve Nutt 08 Jan '08

08 Jan '08

I've created a device with the boost::iostreams::dual_seekable category to use with the boost::iostreams::stream class, but I'm having trouble implementing seek. When I call seekp on the stream, seek on my device is called with a which (openmode) of both BOOST_IOS::in and BOOST_IOS::out. I would have expected for a call to seekp the which parameter in the seek call on my device would be BOOST_IOS::out, and calling seekg would set which to BOOST_IOS::in. The following code in boost\iostream\detail\streambuf\indirect_streambuf.hpp ignores the which argument passed and passes on through both in and out. template<typename T, typename Tr, typename Alloc, typename Mode> inline typename indirect_streambuf<T, Tr, Alloc, Mode>::pos_type indirect_streambuf<T, Tr, Alloc, Mode>::seekpos (pos_type sp, BOOST_IOS::openmode) { return seek_impl( position_to_offset(sp), BOOST_IOS::beg, BOOST_IOS::in | BOOST_IOS::out ); } Is this correct, and if so how do I implement seeking when using a dual head device? Many thanks, Steve Nutt

2 2

[IOStreams] Opinion on test case needed
by Frank Birbacher 06 Jan '08

06 Jan '08

Hi! I had problems using iostreams::stream<file_descriptor> with Windows and MSVC 8 Express. The file_descriptor had a bug and failed seeking. But the error had gone unnoticed. Because of this I wrote a test case for this in hope to discover an error in IOStreams (source: see below). I did not discover one. Instead the test runs fine on linux/gcc 4.1.2 but fails on Windows/MSVC8Express (test run: see below). I based the test on "Standard C++ IOStreams and Locales" (Langer&Kreft), Section 1.3 "The Stream State". And I found the IOStreams library is just using std::iostream for handling the errors. So is MSVC buggy in this regard? Or am I missing something? Source of test case in Trac SVN Browser: http://svn.boost.org/trac/boost/browser/sandbox-branches/birbacher/fix_iost… Result of test case on platform Windows/MSVC 8 Express: > ...using <p..\..\..\bin.v2\libs\iostreams\test\error_test.test\msvc-8.0express\debug>error_test.exe... > testing.capture-output ..\..\..\bin.v2\libs\iostreams\test\error_test.test\msvc-8.0express\debug\error_test.run > ====== BEGIN OUTPUT ====== > Running 12 test cases... > error_test.cpp(81): error in "wrap_nothrow <&test_seekg>": exception thrown by function(stream) > error_test.cpp(66): error in "wrap_nothrow <&test_seekg>": stream still good > error_test.cpp(68): error in "wrap_nothrow <&test_seekg>": stream did not set badbit > error_test.cpp(69): error in "wrap_nothrow <&test_seekg>": stream did not fail > error_test.cpp(71): error in "wrap_nothrow <&test_seekg>": stream does not report failure by operator ! > error_test.cpp(73): error in "wrap_nothrow <&test_seekg>": stream does not report failure by operator void* > error_test.cpp(66): error in "wrap_throw <&test_seekg>": stream still good > error_test.cpp(68): error in "wrap_throw <&test_seekg>": stream did not set badbit > error_test.cpp(69): error in "wrap_throw <&test_seekg>": stream did not fail > error_test.cpp(71): error in "wrap_throw <&test_seekg>": stream does not report failure by operator ! > error_test.cpp(73): error in "wrap_throw <&test_seekg>": stream does not report failure by operator void* > unknown location(0): fatal error in "wrap_throw_delayed<&test_seekg>": std::runtime_error: bad seek > error_test.cpp(73): last checkpoint > error_test.cpp(81): error in "wrap_nothrow <&test_seekp>": exception thrown by function(stream) > error_test.cpp(66): error in "wrap_nothrow <&test_seekp>": stream still good > error_test.cpp(68): error in "wrap_nothrow <&test_seekp>": stream did not set badbit > error_test.cpp(69): error in "wrap_nothrow <&test_seekp>": stream did not fail > error_test.cpp(71): error in "wrap_nothrow <&test_seekp>": stream does not report failure by operator ! > error_test.cpp(73): error in "wrap_nothrow <&test_seekp>": stream does not report failure by operator void* > error_test.cpp(66): error in "wrap_throw <&test_seekp>": stream still good > error_test.cpp(68): error in "wrap_throw <&test_seekp>": stream did not set badbit > error_test.cpp(69): error in "wrap_throw <&test_seekp>": stream did not fail > error_test.cpp(71): error in "wrap_throw <&test_seekp>": stream does not report failure by operator ! > error_test.cpp(73): error in "wrap_throw <&test_seekp>": stream does not report failure by operator void* > unknown location(0): fatal error in "wrap_throw_delayed<&test_seekp>": std::runtime_error: bad seek > error_test.cpp(73): last checkpoint > > *** 24 failures detected in test suite "Master Test Suite" > > EXIT STATUS: 201 > ====== END OUTPUT ====== Any opinions on this? Any suggestions on what to do about it? Frank

4 16

[iostreams] understanding the models
by Benjamin Collins 05 Jan '08

05 Jan '08

I'm trying to write a device for iostreams, but I'm having a hard time understanding how the current sources and the models explained in the documentation are in agreement. Maybe I'm shooting myself in the foot by never being able to work on this for more than 30 minutes at a time and having that 30 minutes be late at night, but... Anyway, it seems odd to me - for example, the boost::iostreams::mapped_file_source class has the following category of source_tag + direct_tag + closable_tag, and has member functions open(), size(), data(), begin(), and end(). None of these are particularly surprising. After all - a mapped_file has to be open()ed at some point. But under what model as defined in the docs? In other words, how and when is that function called, and how does a stream_buffer know he can call open() on a particular device? Also, the documentation indicates that a source device needs to support the read() function - but mapped_file_source has no read() (maybe because it is convertible to 'direct'?). -- Benjamin A. Collins <ben.collins(a)acm.org> http://bloggoergosum.us

2 1

[iostreams] Patch for latest CodeGear (Borland) compiler
by Nicola Musatti 05 Jan '08

05 Jan '08

Hallo, The patch in http://svn.boost.org/trac/boost/ticket/1294 improves support for the latest version of bcc32, 5.9.2 . If I don't receive objections in two or three days I'll apply it. Cheers, Nicola -- Nicola.Musatti <at> gmail <dot> com Home: http://nicola.musatti.googlepages.com/home Blog: http://wthwdik.wordpress.com/

2 1

Performance of various mutex implementations on Linux
by Phil Endecott 02 Jan '08

02 Jan '08

Dear All, I have been benchmarking different mutex implementations on Linux. The variations that I have considered are the following: NO_LOCKING: - no mutex at all, as a baseline for comparison. SPINLOCK - uses an atomic operation and a tight loop. YIELD - uses an atomic operation like SPINLOCK, and calls sched_yield() if the lock is held by another thread. PTHREADS - uses the pthread_mutex_*() functions; this is what the current Boost mutexes do (as I understand it). The pthread_mutex functions are implemented using the futex() system call. FUTEX - this is my own implementation of the same algorithm that PTHREADS uses internally, based on the futex() system call, but it uses inlining to avoid a function call in the un-contended case. Apart from the PTHREADS method, these should all be considered unproven code; if anyone would like to review them for any subtle concurrency issues or other problems you are very welcome to do so. For background details of futexes, I suggest Ullrich Drepper's paper "Futexes Are Tricky". All of these results were obtained with g++ 4.1.2 with -O3. Data Structure Sizes -------------------- The size of the mutex object, in bytes, is as follows: NO_LOCKING: 0 SPINLOCK: 4 [*] YIELD: 4 [*] FUTEX: 4 [*] PTHREADS: 24 [*] in all of these cases I believe that a single byte, or perhaps even a single bit in the case of SPINLOCK and YIELD, would be sufficient; there may be some obscure alignment issue though. I believe that the large size of the pthreads mutex object is because it supports a number of variants such as recursive and checking mutexes, and timeouts, etc. all in the same structure. My other implementations are all very basic with none of these features. Execution Time, Non-Contended Case ---------------------------------- In many uses mutexes are rarely contended, so optimising the speed of the uncontended locking and unlocking operations is a common objective. I have measured this overhead on three machines: egypt: 1GHz VIA C3 x86 uniprocessor linux-2.6.19 ghana: 266MHz Intel XScale ARM uniprocessor linux-2.6.19 atum: 3.6GHz Intel Pentium 4 x86 2 processor linux-2.6.17 The times in nanoseconds for one lock and one unlock are: egypt ghana atum NO_LOCKING 0 0 0 SPINLOCK 17 30 5.7 YIELD 13 38 6.4 FUTEX 50 95 26 PTHREADS 115 986 39 Execution Time, Contended Case ------------------------------ See my detailed results, linked below, for numbers (the runtimes given are for 10^8 operations). The execution time in the contended case includes both the execution time of the lock and unlock functions and the time spent waiting for the lock to become available; it's not straightforward to distinguish these and I haven't tried to. The broad pattern is that SPINLOCK, naturally, is very slow, and the other methods are relatively fast. In my benchmark, YIELD does better than FUTEX, but this may well be a feature of the structure of my benchmark; I would expect FUTEX to do better with large numbers of threads. PTHREADS gets worse more quickly than other other methods as the number of mutexes and threads increases, perhaps because the larger size of its structures is less cache-friendly. Code Size --------- The code size overhead in bytes for each method is as follows. (Note that this is with -O3, not -Os. My impression with -Os is that it does even less inlining that I expected, e.g. even a static function that's only called once is not inlined. I may investigate this further.) x86 ARM NO_LOCKING 0 0 SPINLOCK 42 32 YIELD 31 56 FUTEX 66 96 PTHREADS 25 28 Observations ------------ I suggest that we seriously consider using an inline futex implementation of mutexes as the default choice on Linux, rather than calling the pthread_mutex functions. This is about twice as fast (on x86) and saves a lot of data size. On the other hand, it increases code size. The code size and speed can be improved further by using a simpler method that just calls sched_yield(). However, this method is less suitable for use as a default implementation because it will have worse complexity in applications with contention: futex() knows which thread to wake up next, while sched_yield() doesn't. ARM Observations ---------------- Most ARM processors have only one atomic instruction, swap. My implementations of SPINLOCK, YIELD and FUTEX make use of this. PTHREADS has abysmal performance on ARM; I suspect, but have not yet checked, that it somehow emulates the atomic instructions that it doesn't have by calling into the kernel. You can find all of the mutex implementation and benchmarking scripts and the full results here: https://svn.chezphil.org/mutex_perf/trunk/ I hope this is of interest. Regards, Phil.

4 4

Formal review request: Flyweight
by Joaquín Mª López Muñoz 19 Dec '07

19 Dec '07

Hello, I request that the Candidate Boost Flyweight Library be accepted into the review queue. The latest public version of the library is described at http://lists.boost.org/Archives/boost/2007/11/130274.php There is no pre-appointed review manager for this library. Thank you, Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

7 12

[lib maintainers] Please identify headers in boost-root/boost
by Beman Dawes 18 Dec '07

18 Dec '07

For the headers that appear in boost-root/boost, rather than a library specific subdirectory, it is very helpful if they self-identify what library they belong to. This allows scripts to associate the header with a library. The usual way to do this is to include a comment. A typical example: // See http://www.boost.org/libs/config for documentation The exact format doesn't matter. What does matter is that the library name be found somewhere in the file, immediately following a string which contains "/libs/". Below is the result of doing: grep -L "/libs/" *.h* I'd appreciate maintainers adding a library identifying comment to these files. Otherwise I'll do it, which carries a slight risk that I'll get it wrong. Thanks, --Beman aligned_storage.hpp asio.hpp bimap.hpp blank.hpp blank_fwd.hpp circular_buffer.hpp circular_buffer_fwd.hpp date_time.hpp dynamic_property_map.hpp foreach.hpp function_equal.hpp function_output_iterator.hpp get_pointer.hpp implicit_cast.hpp is_placeholder.hpp iterator_adaptors.hpp lexical_cast.hpp mpi.hpp non_type.hpp none.hpp none_t.hpp optional.hpp parameter.hpp pfto.hpp pointer_cast.hpp program_options.hpp smart_cast.hpp spirit.hpp state_saver.hpp strong_typedef.hpp thread.hpp type.hpp variant.hpp wave.hpp

17 95

[serialization] serialization used in multiple DLL problems
by Felipe Magno de Almeida 17 Dec '07

17 Dec '07

Hi, I was able to make this work on serialization after some minor modifications: PS: pseudo-code DLL1: header.h BOOST_IS_ABSTRACT(base); class base { virtual ~base() {} // serialization }; source.cpp class derived : base { // serialization }; BOOST_CLASS_EXPORT(derived); void serialize(oarchive& ar) { boost::shared_ptr<derived> p(new derived); ar << p; } DLL2: std::stringstream stm; { oarchive oar(stm); serialize(oar); } { iarchive iar(stm); boost::shared_ptr<base> p; iar >> p; // deserializing a derived class defined in DLL1 in DLL2 } I had to compile serialization as a lib. And making tk_map and kt_map global objects (not sure if this is necessary though), removing the assertion about multiple inserts in static insert function of both [kt][tk]_map's and commenting line 315 of basic_oarchive: //if((cobject_info_set.size() > original_count)){ +{ I don't know the internals of serialization enough to say this is anything near correct. And I'll still need to test more, which I will do, since I need this functionality very much. Regards, -- Felipe Magno de Almeida

2 5

[mpi] SEEK_* defined, but shouldn't be
by Benjamin Collins 12 Dec '07

12 Dec '07

A common problem that occurs when building mpich2 is that SEEK_SET, SEEK_CUR, and SEEK_END are all defined. mpich2 (specifically mpicxx.h) requires them to be unset to avoid problems caused by both the MPI interface and stdio using them. Anyway, the suggested solution is to #undef each macro before #include <mpi.h>. I tried that in the mpi library, and it still failed to build. I also tried adding <define>MPICH_IGNORE_CXX_SEEK=1 to the mpi Jamfile. Alas, it didn't fix the problem either. The failure occurs when building release/threading-multi/collective.o How are users of mpich2+boost::mpi getting around this problem? -- Benjamin A. Collins <aggieben(a)gmail.com> http://bloggoergosum.us

3 4

[test] symbol not defined: execution_exception
by Dave Jenkins 11 Dec '07

11 Dec '07

When I include header <boost/test/unit_test.hpp> in my program, the linker complains that symbol "execution_exception" is undefined. I looked at the compiler output and found that class "execution_exception" is a parameter on "virtual void exception_caught(execution_exception const&)" in file test_observer.hpp. It is declared but never defined. I added "#include <boost/test/execution_monitor.hpp>" to file test_observer.hpp and the linker error went away. Is this the correct way to fix the problem? Thanks, Dave Jenkins

2 3