March 2005 - Boost - lists.preview.boost.org

[testing] Possible changes to testing..
by Rene Rivera 09 Mar '05

09 Mar '05

All, I've been reading all the testing related post and not having time to respond to them. So I decided to talk about what might be done to improve things in one big posting. For along time now one of my objectives has been to use BuildBot (http://buildbot.sf.net/) to improve the management of running the regression tests. For those who don't feel like reading about that software here's a quick summary: It is a client-server based configuration where the test server controls the test clients telling them what to do for test. In this arrangement the client do not directly control what they are testing, nor how. All the clients provide is an execution environment for the testing. The server is configured to control the various clients as to what they test, including how to get the source, and when they test. The testing is change driven instead of time driven as we currently have. This means that each time a change occurs on the source (CVS in our case) the server decides to tell the clients to run tests. Because of this direct communication to the clients the server is able to give direct feedback as to what the testing is doing. This includes a live display of test progress, down to a dynamic view of the test log. Some of the issues and how this step might help. And other changes I've thought about might help.. *Reproducibility* BuildBot controls what the checkouts are and what triggered the build. You can see this by selecting one of the yellow "Build #" links on the BuildBot display. If there's ever a question about what code produced an error one can get the specific version of the tree to attempt reproduction. The history of what tests have run is kept. Which means that we can finally answer the dreaded "When did that start breaking?" question. *Scalability* One big change I'd like to see is the breakup of running tests to make it possible for testers to run tests on subsets of libraries, or even individual libraries. For example there might be some testers which have a special interest in a particular library, Boost.Python comes to mind. It would be ideal to make it possible for them to only run those tests, and to go through the extra steps of doing the Python setup. Also for some popular platforms, it becomes possible to get much faster response rates from testing if, for example, we partition the library testing space throughout those platforms and they would test the libraries in parallel. For this to happen some significant organizational changes need to happen to the tests. As it currently stands such a division is not possible because of the way tests are defined and organized. We have a single test point, status/Jamfile, which a) points to the rest of the test points, libs/*/test/Jamfile, and b) defines it's own tests. The problem is that there is a conflict, for some libraries, between the tests defined in status/Jamfile, and the tests defined in libs/*/test/Jamfile. For example Boost.Config has a reduced set of test in status/Jamfile. This situation is likely a reaction to reduce test times to something manageable. And I understand that library authors would need to have a place to run their own set of comprehensive tests. My proposal is to create a set of canonical tests that comprise the regression testing suite independent of the library author's tests. This set of tests would be structured so that it's possible to run each libraries tests independent of others. It would be adding this type of structure: boost-root/tests/<library>/Jamfile boost-root/tests/<library>/<sub-library>/Jamfile boost-root/tests/<library>/<some-other-grouping>/Jamfile So for example a tester could say she only wants to test python/*, or numeric/ublas/*, etc. *Fragility* Simply put because BuildBot exposes the complete testing procedure it becomes much easier to see where problems occur in the testing. Also, because there is a central server managing the procedures it's more likely that problems can be solved at that one location instead of opaquely trying to fix the clients. With the, hopefully, additional distributed testers it becomes harder for testing to break down completely. Another aspect we need to address is fragility of the server. Ideally we would have multiple BuildBot servers to add redundancy. To make such a multi-server, multi-client setup resource efficient we would need to manage the distribution of testing between them. *Resources* The only predictable way to address the resource usage, is to distribute the testing so we can create more capacity. Breaking up the tests is the only way I can see to move there. It was already suggested that only slicing the testing to single toolsets would help. But that doesn't really address the problems. For example it would still be prohibitive for me to run tests on my G3 Mac for CW-8.3 because it's a slow machine and it would take days to run just one cycle of tests making the results useless. But it would be possible for me to run a minimal set of tests, for example Boost.Config and other basic tests. Restructuring also brings up the possibility of moderating the set of tests that make it into the regression suite. Right now we are in a position of asking library authors to define what the tests for the library. We can't moderate what gets into testing, or what needs go out. We need some procedure for reviewing tests, and some form of approval to get tests into the regression system. I'm not saying that authors would not be able to do additional testing, just that we need to define what the standard set of tests are so that we can concentrate our testing efforts. It would still be possible to set up additional resources to run "experimental" tests. *Response* The gain from segmentation and distribution of testing is hopefully obvious ;-) But another advantage of using BuildBot is that we are not tied to waiting for the XSLT processing to see results. Sure the results are not going to be as incredibly well organized as the Meta-Comm results but they are immediately available. So if there is a significant delay in the processing, because of load or breakage, we can still continue working. *Releases* Managing the testing for a release was brought up a many times. And it's clear that requiring testers to do manual changes is just not working. For BuildBot having the control of what is tested on the server means that at any point one, or some small number, person(s) can make the switch to have testing resources devoted to release testing. One possibility that the finner grain testing allows for is to limit the testing for a release to the required set of toolsets and platforms. OK, that's enough rambling... I know I haven't mentioned many other items raised, particularly about bjam and Boost.Build. I'll leave that for another post. I just need to get back to setting up the BuildBot server now ;-) -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com - 102708583/icq

2 1

RE: [boost] thread safety in boost date time
by Yitzhak Sapir 09 Mar '05

09 Mar '05

> From: Jeff Garland > > On Tue, 8 Mar 2005 11:34:15 +0200, Yitzhak Sapir wrote > > Boost date_time apparently has a thread safety problem in linux in > > that it calls localtime() in boost/date_time/time_clock.hpp and > > related functions rather than localtime_r() and related functions. > > This is even though BOOST_HAS_PTHREADS is defined by configure in > > user.hpp. > > That's true -- date_time doesn't do anything to ensure thread > safety. I'm a little reluctant to protect these functions since the > user can determine better if they need to take the associated > performance hit. Of course this needs to be documented, which > it isn't now. Thoughts? Ok, having investigated the problem a little more, I get the following.. The problem comes down to the fact that localtime/gmtime call tzset(), which in turn calls getenv("TZ"). This call isn't done in the _r versions which use pretty much the same code. getenv() appears to be not thread safe, but I'm not sure why. This does indicate, however, that probably the _r versions are a little quicker (because they don't call getenv()). Running localtime (no _r) in loops in four threads on a multi cpu machine resulted in no segmentation fault. Maybe something else is interfering with getenv() in unforseeable ways? boost is not the only one making these calls. log4cxx is another third party we use and it also makes these calls. It also calls wcsftime, a version of strftime, which appears to always call tzset()/getenv(). So, wcsftime/strftime appear to be not threadsafe either. For the record, we use libgcc 3.3.3 and glibc 3.2.3. The libgcc is a by product of compiling an equivalent gcc on all machines for distcc purposes, while the glibc is the version that was installed with the operating system. I am wondering if this version mismatch could be the cause of the problems, but it seems the getenv.c hasn't changed in glibc for 2 years. A link to an example posting about a problem that seems very similar: http://lists.gnu.org/archive/html/bug-glibc/2002-11/msg00092.html If it is to be documented and not fixed, it should probably be documented in a central place for all boost. (This is a good idea regardless). I think it should be fixed, given that the main difference is using a static buffer vs a stack allocated buffer and that the current version does more work (damage?) than the reentrant version. Yitzhak Sapir

2 1

copy_if, composition, and proper predicate
by Michael Goldshteyn 08 Mar '05

08 Mar '05

Given: // Definitions class Element { public: const std::string &GetTag() const; }; typedef std::vector<Element> ElementContainer; // Now here is what is required ElementContainer elements; ... ElementContainer matchingElements; std::string matchingTag("abc"); copy_if(elements.begin(),elements.end(),back_inserter(matchingElements)),pred); -- What is pred? Assuming that iElements represents the iterator inside copy_if, pred should be something like: iElements->GetTag()==matchingTag I could go about this by creating a new function object class/struct, which takes matchingTag as its sole constructor argument and does the comparison inside operator(). But, I want to use in place composition using binders, etc... to create the pred, so as not to have to create a function object class/struct. The question is, "What piece of code should pred be?" // Almost does the right thing, but it's only *iElements==matchingTag, I need iElements->GetTag()==matchingTag bind_2nd(equal_to,matchingTag) This seems like a very common operation. How can I compose the predicate, in place to do this? If there are multiple solutions, I would like to hear them as well! Thanks, Michael

2 1

Archive stuck
by Gennadiy Rozental 08 Mar '05

08 Mar '05

Sorry I know I was given direct address. But I couldn't locate it for some reason. Gennadiy

1 1

Re: [boost] Re: What I Hate About How We Abstract System Dependence
by brangdon＠cix.compulink.co.uk 08 Mar '05

08 Mar '05

In-Reply-To: <422D79A6.4050502(a)aaronwl.com> aaronrabiddog51(a)aaronwl.com (Aaron W. LaFramboise) wrote (abridged): > > I have always thought that concrete implementations of portable > > libraries should supply users a way to get at the OS-specific data so > > they can use platform-specific facilities on them if neccessary. > [...] > It seems to me that, by doing this, we are throwing away so many > correctness guarantees. To me, this seems sort of like taking > shared_ptr, and making it implicitly convertable to a pointer. Surely it's more like taking shared_ptr and giving it an explicit get() function? You can break invariants by doing stuff like: delete p.get(); but people are unlikely to do that by accident. Access to the underlying raw pointer is essential for inter-operating with software which isn't smart-pointer aware. So it is with OS handles. -- Dave Harris, Nottingham, UK

1 0

Re: [boost] Re: What I Hate About How We Abstract System Dependence
by brangdon＠cix.compulink.co.uk 08 Mar '05

08 Mar '05

In-Reply-To: <loom.20050308T013051-569(a)post.gmane.org> belvis(a)pacbell.net (Bob Bell) wrote (abridged): > Another possibility is to provide another level of indirection. For > example, boost::thread can have an accessor which returns a > boost::platform_thread, which has the OS thread ID, plus member > functions which mimic OS operations not reflected in the portable > (boost::thread) API. The advantage of this approach is that > boost::platform_thread member functions may be able to preserve > invariants that otherwise might be broken if the user was allowed > to manipulate the underlying OS thread ID itself, and if the coverage > of the boost::platform_thread member functions is large enough, it > might not even be necessary to expose the OS thread ID at all. I don't think the coverage could never be large enough. We should always allow for 3rd parties which are not boost-aware, and which need to be passed low-level OS handles. -- Dave Harris, Nottingham, UK

1 0

[iostreams] names for non-blocking character properties
by Jonathan Turkanis 08 Mar '05

08 Mar '05

Hi All, Everybody likes naming questions, right? The thread "[iostreams] Major interface changes planned" discussed having the get() function return a type (basic_character, character or wcharacter) which can hold either a character, an EOF indication or an EAGAIN indication. My question is what I should call the functions which query the state of a basic_character. My first idea was to use bool good(character c) - returns true if c represents a character rather than EOF or EAGAIN bool eof(chacrater c) - returns true if c represent EOF bool fail(character c) - returns true if c represents EAGAIN. These functions have the same names as member functions of basic_ios. However, since fail() above doesn't mean exactly the same thing as basic_ios::fail(), I now think this is a bad idea. Here are the names I've thought of: good: good, is_good eof: eof, is_eof fail: fail, eagain, is_eagain, would_block, should_retry I also need names for the two functions called by implementations of get() to return characters representing EOF and EAGAIN. E.g., template<typename Source> character get(Source& src) { ... if ( ... ) return eof(); else return eagain(); } (They will actually return instances of types convertible to basic_character.) What do you think? Jonathan

3 7

[shmem] New Shmem snapshot ready in Boost Files section
by Ion Gaztañaga 08 Mar '05

08 Mar '05

Hello to all, I've just uploaded a new version of Shmem library (removed Boost from name because of the new naming policy) to Boost Files (http://groups.yahoo.com/group/boost/files/) which includes built html help files. Snapshot file: Shmem.2005-03-06.tar.bz2. Changes and features from last version: -> Name changes. -> Improved compatibility with unordered_map/set family. Soon we will be able to place unordered containers in shared memory. Thanks, Daniel. -> Templatized index types for named shared memory object allocator -> New private segregated storage STL allocator for shared memory. -> Bug fixes -> Improved documentation and code comments. For the next version I plan to improve code comments, documentation and tests. Please, if there is any comment or suggestion, let me know. Regards, /Ion

2 1

Is (void) still means ()? Is that syntax deprecated?
by Pavel Chikulaev 08 Mar '05

08 Mar '05

ISO/IEC 14882:1998(E) 8.3.5 (2) says: The parameter list (void) is equivalent to the empty parameter list. Is there any paper that shows that syntax (void) deprecated? I am asking about it because I'm trying to remove (void) syntax in Visual Studio's 2005 IDE Class View and Add Member Wizard. Is there any ideas how to remove it? P.S. As for me, It is really annoying.

1 0

[Boost.Regex] Not compiling without exceptions.
by larsbj＠gullik.net 08 Mar '05

08 Mar '05

In cpp_regex_traits.cpp line 76 there is a re-throw. This does not work when compiling without exceptions. I see that gcc have a #ifndef __EXCEPTIONS # define __throw_exception_again #else # define __throw_exception_again throw #endif for this case. Something similar should work for boost as well. To allow compiling I did this: --- ../../boost/boost/throw_exception.hpp 2004-08-05 17:24:43.000000000 +0200 +++ boost/throw_exception.hpp 2005-03-08 03:04:27.066704153 +0100 @@ -32,6 +32,8 @@ void throw_exception(std::exception const & e); // user defined +#define BOOST_THROW_EXCEPTION_AGAIN + #else template<class E> inline void throw_exception(E const & e) @@ -39,6 +41,8 @@ throw e; } +#define BOOST_THROW_EXCEPTION_AGAIN throw + #endif } // namespace boost -- Lgb

3 3