Guillaume Melquiond writes:
Hi,
Hi Guillaume,
If you take a look at the regression logs [1], you will see a few problems.
First, a lot of test types [2] are missing. The "bind" or "date-time" libraries don't have any value in the test type column. Other libraries like the "array" library do have values in this column, however.
This was fixed in the CVS about two weeks ago (see http://tinyurl.com/36v6h#dont_redirect, for instance); simple rebuilding "process_jam_log" executable will take care of it (I would have posted a notice about this fix if I didn't forget that unlike us not everybody rebuilds "process_jam_log" on every run).
Second, the name of some "nested" libraries are wrong. For example, the "algorithm/string" is shown as "algorithm", the "numeric/conversion" and "numeric/interval" libraries are both mixed under "numeric". But the "numeric/ublas" is correctly named.
This is still an open issue.
Third, expected failures are not correctly shown in the advanced regression logs [3] since some library names are wrong.
Consequently, this one is still open as well.
Fourth, some libraries are compiled in the wrong directories. Instead of being compiled in "bin/boost/libs/whatever/test/something.test", they are directly compiled in "bin/boost/status/something.test" [4]. If you run the regression tests, you will see what I mean.
I wouldn't say that they are "wrong", because that implies that this is a newly introduced behavior (read "bug"), and it's not the case. Basically, whether the test results go into "bin/boost/status" or "bin/boost/libs/" is determined by the age of the library: the new libraries that provide their own Jamfile for testing and use bjam's "subinclude" feature to bring it into "boost/status/jamfile" are put in "bin/boost/libs/<library-name>" location, and the libraries that are listing their test cases directly in the "status/jamfile" are put in "bin/boost/status". Granted, the latter scheme is deficient and makes it unnecessarily hard to post-process the test results. How about we simply factor out these "inline" test-suites into the corresponding Jamfiles, and be done with the issue?
It's quite possible that the problems for the first three points are related to the fourth.
The library name issue is not; it's simply a matter of tweaking "process_jam_log" to extract it correctly.
Indeed, it seems only libraries wrongly compiled in "bin/boost/status/something.test" get to be correctly identified. It's the case for the "array" and "numeric/ublas" libraries for example.
I don't understand how regression testing takes place into Boost. From my humble point of view, it is quite a mess.
The particular part that you've stumbled upon most definitely is, and it's about time to get it cleaned.
I didn't understand why some libraries were compiled in the wrong place; I suppose it has to do with bjam and their Jamfiles. I did however find a way to correct the first two bugs (and probably the third). The test_log.xml are indeed wrongly generated by process_jam_log.
So here is a patch to process_jam_log.cpp so that it correctly generates the test_log.xml files. Unfortunately I don't understand at all some parts of this program, so I only corrected what seemed obviously wrong to me.
Unfortunately, the patch is no good. In particular, it breaks the case when there are multiple identically named tests in different libraries. If you were trying to solve the composite library name issue ("algorithm" vs. "algorithm/string"), I suggest we first do the Jamfile refactoring I talked about above; it would allow us to simplify the code significantly, and then we can tackle this one.
But maybe the bugs were somewhere else and I only designed a lame workaround? Could the author of these lines shed some light on what their purpose was supposed to be?
The purpose is to correctly match the test results with a corresponding test case from the right library, in presence of the discussed variability in the output directory structure.
In the meantime, maybe the regression testers could clear their "bin" directory (so that all the test_log.xml are created from scratch) and apply the patch?
Due to the above, I don't think it's a good idea.
Since Boost 1.31 is supposed to ship in less than 20 days, it would be great if the regression logs were correctly generated.
But this one is definitely worth fighting for. I'm going to look into the library name issue tomorrow. -- Aleksey Gurtovoy MetaCommunications Engineering