Regression testing goes horribly wrong! (+ patch)
Hi, If you take a look at the regression logs [1], you will see a few problems. First, a lot of test types [2] are missing. The "bind" or "date-time" libraries don't have any value in the test type column. Other libraries like the "array" library do have values in this column, however. Second, the name of some "nested" libraries are wrong. For example, the "algorithm/string" is shown as "algorithm", the "numeric/conversion" and "numeric/interval" libraries are both mixed under "numeric". But the "numeric/ublas" is correctly named. Third, expected failures are not correctly shown in the advanced regression logs [3] since some library names are wrong. Fourth, some libraries are compiled in the wrong directories. Instead of being compiled in "bin/boost/libs/whatever/test/something.test", they are directly compiled in "bin/boost/status/something.test" [4]. If you run the regression tests, you will see what I mean. It's quite possible that the problems for the first three points are related to the fourth. Indeed, it seems only libraries wrongly compiled in "bin/boost/status/something.test" get to be correctly identified. It's the case for the "array" and "numeric/ublas" libraries for example. I don't understand how regression testing takes place into Boost. From my humble point of view, it is quite a mess. I didn't understand why some libraries were compiled in the wrong place; I suppose it has to do with bjam and their Jamfiles. I did however find a way to correct the first two bugs (and probably the third). The test_log.xml are indeed wrongly generated by process_jam_log. So here is a patch to process_jam_log.cpp so that it correctly generates the test_log.xml files. Unfortunately I don't understand at all some parts of this program, so I only corrected what seemed obviously wrong to me. But maybe the bugs were somewhere else and I only designed a lame workaround? Could the author of these lines shed some light on what their purpose was supposed to be? In the meantime, maybe the regression testers could clear their "bin" directory (so that all the test_log.xml are created from scratch) and apply the patch? Since Boost 1.31 is supposed to ship in less than 20 days, it would be great if the regression logs were correctly generated. Thanks. Best regards, Guillaume [1] For example, a small regression log: http://boost.sourceforge.net/regression-logs/cs-AIX.html All recent regression logs are affected. [2] "run", "compile", "compile_fail", etc [3] Some of the yellow cells should actually be gray in this log: http://boost.sourceforge.net/regression-logs/cs-win32_metacomm/developer/sum... [4] For example, look at the compile line used for: http://boost.sourceforge.net/regression-logs/cs-AIX-links.html#lambda-bll_an... Index: process_jam_log.cpp =================================================================== RCS file: /cvsroot/boost/boost/tools/regression/process_jam_log.cpp,v retrieving revision 1.25 diff -u -r1.25 process_jam_log.cpp --- process_jam_log.cpp 28 Jun 2004 03:04:21 -0000 1.25 +++ process_jam_log.cpp 9 Jul 2004 17:38:59 -0000 @@ -241,11 +241,11 @@ } test_info info; - test2info_map::iterator itr( test2info.find( library_name + "/" + test_name ) ); + test2info_map::iterator itr( test2info.find( test_name ) ); if ( itr != test2info.end() ) info = itr->second; - if ( library_name.empty() ) + if ( !info.file_path.empty() ) library_name = test_path_to_library_name( info.file_path ); if ( info.type.empty() ) @@ -511,8 +511,6 @@ line.find( "\"", pos+3 )-pos-3 ); convert_path_separators( info.file_path ); if ( info.file_path.find( "libs/libs/" ) == 0 ) info.file_path.erase( 0, 5 ); - if ( test_name.find( "/" ) == string::npos ) - test_name = "/" + test_name; test2info.insert( std::make_pair( test_name, info ) ); // std::cout << test_name << ", " << info.type << ", " << info.file_path << "\n"; }
Guillaume Melquiond <guillaume.melquiond@ens-lyon.fr> writes:
So here is a patch to process_jam_log.cpp so that it correctly generates the test_log.xml files. Unfortunately I don't understand at all some parts of this program, so I only corrected what seemed obviously wrong to me. But maybe the bugs were somewhere else and I only designed a lame workaround? Could the author of these lines shed some light on what their purpose was supposed to be?
I'm afraid Beman is unavailable to reply before the next release. We'll need to muddle on the best we can without him. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
Le ven 09/07/2004 à 21:00, David Abrahams a écrit :
Guillaume Melquiond <guillaume.melquiond@ens-lyon.fr> writes:
So here is a patch to process_jam_log.cpp so that it correctly generates the test_log.xml files. Unfortunately I don't understand at all some parts of this program, so I only corrected what seemed obviously wrong to me. But maybe the bugs were somewhere else and I only designed a lame workaround? Could the author of these lines shed some light on what their purpose was supposed to be?
I'm afraid Beman is unavailable to reply before the next release. We'll need to muddle on the best we can without him.
That's too bad. I guess I will wait till Tuesday so that people have a bit of time to review the proposed patch. If there is no (negative) reaction, I will then commit it and wait for the complaints. Regards, Guillaume
David Abrahams writes:
Guillaume Melquiond <guillaume.melquiond@ens-lyon.fr> writes:
So here is a patch to process_jam_log.cpp so that it correctly generates the test_log.xml files. Unfortunately I don't understand at all some parts of this program, so I only corrected what seemed obviously wrong to me. But maybe the bugs were somewhere else and I only designed a lame workaround? Could the author of these lines shed some light on what their purpose was supposed to be?
I'm afraid Beman is unavailable to reply before the next release. We'll need to muddle on the best we can without him.
Why "muddle"? It's totally under control. -- Aleksey Gurtovoy MetaCommunications Engineering
Aleksey Gurtovoy <agurtovoy@meta-comm.com> writes:
David Abrahams writes:
Guillaume Melquiond <guillaume.melquiond@ens-lyon.fr> writes:
So here is a patch to process_jam_log.cpp so that it correctly generates the test_log.xml files. Unfortunately I don't understand at all some parts of this program, so I only corrected what seemed obviously wrong to me. But maybe the bugs were somewhere else and I only designed a lame workaround? Could the author of these lines shed some light on what their purpose was supposed to be?
I'm afraid Beman is unavailable to reply before the next release. We'll need to muddle on the best we can without him.
Why "muddle"? It's totally under control.
Just a figure of speech. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
Guillaume Melquiond writes:
Hi,
Hi Guillaume,
If you take a look at the regression logs [1], you will see a few problems.
First, a lot of test types [2] are missing. The "bind" or "date-time" libraries don't have any value in the test type column. Other libraries like the "array" library do have values in this column, however.
This was fixed in the CVS about two weeks ago (see http://tinyurl.com/36v6h#dont_redirect, for instance); simple rebuilding "process_jam_log" executable will take care of it (I would have posted a notice about this fix if I didn't forget that unlike us not everybody rebuilds "process_jam_log" on every run).
Second, the name of some "nested" libraries are wrong. For example, the "algorithm/string" is shown as "algorithm", the "numeric/conversion" and "numeric/interval" libraries are both mixed under "numeric". But the "numeric/ublas" is correctly named.
This is still an open issue.
Third, expected failures are not correctly shown in the advanced regression logs [3] since some library names are wrong.
Consequently, this one is still open as well.
Fourth, some libraries are compiled in the wrong directories. Instead of being compiled in "bin/boost/libs/whatever/test/something.test", they are directly compiled in "bin/boost/status/something.test" [4]. If you run the regression tests, you will see what I mean.
I wouldn't say that they are "wrong", because that implies that this is a newly introduced behavior (read "bug"), and it's not the case. Basically, whether the test results go into "bin/boost/status" or "bin/boost/libs/" is determined by the age of the library: the new libraries that provide their own Jamfile for testing and use bjam's "subinclude" feature to bring it into "boost/status/jamfile" are put in "bin/boost/libs/<library-name>" location, and the libraries that are listing their test cases directly in the "status/jamfile" are put in "bin/boost/status". Granted, the latter scheme is deficient and makes it unnecessarily hard to post-process the test results. How about we simply factor out these "inline" test-suites into the corresponding Jamfiles, and be done with the issue?
It's quite possible that the problems for the first three points are related to the fourth.
The library name issue is not; it's simply a matter of tweaking "process_jam_log" to extract it correctly.
Indeed, it seems only libraries wrongly compiled in "bin/boost/status/something.test" get to be correctly identified. It's the case for the "array" and "numeric/ublas" libraries for example.
I don't understand how regression testing takes place into Boost. From my humble point of view, it is quite a mess.
The particular part that you've stumbled upon most definitely is, and it's about time to get it cleaned.
I didn't understand why some libraries were compiled in the wrong place; I suppose it has to do with bjam and their Jamfiles. I did however find a way to correct the first two bugs (and probably the third). The test_log.xml are indeed wrongly generated by process_jam_log.
So here is a patch to process_jam_log.cpp so that it correctly generates the test_log.xml files. Unfortunately I don't understand at all some parts of this program, so I only corrected what seemed obviously wrong to me.
Unfortunately, the patch is no good. In particular, it breaks the case when there are multiple identically named tests in different libraries. If you were trying to solve the composite library name issue ("algorithm" vs. "algorithm/string"), I suggest we first do the Jamfile refactoring I talked about above; it would allow us to simplify the code significantly, and then we can tackle this one.
But maybe the bugs were somewhere else and I only designed a lame workaround? Could the author of these lines shed some light on what their purpose was supposed to be?
The purpose is to correctly match the test results with a corresponding test case from the right library, in presence of the discussed variability in the output directory structure.
In the meantime, maybe the regression testers could clear their "bin" directory (so that all the test_log.xml are created from scratch) and apply the patch?
Due to the above, I don't think it's a good idea.
Since Boost 1.31 is supposed to ship in less than 20 days, it would be great if the regression logs were correctly generated.
But this one is definitely worth fighting for. I'm going to look into the library name issue tomorrow. -- Aleksey Gurtovoy MetaCommunications Engineering
Le sam 10/07/2004 à 10:01, Aleksey Gurtovoy a écrit :
Guillaume Melquiond writes:
Hi,
Hi Guillaume,
If you take a look at the regression logs [1], you will see a few problems.
First, a lot of test types [2] are missing. The "bind" or "date-time" libraries don't have any value in the test type column. Other libraries like the "array" library do have values in this column, however.
This was fixed in the CVS about two weeks ago (see http://tinyurl.com/36v6h#dont_redirect, for instance); simple rebuilding "process_jam_log" executable will take care of it (I would have posted a notice about this fix if I didn't forget that unlike us not everybody rebuilds "process_jam_log" on every run).
That's strange. I do use the CVS version and I did rebuild the process_jam_log. Moreover, anybody that uses the run_tests.sh script to run the regression testing should have an up-to-date version of process_jam_log ("step 2: rebuild the regression test helper programs if required"). I think your fix was sufficient was for you but not for everybody (don't ask me why :).
Second, the name of some "nested" libraries are wrong. For example, the "algorithm/string" is shown as "algorithm", the "numeric/conversion" and "numeric/interval" libraries are both mixed under "numeric". But the "numeric/ublas" is correctly named.
This is still an open issue.
It's because process_jam_log only try to look if a "sublibs" directory exists when it does not have a library name. It does not have a library name when the test lies in "bin/boost/status", it is the case for "ublas" for example. However, for "conversion", the test lies in "bin/boost/libs/numeric/conversion/test". So the program has a library name which is "numeric" and it does not bother testing if "sublibs" exists. It's what my patch does, I force process_jam_log to always scan for a "sublibs" directory: - if ( library_name.empty() ) + if ( !info.file_path.empty() ) library_name = test_path_to_library_name( info.file_path ); As you can see, process_jam_log was only testing for "sublibs" if library_name is empty. That was obviously wrong.
Third, expected failures are not correctly shown in the advanced regression logs [3] since some library names are wrong.
Consequently, this one is still open as well.
Fourth, some libraries are compiled in the wrong directories. Instead of being compiled in "bin/boost/libs/whatever/test/something.test", they are directly compiled in "bin/boost/status/something.test" [4]. If you run the regression tests, you will see what I mean.
I wouldn't say that they are "wrong", because that implies that this is a newly introduced behavior (read "bug"), and it's not the case.
Basically, whether the test results go into "bin/boost/status" or "bin/boost/libs/" is determined by the age of the library: the new libraries that provide their own Jamfile for testing and use bjam's "subinclude" feature to bring it into "boost/status/jamfile" are put in "bin/boost/libs/<library-name>" location, and the libraries that are listing their test cases directly in the "status/jamfile" are put in "bin/boost/status".
Granted, the latter scheme is deficient and makes it unnecessarily hard to post-process the test results. How about we simply factor out these "inline" test-suites into the corresponding Jamfiles, and be done with the issue?
... Probably... If you say so... Sorry, I'm quite fond of the simplicity of Makefiles, and whenever it comes to Jamfiles I'm a bit lost :-).
It's quite possible that the problems for the first three points are related to the fourth.
The library name issue is not; it's simply a matter of tweaking "process_jam_log" to extract it correctly.
Indeed, it seems only libraries wrongly compiled in "bin/boost/status/something.test" get to be correctly identified. It's the case for the "array" and "numeric/ublas" libraries for example.
I don't understand how regression testing takes place into Boost. From my humble point of view, it is quite a mess.
The particular part that you've stumbled upon most definitely is, and it's about time to get it cleaned.
I didn't understand why some libraries were compiled in the wrong place; I suppose it has to do with bjam and their Jamfiles. I did however find a way to correct the first two bugs (and probably the third). The test_log.xml are indeed wrongly generated by process_jam_log.
So here is a patch to process_jam_log.cpp so that it correctly generates the test_log.xml files. Unfortunately I don't understand at all some parts of this program, so I only corrected what seemed obviously wrong to me.
Unfortunately, the patch is no good. In particular, it breaks the case when there are multiple identically named tests in different libraries. If you were trying to solve the composite library name issue ("algorithm" vs. "algorithm/string"), I suggest we first do the Jamfile refactoring I talked about above; it would allow us to simplify the code significantly, and then we can tackle this one.
I don't think my patch is that bad. I agree with your point about multiple identically named tests. I was under the same impression. But process_jam_log can't actually deal with them, because it does around line 520: test2info.insert( std::make_pair( test_name, info ) ); test2info is a map and test_name is the short name of test. There can only be one test with a given name in Boost. So maybe we should replace it with something like: test2info.insert( std::make_pair( library_name + '/' + test_name, info ) ); What do you think about it?
But maybe the bugs were somewhere else and I only designed a lame workaround? Could the author of these lines shed some light on what their purpose was supposed to be?
The purpose is to correctly match the test results with a corresponding test case from the right library, in presence of the discussed variability in the output directory structure.
But it still works after my patch is applied, so...
In the meantime, maybe the regression testers could clear their "bin" directory (so that all the test_log.xml are created from scratch) and apply the patch?
Due to the above, I don't think it's a good idea.
Since Boost 1.31 is supposed to ship in less than 20 days, it would be great if the regression logs were correctly generated.
But this one is definitely worth fighting for. I'm going to look into the library name issue tomorrow.
I will write another patch so that test names are adjusted with their library name (the bad one should be enough at first, it would remove the first line change of my initial patch, I think) in test2info in process_jam_log. Regards, Guillaume
Le sam 10/07/2004 à 10:50, Guillaume Melquiond a écrit :
I will write another patch so that test names are adjusted with their library name (the bad one should be enough at first, it would remove the first line change of my initial patch, I think) in test2info in process_jam_log.
So here is the new patch. I have created a new function get_bad_library_name. Since it's the "bad" library name, there is still a problem with multiple test with identical name if they lie in the same sub-directory. But in the meantime, it will do... Regards, Guillaume
Guillaume Melquiond writes:
Le sam 10/07/2004 à 10:01, Aleksey Gurtovoy a écrit :
Guillaume Melquiond writes:
Hi,
Hi Guillaume,
If you take a look at the regression logs [1], you will see a few problems.
First, a lot of test types [2] are missing. The "bind" or "date-time" libraries don't have any value in the test type column. Other libraries like the "array" library do have values in this column, however.
This was fixed in the CVS about two weeks ago (see http://tinyurl.com/36v6h#dont_redirect, for instance); simple rebuilding "process_jam_log" executable will take care of it (I would have posted a notice about this fix if I didn't forget that unlike us not everybody rebuilds "process_jam_log" on every run).
That's strange. I do use the CVS version and I did rebuild the process_jam_log.
Did you do a clean run?
Moreover, anybody that uses the run_tests.sh script to run the regression testing should have an up-to-date version of process_jam_log ("step 2: rebuild the regression test helper programs if required"). I think your fix was sufficient was for you but not for everybody (don't ask me why :).
Given that we remove all binaries and sources and rebuild everything from scratch on every run, I'm inclined to think that our fix works and that the problems you see are caused by a stale environment. If you still can reproduce the problem on a clean state, you can always step through the process_jam_log code in debugger and see how these empty test types appear there.
Second, the name of some "nested" libraries are wrong. For example, the "algorithm/string" is shown as "algorithm", the "numeric/conversion" and "numeric/interval" libraries are both mixed under "numeric". But the "numeric/ublas" is correctly named.
This is still an open issue.
It's because process_jam_log only try to look if a "sublibs" directory exists when it does not have a library name. It does not have a library name when the test lies in "bin/boost/status", it is the case for "ublas" for example. However, for "conversion", the test lies in "bin/boost/libs/numeric/conversion/test". So the program has a library name which is "numeric" and it does not bother testing if "sublibs" exists. It's what my patch does, I force process_jam_log to always scan for a "sublibs" directory:
- if ( library_name.empty() ) + if ( !info.file_path.empty() ) library_name = test_path_to_library_name( info.file_path );
As you can see, process_jam_log was only testing for "sublibs" if library_name is empty. That was obviously wrong.
OK, indeed, this is a right change. In fact, it fixes the issue entirely. Tested and committed, will be reflected in the next run. [..]
Unfortunately, the patch is no good. In particular, it breaks the case when there are multiple identically named tests in different libraries. If you were trying to solve the composite library name issue ("algorithm" vs. "algorithm/string"), I suggest we first do the Jamfile refactoring I talked about above; it would allow us to simplify the code significantly, and then we can tackle this one.
I don't think my patch is that bad. I agree with your point about multiple identically named tests. I was under the same impression.
But process_jam_log can't actually deal with them,
It definitely can, please see below.
because it does around line 520:
test2info.insert( std::make_pair( test_name, info ) );
test2info is a map and test_name is the short name of test.
Nope, it's not. It's the name extracted from one of the top lines of the bjam log, and the lines corresponding to the "new" test cases do contain the library name: boost-test(RUN) "utility/ref_test" : "libs\utility\ref_test.cpp" boost-test(RUN) "utility/addressof_test" : "libs\utility\addressof_test.cpp" ...
There can only be one test with a given name in Boost.
Not true, see "preprocessor/test/arithmetic.cpp" and "mpl/test/arithmetic.cpp", for instance.
So maybe we should replace it with something like:
test2info.insert( std::make_pair( library_name + '/' + test_name, info ) );
What do you think about it?
That's basically what is already there. -- Aleksey Gurtovoy MetaCommunications Engineering
Le dim 11/07/2004 à 04:46, Aleksey Gurtovoy a écrit : [...]
This was fixed in the CVS about two weeks ago (see http://tinyurl.com/36v6h#dont_redirect, for instance); simple rebuilding "process_jam_log" executable will take care of it (I would have posted a notice about this fix if I didn't forget that unlike us not everybody rebuilds "process_jam_log" on every run).
That's strange. I do use the CVS version and I did rebuild the process_jam_log.
Did you do a clean run?
Yes. It's what my sentence was supposed to meant. [...]
Given that we remove all binaries and sources and rebuild everything from scratch on every run, I'm inclined to think that our fix works and that the problems you see are caused by a stale environment.
Seems like your fix never get commited if it does fix for you.
If you still can reproduce the problem on a clean state, you can always step through the process_jam_log code in debugger and see how these empty test types appear there.
I did. The reason is quite simple. And it's what I have been explaining in my previous mails: test_info is not created with the same name that the name used to access it later. The test type is set at creation time, and the test_log.xml is used at a later time. So they are no more related since the test type cannot be retrieved during the second access. [...]
But process_jam_log can't actually deal with them,
It definitely can, please see below.
Please see below why it definitely cannot.
because it does around line 520:
test2info.insert( std::make_pair( test_name, info ) );
test2info is a map and test_name is the short name of test.
Nope, it's not. It's the name extracted from one of the top lines of the bjam log, and the lines corresponding to the "new" test cases do contain the library name:
boost-test(RUN) "utility/ref_test" : "libs\utility\ref_test.cpp" boost-test(RUN) "utility/addressof_test" : "libs\utility\addressof_test.cpp" ...
boost-test(RUN) "ref_test" : "libs/utility/ref_test.cpp" boost-test(RUN) "addressof_test" : "libs/utility/addressof_test.cpp" Maybe you did patch some Jamfiles or bjam? It would explain why your tests work. And why the tests of all other regression testers fail in a similar way. So all I said is still valid. [...]
So maybe we should replace it with something like:
test2info.insert( std::make_pair( library_name + '/' + test_name, info ) );
What do you think about it?
That's basically what is already there.
No it isn't unfortunately, since the library_name is not in the regress.log file when using the CVS version. My second patch is *really* required so that process_jam_log correctly works. So, unless you find and commit your patch that puts library_name in regress.log, I will commit my second patch. Regards, Guillaume
Guillaume Melquiond writes: Le dim 11/07/2004 à 04:46, Aleksey Gurtovoy a écrit :
[...]
This was fixed in the CVS about two weeks ago (see http://tinyurl.com/36v6h#dont_redirect, for instance); simple rebuilding "process_jam_log" executable will take care of it (I would have posted a notice about this fix if I didn't forget that unlike us not everybody rebuilds "process_jam_log" on every run).
That's strange. I do use the CVS version and I did rebuild the process_jam_log.
Did you do a clean run?
Yes. It's what my sentence was supposed to meant.
OK.
Given that we remove all binaries and sources and rebuild everything from scratch on every run, I'm inclined to think that our fix works and that the problems you see are caused by a stale environment.
Seems like your fix never get commited if it does fix for you.
It's not the same fix, but you are actually right, please see below.
If you still can reproduce the problem on a clean state, you can always step through the process_jam_log code in debugger and see how these empty test types appear there.
I did. The reason is quite simple. And it's what I have been explaining in my previous mails: test_info is not created with the same name that the name used to access it later. The test type is set at creation time, and the test_log.xml is used at a later time. So they are no more related since the test type cannot be retrieved during the second access.
OK, understood. It all works for us because we apply an [uncommitted] patch to "testing.jam" that modifies bjam output to prefix a test case name with a library name (when appropriate).
[...]
But process_jam_log can't actually deal with them,
It definitely can, please see below.
Please see below why it definitely cannot.
because it does around line 520:
test2info.insert( std::make_pair( test_name, info ) );
test2info is a map and test_name is the short name of test.
Nope, it's not. It's the name extracted from one of the top lines of the bjam log, and the lines corresponding to the "new" test cases do contain the library name:
boost-test(RUN) "utility/ref_test" : "libs\utility\ref_test.cpp" boost-test(RUN) "utility/addressof_test" : "libs\utility\addressof_test.cpp" ...
boost-test(RUN) "ref_test" : "libs/utility/ref_test.cpp" boost-test(RUN) "addressof_test" : "libs/utility/addressof_test.cpp"
Ouch! We've fixed this one so long ago that I totally forgot that it's still uncommitted. Sorry for the confusion it caused! It's in the CVS now. Rebuilding everything should bring us to the desired state, at last.
Maybe you did patch some Jamfiles or bjam?
"boost\tools\build\v1\testing.jam"
It would explain why your tests work. And why the tests of all other regression testers fail in a similar way.
It definitely does.
[...]
So maybe we should replace it with something like:
test2info.insert( std::make_pair( library_name + '/' + test_name, info ) );
What do you think about it?
That's basically what is already there.
No it isn't unfortunately, since the library_name is not in the regress.log file when using the CVS version. My second patch is *really* required so that process_jam_log correctly works.
Part of your patch that I already committed + our fix to "testing.jam", yes. Your full second patch, no. The reason we append the library name in "testing.jam" instead of doing it in "process_jam_log" is that at the point of parsing the old bjam log's boost-test(RUN) "test1" : "libs\numeric\ublas\test1\test1.cpp" "libs\numeric\ublas\test1\test11.cpp" line you have no clue whether the test's *output directory* will be bin\boost\libs\numeric\ublas\test1.test or bin\boost\status\test1.test Therefore you cannot, in all cases, reliably construct the corresponding 'test2info' key so that it's retrievable later. "testing.jam" has this information.
So, unless you find and commit your patch that puts library_name in regress.log, I will commit my second patch.
Please understand that the only reason for my unwillingness to commit it at once is that I didn't feel that we got to the root of the issue (then). Thank you for your patience, -- Aleksey Gurtovoy MetaCommunications Engineering
Le lun 12/07/2004 à 04:22, Aleksey Gurtovoy a écrit : [...]
Maybe you did patch some Jamfiles or bjam?
"boost\tools\build\v1\testing.jam"
Great! I don't know if it is actually necessary (or if it is even possible), but maybe a similar patch should be applied to bjam v2? I have another concern. Maybe I'm overly pessimistic, but I think regression testers should restart their testing from scratch (rm -r boost/bin, or whatever it is called) now that the format has changed, shouldn't they? [...] Regards, Guillaume
Guillaume Melquiond wrote:
I don't know if it is actually necessary (or if it is even possible), but maybe a similar patch should be applied to bjam v2?
I have another concern. Maybe I'm overly pessimistic, but I think regression testers should restart their testing from scratch (rm -r boost/bin, or whatever it is called) now that the format has changed, shouldn't they?
I always do that (i.e. for the platforms IBM/Aix, SGI/Irix and HPUX itanium). toon
Aleksey Gurtovoy wrote:
Part of your patch that I already committed + our fix to "testing.jam", yes. Your full second patch, no. The reason we append the library name in "testing.jam" instead of doing it in "process_jam_log" is that at the point of parsing the old bjam log's
boost-test(RUN) "test1" : "libs\numeric\ublas\test1\test1.cpp" "libs\numeric\ublas\test1\test11.cpp"
line you have no clue whether the test's *output directory* will be
bin\boost\libs\numeric\ublas\test1.test
or
bin\boost\status\test1.test
There's one gotcha. uBLAS actually have duplicate definitions for tests. There's libs/numeric/ublas/test1/Jamfile, which defines a single target of type "exe", and there's status/Jamfile (which will be moved to libs/numeric/ublas, which defines "run" target with exactly the same sources). This duplication is really bad, so it would be desirable to just "refer" to ublas/test1/Jamfile from ublas/Jamfile. This is something that Boost.Build V2 Jamfile written by Michael does. But then the target path will be not bin\boost\libs\numeric\ublas\test1.test but bin\boost\libs\numeric\ublas\test1\test1.test I wonder how regression tools can determine the library name in that case? - Volodya
Vladimir Prus writes:
Aleksey Gurtovoy wrote:
Part of your patch that I already committed + our fix to "testing.jam", yes. Your full second patch, no. The reason we append the library name in "testing.jam" instead of doing it in "process_jam_log" is that at the point of parsing the old bjam log's
boost-test(RUN) "test1" : "libs\numeric\ublas\test1\test1.cpp" "libs\numeric\ublas\test1\test11.cpp"
line you have no clue whether the test's *output directory* will be
bin\boost\libs\numeric\ublas\test1.test
or
bin\boost\status\test1.test
There's one gotcha. uBLAS actually have duplicate definitions for tests. There's libs/numeric/ublas/test1/Jamfile, which defines a single target of type "exe", and there's status/Jamfile (which will be moved to libs/numeric/ublas, which defines "run" target with exactly the same sources).
This duplication is really bad, so it would be desirable to just "refer" to ublas/test1/Jamfile from ublas/Jamfile.
Right. I think it can wait until after the release, though.
This is something that Boost.Build V2 Jamfile written by Michael does. But then the target path will be not
bin\boost\libs\numeric\ublas\test1.test
but
bin\boost\libs\numeric\ublas\test1\test1.test
I wonder how regression tools can determine the library name in that case?
Correctly :). It's still in the path. They already do so for other libraries. -- Aleksey Gurtovoy MetaCommunications Engineering
Aleksey Gurtovoy wrote:
There's one gotcha. uBLAS actually have duplicate definitions for tests. There's libs/numeric/ublas/test1/Jamfile, which defines a single target of type "exe", and there's status/Jamfile (which will be moved to libs/numeric/ublas, which defines "run" target with exactly the same sources).
This duplication is really bad, so it would be desirable to just "refer" to ublas/test1/Jamfile from ublas/Jamfile.
Right. I think it can wait until after the release, though.
Yea, because I don't think V1 can refer to test targets in other Jamfile, and V2 can be only used after release.
This is something that Boost.Build V2 Jamfile written by Michael does. But then the target path will be not
bin\boost\libs\numeric\ublas\test1.test
but
bin\boost\libs\numeric\ublas\test1\test1.test
I wonder how regression tools can determine the library name in that case?
Correctly :). It's still in the path. They already do so for other libraries.
You mean this: local rule get-library-name ( path ) { local match1 = [ MATCH [/\\]libs[/\\](.*)[/\\]test : $(path) ] ; I'm not sure: ghost@zigzag:/tmp$ bjam -f- ECHO [ MATCH [/\\]libs[/\\](.*)[/\\]test : bin\\boost\\libs\\numeric\\ublas\\test1\\test1.test ] ; numeric\ublas\test1 It's numeric\ublas\test1, not numeric\ublas. - Volodya
Vladimir Prus writes:
This is something that Boost.Build V2 Jamfile written by Michael does. But then the target path will be not
bin\boost\libs\numeric\ublas\test1.test
but
bin\boost\libs\numeric\ublas\test1\test1.test
I wonder how regression tools can determine the library name in that case?
Correctly :). It's still in the path. They already do so for other libraries.
You mean this:
local rule get-library-name ( path ) { local match1 = [ MATCH [/\\]libs[/\\](.*)[/\\]test : $(path) ] ;
I'm not sure:
ghost@zigzag:/tmp$ bjam -f- ECHO [ MATCH [/\\]libs[/\\](.*)[/\\]test : bin\\boost\\libs\\numeric\\ublas\\test1\\test1.test ] ; numeric\ublas\test1
It's numeric\ublas\test1, not numeric\ublas.
'get-library-name' will be passed "bin\\boost\\libs\\numeric\\ublas\\test1", though, not "bin\\boost\\libs\\numeric\\ublas\\test1\\test1.test". -- Aleksey Gurtovoy MetaCommunications Engineering
Aleksey Gurtovoy wrote:
local rule get-library-name ( path ) { local match1 = [ MATCH [/\\]libs[/\\](.*)[/\\]test : $(path) ] ;
I'm not sure:
ghost@zigzag:/tmp$ bjam -f- ECHO [ MATCH [/\\]libs[/\\](.*)[/\\]test : bin\\boost\\libs\\numeric\\ublas\\test1\\test1.test ] ; numeric\ublas\test1
It's numeric\ublas\test1, not numeric\ublas.
'get-library-name' will be passed "bin\\boost\\libs\\numeric\\ublas\\test1", though, not "bin\\boost\\libs\\numeric\\ublas\\test1\\test1.test".
Well, first of all, when I create a separate test Jamfile for uBLAS and subinclude it, I get: Cannot extract library name from path ../libs/numeric/ublas This is because when it's in status, 'get-library-name' just returns empty string. When it's not in status, the 'get-library-name' rule is called with ../libs/numeric/ublas and the first regex does not match because there's no "test" in the name, and second regex does not match because there is slash between "numeric" and "ublas". I've applied the attached patch -- since it strictly adds the set of arguments to 'get-library-name' on which that function does not fail, I think it should not cause any problems. Another question whether all libraries should have their test Jamfiles under "test" directory -- which is not what uBLAS does. And now back to original question. As I understand, the 'get-library-name' rules gets the location of *Jamfile*. The primary heuristic is that location is "libs/ <library-name> / test / whatever". Why can't this logic be applied to the name of source file which bjam prints -- and be implemented in process_jam_log. I'm asking because it seems to be that it's better to have smart logic in process_jam_log than in jam. - Volodya
participants (5)
-
Aleksey Gurtovoy -
David Abrahams -
Guillaume Melquiond -
Toon Knapen -
Vladimir Prus