Johan Nilsson wrote:
Sure, there are many potential problems with such an approach:
- Some library tests are likely to also excercise code at the implementation/detail level. - Already limited testing resources (on the other hand, interface changes are likely to be detected by running such tests for a single, reasonably compliant, compiler). - ...
Anyway, this is mostly me thinking out loud; perhaps someone smarter than me could expand on the feasibility of such an idea.
I hope somebody not claiming to be smarter than you is also allowed to comment ;-) An approach similar to this is used by the GDB project -- there are several versions of the interface from GDB to GUI frontends, and there are presently basically two sets of tests -- one is fixed and tests specific version of the interface. The second set tests all the current behaviour, when it's considered good to go, tests for the current behaviour will be copied into a new set, and won't be touched any more. The primary problem with that approach, in case of GDB, is that it's just not practical to conditionalize every bit of output on used interface version, so the testcases are written to tolerate some "good" additional fields of output. And the testcases have to be modified when new "good" additional fields appear, so the maintenance burden increases. Probably, for C++ Boost this is not an issue, and it's possible to freeze the tests of earlier published interface, and don't touch them. But for C++ Boost, the performance is a real problem. E.g, I suspect that two complete Spirit testsuites, for different versions, will be very hard on test resources. - Volodya