On Wed, May 22, 2019 at 5:40 PM Jeff Bonyun
... How will we detect whether liblzma is capable? If it is not capable but specified, will we allow the compression anyway?
I've been investigating this. liblzma defines a version macro, LZMA_VERSION. It looks like liblzma obtained its multithreaded function in 5.1.1 (in 2011). So the idea would be to branch on this macro, and fall back to the currently-used single-threaded function for an older liblzma version. In that case, requesting any number of threads will carry out the operation, but will only use a single thread. This makes it tolerant of different versions, but does mark up the code with a few #if branches. I'm assuming boost goes for tolerance in this case.
Is there also an option to have it use the number of cores the operating system is advertising?
I'm glad you asked! The xz utility, that liblzma is packaged with, does this. But it isn't inherent to liblzma itself, which considers "0" threads to be an error. Copying the approach of xz, I'm intending to use lzma_cputhreads, which is exposed by liblzma. This was added in 5.2.0 (2014); prior to that it used an internal function that I don't think we should/can use. This would necessitate yet another #if branch. Assuming this is acceptable, using "0" threads would indicate that this all-cores method is desired. This matches the xz utility's approach. For liblzma versions before 5.2.0, a value of "0" would be translated into "1" instead. This isn't ideal, as the user clearly wanted more than 1 thread, but I'm not sure we should guess how many the user wanted.
It would still fail in one situation: if your liblzma was built with `--disable-threads`. In this case, the documentation says that lzma_cputhreads is omitted. I would imagine it would result in a link-time error when building Boost.iostreams. I do not have a solution to this problem, because I don't know how to detect this before the link error. The best I can offer is a `#define BOOST_*` to disable it manually if the user runs into this problem.
An alternative is using Boost.Thread, which would solve the problem gracefully. But I was going to avoid pulling in the extra library, and not do this. If you think the Official Boost Way would be to use Boost.Thread, I can change directions.
Fewer dependencies is better, so if lzma exposes something you can use that.
If there is appetite for it, I can make a pull request on github. That's the best way to get started. Be sure to include tests that exercise it.
I'll work on it! The bones are done. It's the tests, particular formatting for boost, and testing-with-different-liblzma versions that needs to be done.
Jeff
Thanks, Jim