[Iostreams] Zlib compression sometimes creates corrupt data on OSX
Hello, I’m having the following problem with my program: If I save data on OSX using a boost::iostreams::filtering_ostream with a boost::iostreams::zlib_compressor() the resulting file is sometimes corrupted. The following example code reproduces the problem: #include <fstream> #include <iostream> #include <vector> #include <boost/iostreams/filtering_stream.hpp> #include <boost/iostreams/filter/zlib.hpp> int main(int argc, char** argv) { if (argc != 2) return 1; std::ifstream file(argv[1]); if (file) { file.seekg(0, std::ios::end); std::streampos length = file.tellg(); file.seekg(0, std::ios::beg); std::vector<char> buffer1(length); std::vector<char> buffer2(length); file.read(&buffer1[0], length); { std::ofstream ofs("test.dat", std::ios_base::binary); boost::iostreams::filtering_ostream fos; fos.push(boost::iostreams::zlib_compressor()); fos.push(ofs); fos.write(&buffer1[0], length); } { std::ifstream ifs("test.dat", std::ios_base::binary); boost::iostreams::filtering_istream fis; fis.push(boost::iostreams::zlib_decompressor()); fis.push(ifs); fis.read(&buffer2[0], length); } std::cout << (buffer1 == buffer2 ? "equal" : "not equal") << std::endl; } } This piece of code reads a file that is specified as command line argument, compresses and saves it to “test.dat”, reloads and decompresses it and then compares the result with the content of the original file. Using it with two files linked below leads to the following output: $ ./bugtest example1_working.txt equal $ ./bugtest example1_not_working.txt not equal i.e. in the second case the compression creates corrupt data. I verified that the problem is in the compression part by trying to decompress test.dat on Linux using zlib-flate which gives an error (incorrect data check). If I compress the file manually on Linux using zlib-flate and decompress it on OSX with code similar to the above, the input file is reproduced correctly. The two files used above are save files of my software. You can download them from: http://neptun.iup.uni-heidelberg.de/files/example1_working.txt http://neptun.iup.uni-heidelberg.de/files/example1_not_working.txt Another pair of files with the problem is: http://neptun.iup.uni-heidelberg.de/files/example2_working.txt http://neptun.iup.uni-heidelberg.de/files/example2_not_working.txt I haven’t been able to find out what is special about the ones causing the problem, so I just put them up as they are. The files are quite similar, but show a few differences on every (or almost every?) line. If needed, I can create other pairs of files showing the same problem. I compiled the above code in the following way: c++ -stdlib=libc++ -std=c++11 test.cpp -I /Users/michael/projects/boost-svn-build/include/ -c c++ -stdlib=libc++ -std=c++11 -o test test.o /Users/michael/projects/boost-svn-build/lib/libboost_iostreams.a -lz The boost libraries were built from svn as follows: ./bootstrap.sh --prefix=../boost-svn-build --with-libraries=iostreams --with-toolset=clang ./b2 -j6 link=static toolset=clang cxxflags="-std=c++11 -stdlib=libc++" linkflags="-stdlib=libc++" install Clang version is: “Apple LLVM version 4.2 (clang-425.0.28) (based on LLVM 3.2svn)” I am running OS X 10.8.2. The same files are compressed correctly with the above code on Linux, compiled with GCC 4.6.3. I think this problem might be a bug in boost or maybe also in the llvm libraries (c.f. https://svn.boost.org/trac/boost/ticket/8150 , which looks like a similar problem). Compiling boost with zlib 1.2.8 instead of using zlib 1.2.5, which is shipped with OSX, leads to the same problem. Both Boost 1.54.0 and the svn version are affected. Should I create a ticket for this on the bug tracker? Or is this just me using the library incorrectly?? Best regards, Michael -- Michael Jung michaeljung@mailhaven.com
On 10/2/2013 11:07 AM, Michael Jung wrote:
Hello,
I’m having the following problem with my program: If I save data on OSX using a boost::iostreams::filtering_ostream with a boost::iostreams::zlib_compressor() the resulting file is sometimes corrupted.
The following example code reproduces the problem:
#include <fstream> #include <iostream> #include <vector> #include <boost/iostreams/filtering_stream.hpp> #include <boost/iostreams/filter/zlib.hpp>
int main(int argc, char** argv) { if (argc != 2) return 1; std::ifstream file(argv[1]);
Maybe add: , std::ios_base::binary
if (file) { file.seekg(0, std::ios::end); std::streampos length = file.tellg(); file.seekg(0, std::ios::beg);
std::vector<char> buffer1(length); std::vector<char> buffer2(length);
file.read(&buffer1[0], length);
{ std::ofstream ofs("test.dat", std::ios_base::binary); boost::iostreams::filtering_ostream fos; fos.push(boost::iostreams::zlib_compressor()); fos.push(ofs);
fos.write(&buffer1[0], length); } { std::ifstream ifs("test.dat", std::ios_base::binary); boost::iostreams::filtering_istream fis; fis.push(boost::iostreams::zlib_decompressor()); fis.push(ifs);
fis.read(&buffer2[0], length); }
std::cout << (buffer1 == buffer2 ? "equal" : "not equal") << std::endl; } }
otherwise search the mailing list for a thread from back in July 2013: "[iostreams] How can I use the Stream Insertion operator (<<) with a filtering_streambuf?" Jeff
participants (2)
-
Jeff Flinn
-
Michael Jung