Re: [Boost-users] Writing large binary files with boost gzip

On Mon, Apr 11, 2011 at 3:35 AM, wrote:

Ok, here's my entire script. As mentioned in the comments it works fine if

...

size=5e5. It also runs fine if size=5e6, but then the resulting file is corrupt (i.e. cannot be gunzipped with the gzip utility). Try it out. I'm using Visual Studio 2008 in a Windows XP machine.

#include #include #include

using namespace std; namespace io = boost::iostreams;

int main() { //Set filename string outfile = "c:/outfile.bin.gz";

//Set filesize int size = int(5e6); // <- If I change this to '5e5' instead of '5e6', everything works just fine.

//Declare memory block to be compressed to file char* memblock = new char [size];

//Create a filtering_ostream out io::filtering_ostream out;

//Assigns the gzip_compressor to out out.push(io::gzip_compressor());

//Assigns out as a file sink out.push(io::file_sink(outfile));

//Write memblock to out out.write(memblock, size);

//Clean up delete[] memblock; io::close(out); //Note, also tried 'out.close();', 'io::close(out, ios_base::out);' and 'close(out);'. Same result.

return 0; }

Works just fine on linux gcc 4.3.2 boost 1.45. with 5e5, 5e6 and 5e7 size. As a guess, I'd delete memblock after closing the out to be on a safe side, maybe it helps.

-- Slava

_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

If anyone is still interested, this is what I got to work after much hair-pulling. Maybe someone who understand streams (std and boost) better than I can explain why this works and my previous attempts created corrupt files when size was large (again, I'm using Windows, VS 2008). Cheers, Anders #include <fstream> #include <iostream> #include <sstream> #include #include #include using namespace std; namespace io = boost::iostreams; int main() { //Trying with stringstream, which can act as both istream and ostream int size = 5700 * 4800 * 2; //This happens to be the actual size of the data I need to write char* memblock = new char [size]; //Data to write stringstream ss(stringstream::in | stringstream::out | stringstream::binary); //Declare ss ss.write(memblock, size); //Write data to ss io::filtering_streambufio::input buf; //Declare buf buf.push(io::gzip_compressor()); //Assign compressor to buf buf.push(ss); //Push ss to buf ofstream out("c:/outfile.gz", ios_base::out | ios_base::binary); //Declare out io::copy(buf, out); //Copy buf to out //Clean up out.close(); delete[] memblock; return 0; } -- Anders Knudby, PhD e-mail: knudby@gmail.com skype ID: anders.knudby Man's mind, once stretched by a new idea, never regains its original dimensions. ~ Oliver Wendell Holmes