Writing large binary files with boost gzip
Hello all, please help! I'm truly banging my head against the wall here. I am trying to write a function that will allow me to write gzipped binary files (~50 MB) and read them again. I have data in memory. Here's my code for writing: namespace io = boost::iostreams; filename = "c:/test.bin.gz"; //My output file int size = 5000000; //Data size in bytes, ~5 MB //Create filtering_ostream io::filtering_ostream out; //Creates a filtering_ostream called out out.push(io::gzip_compressor()); //Assigns the gzip_compressor to out out.push(io::file_sink(filename)); //Assigns a file sink to out char* memblock = new char [size]; //This is my data. In reality memblock will have been created earlier and filled with real data out.write(memblock, size); //Do the writing delete[] memblock; //Clean up As written above the resulting file, c:/test.bin.gz, is corrupt. If I try to decompress it, either with gzip or winrar, I get an error message. However, if I instead set size = 4000000 (~4 MB) (slightly smaller), the resulting file works just fine. My problem therefore is that my actual size is 50 MB... Please help! Anders (not a real programmer)
AMDG On 04/07/2011 12:09 PM, Anders Knudby wrote:
Hello all, please help! I'm truly banging my head against the wall here. I am trying to write a function that will allow me to write gzipped binary files (~50 MB) and read them again. I have data in memory. Here's my code for writing:
namespace io = boost::iostreams;
filename = "c:/test.bin.gz"; //My output file int size = 5000000; //Data size in bytes, ~5 MB
//Create filtering_ostream io::filtering_ostream out; //Creates a filtering_ostream called out out.push(io::gzip_compressor()); //Assigns the gzip_compressor to out out.push(io::file_sink(filename)); //Assigns a file sink to out
char* memblock = new char [size]; //This is my data. In reality memblock will have been created earlier and filled with real data
out.write(memblock, size); //Do the writing
delete[] memblock; //Clean up
As written above the resulting file, c:/test.bin.gz, is corrupt. If I try to decompress it, either with gzip or winrar, I get an error message. However, if I instead set size = 4000000 (~4 MB) (slightly smaller), the resulting file works just fine. My problem therefore is that my actual size is 50 MB...
Have you checked that the stream is closed correctly? I tried on Linux, and it seemed to work fine after a added #includes/main etc. In Christ, Steven Watanabe
Many thanks for the quick response. And apologies for being a total newbie, but how do I close the stream (and how do I check that it has been closed)? With normal iostreams I would just write "out.close();", out being the name of the stream. But that doesn't seem to be the way to do it for filtering_iostreams?!? And I would think it would close when it goes out of scope, but I haven't checked. Any help appreciated. I'm pleased it worked for you on Linux, but I'm largely stuck with Windows for the time being... Anders On Thu, Apr 7, 2011 at 4:28 PM, Steven Watanabe <watanabesj@gmail.com>wrote:
AMDG
On 04/07/2011 12:09 PM, Anders Knudby wrote:
Hello all, please help! I'm truly banging my head against the wall here. I am trying to write a function that will allow me to write gzipped binary files (~50 MB) and read them again. I have data in memory. Here's my code for writing:
namespace io = boost::iostreams;
filename = "c:/test.bin.gz"; //My output file int size = 5000000; //Data size in bytes, ~5 MB
//Create filtering_ostream io::filtering_ostream out; //Creates a filtering_ostream called out out.push(io::gzip_compressor()); //Assigns the gzip_compressor to out out.push(io::file_sink(filename)); //Assigns a file sink to out
char* memblock = new char [size]; //This is my data. In reality memblock will have been created earlier and filled with real data
out.write(memblock, size); //Do the writing
delete[] memblock; //Clean up
As written above the resulting file, c:/test.bin.gz, is corrupt. If I try to decompress it, either with gzip or winrar, I get an error message. However, if I instead set size = 4000000 (~4 MB) (slightly smaller), the resulting file works just fine. My problem therefore is that my actual size is 50 MB...
Have you checked that the stream is closed correctly? I tried on Linux, and it seemed to work fine after a added #includes/main etc.
In Christ, Steven Watanabe _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- Anders Knudby, PhD e-mail: knudby@gmail.com skype ID: anders.knudby Man's mind, once stretched by a new idea, never regains its original dimensions. ~ Oliver Wendell Holmes
Anders Knudby wrote:
Many thanks for the quick response. And apologies for being a total newbie, but how do I close the stream (and how do I check that it has been closed)? With normal iostreams I would just write "out.close();", out being the name of the stream. But that doesn't seem to be the way to do it for filtering_iostreams?!? And I would think it would close when it goes out of scope, but I haven't checked. Any help appreciated. I'm pleased it worked for you on Linux, but I'm largely stuck with Windows for the time being...
Can you post a complete small example demonstrating the problem? The destructor should close the stream. The docs for iostreams::stream shows a close method at http://www.boost.org/doc/libs/1_46_1/libs/iostreams/doc/index.html. Please don't top post. Jeff
On Fri, Apr 8, 2011 at 7:43 AM, Jeff Flinn <TriumphSprint2000@hotmail.com>wrote:
Anders Knudby wrote:
Many thanks for the quick response. And apologies for being a total newbie, but how do I close the stream (and how do I check that it has been closed)? With normal iostreams I would just write "out.close();", out being the name of the stream. But that doesn't seem to be the way to do it for filtering_iostreams?!? And I would think it would close when it goes out of scope, but I haven't checked. Any help appreciated. I'm pleased it worked for you on Linux, but I'm largely stuck with Windows for the time being...
Can you post a complete small example demonstrating the problem? The destructor should close the stream. The docs for iostreams::stream shows a close method at http://www.boost.org/doc/libs/1_46_1/libs/iostreams/doc/index.html.
Please don't top post.
Jeff
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Ok, here's my entire script. As mentioned in the comments it works fine if size=5e5. It also runs fine if size=5e6, but then the resulting file is corrupt (i.e. cannot be gunzipped with the gzip utility). Try it out. I'm using Visual Studio 2008 in a Windows XP machine. #include <boost/iostreams/device/file.hpp> #include <boost/iostreams/filter/gzip.hpp> #include <boost/iostreams/filtering_stream.hpp> using namespace std; namespace io = boost::iostreams; int main() { //Set filename string outfile = "c:/outfile.bin.gz"; //Set filesize int size = int(5e6); // <- If I change this to '5e5' instead of '5e6', everything works just fine. //Declare memory block to be compressed to file char* memblock = new char [size]; //Create a filtering_ostream out io::filtering_ostream out; //Assigns the gzip_compressor to out out.push(io::gzip_compressor()); //Assigns out as a file sink out.push(io::file_sink(outfile)); //Write memblock to out out.write(memblock, size); //Clean up delete[] memblock; io::close(out); //Note, also tried 'out.close();', 'io::close(out, ios_base::out);' and 'close(out);'. Same result. return 0; } -- Anders Knudby
Ok, here's my entire script. As mentioned in the comments it works fine if size=5e5. It also runs fine if size=5e6, but then the resulting file is corrupt (i.e. cannot be gunzipped with the gzip utility). Try it out. I'm using Visual Studio 2008 in a Windows XP machine.
#include <boost/iostreams/device/file.hpp> #include <boost/iostreams/filter/gzip.hpp> #include <boost/iostreams/filtering_stream.hpp>
using namespace std; namespace io = boost::iostreams;
int main() { //Set filename string outfile = "c:/outfile.bin.gz";
//Set filesize int size = int(5e6); // <- If I change this to '5e5' instead of '5e6', everything works just fine.
//Declare memory block to be compressed to file char* memblock = new char [size];
//Create a filtering_ostream out io::filtering_ostream out;
//Assigns the gzip_compressor to out out.push(io::gzip_compressor());
//Assigns out as a file sink out.push(io::file_sink(outfile));
//Write memblock to out out.write(memblock, size);
//Clean up delete[] memblock; io::close(out); //Note, also tried 'out.close();', 'io::close(out, ios_base::out);' and 'close(out);'. Same result.
return 0; }
Works just fine on linux gcc 4.3.2 boost 1.45. with 5e5, 5e6 and 5e7 size. As a guess, I'd delete memblock after closing the out to be on a safe side, maybe it helps. -- Slava
On Mon, Apr 11, 2011 at 3:35 AM, <Viatcheslav.Sysoltsev@h-d-gmbh.de> wrote:
Ok, here's my entire script. As mentioned in the comments it works fine if
size=5e5. It also runs fine if size=5e6, but then the resulting file is corrupt (i.e. cannot be gunzipped with the gzip utility). Try it out. I'm using Visual Studio 2008 in a Windows XP machine.
#include <boost/iostreams/device/file.hpp> #include <boost/iostreams/filter/gzip.hpp> #include <boost/iostreams/filtering_stream.hpp>
using namespace std; namespace io = boost::iostreams;
int main() { //Set filename string outfile = "c:/outfile.bin.gz";
//Set filesize int size = int(5e6); // <- If I change this to '5e5' instead of '5e6', everything works just fine.
//Declare memory block to be compressed to file char* memblock = new char [size];
//Create a filtering_ostream out io::filtering_ostream out;
//Assigns the gzip_compressor to out out.push(io::gzip_compressor());
//Assigns out as a file sink out.push(io::file_sink(outfile));
//Write memblock to out out.write(memblock, size);
//Clean up delete[] memblock; io::close(out); //Note, also tried 'out.close();', 'io::close(out, ios_base::out);' and 'close(out);'. Same result.
return 0; }
Works just fine on linux gcc 4.3.2 boost 1.45. with 5e5, 5e6 and 5e7 size. As a guess, I'd delete memblock after closing the out to be on a safe side, maybe it helps.
-- Slava
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
If anyone is still interested, this is what I got to work after much hair-pulling. Maybe someone who understand streams (std and boost) better than I can explain why this works and my previous attempts created corrupt files when size was large (again, I'm using Windows, VS 2008). Cheers, Anders #include <fstream> #include <iostream> #include <sstream> #include <boost/iostreams/filtering_streambuf.hpp> #include <boost/iostreams/copy.hpp> #include <boost/iostreams/filter/gzip.hpp> using namespace std; namespace io = boost::iostreams; int main() { //Trying with stringstream, which can act as both istream and ostream int size = 5700 * 4800 * 2; //This happens to be the actual size of the data I need to write char* memblock = new char [size]; //Data to write stringstream ss(stringstream::in | stringstream::out | stringstream::binary); //Declare ss ss.write(memblock, size); //Write data to ss io::filtering_streambuf<io::input> buf; //Declare buf buf.push(io::gzip_compressor()); //Assign compressor to buf buf.push(ss); //Push ss to buf ofstream out("c:/outfile.gz", ios_base::out | ios_base::binary); //Declare out io::copy(buf, out); //Copy buf to out //Clean up out.close(); delete[] memblock; return 0; } -- Anders Knudby, PhD e-mail: knudby@gmail.com skype ID: anders.knudby Man's mind, once stretched by a new idea, never regains its original dimensions. ~ Oliver Wendell Holmes
participants (4)
-
Anders Knudby -
Jeff Flinn -
Steven Watanabe -
Viatcheslav.Sysoltsev＠h-d-gmbh.de