Seekg on a gzip'd file...Is this possible?
Hello all, I¹m switching one of my projects over from java to c++. Basically it¹s a gzip file reader, where I¹d like to parse a bunch of numbers (floats, ints, doubles) and in some instances I¹d like to skip some of the data. I¹m a ³noob² to boost and after getting the file reader working using the boost like this: { .... std::ifstream ormodelFile(filenameForTimeStep.c_str(), std::ios_base::in|std::ios_base::binary); boost::iostreams::filtering_istreambuf compressedFileFilter; compressedFileFilter.push(boost::iostreams::gzip_decompressor()); compressedFileFilter.push(ormodelFile); this->inputFile.rdbuf(&compressedFileFilter); //where inputFile is a an istream. ... //do the file reading } I tried to add the skipping parts to the code by simply calling: { ... double value; this->inputFile->seekg(sizeof(double),ios::cur); ... } the fail bit is set on the inputFile istream. Is it possible to do this using boost? I¹m pretty ³green² with boost so I¹m sure it¹s just my misunderstanding of how to use the lib but I couldn¹t find my way around the docs to well. The best I could find was that I need to somehow make my stream ³seekable² but it wasn¹t too clear to me how to do this. Any help would be greatly appreciated. Gerrick
I changed my code a bit but I¹m still getting errors when trying to use boost::iostreams::seek as follows: boost::iostreams::seek(inputStream,position_to_offset(bytesToSkip),BOOST_IO S::cur); What is the correct way to go about this? Gerrick On 3/13/09 9:29 AM, "Gerrick Bivins" <gbivins@objectreservoir.com> wrote:
Hello all, I¹m switching one of my projects over from java to c++. Basically it¹s a gzip file reader, where I¹d like to parse a bunch of numbers (floats, ints, doubles) and in some instances I¹d like to skip some of the data. I¹m a ³noob² to boost and after getting the file reader working using the boost like this: { .... std::ifstream ormodelFile(filenameForTimeStep.c_str(), std::ios_base::in|std::ios_base::binary);
boost::iostreams::filtering_istreambuf compressedFileFilter; compressedFileFilter.push(boost::iostreams::gzip_decompressor()); compressedFileFilter.push(ormodelFile);
this->inputFile.rdbuf(&compressedFileFilter); //where inputFile is a an istream. ... //do the file reading }
I tried to add the skipping parts to the code by simply calling: { ... double value; this->inputFile->seekg(sizeof(double),ios::cur); ... } the fail bit is set on the inputFile istream. Is it possible to do this using boost? I¹m pretty ³green² with boost so I¹m sure it¹s just my misunderstanding of how to use the lib but I couldn¹t find my way around the docs to well.
The best I could find was that I need to somehow make my stream ³seekable² but it wasn¹t too clear to me how to do this. Any help would be greatly appreciated. Gerrick
AMDG Gerrick Bivins wrote:
I changed my code a bit but I¹m still getting errors when trying to use boost::iostreams::seek as follows:
boost::iostreams::seek(inputStream,position_to_offset(bytesToSkip),BOOST_IO S::cur);
What is the correct way to go about this?
gzip_decompressor doesn't support seeking. In Christ, Steven Watanabe
Hi Steven, Thanks for the response. Does that mean that the gzip_decompressor in boost doesn't' support seeking or does gzip in general not support seekeing? Any suggestions for working around this problem? Gerrick On 3/13/09 5:19 PM, "Steven Watanabe" <watanabesj@gmail.com> wrote:
AMDG
Gerrick Bivins wrote:
I changed my code a bit but I¹m still getting errors when trying to use boost::iostreams::seek as follows:
boost::iostreams::seek(inputStream,position_to_offset(bytesToSkip),BOOST_IO S::cur);
What is the correct way to go about this?
gzip_decompressor doesn't support seeking.
In Christ, Steven Watanabe
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On Fri, Mar 13, 2009 at 7:59 PM, Gerrick Bivins <gbivins@objectreservoir.com> wrote:
Hi Steven, Thanks for the response. Does that mean that the gzip_decompressor in boost doesn't' support seeking or does gzip in general not support seekeing? Any suggestions for working around this problem? Gerrick
Gzip steams can only move forward. I see no reason why at least this couldn't be supported other than most people expect seek to be instant, whereas gzip will have to decompress all the way to that point. Does seek have some complexity requirements that make this a bad idea? -- Cory Nelson
Cory Nelson <phrosty@gmail.com> writes:
On Fri, Mar 13, 2009 at 7:59 PM, Gerrick Bivins <gbivins@objectreservoir.com> wrote:
Hi Steven, Thanks for the response. Does that mean that the gzip_decompressor in boost doesn't' support seeking or does gzip in general not support seekeing? Any suggestions for working around this problem? Gerrick
Gzip steams can only move forward. I see no reason why at least this couldn't be supported other than most people expect seek to be instant, whereas gzip will have to decompress all the way to that point. Does seek have some complexity requirements that make this a bad idea?
I would say the fact that it would have to decompress all the way to the seek point is reason enough. As a client, I expect a seek to be instant, or at most kick off some /background/ I/O to fill a buffer. I won't argue either way about whether such a call should be added: but it certainly shouldn't have `seek' in the name if it is. Unless maybe it's named ``expensive_seek'' or similar ;) Cheers, -tom
I'd actually like to "skip" (move forward) in the stream. "Rewind" functionality would not be necessary. As I stated earlier, I'm porting code from java to c++ that has this functionality via java.util.zip.GZIPInputStream. My desired functionality would be something like this: 1) read some header info 2) read meta data that tells me which parts of the data I care about 3) Continue read through file sequentially: a) extract needed data as it is encountered b) use meta data to skipping over unneeded data to the next set of needed data. So maybe a "skip" method would be appropriate? Gerrick On 3/13/09 11:05 PM, "Cory Nelson" <phrosty@gmail.com> wrote:
On Fri, Mar 13, 2009 at 7:59 PM, Gerrick Bivins <gbivins@objectreservoir.com> wrote:
Hi Steven, Thanks for the response. Does that mean that the gzip_decompressor in boost doesn't' support seeking or does gzip in general not support seekeing? Any suggestions for working around this problem? Gerrick
Gzip steams can only move forward. I see no reason why at least this couldn't be supported other than most people expect seek to be instant, whereas gzip will have to decompress all the way to that point. Does seek have some complexity requirements that make this a bad idea?
AMDG Gerrick Bivins wrote:
Thanks for the response. Does that mean that the gzip_decompressor in boost doesn't' support seeking or does gzip in general not support seekeing? Any suggestions for working around this problem
The gzip format makes random access impossible. The only way to seek is to read and throw away the intermediate bytes. In Christ, Steven Watanabe
In Christ, Steven Watanabe
Hello Steven, I recently signed up for the boost mailing list. I'm no template metaprogramming expert, :) I just wanted to announce a couple of things I was working on which might be interesting to fellow boost- library-users. As it is such a high traffic list, I probably will not be able to read it for very long...too many messages! But for the moment, I'm watching what goes by and enjoying learning about some of the ways people are working. While reading, I've noticed you sign your messages "In Christ". I take it by your choice to do so that you are inviting questions or conversations about your religious beliefs. :) One thing I wondered was what effect this has had on your beliefs about software. Have you written anything on this topic? A casual search on Google did not find any public essays attributed to you. (Don Knuth, for instance, has Things a Computer Scientist Rarely Talks About: http://www.amazon.com/Things-Computer-Scientist-Language-Information/dp/1575... ) The ties between religious/moral beliefs and conduct in technology pursuits is a subject of interest to me. So if you have anything you'd like to share I would read it. Thank you! Brian Los Angeles, CA http://hostilefork.com (P.S. Subject is an play on words of a Ministry song title, "Jesus Built My Hotrod", which I listened to when I was much younger: http://en.wikipedia.org/wiki/Jesus_Built_My_Hotrod )
participants (5)
-
Cory Nelson -
Gerrick Bivins -
Hostile Fork -
Steven Watanabe -
tom fogal