[Boost.Iostreams] Question about gzip codec and dual_use filter
Hello, I'm just wondering why zlib and bzip2 codec are "dual_use" filter but not gzip? Can anybody enlighten me on this restriction? In fact, I have the following use case: I need to manage an "iostream" instance for interfacing with an external library but I would like to be able to support the GZIP codec during writing/reading operations, that is, I must deal with the following "external" serializer engine: SerializerEngine ar(std::iostream& ios, OpeningMode mode) The SerializerEngine interface provides some read/write raw operations. For doing that, I've just tried to implement the following class: class gzfstream: public boost::iostreams::filtering_streamboost::iostreams::dual_use { public: template <class Path> gzfstream(const Path& file_path, std::ios_base::openmode mode): _file(file_path, mode) { if( _file.fail() ) return ; namespace fs = boost::filesystem ; namespace io = boost::iostreams ; const typename Path::string_type& ext = fs::extension(file_path) ; if( gz_extension_exists(ext) ) { const bool in_op = (mode & std::ios::in) != 0 ; const bool out_op = (mode & std::ios::out) != 0 ; if( in_op && out_op ) throw std::logic_error("can't support input/output operation with codec") ; if( out_op ) push(io::gzip_compressor()) ; else push(io::gzip_decompressor()) ; } push(_file) ; } ~gzfstream() { reset() ; } private: boost::filesystem::fstream _file ; bool gz_extension_exists(const std::string& ext) { return ext.size() == 3 && ext == ".gz" ; } bool gz_extension_exists(const std::wstring& ext) { return ext.size() == 3 && ext == L".gz" ; } } ; But this class doesn't compile only for GZIP codec because, as explained in the documentation, GZIP doesn't support dual_use concept - contrary to the ZLib and BZip2 codecs. Is there any issue to deal with my use case? Any help will be appreciate? Best regards, Marc VIALA
Marc Viala Perso wrote:
Hello,
I'm just wondering why zlib and bzip2 codec are "dual_use" filter but not gzip? Can anybody enlighten me on this restriction?
Yeah, I'm probably the only one who can enlighten you. It was sheer laziness on my part, because the implementation looked like it would be pretty messy and I thought it would typically be used with input streams. There's no good excuse for this -- sorry. I've created a ticket to fix it for 1.36 (http://svn.boost.org/trac/boost/ticket/1579). It's too late for 1.35, but I may post a patch.
In fact, I have the following use case:
...
class gzfstream: public boost::iostreams::filtering_streamboost::iostreams::dual_use
You should never instantiate a
filtering_streamboost::iostreams::dual_use. DualUse is not a real
mode, it's a pseudo mode, provided as a convenience for filter
implementors. I should put in a STATIC_ASSERT somewhere to prevent this.
If you need to use an external interface that expects an iostream, you
could use a filtering_stream<bidirectional>, together with
boost::iostreams::combination (http://tinyurl.com/24zwqy) to producing a
bidirectional filter. Depending on your use case, which I don't fully
understand, you could combine two compression filters to produce a
bidirectional compression filter, or a compression filter with a dummy
filter that always throws, to produce a bidirectional stream that at
runtime can only be used for unidirectional i/o.
If you need to use a gzip_decompressor for output, you should be able to
use
boost::iostreams::inverse
Best regards,
Marc VIALA
-- Jonathan Turkanis CodeRage http://www.coderage.com
Marc Viala Perso wrote:
Hello,
I'm just wondering why zlib and bzip2 codec are "dual_use" filter but not gzip? Can anybody enlighten me on this restriction?
Yeah, I'm probably the only one who can enlighten you. It was sheer laziness on my part, because the implementation looked like it would be pretty messy and I thought it would typically be used with input streams. There's no good excuse for this -- sorry. I've created a ticket to fix it for 1.36 (http://svn.boost.org/trac/boost/ticket/1579). It's too late for 1.35, but I may post a patch.
[Marc Viala] Ok, thanks for this explanation I'm pleased for hearing that I'm not the only lazy developer ;-). Right now, the patch is not necessary (see below) so I'll be able to wait for Boost 1.36.
In fact, I have the following use case:
...
class gzfstream: public boost::iostreams::filtering_streamboost::iostreams::dual_use
You should never instantiate a filtering_streamboost::iostreams::dual_use. DualUse is not a real mode, it's a pseudo mode, provided as a convenience for filter implementors. I should put in a STATIC_ASSERT somewhere to prevent this.
[Marc Viala] Oups, sorry but maybe it will be very valuable to add some explanations on the Iostreams documentation to explain this type of restriction (and to add also the STATIC_ASSERT). More generally, my feelings after some days diving into the Iostream documentation, is that adding some use cases or examples would be relevant. Your library is very important/great in my sense and I'm sure that I'm not able to embrace all its features up to now, maybe due to the documentation.
If you need to use an external interface that expects an iostream, you could use a filtering_stream<bidirectional>, together with boost::iostreams::combination (http://tinyurl.com/24zwqy) to producing a bidirectional filter. Depending on your use case, which I don't fully understand, you could combine two compression filters to produce a bidirectional compression filter, or a compression filter with a dummy filter that always throws, to produce a bidirectional stream that at runtime can only be used for unidirectional i/o.
[Marc Viala] Ok, I've implemented your suggestion and its works fine. Just to be sure that I didn't miss anything, do you confirm the implementation hereafter? class gzfstream: public boost::iostreams::filtering_streamboost::iostreams::bidirectional { public: template <class Path> gzfstream(const Path& file_path, std::ios_base::openmode mode): _file(file_path, mode) { namespace fs = boost::filesystem ; namespace io = boost::iostreams ; const typename Path::string_type& ext = fs::extension(file_path) ; if( gz_extension_exists(ext) ) push(io::combine(io::gzip_decompressor(), io::gzip_compressor())) ; push(_file) ; } ~gzfstream() { reset() ; } static bool gz_extension_exists(const std::string& ext) { return ext.size() == 3 && ext == ".gz" ; } static bool gz_extension_exists(const std::wstring& ext) { return ext.size() == 3 && ext == L".gz" ; } private: boost::filesystem::fstream _file ; } ; Jonathan thanks again for your library and your very quick help. Best regards, Marc VIALA
Marc Viala Perso wrote:
You should never instantiate a filtering_streamboost::iostreams::dual_use. DualUse is not a real mode, it's a pseudo mode, provided as a convenience for filter implementors. I should put in a STATIC_ASSERT somewhere to prevent this.
[Marc Viala] Oups, sorry but maybe it will be very valuable to add some explanations on the Iostreams documentation to explain this type of restriction (and to add also the STATIC_ASSERT).
I certainly will. It never occurred to me that someone might try to
instantiate filtering_stream
More generally, my feelings after some days diving into the Iostream documentation, is that adding some use cases or examples would be relevant. Your library is very important/great in my sense and I'm sure that I'm not able to embrace all its features up to now, maybe due to the documentation.
That's a reasonable request. However, I did provide a large amount of documentation, including a tutorial, user guide, and reference. If something is not explained well, it's probably because it seems obvious to me, as the library author. So it would be very helpful for you to point out areas that need more examples.
If you need to use an external interface that expects an iostream, you could use a filtering_stream<bidirectional>, together with boost::iostreams::combination (http://tinyurl.com/24zwqy) to producing a bidirectional filter. Depending on your use case, which I don't fully understand, you could combine two compression filters to produce a bidirectional compression filter, or a compression filter with a dummy filter that always throws, to produce a bidirectional stream that at runtime can only be used for unidirectional i/o.
[Marc Viala] Ok, I've implemented your suggestion and its works fine. Just to be sure that I didn't miss anything, do you confirm the implementation hereafter?
class gzfstream: public boost::iostreams::filtering_streamboost::iostreams::bidirectional { public: template <class Path> gzfstream(const Path& file_path, std::ios_base::openmode mode): _file(file_path, mode) { namespace fs = boost::filesystem ; namespace io = boost::iostreams ;
const typename Path::string_type& ext = fs::extension(file_path) ; if( gz_extension_exists(ext) ) push(io::combine(io::gzip_decompressor(), io::gzip_compressor())) ; push(_file) ; }
~gzfstream() { reset() ; }
static bool gz_extension_exists(const std::string& ext) { return ext.size() == 3 && ext == ".gz" ; }
static bool gz_extension_exists(const std::wstring& ext) { return ext.size() == 3 && ext == L".gz" ; }
private: boost::filesystem::fstream _file ; } ;
Looks good. The call to reset() in the destructor is unnecessary, but harmless.
Jonathan thanks again for your library and your very quick help.
np
Best regards,
Marc VIALA
-- Jonathan Turkanis CodeRage http://www.coderage.com
More generally, my feelings after some days diving into the Iostream documentation, is that adding some use cases or examples would be relevant. Your library is very important/great in my sense and I'm sure that I'm not able to embrace all its features up to now, maybe due to the documentation.
That's a reasonable request. However, I did provide a large amount of documentation, including a tutorial, user guide, and reference. If something is not explained well, it's probably because it seems obvious to me, as the library author. So it would be very helpful for you to point out areas that need more examples.
Yes, there is quite a bit of good documentation. Thanks for that. I think a small section in the tutorial (with a simple example) on implementing symmetric_filters might be useful. I recall this has come up a few times on the list. It is hard for new users to tease out the nuances of the iostreams aspects of symmetric_filters by only looking at the supplied zlib and bzip2 filter implementations.
That's a reasonable request. However, I did provide a large amount of documentation, including a tutorial, user guide, and reference. If something is not explained well, it's probably because it seems obvious to me, as the library author. So it would be very helpful for you to point out areas that need more examples.
I haven't posted to this list in a while, but I noticed this discussion going on. I actually made use of the boost::iostream stuff a few months back and I can confirm that, while the documentation is plentiful, and good for providing high level conceptual overview, it lacks some pretty important details. What I found most poignantly lacking was documentation of exactly what interfaces filters, source and sinks should implement to be used in the various streams. This was spread out throughout the docs, instead of being summarized in one place. I figured it out eventually, but it felt like a lot of guesswork, and piecing things together from various chunks of documentation and header files. Honestly, it made me reconsider template programming altogether, since it made me realize that the lack of an explicit interface can be seriously frustrating and we don't have a standardized way of documenting implicit interfaces yet. Also, I noticed some interesting behaviors. Calling sync on a stream, didn't seem to call flush on my filter, but the destructor did, causing some interesting side effects. - mike
eg wrote:
I think a small section in the tutorial (with a simple example) on implementing symmetric_filters might be useful. I recall this has come up a few times on the list. It is hard for new users to tease out the nuances of the iostreams aspects of symmetric_filters by only looking at the supplied zlib and bzip2 filter implementations.
Michael Linck wrote:
I haven't posted to this list in a while, but I noticed this discussion going on. I actually made use of the boost::iostream stuff a few months back and I can confirm that, while the documentation is plentiful, and good for providing high level conceptual overview, it lacks some pretty important details. What I found most poignantly lacking was documentation of exactly what interfaces filters, source and sinks should implement to be used in the various streams. This was spread out throughout the docs, instead of being summarized in one place. I figured it out eventually, but it felt like a lot of guesswork, and piecing things together from various chunks of documentation and header files. Honestly, it made me reconsider template programming altogether, since it made me realize that the lack of an explicit interface can be seriously frustrating and we don't have a standardized way of documenting implicit interfaces yet. Also, I noticed some interesting behaviors. Calling sync on a stream, didn't seem to call flush on my filter, but the destructor did, causing some interesting side effects.
- mike
Thanks, guys. I will try to improve the docs to address these concerns. -- Jonathan Turkanis CodeRage http://www.coderage.com
bounces@lists.boost.org] De la part de Jonathan Turkanis
eg wrote:
I think a small section in the tutorial (with a simple example) on implementing symmetric_filters might be useful. I recall this has come up a few times on the list. It is hard for new users to tease out the nuances of the iostreams aspects of symmetric_filters by only looking at the supplied zlib and bzip2 filter implementations.
Michael Linck wrote:
I haven't posted to this list in a while, but I noticed this discussion going on. I actually made use of the boost::iostream stuff a few months back and I can confirm that, while the documentation is plentiful, and good for providing high level conceptual overview, it lacks some pretty important details. What I found most poignantly lacking was documentation of exactly what interfaces filters, source and sinks should implement to be used in the various streams. This was spread out throughout the docs, instead of being summarized in one place. I figured it out eventually, but it felt like a lot of guesswork, and piecing things together from various chunks of documentation and header files. Honestly, it made me reconsider template programming altogether, since it made me realize that the lack of an explicit interface can be seriously frustrating and we don't have a standardized way of documenting implicit interfaces yet. Also, I noticed some interesting behaviors. Calling sync on a stream, didn't seem to call flush on my filter, but the destructor did, causing some interesting side effects.
- mike
Thanks, guys. I will try to improve the docs to address these concerns.
[Marc Viala] This is a good new; just for giving a feedback about the documentation, I'm sharing the same feeling that Micheal. Marc
participants (4)
-
eg
-
Jonathan Turkanis
-
Marc Viala Perso
-
Michael Linck