Re: [boost] Re: iostreams library review
I downloaded the library from the yahoo link you mentioned in the review request. When trying to use 'filtered_stream' I ran into some compilation issues (RedHat 9, g++ 3.3) and before looking into it I wanted to know the review status. Knowing that the library is in the review pipeline I will give it another try. Some time ago I developed simple iostream adapter 'library' that allows to wrap IO device (sockets, pipes, etc) in std::[i,o,io]stream that is very similar to your sink/source technique. One difference is that the adapter is a template class parameterized by the source/sink types. Since your library offers this functionality (plus much more) I am very pleased it will become part of boost soon. technews@kangarool ogic.com To: boost@lists.boost.org Sent by: cc: boost-bounces@list Subject: [boost] Re: iostreams library review s.boost.org 05/29/04 06:41 PM Please respond to boost <slavek.f.kuzniar@jpmorgan.com> wrote in message news:OFF5E3130C.48F086BB-ON85256EA2.004FA2A3@ny.jpmorgan.com...
On Boost mailing archives I found a review request for Iostreams Library by Jonathan Turkanis. I did not see any follow-up and the library is not part of latest release so I suspect the review did not happen. Anyone knows what is the fate of this library?
Thanks for your interest! The previous version available at http://www.xmission.com/~turkanis/iostreams/. I've since made some major simplifications and improvements, and am scrambing to update the docs. It now contains support for memory-mapped files (from Craig Henderson) and for compression and decompression in the zlib, gzip and bzip2 formats. Jonathan _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of J.P. Morgan Chase & Co., its subsidiaries and affiliates.
slavek.f.kuzniar@jpmorgan.com writes:
The previous version available at http://www.xmission.com/~turkanis/iostreams/.
I've since made some major simplifications and improvements, and am scrambing to update the docs. It now contains support for memory-mapped files (from Craig Henderson) and for compression and decompression in the zlib, gzip and bzip2 formats.
The thing I wanted recently was an "offset streambuf", an adapter for another streambuf that presents an "offset view" (i.e. with respect to seeking) of the underlying streambuf. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
"David Abrahams" <dave@boost-consulting.com> wrote in message news:uisebatkh.fsf@boost-consulting.com...
slavek.f.kuzniar@jpmorgan.com writes:
The previous version available at http://www.xmission.com/~turkanis/iostreams/.
I've since made some major simplifications and improvements, and am scrambing to update the docs. It now contains support for memory-mapped files (from Craig Henderson) and for compression and decompression in the zlib, gzip and bzip2 formats.
The thing I wanted recently was an "offset streambuf", an adapter for another streambuf that presents an "offset view" (i.e. with respect to seeking) of the underlying streambuf.
This sounds like it might be a job for a 'SeekableFilter' -- something I included support for even though I didn't have any use for it. The interface for a seekable filter is basically this: struct seekable_filter { // some typedefs streamsize read(char* s, streamsize n, streambuf& next); void write(const char* s, streamsize n, streambuf& next); streamoff seek(streamoff off, ios_base::seekdir way, streambuf& next); }; Here 'next' is the streambuf being filtered. Any number of filters can be chained. You take a filtering_streambuf<seekable>, add your filter, then add the underlying streambuf: filtering_streambuf<seekable> filtered; filtered.push(seekable_filter()); filtered.push(sbuf); What exactly did you want the offset streambuf to do? I'd be interested to see if I can implement it using the above framework. It would be a nice example of a concept which so far I haven't used. Jonathan
"Jonathan Turkanis" <technews@kangaroologic.com> writes:
"David Abrahams" <dave@boost-consulting.com> wrote in message news:uisebatkh.fsf@boost-consulting.com...
slavek.f.kuzniar@jpmorgan.com writes:
The previous version available at http://www.xmission.com/~turkanis/iostreams/.
I've since made some major simplifications and improvements, and am scrambing to update the docs. It now contains support for memory-mapped files (from Craig Henderson) and for compression and decompression in the zlib, gzip and bzip2 formats.
The thing I wanted recently was an "offset streambuf", an adapter for another streambuf that presents an "offset view" (i.e. with respect to seeking) of the underlying streambuf.
This sounds like it might be a job for a 'SeekableFilter' -- something I included support for even though I didn't have any use for it. The interface for a seekable filter is basically this:
struct seekable_filter { // some typedefs streamsize read(char* s, streamsize n, streambuf& next); void write(const char* s, streamsize n, streambuf& next); streamoff seek(streamoff off, ios_base::seekdir way, streambuf& next); };
Here 'next' is the streambuf being filtered. Any number of filters can be chained.
I didn't really think of this as a "filtering" operation, but anyway...
You take a filtering_streambuf<seekable>, add your filter, then add the underlying streambuf:
filtering_streambuf<seekable> filtered; filtered.push(seekable_filter()); filtered.push(sbuf);
What exactly did you want the offset streambuf to do?
I wanted to represent a region of offsets in the underlying streambuf as the whole stream. So, for example, void operate_on_region(streambuf& s) { region_streambuf s1(s, 500, 1000); // a stream over 500 bytes of s s1.pubseekpos(10); // I probably don't have this call quite right traits::int_type c = s1.sbumpc(); // reads byte 510 from s } Obviously I would want it to work for all the other seeking operations.
I'd be interested to see if I can implement it using the above framework.
Me too ;-) -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
"David Abrahams" <dave@boost-consulting.com> wrote in message news:ur7syb23h.fsf@boost-consulting.com...
"Jonathan Turkanis" <technews@kangaroologic.com> writes:
"David Abrahams" <dave@boost-consulting.com> wrote in message
The thing I wanted recently was an "offset streambuf", an adapter for another streambuf that presents an "offset view" (i.e. with respect to seeking) of the underlying streambuf.
This sounds like it might be a job for a 'SeekableFilter' -- something I included support for even though I didn't have any use for it. The interface for a seekable filter is basically this:
<snip>
Here 'next' is the streambuf being filtered. Any number of filters can be chained.
I didn't really think of this as a "filtering" operation, but anyway...
Sure, I'm just using 'filtering' as a catch-all term <snip>
What exactly did you want the offset streambuf to do?
I wanted to represent a region of offsets in the underlying streambuf as the whole stream. So, for example,
void operate_on_region(streambuf& s) { region_streambuf s1(s, 500, 1000); // a stream over 500 bytes of s s1.pubseekpos(10); // I probably don't have this call quite right traits::int_type c = s1.sbumpc(); // reads byte 510 from s }
This fits nicely into the framework, with one caveat, noted below.
struct seekable_filter { // some typedefs streamsize read(char* s, streamsize n, streambuf& next); void write(const char* s, streamsize n, streambuf& next); streamoff seek(streamoff off, ios_base::seekdir way, streambuf& next); };
The filter would start by seeking to the first offset. Reads and writes would delegate to next after adjusting the streamsize argument, if needed. Seeks using ios_bas::cur would delegate to next, after checking that they stay within the given bounds. Seeks using ios_base::beg or ios_base::end would adjust the streamoff argument before delegating. The caveat is that seeking with respect to stream positions -- i.e. values of type traits_type::pos_type rather than traits_type::off_type -- only works as expected for positions which are equivalent to integral offsets. Free ranging random access doesn't make much sense in multibyte streams, anyway. There's a more limited type of random access which is possible in multibyte streams, in which you can save a position and go back to it later. I've chosen not to support this, however. Jonathan
"Jonathan Turkanis" <technews@kangaroologic.com> writes:
The caveat is that seeking with respect to stream positions -- i.e. values of type traits_type::pos_type rather than traits_type::off_type -- only works as expected for positions which are equivalent to integral offsets.
You lost me here. What positions aren't?
Free ranging random access doesn't make much sense in multibyte streams, anyway. There's a more limited type of random access which is possible in multibyte streams, in which you can save a position and go back to it later. I've chosen not to support this, however.
Do you mean variable-length-character streams? I definitely want to do regular random access on streams of wchar_t, which I suppose are "multibyte". -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
"David Abrahams" <dave@boost-consulting.com> wrote in message news:ubrk18u0m.fsf@boost-consulting.com...
"Jonathan Turkanis" <technews@kangaroologic.com> writes:
The caveat is that seeking with respect to stream positions -- i.e. values of type traits_type::pos_type rather than traits_type::off_type -- only works as expected for positions which are equivalent to integral offsets.
You lost me here. What positions aren't?
positions that record an offset and a code conversion state.
Free ranging random access doesn't make much sense in multibyte streams, anyway. There's a more limited type of random access
which
is possible in multibyte streams, in which you can save a position and go back to it later. I've chosen not to support this, however.
Do you mean variable-length-character streams?
I mean streams of characters as they are read from a file with a multibyte encoding -- an encoding in which a single character may be represented by a sequence of bytes, possibly of varying length -- before code conversion is performed.
I definitely want to do regular random access on streams of wchar_t, which I suppose are "multibyte".
Random access in wide streams is generally fine. (However, if the wide stream is based on an underlying multibyte stream, and the member function encoding() of the codecvt used to perform the conversion returns a value <= 0, then random-access is rather inefficient.) (Maybe I shouldn't have confused the issues by bringing up code conversion. It is an issue, however, whenever you use pubseekpos, since positions don't exactly correspond to offsets.) The offsetbuf could be defined like this (assuming a narrow stream, for simplicity): class offsetbuf : public streambuf_facade<offset_filter> { public: offsetbuf(std::streambuf& sb, std::streamoff first, std::streamoff last) { set_next(sb); open(offset_filter(first, last)); } }; Here offset_filter is the filter I described before. Jonathan
"Jonathan Turkanis" <technews@kangaroologic.com> writes:
"David Abrahams" <dave@boost-consulting.com> wrote in message news:ubrk18u0m.fsf@boost-consulting.com...
"Jonathan Turkanis" <technews@kangaroologic.com> writes:
The caveat is that seeking with respect to stream positions -- i.e. values of type traits_type::pos_type rather than traits_type::off_type -- only works as expected for positions which are equivalent to integral offsets.
You lost me here. What positions aren't?
positions that record an offset and a code conversion state.
Oh. Yeah, I know nothing about that stuff.
Free ranging random access doesn't make much sense in multibyte streams, anyway. There's a more limited type of random access which is possible in multibyte streams, in which you can save a position and go back to it later. I've chosen not to support this, however.
Do you mean variable-length-character streams?
I mean streams of characters as they are read from a file with a multibyte encoding -- an encoding in which a single character may be represented by a sequence of bytes, possibly of varying length -- before code conversion is performed.
OK.
I definitely want to do regular random access on streams of wchar_t, which I suppose are "multibyte".
Random access in wide streams is generally fine. (However, if the wide stream is based on an underlying multibyte stream, and the member function encoding() of the codecvt used to perform the conversion returns a value <= 0, then random-access is rather inefficient.)
:-)
(Maybe I shouldn't have confused the issues by bringing up code conversion. It is an issue, however, whenever you use pubseekpos, since positions don't exactly correspond to offsets.)
Ugh.
The offsetbuf could be defined like this (assuming a narrow stream, for simplicity):
class offsetbuf : public streambuf_facade<offset_filter> { public: offsetbuf(std::streambuf& sb, std::streamoff first, std::streamoff last) { set_next(sb); open(offset_filter(first, last)); } };
Here offset_filter is the filter I described before.
If you say so. Anyway, the library would be useful to me if it made that possible. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
"David Abrahams" <dave@boost-consulting.com> wrote in message news:upt8fxspr.fsf@boost-consulting.com...
"Jonathan Turkanis" <technews@kangaroologic.com> writes:
"David Abrahams" <dave@boost-consulting.com> wrote in message news:ubrk18u0m.fsf@boost-consulting.com...
"Jonathan Turkanis" <technews@kangaroologic.com> writes:
The caveat is that seeking with respect to stream positions -- i.e. values of type traits_type::pos_type rather than traits_type::off_type -- only works as expected for positions which are equivalent to integral offsets.
You lost me here. What positions aren't?
positions that record an offset and a code conversion state.
Oh. Yeah, I know nothing about that stuff.
You're better off that way -- trust me ;-) <snip>
The offsetbuf could be defined like this (assuming a narrow
stream,
for simplicity):
class offsetbuf : public streambuf_facade<offset_filter> { public: offsetbuf(std::streambuf& sb, std::streamoff first, std::streamoff last) { set_next(sb); open(offset_filter(first, last)); } };
Here offset_filter is the filter I described before.
If you say so.
Well, I realize I wrote a bunch of stuff without explaning it, and the documentation is not even available, so I shouldn't expect it to make any sense. I guess my point was that it can be done with a few lines of code.
Anyway, the library would be useful to me if it made that possible.
Good. I think I'll add it as an example application, or maybe as part of the core library. Jonathan
<slavek.f.kuzniar@jpmorgan.com> wrote in message news:OFBB047E1B.B12BDF48-ON85256EA6.004E93AD@ny.jpmorgan.com...
I downloaded the library from the yahoo link you mentioned in the
review
request. When trying to use 'filtered_stream' I ran into some compilation issues (RedHat 9, g++ 3.3) and before looking into it I wanted to know the review status. Knowing that the library is in the review pipeline I will give it another try.
Thanks for trying it out. I'm sorry to hear about the compilation problems. gcc 3.3 on Linux is defintely one of the supported platforms. Last I rememeber it passed all the regression tests. Normally, I would ask you to post your code and error messages, but I think in this case it doesn't make much sense to patch an old version of the library. I'd like to put up the new version and have you try it out.
Some time ago I developed simple iostream adapter 'library' that allows to wrap IO device (sockets, pipes, etc) in std::[i,o,io]stream that is very similar to your sink/source technique. One difference is that the adapter is a template class parameterized by the source/sink types.
This is the approach I take in the latest version (actually its present in the previous version but sort of hidden). There are two main streambuf and stream templates -- streambuf_facade and stream_facade -- both parameterized by a source/sink. (The names are stolen from Boost.Iterator, of course.) For filtering there are also two templates, filtering_streambuf and filtering_stream, parameterized by a 'mode' (input, output, etc.) and by character type. Basically, everything is simpler to use, and more flexible. I've also improved the implementation. As an experiment I replaced tag dispatch with a technique using is_convertible and simulated partial specialization. To my amazement I found about a 30% speed-up on como and found that Borland 5.6.4 runs about twice as fast.
Since your library offers this functionality (plus much more) I am very pleased it will become part of boost soon.
Thanks for the encouragement. I certainly hope it will become part of boost. Best Regards, Jonathan
participants (3)
-
David Abrahams -
Jonathan Turkanis -
slavek.f.kuzniarļ¼ jpmorgan.com