Ares Lagae wrote:

There were not that much responses to my query. Should I conclude that there is no interest in a binary_iostreams library ?

I'd give it a bit more time and maybe even a second post in a month or 2. This mailing list gets a ton of mail and sometimes people don't get a chance to reply immediately. I'm interested myself, but I don't have time to really look at it currently. Jeff

Richard Hadsell wrote:

myself, but one of my colleagues tried to use it for binary files. He found it very difficult (or impossible) to use the insertion and extraction operators to handle binary format. I don't know how he solved the problem, but I will suggest that he look at a new binary_iostreams library, if it is ever accepted and released.

The binary_iostreams library does not try to modify the traditional iostreams classes, because this is indeed very difficult or impossible. The binary_iostreams library is a new hierarchy of classes developed to be very similar to the iostreams hierarchy. The actual implementation is quite simple. Best regards, -- Ares Lagae Computer Graphics Research Group, Katholieke Universiteit Leuven http://www.cs.kuleuven.be/~ares/

I just sent an email about the use of streambuf's in this library. I think that compression would work well as a streambuf filter. That is, something like class compresionbuf : streambuf_type { public: compressionbuf(streambuf* underlying); .... } This would apply the compression algoritym (whatever you want to use) bytes put into the compressionbuf would be compressed according to what ever algorithm you're using and then sent to underlying. Similarly requests to get bytes from compressionbuf would get bytes from underlying and apply whatever decompression algorithm is desired. Note that such a class would be usable with both iostream and binary_iostream. On May 30, 2007, at 9:56 AM, Richard Hadsell wrote:

Ares Lagae wrote:

...

The binary_iostreams library does not try to modify the traditional iostreams classes, because this is indeed very difficult or impossible. The binary_iostreams library is a new hierarchy of classes developed to be very similar to the iostreams hierarchy. The actual implementation is quite simple.

As long as it can read and write compressed data in binary format, we'll be happy. Maybe it was the compression filters that seemed to be incompatible with binary format.

-- Dick Hadsell 914-259-6320 Fax: 914-259-6499 Reply-to: hadsell@blueskystudios.com Blue Sky Studios http://www.blueskystudios.com 44 South Broadway, White Plains, NY 10601

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/ listinfo.cgi/boost

Jerry Schwarz wrote:

For what its worth, I always envisioned alternative top level IO classes using streambuf's as the underlying transport. The central class of the iostream library is really streambuf and I've long regretted not calling the library "streambuf". I always encourage people to use streambuf's directly when all they want to do is transport bytes.

So I like the idea of this library. It is more consistent with my ideas of how "iostream" should be used than is using istream or ostream directly to read and write binary files. But I would prefer it if binary_istream, binary_ostream and binary_iostream had a constructor that took a streambuf argument and that (at least optionall) ownership would be transfered to the streambuf. That is the stream would be responsible for freeing it.

The C++ streambuf class is still character-oriented, not byte-oriented. In particular, it needs a char_traits parameter (and the associated state_type typedef and length operation, both of which are meaningless for bytes), the importance of which I question anyway. What's worse, it's the stream buffer that's responsible for converting between the "internal character representation" and the "external character representation". This means that, unless it's in binary mode (and perhaps even then, partially), the stream buffer does character set conversion (if it's a wide stream, or perhaps some exotic system that enforces a specific encoding for text files) and newline conversion. Another example of the character-orientation of the buffers is that there is a stringstream, not a vectorstream. Having binary streams that can work with classical stream buffers is good for retaining existing work, but I consider it unacceptable as the mandatory underlying transport. I think a new I/O system should be built in layers like this: lowest) The byte transport. It always seemed weird and a severe shortcoming to me that C++ basically assumed the base for external data to be text. The basic underlying unit of external data should be the byte (i.e. unsigned char in the C++ type system). To represent this level, I think a source/sink/filter stack system like Boost.IOStreams would be appropriate, but oriented strictly towards binary. Buffers are on this level. --) On top of the byte transport, there is the binary I/O layer, akin to the proposed binary_iostream here. It handles the fun part of binary I/O, such as assembling bytes into multibyte types (endianness) or rearranging the internal data within bytes (also endianness, on systems where char is large), and perhaps even converting to and from a canonical external float representation. --) Aside: Binary serialization of objects could build on the binary I/O layer. --) On top of the binary I/O comes the text conversion layer. This should be a comprehensive character set and encoding library, capable of converting between the internal character set(s) and external representations. The external encoding should be chosen by compiler default, locale or explicitly passed name. This builds on the binary I/O and not directly the byte transport because the base type of a character representation can be a multi-byte entity, e.g. in the UTF-16 encoding with 8-bit bytes. --) At this point, another filter layer could be inserted, to have filters that work on the text level. This layer could be made responsible for handling line ending conversion. highest) Finally, once we've got text, we add a formatting layer like the current iostreams. This gives each layer a clearly defined function, and the programmer a great choice between ease of use and flexibility. While we're at it, the interface should consider the possibility of non-blocking and/or async I/O. Sebastian Redl

I agree with you that the current streambuf is a mess. If you want to create a new binary_streambuf and build your library on that I wouldn't object. But you didn't do that, you built a library without any other transport mechanism. At heart my suggestion is to have binary_streams built based on some transport mechanism, and have that mechanism (whatever it is) highly visible. On May 31, 2007, at 2:09 AM, Sebastian Redl wrote:

Jerry Schwarz wrote:

...

For what its worth, I always envisioned alternative top level IO classes using streambuf's as the underlying transport. The central class of the iostream library is really streambuf and I've long regretted not calling the library "streambuf". I always encourage people to use streambuf's directly when all they want to do is transport bytes.

So I like the idea of this library. It is more consistent with my ideas of how "iostream" should be used than is using istream or ostream directly to read and write binary files. But I would prefer it if binary_istream, binary_ostream and binary_iostream had a constructor that took a streambuf argument and that (at least optionall) ownership would be transfered to the streambuf. That is the stream would be responsible for freeing it.

The C++ streambuf class is still character-oriented, not byte- oriented. In particular, it needs a char_traits parameter (and the associated state_type typedef and length operation, both of which are meaningless for bytes), the importance of which I question anyway. What's worse, it's the stream buffer that's responsible for converting between the "internal character representation" and the "external character representation". This means that, unless it's in binary mode (and perhaps even then, partially), the stream buffer does character set conversion (if it's a wide stream, or perhaps some exotic system that enforces a specific encoding for text files) and newline conversion. Another example of the character-orientation of the buffers is that there is a stringstream, not a vectorstream.

Having binary streams that can work with classical stream buffers is good for retaining existing work, but I consider it unacceptable as the mandatory underlying transport.

I think a new I/O system should be built in layers like this:

lowest) The byte transport. It always seemed weird and a severe shortcoming to me that C++ basically assumed the base for external data to be text. The basic underlying unit of external data should be the byte (i.e. unsigned char in the C++ type system). To represent this level, I think a source/sink/filter stack system like Boost.IOStreams would be appropriate, but oriented strictly towards binary. Buffers are on this level.

--) On top of the byte transport, there is the binary I/O layer, akin to the proposed binary_iostream here. It handles the fun part of binary I/O, such as assembling bytes into multibyte types (endianness) or rearranging the internal data within bytes (also endianness, on systems where char is large), and perhaps even converting to and from a canonical external float representation.

--) Aside: Binary serialization of objects could build on the binary I/O layer.

--) On top of the binary I/O comes the text conversion layer. This should be a comprehensive character set and encoding library, capable of converting between the internal character set(s) and external representations. The external encoding should be chosen by compiler default, locale or explicitly passed name. This builds on the binary I/O and not directly the byte transport because the base type of a character representation can be a multi-byte entity, e.g. in the UTF-16 encoding with 8-bit bytes.

--) At this point, another filter layer could be inserted, to have filters that work on the text level. This layer could be made responsible for handling line ending conversion.

highest) Finally, once we've got text, we add a formatting layer like the current iostreams.

This gives each layer a clearly defined function, and the programmer a great choice between ease of use and flexibility.

While we're at it, the interface should consider the possibility of non-blocking and/or async I/O.

Sebastian Redl _______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/ listinfo.cgi/boost

Sebastian Redl <sebastian.redl@getdesigned.at> writes:

Jerry Schwarz wrote:

...

I agree with you that the current streambuf is a mess. If you want to create a new binary_streambuf and build your library on that I wouldn't object. But you didn't do that, you built a library without any other transport mechanism. At heart my suggestion is to have binary_streams built based on some transport mechanism, and have that mechanism (whatever it is) highly visible.

You're confusing me with the binary_iostreams author. I didn't build any library.

Elaborating on my previous post, here's a quick diagram I drew of the system I envision. http://windmuehlgasse.getdesigned.at/architecture.png

Your diagram is reasonable. I have several comments about issues relating to this design, some of which you have suggested: It needs to be considered how an asynchronous I/O interface will fit into this framework. It also needs to be considered how operations like seeking fit into this framework. For efficiency reasons, particularly in supporting seeking, it is probably important that buffering filters can be applied directly at the character level as well as at the byte level. It is also probably important to support stackable character sources and sinks in a similar way that stackable byte sources are to be supported. (For instance, the user may want to use std::vector<character_type> or, more generally, an iterator range over a sequence of character_type, as a source of characters.) -- Jeremy Maitin-Shepard

Ares Lagae wrote:

* classes binary_istringstream, binary_ostringstream and binary_stringstream

The classes binary_istringstream, binary_ostringstream and binary_stringstream allow to read/write to a string that serves as buffer for the binary data. (althought a streambuf that writes to std::vector<unsigned char> would be more suited).

Indeed, basing this off basic_stringbuf (I suppose this is what you're doing) is most peculiar for binary data. Is there a specific reason why you don't want to write a stream buffer backed by a std::vector? These babies aren't particularly hard to write.

* << and >> operators for string types

The stream classes now only support fundamental data types. I am not sure how and wether to handle strings. For example, a regular char* string can be output as a null terminated array of char's, or as a size and a non null-terminated array of char's. Although I prefer the first solution (more portable, no problems with the size of the size type in the second solution), I am not sure if the library should make such a decision.

I fully agree with this decision. Writing strings is IMO a serialization issue and should not be part of a low-level binary I/O module.

* compression

I am also planning on adding a compressing_streambuf class, that can be used in both the IOStreams and the binary_iostreams library.

What kind of compression? Would it depend on external libraries? (zlib, libbzip2, ...) Is it worth the effort when Boost.IOStreams already supports compression?

* design

I agree with a lot of the comments on design of I/O libraries in the previous posts. However, I do not have the time to design and implement an improved I/O system for C++,

Would you be willing to help me in doing so? Sebastian Redl