On Sun, Apr 24, 2011 at 3:18 PM, Vicente BOTET <vicente.botet@wanadoo.fr> wrote:

Hi,

...

...

int main() { fstream f("binary_stream_example.dat", std::ios_base::trunc | std::ios_base::in | std::ios_base::out | std::ios_base::binary);

int32_t x = 0x01020304; int32_t y = 0;

f << bin(x); // write 4 bytes f.seekg(0); f >> bin(y); // read 4 bytes

BOOST_ASSERT(x == y);

return 0; }

I guess that we can not mix binary and non binary formats. How a binary stream woks for non binary codes?

It works fine, as long as the formats are such that there is never a question as to what you are dealing with. You can't do line oriented operations on portions that have binary values that might be mistaken for newline characters.

I don·t know if a manipulator that changes the mode globaly can be defined so the following will work

f << bin << x1 << x2; // write 4+4 bytes f.seekg(0); f >> bin >> y1 >> y2; // read 4+4 bytes

I share Steven's concern that bin in this usage could be misleading. I also don't see how to do that without modifying the standard library. bin() isn't meant to replace read() or write(); they still have their place, and can be used to implement read() or write() templates that can handle ranges or structs, among other things. I've done that several times, but don't have a Boost quality implementation ready just yet. --Beman

On 20:59, Wouter Van Alboom wrote:

...

Is there interest in this for Boost?

Yes, this can be usefull.

...

It seems way too small and simple to be a whole library itself.

Should it also address Endianness?

I too think it would be very useful, especially if endianess would be addressed. Regards, Anders Dalvander -- WWFSMD?

On 25/04/11 07:01, Michael Caisse wrote:

On 04/24/2011 06:43 PM, Marsh Ray wrote:

...

On 04/24/2011 07:12 AM, Beman Dawes wrote:

...

Is there interest in this for Boost?

Rather than a machine and compiler-dependent binary IO facility based on iostreams, I'd rather have a simple, minimal, portable binary streaming interface that handled endianness and alignment well for wire-representation defined structures.

I use Spirit Qi/Karma for this exact thing. When paired with Asio it can be a very convenient solution.

As I recall spirit uses the proposed boost endian library underneath for this which of course causes a problem if you are using boost endian directly yourself (I add it to my current tree) since spirit hasn't placed endian into a detail namespace. Jamie

On 04/28/2011 10:15 PM, Scott McMurray wrote:

On Sun, Apr 24, 2011 at 18:43, Marsh Ray<marsh@extendedsubset.com> wrote:

...

Rather than a machine and compiler-dependent binary IO facility based on iostreams, I'd rather have a simple, minimal, portable binary streaming interface that handled endianness and alignment well for wire-representation defined structures.

<iostream> brings in a huge header tree with 99% unnecessary functionality for binary protocols. The more efficient the headers, the better. Functions that didn't need to read could just include the header for writing, and vice-versa.

Would you be fine with it working on streambufs? That way you'd only need<streambuf>...

The last time I messed with streambufs (some years ago) I came to the conclusion that they could not be accessed multithreadedly in a natural way. I tried to define my own custom ostream channel and it took hundreds of lines of code to get something that just functional and not optimized. IIRC, streamsize was defined to be a signed int and not 64-bit clean. But after preprocessing, <streambuf> adds only 1320 non-comment non-blank lines in addition to the 6529 I get just from including <string> which I don't try to live without. Going all the way with <fstream> and <iomanip> it come to 13783 NCNB lines, although they take just 200 ms to parse on g++ 4.6.0. Although I complain about header size and compile time, my problem is probably just conceptual. Probably I'm subconsciously terrified of locale dependencies or something. All I really need is: // Binary write operation. May block. // Throws something derived from std::exception if output is // known to have failed. typedef boost::function< void(uint8_t const * begin, uint8_t const *end) > write_data_fn_t; So my code can write: void some_code_emitting_bytes(write_data_fn_t data_out) { vector<uint8_t> buf(256, 0); // ... code involving buf ... data_out(&buf[0], &buf[0] + buf.size()); } Actually, the other day I wrote something slightly more complicated. It went like this: // A function object representing write operations on an output stream. // // Intended to be an easy to use interface code needing to write data. // Several overloads are provided for easy writing of common source data types. // // It should be cheap to copy, so functions may accept it as a non-ref parameter. // // This type is also intended to be easy to construct on-the-fly at the call site, // even implicitly. The caller need only supply a function-like object taking a // begin and one-past-end char pointer pair and throwing something derived from // std::exception on error. // The implementation should avoid gratuitous copy operations, but callers may // still wish to attempt to write data in reasonably-sized chunks to avoid // excessive function call overhead. struct write_data_fxr { //----- Construction. // Construct this type from an even simpler function object interface. // Intended to provide an implicit conversion from compatible functions. typedef boost::function<void (char const * beg, char const * end)> write_data_fn_t; write_data_fxr(write_data_fn_t fn); //----- Writing data. // Even though these operations don't modify the object per se, they are declared // non-const on the theory that they are non-idempotent with respect to the // underlying output stream. // These operations throw something derived from std::exception to indicate error. // Whether or not the operation is re-tryable or not is unspecified. // void operator () (std::string const &); void operator () (std::vector<signed char> const &); void operator () (std::vector<unsigned char> const &); void operator () ( signed char const * beg, signed char const * end); void operator () (unsigned char const * beg, unsigned char const * end); private: write_data_fn_t fn_; }; Allowing my code to look like: void some_code_emitting_bytes(write_data_fxr data_out) { vector<uint8_t> buf(256, 0); // ... code involving buf ... data_out(buf); } - Marsh

On 25/04/11 14:08, Daniel James wrote:

On 24 April 2011 13:12, Beman Dawes<bdawes@acm.org> wrote:

...

It seems way too small and simple to be a whole library itself. Are there any ideas where it should live and what namespace it should be in?

'boost/io' seem like the right place. http://www.boost.org/libs/io/doc/ says, "This sub-library should contain various items to use with/for the standard I/O library".

+1

On 29.04.2011 11:52, Mathias Gaunard wrote:

...

int main() { fstream f("binary_stream_example.dat", std::ios_base::trunc | std::ios_base::in | std::ios_base::out | std::ios_base::binary);

int32_t x = 0x01020304; int32_t y = 0;

f<< bin(x); // write 4 bytes f.seekg(0); f>> bin(y); // read 4 bytes

I think it's a bad idea.

If you want to do that kind of thing, you should use the streambuf abstraction directly, along with its sgetn and sputn member functions. It will be much more efficient.

I am not agree. (If you *want* to use the streambuf directly, you should :) But don't force everyone to use this low-level tool.) I think binary stream IO shouldn't be implemented on the manipulator-like basis, an independent raw_stream class with minimal and full functionality is better. Then below I don't tied to proposed syntax (on the manipulator-like basis). A stream provide automatic "type deduction" / type safety, therefore an user shouldn't specify the type by hand: double d; char c; raw_stream raw; raw << d << c; raw >> d >> c; but with streambuf it should: std::streambuf* buf = . . . ; double d = . . . ; buf->sputn(reinterpret_cast<char const*>(&d), sizeof( double )); int i = . . . ; buf->sputn(reinterpret_cast<char const*>(&i), sizeof( int )); It's error-prone; and the user to simplify its job should write template wrapper like: template<typename T> void put(std::streambuf* buf, T value) { buf->sputn(reinterpret_cast<char const*>(&value), sizeof( T )); } template<typename T> T get(std::streambuf* buf) { T value; buf->sgetn(reinterpret_cast<char*>(&value), sizeof( T )); return value; } ..but a similar "type deduced" / type-safe code already written for stream (in more convenient manner) and placed in library for general use. sgetn() / sputn() functions doesn't support endianess directly, the user should use something like reverse_iterator<char*> explicitly. In stream's environment endianess can be implemented on the manipulator basis: raw_stream raw; raw << ordering::little_endian << 1.0 << 'X' << 127; double d; char c; int i; raw >> d >> c >> ordering::big_endian >> i; std::cout << d << " - " << c << " - " << std::hex << std::showbase << i << std::endl; // out: 1 - X - 0x7f000000 // on a little-endian machine Streams are highly extensible with a well-known to every C++ programmer interface; library can provide shift operators overloaded for std::vector<>, std::basic_string<>, std::deque<> and std::list<> in a separate library's header: std::vector<int> v; raw << v; // trivial implementation raw >> v; raw << some_user_type(); // if operator << (raw_stream&, some_user_type) exists and found via ADL With streambuf it's not so convenient. -- - thank you for your indulgence about my poor english :)

On 30.04.2011 14:32, Mathias Gaunard wrote:

On 29/04/2011 13:13, Max Sobolev wrote:

...

In stream's environment endianess can be implemented on the manipulator basis:

raw_stream raw; raw << ordering::little_endian << 1.0 << 'X' << 127; double d; char c; int i; raw >> d >> c >> ordering::big_endian >> i;

And what would you construct raw_stream from?

from std::string or nothing as std::stringstream does. ..At the moment I have some draft of the raw_stream implementation: its interface is very close to std::stringstream's one, and (now) raw_stream use std::stringbuf as its stream-buffer.

On Fri, Apr 29, 2011 at 3:52 AM, Mathias Gaunard <mathias.gaunard@ens-lyon.org> wrote:

On 24/04/2011 14:12, Beman Dawes wrote:

...

There have been many requests on Boost and various C++ newsgroups for unformatted binary I/O. For example, in 2003 Neal Becker wrote:

     I wonder if anyone has code for implementing unformatted I/O? What I have in mind is      for the simple case where the application that reads data knows the data types, so      this is not as complicated as the general marshalling situation.

This proposal provides a very simple solution that works with standard library input and output streams. The one caveat is that the stream must be opened with filemode std::ios_base::binary to avoid certain data values being treated as line endings.

int main() {   fstream f("binary_stream_example.dat",     std::ios_base::trunc | std::ios_base::in | std::ios_base::out | std::ios_base::binary);

  int32_t x = 0x01020304;   int32_t y = 0;

  f<<  bin(x);  // write 4 bytes   f.seekg(0);   f>>  bin(y);   // read 4 bytes

I think it's a bad idea.

If you want to do that kind of thing, you should use the streambuf abstraction directly, along with its sgetn and sputn member functions. It will be much more efficient.

Interesting. But what specifically do you think is a bad idea? The whole idea of doing binary (aka raw) I/O to a stream or just the way I have implemented it using read() and write()? --Beman