Re:Re:Re: serialization library #17
Neal D. Becker wrote:
Thanks. Actually, though, what I'm talking about is a different archive type, not a data structure. Right now binary data is saved/restored using read/write. mmap is potentially much more efficient for large data structures. OTOH, there is one big problem for it, which is that mmap can only be applied to one contiguous memory region. I believe you would have to serialize one object to one file, and if you want more objects you would need multiple files to map them to. Still, might be interesting.
Hmmm - wouldn't be better to implement a new derivation of std::;basic_streambuf for memory mapped i/o. This would be useful for a lot more than serialization. Of course, all the archive already provided are implemented in terms of streams so they would automatically inherit all the features of memory mapped i/o. The whole topic of custom stream_buf versions or better yet, composable stream_buf adaptors has been under appreciated in my view. I know there was some discussion and work done. I don't know what happened. I suspect it turned out to be "too big" to finish to boost standards in the amount of time that the developer could allocate. I do much appreciate feedback on the serialization library on this list. It is very helpful and encouraging. Note that others have made significant efforts well. These are people that have been experimenting with the library in their own projects. They have been very patient and helpful. Among others I would mention Pavel Vozenilek, Vladimir Prus, David Tonge, and Jeff Finn. Robert Ramey
The whole topic of custom stream_buf versions or better yet, composable stream_buf adaptors has been under appreciated in my view. I know
"Robert Ramey" <ramey@rrsd.com> wrote in message: there was
some discussion and work done. I don't know what happened. I suspect it turned out to be "too big" to finish to boost standards in the amount of time that the developer could allocate.
Sometime last summer I posted a small library for creating custom streams and stream buffers. You suggested I extended it to incorporate filtering, which I did. It's in the formal review queue. I'm not sure it's what you had in mind, but you can take a look at the online version here: http://www.xmission.com/~turkanis/iostreams/. One type of streambuf I chose not to support, because I didn't think it would be used much, was a streambuf whose underlying sequence of characters is presented as a fixed in-memory array. (Daryle Walker provdies one in his i/o library.) The application with memory mapped files seems important enough that I think I should add such a facility, which would be easy to do. I'm trying to decide the most elegant way. Jonathan
On Tue, 2 Mar 2004 12:38:41 -0700, Jonathan Turkanis wrote
Sometime last summer I posted a small library for creating custom streams and stream buffers. You suggested I extended it to incorporate filtering, which I did. It's in the formal review queue. I'm not sure it's what you had in mind, but you can take a look at the online version here: http://www.xmission.com/~turkanis/iostreams/.
One type of streambuf I chose not to support, because I didn't think it would be used much, was a streambuf whose underlying sequence of characters is presented as a fixed in-memory array. (Daryle Walker provdies one in his i/o library.) The application with memory mapped files seems important enough that I think I should add such a facility, which would be easy to do. I'm trying to decide the most elegant way.
I think it would be an excellent testcase for your library. In case you missed it earlier in the thread there is already a library from Craig Henderson in the boost-sandbox (memmap) that portably allocates memory mapped files. You should be able to build on top of that work.... Jeff
"Jeff Garland" <jeff@crystalclearsoftware.com> wrote in message news:20040302201415.M97761@crystalclearsoftware.com...
On Tue, 2 Mar 2004 12:38:41 -0700, Jonathan Turkanis wrote
One type of streambuf I chose not to support, because I didn't
think
it would be used much, was a streambuf whose underlying sequence of characters is presented as a fixed in-memory array. (Daryle Walker provdies one in his i/o library.) The application with memory mapped files seems important enough that I think I should add such a facility, which would be easy to do. I'm trying to decide the most elegant way.
I think it would be an excellent testcase for your library. In case you missed it earlier in the thread there is already a library from Craig Henderson in the boost-sandbox (memmap) that portably allocates memory mapped files. You should be able to build on top of that work....
Thanks, this looks like a good idea. I'll just have to write some connective tissue to integrate Craig's work into my framework. Jonathan
Jonathan Turkanis wrote:
"Jeff Garland" <jeff@crystalclearsoftware.com> wrote in message news:20040302201415.M97761@crystalclearsoftware.com...
On Tue, 2 Mar 2004 12:38:41 -0700, Jonathan Turkanis wrote
One type of streambuf I chose not to support, because I didn't
think
it would be used much, was a streambuf whose underlying sequence of characters is presented as a fixed in-memory array. (Daryle Walker provdies one in his i/o library.) The application with memory mapped files seems important enough that I think I should add such a facility, which would be easy to do. I'm trying to decide the most elegant way.
I think it would be an excellent testcase for your library. In case you missed it earlier in the thread there is already a library from Craig Henderson in the boost-sandbox (memmap) that portably allocates memory mapped files. You should be able to build on top of that work....
Thanks, this looks like a good idea. I'll just have to write some connective tissue to integrate Craig's work into my framework.
On a slightly different thought: if you use a special allocator with a container, such as vector, that is mmap-based, then serialization would be trivial. I wonder if we can use this?
Neal D. Becker wrote:
On a slightly different thought: if you use a special allocator with a container, such as vector, that is mmap-based, then serialization would be trivial. I wonder if we can use this?
This is the moral equivalent of fwrite'ing the vector to disk. It can work... for a while... if you don't upgrade the compiler and never change the object layout, or if you don't mind all your files being invalidated. Some compilers use such a scheme for their precompiled headers, I believe. You don't really need a serialization library for this, of course. ;-)
On Wed, 3 Mar 2004 15:08:52 +0200, Peter Dimov wrote
Neal D. Becker wrote:
On a slightly different thought: if you use a special allocator with a container, such as vector, that is mmap-based, then serialization would be trivial. I wonder if we can use this?
You can, but only very carefully. The problem is that pointers are not handled correctly. You cannot assume that an mmap file is reloaded in the same memory location and hence pointers between objects will be wrong (including vtable pointers). This is why object databases like ObjectStore and Objectivity 'swizzle' pointer references when they reload a page from disk.
This is the moral equivalent of fwrite'ing the vector to disk. It can work... for a while... if you don't upgrade the compiler and never change the object layout, or if you don't mind all your files being invalidated. Some compilers use such a scheme for their precompiled headers, I believe.
Exactly. So actually I think serializing into the memory mapped file provides a very interesting data store for a broad range of purposes. Using allocators is much more dangerous. Jeff
participants (5)
-
Jeff Garland -
Jonathan Turkanis -
Neal D. Becker -
Peter Dimov -
Robert Ramey