Interest in a simple buffer abstraction?
Hi, While working on the Boost profile for ODB C++ ORM[1], I was looking for a Boost type to map the database BLOB type to. I hoped to find something that would encapsulate the memory management and provide basic operations like copy (memcpy), append, set all bytes in a buffer to a specific value (memset), etc. Something that I could use like this: class packet { address to_; boost::buffer payload_; public: packet (const address& to, const void* data, std::size_t size) to_ (to), data_ (data, size) { } const boost::buffer& payload () const { return payload_; } }; class socket { public: void send (const packet& p) { ... s_.send (p.payload ().data (), p.payload ().size ()) } }; I found a number of buffer types in asio (asio/buffer.hpp) but they don't do memory management. I also found auto_buffer in implementation details of signal2 (signal2/detail/auto_buffer.hxx) and buffer in implementation details of iostream (iostream/detail/buffer.hxx). But they appear to be purpose-built for the libraries in question. So, unless I missed something obvious, I would like to propose a simple buffer abstraction for inclusion into Boost. Here are some additional motivations: - Seeing that there is a number of library-specific implementations, it is quite clear that such a class is often needed. - If we have a "top-level" buffer class, various i/o libraries could use it in their interfaces which would lead to better library interoperability (i.e., one could receive data from a socket into a buffer using asio and then pass that buffer to iostream or interprocess without any conversions). - A top-level buffer class will be fairly light-weight (perhaps even header-only). Right now, if I am to use a buffer implementation from say, asio, I need to install and potentially link to that library. If the community believes this is a good idea, I will next send the buffer interface for review/discussion. Let me know what you think. [1] http://www.codesynthesis.com/products/odb/ Boris
On 27/01/2011 11:53, Boris Kolpackov wrote:
Hi,
While working on the Boost profile for ODB C++ ORM[1], I was looking for a Boost type to map the database BLOB type to. I hoped to find something that would encapsulate the memory management and provide basic operations like copy (memcpy), append, set all bytes in a buffer to a specific value (memset), etc. Something that I could use like this:
class packet { address to_; boost::buffer payload_;
public: packet (const address& to, const void* data, std::size_t size) to_ (to), data_ (data, size) { }
const boost::buffer& payload () const { return payload_; } };
class socket { public: void send (const packet& p) { ... s_.send (p.payload ().data (), p.payload ().size ()) } };
How is that any different from std::vector<char>?
Hi Mathias, Mathias Gaunard <mathias.gaunard@ens-lyon.org> writes:
How is that any different from std::vector<char>?
Ah, that's right, I forgot to cover this case. The main problem with using vector<char> as a binary buffer is the unnatural interface. Provided: typedef std::vector<char> vbuffer; Compare: buffer vbuffer -------------------------------------------------------------------- | buffer b (data, size); | vbuffer b (data, data + size); | const char* p = b.data (); | const char* p = &b[0]; | b.copy (data, size); | b.assign (data, data + size); | b.append (data, size); | b.insert (b.end (), data, data + size); Also, you cannot pass the data to vector as void* which will be possible with the buffer class, for example: void* data; ... buffer b (data, size); // Ok. vbuffer vb (data, data + size); // Error. Plus, some natural to have functions are not available in vector, for example: buffer b (size, capacity); buffer b (data, size, capacity); Also, for a binary buffer it is nice to be able to assume ownership of an existing raw buffer and to detach the underlying raw buffer. Finally, the implementation of std::vector<char> may not be as efficient (e.g., using loops to copy elements instead of memcpy, etc). So, to summarize, yes, it is possible to use vector as a binary buffer (and a lot of people do so), but a separate buffer class will be able to provide a more natural interface and also some additional functionality. Boris -- Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog Compiler-based ORM system for C++ http://codesynthesis.com/products/odb Open-source XML data binding for C++ http://codesynthesis.com/products/xsd XML data binding for embedded systems http://codesynthesis.com/products/xsde
On 27/01/2011 13:37, Boris Kolpackov wrote:
const char* p = b.data (); | const char* p =&b[0];
That's actually undefined behaviour if b doesn't have at least 1 element. In C++0x std::vector has a data() member just like your buffer type.
b.copy (data, size); | b.assign (data, data + size);
How about std::vector<char> a; std::vector<char> b = a;
| b.append (data, size); | b.insert (b.end (), data, data + size);
Appending is more of a string thing. Are you sure you want this?
Also, you cannot pass the data to vector as void* which will be possible with the buffer class, for example:
void* data; ... buffer b (data, size); // Ok. vbuffer vb (data, data + size); // Error.
Beware of the strict aliasing rules.
Plus, some natural to have functions are not available in vector, for example:
buffer b (size, capacity); buffer b (data, size, capacity);
Also, for a binary buffer it is nice to be able to assume ownership of an existing raw buffer and to detach the underlying raw buffer.
And how is that allowed by your interface? Also, how do you take care of how the memory is allocated when copying said buffer?
Finally, the implementation of std::vector<char> may not be as efficient (e.g., using loops to copy elements instead of memcpy, etc).
std::vector can detect PODs.
Hi Mathias, Mathias Gaunard <mathias.gaunard@ens-lyon.org> writes:
In C++0x std::vector has a data() member just like your buffer type.
True, thought, that's not the only problem with vector's interface when used as buffer. Probably just the ugliest.
b.copy (data, size); | b.assign (data, data + size);
How about std::vector<char> a; std::vector<char> b = a;
Not sure I follow what you are trying to say here.
b.append (data, size); | b.insert (b.end (), data, data + size);
Appending is more of a string thing. Are you sure you want this?
I don't see why not, but I am open to change my mind if there is a good reason not to provide it.
void* data; ... buffer b (data, size); // Ok. vbuffer vb (data, data + size); // Error.
Beware of the strict aliasing rules.
Seeing that we will just pass the pointer to memcpy, I think we will be safe.
Also, for a binary buffer it is nice to be able to assume ownership of an existing raw buffer and to detach the underlying raw buffer.
And how is that allowed by your interface?
There would a special constructor for that: buffer (char* data, size_t size, size_t capacity, bool assume_ownership);
Also, how do you take care of how the memory is allocated when copying said buffer?
You probably mean "freeing said buffer" instead of "copying said buffer". The simplest approach would be to stipulate that such a buffer should be allocated with operator new[] and freed with operator delete[]. Boris -- Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog Compiler-based ORM system for C++ http://codesynthesis.com/products/odb Open-source XML data binding for C++ http://codesynthesis.com/products/xsd XML data binding for embedded systems http://codesynthesis.com/products/xsde
On 27/01/2011 15:10, Boris Kolpackov wrote:
You probably mean "freeing said buffer" instead of "copying said buffer". The simplest approach would be to stipulate that such a buffer should be allocated with operator new[] and freed with operator delete[].
No, I did mean copy, as in when you copy the buffer object.
Hi Mathias, Mathias Gaunard <mathias.gaunard@ens-lyon.org> writes:
No, I did mean copy, as in when you copy the buffer object.
In this case I don't see a problem. The buffer class will provide the "deep copy" semantics, just like std::vector. Boris -- Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog Compiler-based ORM system for C++ http://codesynthesis.com/products/odb Open-source XML data binding for C++ http://codesynthesis.com/products/xsd XML data binding for embedded systems http://codesynthesis.com/products/xsde
On 27/01/2011 15:10, Boris Kolpackov wrote:
You probably mean "freeing said buffer" instead of "copying said buffer". The simplest approach would be to stipulate that such a buffer should be allocated with operator new[] and freed with operator delete[].
Oh and calling delete[] (or delete for that matter) on a different type than the one that was used for new[]/new (or a non-base type if the destructor is virtual) is undefined behaviour.
Mathias Gaunard wrote:
On 27/01/2011 13:37, Boris Kolpackov wrote:
b.copy (data, size); | b.assign (data, data + size);
How about std::vector<char> a; std::vector<char> b = a;
The point was succinctness for the purpose at hand.
b.append (data, size); | b.insert (b.end (), data, data + size);
Appending is more of a string thing. Are you sure you want this?
Imagine reading several times from a socket to try to fill a buffer before returning. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
Boris Kolpackov-2 wrote:
So, unless I missed something obvious, I would like to propose a simple buffer abstraction for inclusion into Boost. Here are some additional motivations:
- Seeing that there is a number of library-specific implementations, it is quite clear that such a class is often needed.
- If we have a "top-level" buffer class, various i/o libraries could use it in their interfaces which would lead to better library interoperability (i.e., one could receive data from a socket into a buffer using asio and then pass that buffer to iostream or interprocess without any conversions).
- A top-level buffer class will be fairly light-weight (perhaps even header-only). Right now, if I am to use a buffer implementation from say, asio, I need to install and potentially link to that library.
If the community believes this is a good idea, I will next send the buffer interface for review/discussion.
Let me know what you think.
I'm working on a frame concept, which is something like a bidirectional buffer. It allows to prepend and append binary data, and obtain/remove the header and the trailer. In addition we can split a frame in fragments and join fragments to make a frame. All these features are needed when working with protocol stacks. Best, Vicente -- View this message in context: http://boost.2283326.n4.nabble.com/Interest-in-a-simple-buffer-abstraction-t... Sent from the Boost - Dev mailing list archive at Nabble.com.
On Thu, Jan 27, 2011 at 10:49 PM, Vicente Botet <vicente.botet@wanadoo.fr> wrote:
I'm working on a frame concept, which is something like a bidirectional buffer. It allows to prepend and append binary data, and obtain/remove the header and the trailer.
In addition we can split a frame in fragments and join fragments to make a frame.
All these features are needed when working with protocol stacks.
+1 Something like this is very welcome indeed. -- Dean Michael Berris about.me/deanberris
On Thu, Jan 27, 2011 at 6:49 AM, Vicente Botet <vicente.botet@wanadoo.fr> wrote:
I'm working on a frame concept, which is something like a bidirectional buffer. It allows to prepend and append binary data, and obtain/remove the header and the trailer.
In addition we can split a frame in fragments and join fragments to make a frame.
All these features are needed when working with protocol stacks.
Best, Vicente
You'll have a winner if you expand that to allow bidirectional iteration of both bytes and fragments (and then a fragment is just a raw array giving random-access iteration within them). VERY useful for high-performance incremental parsing, I would definitely give my +1 for boost inclusion. -- Cory Nelson http://int64.org
Cory Nelson wrote:
On Thu, Jan 27, 2011 at 6:49 AM, Vicente Botet <vicente.botet@wanadoo.fr> wrote:
I'm working on a frame concept, which is something like a bidirectional buffer. It allows to prepend and append binary data, and obtain/remove the header and the trailer.
In addition we can split a frame in fragments and join fragments to make a frame.
All these features are needed when working with protocol stacks.
You'll have a winner if you expand that to allow bidirectional iteration of both bytes and fragments (and then a fragment is just a raw array giving random-access iteration within them). VERY useful for high-performance incremental parsing, I would definitely give my +1 for boost inclusion.
I wanted to hide whether the memory is continuous or not. Only when the user needs continuous memory the library will ensure it on the resulting data. Of course this continuous memory will have random access. Could you give an example of how you want to use the features you seem to need? This will help me to see if the current interface is enough or if it must be extended. Thanks in advance, Vicente -- View this message in context: http://boost.2283326.n4.nabble.com/Interest-in-a-simple-buffer-abstraction-t... Sent from the Boost - Dev mailing list archive at Nabble.com.
On Sat, Jan 29, 2011 at 2:59 PM, Vicente Botet <vicente.botet@wanadoo.fr> wrote:
Cory Nelson wrote:
On Thu, Jan 27, 2011 at 6:49 AM, Vicente Botet <vicente.botet@wanadoo.fr> wrote:
I'm working on a frame concept, which is something like a bidirectional buffer. It allows to prepend and append binary data, and obtain/remove the header and the trailer.
In addition we can split a frame in fragments and join fragments to make a frame.
All these features are needed when working with protocol stacks.
You'll have a winner if you expand that to allow bidirectional iteration of both bytes and fragments (and then a fragment is just a raw array giving random-access iteration within them). Â VERY useful for high-performance incremental parsing, I would definitely give my +1 for boost inclusion.
I wanted to hide whether the memory is continuous or not. Only when the user needs continuous memory the library will ensure it on the resulting data. Of course this continuous memory will have random access.
Could you give an example of how you want to use the features you seem to need? This will help me to see if the current interface is enough or if it must be extended.
I've designed a similar buffer class before, with I/O and incremental parsing in mind. In addition to what you describe, it also had reserve(size_t), commit(size_t), release(size_t), and access to the reserved area. I'm still not 100% sure if this type of access is appropriate for a general buffer, but allowing it would not cost anything to those who aren't using it and it would reduce a ton of code duplication for a specific io::buffer, since they're basically the same but with the internals a bit more exposed. 1) Incremental parsing. enum result { ok, need_more }; template<typename Iterator> result parse(Iterator first, Iterator last); I can call parse() on the first fragment (just using pointers for iterators) for high performance. Only if the first fragment doesn't contain enough to parse (returning need_more) will I call parse() on the entire buffer, which will be slower (like dequeue iterators) but rare. If that returns need_more still, then I'll perform more I/O to fill the buffer with more data. Think of, eg. HTTP or XML, where the individual bits (like a header or <element>) are quite small but you'll want to read in large page-sized fragments to reduce I/O calls. A single fragment will hold a lot of things and there's no reason to use slow iterators to get them. 2) Scatter/gather I/O. I will typically have a pool of page-sized fragments. A slow stream might only work on one or two fragments at a time, but for a fast one I might want to reduce I/O calls and work with a lot of fragments at once. If a stream is coming in fast, I can reserve() 64KB worth of fragments, request a scatter into the reserved area, then commit() whatever amount of data it actually gave me. Whatever didn't get committed will still be usable for the next I/O. I then release() however much data I was able to parse. This can be implemented pretty cleanly with hierarchical iterators, like so: class fragment { T* begin(); T* end(); }; class iterator { fragment& fragment(); }; class buffer { iterator begin(); // go over committed range only. iterator end(); iterator reserved_begin(); // go over reserved (uninitialized/unconstructed) range only. iterator reserved_end(); iterator to_iterator(fragment &f, T* iter); // creates a slower iterator from a fragment iterator. size_type size(); // committed size. size_type reserved_size(); size_type capacity(); // size() + reserved_size(). size_type reserve(size_type minsize); // ensures there are at least 'minspace' elements reserved, and returns the new reserved_size(). size_type commit(size_type size); // commits 'size' elements from the reserved space, returns the new size(). size_type release(size_type size); // releases 'size' elements from committed space, returns the new size(). }; -- Cory Nelson http://int64.org
Hi Vicente, Vicente Botet <vicente.botet@wanadoo.fr> writes:
I'm working on a frame concept, which is something like a bidirectional buffer. It allows to prepend and append binary data, and obtain/remove the header and the trailer.
In addition we can split a frame in fragments and join fragments to make a frame.
That sounds like a multi-block buffer, i.e., a buffer that instead of one contiguous memory block manages a sequence of such blocks. While this would allow more efficient resizing, prepending, etc., the interface becomes somewhat more complicated. But it can be another implementation, in addition to the simple buffer. I didn't want to mention this possibility initially in order to keep things simple, but it seem people here are more interested in the complex than simple ;-). Boris -- Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog Compiler-based ORM system for C++ http://codesynthesis.com/products/odb Open-source XML data binding for C++ http://codesynthesis.com/products/xsd XML data binding for embedded systems http://codesynthesis.com/products/xsde
Boris Kolpackov-2 wrote:
Hi Vicente,
Vicente Botet <vicente.botet@wanadoo.fr> writes:
I'm working on a frame concept, which is something like a bidirectional buffer. It allows to prepend and append binary data, and obtain/remove the header and the trailer.
In addition we can split a frame in fragments and join fragments to make a frame.
That sounds like a multi-block buffer, i.e., a buffer that instead of one contiguous memory block manages a sequence of such blocks. While this would allow more efficient resizing, prepending, etc., the interface becomes somewhat more complicated. But it can be another implementation, in addition to the simple buffer.
A fragment has continuous memory and allows to prepend and append respecting its initial capacity. A frame can be non-continuous.
I didn't want to mention this possibility initially in order to keep things simple, but it seem people here are more interested in the complex than simple ;-).
I'm talking of what I have found as real needs in concrete projects. The simple buffer functionality you are presenting don't satisfies my needs. This doesn't mind that the abstraction don't have other uses, but instead of appending what I use to need is to prepend headers. Best, Vicente -- View this message in context: http://boost.2283326.n4.nabble.com/Interest-in-a-simple-buffer-abstraction-t... Sent from the Boost - Dev mailing list archive at Nabble.com.
Boris Kolpackov wrote:
I hoped to find something that would encapsulate the memory management and provide basic operations like copy (memcpy), append, set all bytes in a buffer to a specific value (memset), etc. Something that I could use like this:
class packet { address to_; boost::buffer payload_;
public: packet (const address& to, const void* data, std::size_t size) to_ (to), data_ (data, size)
^^^^^ payload_?
{ }
const boost::buffer& payload () const { return payload_; } }; [snip]
I would like to propose a simple buffer abstraction for inclusion into Boost. Here are some additional motivations:
- Seeing that there is a number of library-specific implementations, it is quite clear that such a class is often needed.
- If we have a "top-level" buffer class, various i/o libraries could use it in their interfaces which would lead to better library interoperability (i.e., one could receive data from a socket into a buffer using asio and then pass that buffer to iostream or interprocess without any conversions).
- A top-level buffer class will be fairly light-weight (perhaps even header-only). Right now, if I am to use a buffer implementation from say, asio, I need to install and potentially link to that library.
The idea has merit, but I wonder if an adaptor approach would be better. That is, permit construction of your buffer type from various other storage types, like std::vector, boost::array, char [], etc., and simply adapt them to a common interface for use by other code. Memory management would be outside the scope of the adaptor as it exists merely to overlay an abstraction on the specific storage medium in use, but that also eliminates the issues Mathias is raising. That restricts the API a bit from your vision, but appending, copying, iterators, etc. are still possible. Storage lifetime must exceed that of the adaptor of course. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
- A top-level buffer class will be fairly light-weight (perhaps even header-only). Right now, if I am to use a buffer implementation from say, asio, I need to install and potentially link to that library.
You don't need to link to asio to use its buffer API. If you can use all other Boost header only libs, you can use asio as well. Personally, I like the approach taken by asio, I wished it were used in more places (Boost,interprocess comes to mind). What's missing in asio's buffer interface that you would like to add? Inventing another unmanaged buffer abstraction seems not the right way to go IMO.
- christian
Hi Christian, Christian Holmquist <c.holmquist@gmail.com> writes:
What's missing in asio's buffer interface that you would like to add?
Memory management and common functions such as copy, append, etc. Asio buffers are just wrappers for the void*, size_t pair. Boris -- Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog Compiler-based ORM system for C++ http://codesynthesis.com/products/odb Open-source XML data binding for C++ http://codesynthesis.com/products/xsd XML data binding for embedded systems http://codesynthesis.com/products/xsde
On 28 January 2011 04:13, Boris Kolpackov <boris@codesynthesis.com> wrote:
Hi Christian,
Christian Holmquist <c.holmquist@gmail.com> writes:
What's missing in asio's buffer interface that you would like to add?
Memory management and common functions such as copy, append, etc. Asio buffers are just wrappers for the void*, size_t pair.
Then std::vector<char> is a better option. You wrote earlier that the code is a bit ugly when using the std::vector in certain circumstances, but I'm sure people will argue the same with a new invention. I don't understand why you say std::vector doesn't have copy and append? std::vector<char> a; std::vector<char> b; a = b; // copy a.insert(a.end(), b.begin(), b.end()); // append - Christian
Hi Robert, "Stewart, Robert" <Robert.Stewart@sig.com> writes:
The idea has merit, but I wonder if an adaptor approach would be better. That is, permit construction of your buffer type from various other storage types, like std::vector, boost::array, char [], etc., and simply adapt them to a common interface for use by other code.
This way I would have to write: class packet { packet (void* data, size_t size) : data_ (data_storage_, data, size) { } std::vector data_storage_; boost::buffer data_; }; Which i find quite hairy. Boris -- Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog Compiler-based ORM system for C++ http://codesynthesis.com/products/odb Open-source XML data binding for C++ http://codesynthesis.com/products/xsd XML data binding for embedded systems http://codesynthesis.com/products/xsde
Boris Kolpackov wrote:
"Stewart, Robert" <Robert.Stewart@sig.com> writes:
The idea has merit, but I wonder if an adaptor approach would be better.
This way I would have to write:
class packet { packet (void* data, size_t size) : data_ (data_storage_, data, size) { }
std::vector data_storage_;
std::vector<char>, I presume.
boost::buffer data_;
Why would you have both?
};
You either have this: packet::packet(void const * _data, size_t _size) : data_storage_(static_cast<char const *>(_data), static_cast<char const *>(_data) + _size) { } which means that the data is copied into data_storage_, or you have this: packet::packet(boost::buffer & _buffer) : data_(_buffer) { } which requires that some other code create a std::vector<char> or other container for the packet to use. This, like any type erasing mechanism, isolates packet from the actual storage type. The client can create a std::vector<char>, allocate an array of char on the stack, etc., and packet will work with them all. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
Hi Robert, "Stewart, Robert" <Robert.Stewart@sig.com> writes:
std::vector<char> data_store_; boost::buffer data_;
Why would you have both?
Because I want the packet to own the data and to be able to access it using a natural interface. With your approach I would have to write gratuitous code like this.
which requires that some other code create a std::vector<char> or other container for the packet to use.
This goes down the slippery slop of someone else owning the memory and making sure it is not gone while other objects still reference it. If that's the functionality you want, then why bother with the buffer class at all? Simply store a pair: const void*, size_t; it will be just as convenient. Boris -- Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog Compiler-based ORM system for C++ http://codesynthesis.com/products/odb Open-source XML data binding for C++ http://codesynthesis.com/products/xsd XML data binding for embedded systems http://codesynthesis.com/products/xsde
Boris Kolpackov wrote:
"Stewart, Robert" <Robert.Stewart@sig.com> writes:
std::vector<char> data_store_; boost::buffer data_;
Why would you have both?
Because I want the packet to own the data and to be able to access it using a natural interface. With your approach I would have to write gratuitous code like this.
No you wouldn't. As I showed in the portion of my reply that you cut, if your packet owns its data, then it would have a container of that data. The point of my idea is that you can create a packet type that *doesn't* own the data or even know how the data is stored because the adaptor would abstract the underlying storage type and provide a useful interface permitting the packet to append, copy, etc. I understand that your idea is for a type that owns the data. I'm proposing an alternative view of the problem.
which requires that some other code create a std::vector<char> or other container for the packet to use.
This goes down the slippery slop of someone else owning the memory and making sure it is not gone while other objects still reference it.
When you want performance, you don't want to copy data. Your model requires that the data be copied. Mine references data allocated elsewhere. As I noted, that does indeed mean that the reference mustn't outlive the allocation, but with that responsibility comes improved performance and greater flexibility.
If that's the functionality you want, then why bother with the buffer class at all? Simply store a pair: const void*, size_t; it will be just as convenient.
As I noted (in what you snipped), the adaptor type can still provide the additional functionality you had proposed: appending, copying, etc. You get none of that with such a pair. When sharing a buffer among libraries, using my suggestion, memory need only be allocated in one place whilst various functions from different libraries can all reference that memory using a type-erased, feature rich, standardized reference to it. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
Hi Robert, "Stewart, Robert" <Robert.Stewart@sig.com> writes:
No you wouldn't. As I showed in the portion of my reply that you cut, if your packet owns its data, then it would have a container of that data.
Yes, which would involve having a data member of the std::vector<char> type. Let me put it this way: if I want *both* to own the data and have a convenient interface for it, I would have to have two members, as I showed in my original code. Is that not so?
When you want performance, you don't want to copy data. Your model requires that the data be copied.
If you want unsafe sharing semantics, you can always pass/store references to the buffer instead of making a copy. For example: class packet { packet (const buffer& p): payload_ (p) {} buffer& payload_; }; You also have the option of safe sharing with something like shared_ptr. Boris -- Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog Compiler-based ORM system for C++ http://codesynthesis.com/products/odb Open-source XML data binding for C++ http://codesynthesis.com/products/xsd XML data binding for embedded systems http://codesynthesis.com/products/xsde
Boris Kolpackov wrote:
"Stewart, Robert" <Robert.Stewart@sig.com> writes:
No you wouldn't. As I showed in the portion of my reply that you cut, if your packet owns its data, then it would have a container of that data.
Yes, which would involve having a data member of the std::vector<char> type. Let me put it this way: if I want *both* to own the data and have a convenient interface for it, I would have to have two members, as I showed in my original code. Is that not so?
No. If that remains the problem you wish to solve -- and I'm trying to suggest an alternative, obviously -- then you would use something like your original proposal if you really dislike std::vector<char> or other alternatives.
When you want performance, you don't want to copy data. Your model requires that the data be copied.
If you want unsafe sharing semantics, you can always pass/store references to the buffer instead of making a copy. For example:
class packet { packet (const buffer& p): payload_ (p) {} buffer& payload_; };
You also have the option of safe sharing with something like shared_ptr.
Of course those are options, but then you're imposing a new container on clients of packet and on other libraries. Consider some existing code that uses a std::vector<char>. You would require that it be changed to use your buffer type. Suppose that there were other APIs that code calls expect a std::vector<char>. Code that uses your packet type and those other APIs would be forced to copy the data. With my adaptor suggestion, packet can use the adaptor type, the client can still create a std::vector<char>, use your packet type, and still use the other APIs. Extend that to other ways one can allocate a buffer. Suppose there's code that uses some API that allocates a buffer which it reclaims later, but exposes for client code to access. That buffer could also be adapted for use with packet, again without need to copy the data to a new data structure. Does this make sense? _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
Hi Robert, "Stewart, Robert" <Robert.Stewart@sig.com> writes:
Of course those are options, but then you're imposing a new container on clients of packet and on other libraries.
Consider some existing code that uses a std::vector<char>. You would require that it be changed to use your buffer type.
I think requiring users of your API to pass buffers as vector<char> only is crazy. Any existing code that tries to be universally usable will at least allow for the (void*, size_t) pair as the buffer representation which will work just fine with the buffer abstraction I am proposing.
Does this make sense?
Yes, it does. We are trying to solve orthogonal problems here. I want a simple and convenient buffer abstraction that will manage the memory for me and provide a natural interface. You are trying to find a way to work with various buffer representations (of which my buffer class is just one instance) in a uniform way. You are viewing this from the library designer's point of view ("How can I design my socket class so I don't have to deal with all these different buffer representations?"). I am think from the user's point of view ("What can I use to manage the buffer memory for me and make my code succinct?"). Boris -- Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog Compiler-based ORM system for C++ http://codesynthesis.com/products/odb Open-source XML data binding for C++ http://codesynthesis.com/products/xsd XML data binding for embedded systems http://codesynthesis.com/products/xsde
Boris Kolpackov wrote:
"Stewart, Robert" <Robert.Stewart@sig.com> writes:
Of course those are options, but then you're imposing a new container on clients of packet and on other libraries.
Consider some existing code that uses a std::vector<char>. You would require that it be changed to use your buffer type.
I think requiring users of your API to pass buffers as vector<char> only is crazy.
Crazy or not, such things are often done.
Any existing code that tries to be universally usable will at least allow for the (void*, size_t) pair as the buffer representation which will work just fine with the buffer abstraction I am proposing.
That interface is more error prone. With std::vector<char>, the memory and size are intrinsically related. However, that's beside the point. There's a great deal of extant code that uses various ways to allocate and pass buffers. Your buffer class provides slightly simpler syntax than std::vector<char>, but is non-standard, so I'm not sure it will gain much traction. Your examples of similar types in other Boost libraries suggest that something might be needed, but I suggest that those libraries should use something like my adaptor idea. If they were changed accordingly, then the onus falls on clients of those libraries to create a suitable buffer any way that makes sense to the client (and can be or has been adapted). If the library needs to create a buffer, the buffer can be of any type since the adaptor can be used everywhere it is referenced to provide the simpler, common interface. Obviously, my adaptor idea and your buffer type are not mutually exclusive, but if the adaptor provided the interface you wanted, then you wouldn't need to have your buffer type but could use any number of other types as appropriate to the context. Clearly, if you have a function or a class that creates and manipulates the buffer, then the adaptor approach is a little less direct as one would need both the container and the adaptor in the same scope (as you showed in an earlier variant of the example packet class). However, if the adaptor were the one way to get the nice interface you're after, and it provides a means of using many different containers with various functions and classes, including your packet class, doesn't that seem cleaner overall? _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
Hi Robert, "Stewart, Robert" <Robert.Stewart@sig.com> writes:
If they were changed accordingly, then the onus falls on clients of those libraries to create a suitable buffer any way that makes sense to the client (and can be or has been adapted).
BTW, do you have any ideas about how such an adapter can be implemented that is fast (virtual functions are probably not an option), easy to use, and extensible?
If the library needs to create a buffer, the buffer can be of any type since the adaptor can be used everywhere it is referenced to provide the simpler, common interface.
Of course, if the library needs to return a buffer, ownership issues arise again.
Clearly, if you have a function or a class that creates and manipulates the buffer, then the adaptor approach is a little less direct as one would need both the container and the adaptor in the same scope (as you showed in an earlier variant of the example packet class). However, if the adaptor were the one way to get the nice interface you're after, and it provides a means of using many different containers with various functions and classes, including your packet class, doesn't that seem cleaner overall?
I fail to see why we can't have both, seeing that they are not mutually exclusive. Boris -- Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog Compiler-based ORM system for C++ http://codesynthesis.com/products/odb Open-source XML data binding for C++ http://codesynthesis.com/products/xsd XML data binding for embedded systems http://codesynthesis.com/products/xsde
Boris Kolpackov wrote:
"Stewart, Robert" <Robert.Stewart@sig.com> writes:
If they were changed accordingly, then the onus falls on clients of those libraries to create a suitable buffer any way that makes sense to the client (and can be or has been adapted).
BTW, do you have any ideas about how such an adapter can be implemented that is fast (virtual functions are probably not an option), easy to use, and extensible?
Yes. I've done something similar for strings, which adapts a char * plus a length, an iterator pair, a std::vector<char>, etc., all of which are presumed to refer to an empty buffer initially, and provides for the ability to append and assign (within the initial capacity), provides iterators (char const *'s), capacity and length queries, etc. from a pointer, a length, and a capacity. To permit adaptation to other types would require defining a concept for extracting a capacity and pointer to the buffer start, such that a templated constructor can apply the appropriate concept functions to capture the needed values.
If the library needs to create a buffer, the buffer can be of any type since the adaptor can be used everywhere it is referenced to provide the simpler, common interface.
Of course, if the library needs to return a buffer, ownership issues arise again.
Maybe. It may be that the library alone knows the needed buffer size, in which case its interface could be complicated by requiring the user to ask for the size, create a buffer, and then call the function to load the buffer (which must assume or verify the size). In that case, yes, the library might need to allocate memory and return something that manages its lifetime. However, that could be a simple memory allocation managed by, say, boost::shared_ptr, and the caller could use the adaptor to get the desired high level interface. There are other ways of doing such things, but you get the idea.
Clearly, if you have a function or a class that creates and manipulates the buffer, then the adaptor approach is a little less direct as one would need both the container and the adaptor in the same scope (as you showed in an earlier variant of the example packet class).
I fail to see why we can't have both, seeing that they are not mutually exclusive.
I'm attempting to show that there may really be no need for your class. If the adaptor approach works nicely and provides better genericity, it seems like the better direction. You may yet find a compelling case for your idea, but given the other push back, I thought the adaptor approach might be ideal. _____ Rob Stewart robert.stewart@sig.com Software Engineer, Core Software using std::disclaimer; Susquehanna International Group, LLP http://www.sig.com IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
participants (7)
-
Boris Kolpackov -
Christian Holmquist -
Cory Nelson -
Dean Michael Berris -
Mathias Gaunard -
Stewart, Robert -
Vicente Botet