April 2006 - Boost - lists.preview.boost.org

[asio] Draft version uploaded to vault
by Christopher Kohlhoff 08 May '06

08 May '06

Hello all, I have just uploaded a draft version of Boost.Asio to the vault (in the "Input - Output" folder). I'm particularly interested in feedback on the handler-based custom memory allocation and IPv4/v6 protocol independence. Please note: this is not a properly tested release. Caveat emptor. It has the following known issues: - It uses getaddrinfo/getnameinfo, but these functions are not thread-safe on Mac OS X. They will be replaced by thread-safe emulation using getipnodebyname and getipnodebyaddr. - Although builds using MSVC or Borland C++ automatically emulate getaddrinfo/getnameinfo on Windows 2000 and earlier, MinGW does not. Therefore MinGW currently only supports Windows XP and Windows Server 2003 targets. - It has not been compiled or tested at all on Solaris. It has been tested on Mac OS X 10.4, Debian Sarge with 2.6 linux kernel and Windows XP. Cheers, Chris Changes since 0.3.6 =================== (Excluding minor bug fixes and in no particular order.) * Renamed demuxer to io_service. * All services inherit from a common base io_service::service. As part of this change, the demuxer::get_service() member template function has been replaced with three free template functions: use_service(), add_service() and has_service(). * Resource classes inherit from basic_io_object. Sockets, timers, etc inherit from basic_io_object<Service>. This class automatically obtains the correct service object from the io_service, and provides the typedefs and functions that all objects must provide (e.g. an io_service() member function for obtaining the associated io_service object). * All handler objects destroyed during io_service destruction. Previously, if you did not allow demuxer::run() to continue until it had no more work to do, and then destroyed the demuxer, uninvoked handlers would never be destroyed. All stored handler objects that are "owned" by the io_service or any associated services are now destroyed automatically from inside the io_service. This allows quick, clean shutdown of programs using io_service::interrupt(). This handler cleanup has been implemented using "two-phase" service shutdown. Every service that inherits from io_service::service must implement a virtual function called shutdown_service(). The service's implementation of this function is required to destroy all copies of handlers that it owns (directly or indirectly). The cleanup must be done in a separate phase prior to the destruction of the service objects, since the handler destructors may try to access other services. * Renamed locking_dispatcher to strand. A strand is defined as a strictly sequential invocation of event handlers (i.e. no concurrent invocation). Use of strands allows execution of code in a multithreaded program without the need for explicit locking (e.g. using mutexes). Strands may be either implicit or explicit, as illustrated by the following alternative approaches: - Calling io_service::run() from only one thread means all event handlers execute in an implicit strand, due to the io_service's guarantee that handlers are only invoked from inside run(). - Where there is a single chain of asynchronous operations associated with a connection (e.g. in a half duplex protocol implementation like HTTP) there is no possibility of concurrent execution of the handlers. This is an implicit strand. - An explicit strand is an instance of asio::strand. All event handler function objects need to be wrapped using asio::strand::wrap() or otherwise posted/dispatched through the asio::strand object. * Add macros to disable platform-specific implementations. The following macros can be used to disable platform-specific implementation code and revert to portable select-based asynchronous I/O emulation: - BOOST_ASIO_DISABLE_EPOLL - BOOST_ASIO_DISABLE_IOCP - BOOST_ASIO_DISABLE_KQUEUE * Handler-based custom memory allocation. Many asynchronous operations need to allocate an object to store state associated with the operation. For example, the Win32 implementation needs OVERLAPPED-derived objects to pass to Win32 API functions. Furthermore, programs typically contain easily identifiable chains of asynchronous operations. A half duplex protocol implementation (e.g. an HTTP server) would have a single chain of operations per client (receives followed by sends). A full duplex protocol implementation would have two chains executing in parallel. Programs should be able to leverage this knowledge to reuse memory for all asynchronous operations in a chain. Given a copy of a user-defined Handler object h, if asio needs to allocate memory associated with that handler it will execute the code: void* pointer = asio_handler_allocate(size, &h); Similarly, to deallocate the memory it will execute: asio_handler_deallocate(pointer, &h); These functions are located using ADL. The asio library provides default implementations of the above functions in the global namespace: void* asio_handler_allocate(size_t, ...); void asio_handler_deallocate(void*, ...); which are implemented in terms of ::operator new() and ::operator delete() respectively. The implementation guarantees that the deallocation will occur before the associated handler is invoked, which means the memory is ready to be reused for any new asynchronous operations started by the handler. Custom memory allocation support is not yet used throughout the asio implementation. In particular, the reactor-based emulation still uses dynamically allocated objects. It is fully implemented for the Win32-specific I/O completion port code. The custom memory allocation support has also been disabled for Borland C++, since it generated obscure compiler errors. See libs/asio/example/allocation/server.cpp for an example of how to use the handler-based custom memory allocation. * Allocator template parameter removed. Removing the Allocator template parameter reduces the number of core types in asio and makes it easier to develop custom services (i.e. you don't have to worry what Allocator object the io_service has been templated on). It should also make it simpler to create a shared or static library-based implementation. A combination of handler-based custom memory allocation, and the ability for the asio implementation to embed objects directly into object internals (e.g. socket implementations), should minimise dynamic memory allocations. If there is a demonstrable need, I may consider adding an io_service::service-based allocator to allow memory allocation to be further customised on a per-io_service-object basis. * Add support for building without thread support. POSIX platforms only. The Windows implementation still requires threads. * Async waits cancelled by deadline_timer expires_*() setters. Outstanding asynchronous wait operations are now automatically cancelled if you call expires_at() or expires_from_now() to modify the expiry time of a deadline_timer. The previous documented behaviour was that it was "undefined" to modify the expiry time while there were outstanding waits. * Support for reference-counted buffers. The Const_Buffers and Mutable_Buffers concepts have been modified to allow reference counted buffers to be used. In particular: - The {Const|Mutable}_Buffers::iterator typedef and non-const begin()/end() functions have been removed from the concepts' requirements. Implementations of the concepts now need only provide const_iterator support, and do not need to allow in-place modification of the elements. - The asio implementation guarantees that at least one copy of the buffers object will be maintained for as long as the system may need access to the data. - The asio implementation explicitly converts the dereferenced const_iterator into mutable_buffer or const_buffer as needed. See the libs/asio/example/buffers/reference_counted.cpp example program. * Strongly-typed socket classes. Rather than asio::stream_socket, asio::datagram_socket and asio::socket_acceptor classes that were used for all protocols, these socket classes are now specific to each protocol. For example, we now have: ip::tcp::socket ip::tcp::acceptor ip::udp::socket and in the future we might have: unix::stream_protocol::socket bluetooth::rfcomm::socket and so on. These protocol-specific classes are simply typedefs for templates instantiated on the corresponding Protocol class. E.g.: template <typename Protocol> class stream_socket_service; template <typename Service> class basic_stream_socket; class tcp { ... typedef stream_socket_service<tcp> socket_service; typedef basic_stream_socket<socket_service> socket; ... }; New protocols can be added by creating classes like tcp above. * Use inheritance in socket types to prevent "layer violations". The basic_stream_socket and basic_datagram_socket templates now inherit from basic_socket<Service>. This class includes all functions that are common to stream and datagram sockets (everything except send/receive and friends). Stream layers such as ssl::stream return a reference to basic_socket<> from the lowest_layer() function. * IPv4 and IPv6 protocol independence. Socket, endpoint and address classes for IP (i.e. TCP and UDP) are independent of the particular IP version (4 or 6) used. The ipv4 namespace has been removed. For example: - The ipv4::address class has become IP version-independent ip::address. If you need to manipulate addresses for a specific IP version use ip::address_v4 or ip::address_v6. - The protocol classes are now ip::tcp and ip::udp, and the corresponding endpoint types are ip::tcp::endpoint and ip::udp::endpoint. - To open a socket you must specify the protocol version, e.g.: ip::tcp::socket socket(io_service); ... socket.open(ip::tcp::v4()); // IPv4 ... socket.open(ip::tcp::v6()); // IPv6 Note: if you already have an endpoint you can avoid this by using the protocol associated with the endpoint object: socket.open(endpoint.protocol()); - When constructing an endpoint with just a port number, you must specify the IP version: ip::tcp::endpoint(ip::tcp::v4(), 12345); The development of programs that are independent of the IP version is further aided by the new resolver interface outlined below. * Resolver replaces ipv4::host_resolver. The ipv4::host_resolver and ipv4::host classes have been removed in favour of an endpoint-oriented interface that can support different protocols. The functionality of this new interface is based on getaddrinfo and getnameinfo. Use the resolver by constructing a query, calling resolve() (or async_resolve()), and using the returned forward-only iterator to enumerate the endpoints that match the query. For example: ip::tcp::resolver resolver(io_service); ip::tcp::resolver::query query("foobar.com", "http"); ip::tcp::resolver::iterator iter = resolver.resolve(query); ip::tcp::resolver::iterator end; while (iter != end) { ip::tcp::endpoint ep = *iter; std::cout << iter->endpoint() << std::endl; std::cout << iter->host_name() << std::endl; std::cout << iter->service_name() << std::endl; ++iter; } The asynchronous equivalent is: void my_class::start_resolve() { ip::tcp::resolver resolver(io_service); ip::tcp::resolver::query query("foobar.com", "http"); resolver.async_resolve(query, boost::bind(&my_class::resolve_handler, this, asio::placeholders::error, asio::placeholders::iterator)); } void my_class::resolve_handler(const asio::error& e, ip::tcp::resolver::iterator iter) { if (!e) { ip::tcp::resolver::iterator end; while (iter != end) { ... ++iter; } } } There are overloads of resolve()/async_resolve() that take an endpoint and return an iterator. These are intended for reverse lookups. Since the ipv4::host_resolver had a local_host() function that returned information about the local system, a new function ip::host_name() has been added which returns a string containing the name of the system. * Multicast socket options renamed. The IP-specific socket options for multicast have been renamed. They are now also independent of the IP version. The name changes are: - add_membership is now join_group - drop_membership is now leave_group - time_to_live is now hops These multicast socket options are now found in the ip::multicast namespace. * String-to-address conversion now uses from_string function. This change was driven by three things: - The need to indicate an error without throwing if a conversion from string to address fails. - String literal addresses would not be common and so support for them does not need to be convenient. - It is more common to use a resolver to convert strings into complete endpoints. To convert a string into an address you would now write: ip::address address = ip::address::from_string("1.2.3.4"); which throws an asio::error on failure, or: asio::error error; ip::address address = ip::address::from_string( "1.2.3.4", asio::assign_error(error)); to convert without throwing. Similar functions exist on the ip::address_v4 and ip::address_v6 classes. * IPv4 and IPv6 addresses can be converted to and from bytes. The ip::address_v4 and ip::address_v6 classes contain: - A typedef called bytes_type, which is a boost::array<unsigned char, N>. For IPv4, N is 4. For IPv6, N is 16. - A constructor that takes a bytes_type value. - A function to_bytes() that returns a bytes_type value. * Order of endpoint constructor params reversed. The old ipv4::tcp::endpoint class (and udp equivalent) had a constructor that took a port and address in that order. The new ip::tcp::endpoint and ip::udp::endpoint classes have the order reversed, since that is more idiomatic. * Change position of flags param on send_to, receive_from. The order of parameters on the basic_datagram_socket's send_to, async_send_to, receive_from and async_receive_from functions have been changed to put the endpoint before the message flags. Overloads of these functions have been added that do not include the flags parameter, which gives the flags a default value of 0. A similar change has been made for the send, async_send, receive and async_receive functions on basic_datagram_socket and basic_stream_socket. * Don't globally block SIGPIPE if an alternative is available. On Linux, the MSG_NOSIGNAL flag is now passed to the sendmsg function. On Mac OS X, the SOL_SOCKET/SO_NOSIGPIPE socket option is set on all sockets when they are created. The SIGPIPE signal is globally ignored on Solaris. * Reactor implementations run operations until complete. The reactor socket implementation sets non-blocking I/O for asynchronous emulation (non-blocking mode is enabled for a socket the first time an async call is made on that socket). Operations are now restarted automatically if the asynchronous operation returns would_block or try_again, meaning that asynchronous operations are guaranteed never to return these errors. Changes planned for 0.3.7 ========================= * Fix tutorials/examples to not use "C style" code. * Emulation of getaddrinfo/getnameinfo on platforms where it is unavailable or broken. Changes planned for post-0.3.7 ============================== * Iostreams support. * Split_error handler function object. * Overloads of io_service::run() that take a timeout parameter. This is to allow calls to run() to be interleaved with other application code. The run() function will return an enum value indicating why it finished (e.g. timed_out, out_of_work, interrupted, etc). * Polymorphic wrapper for the Dispatcher concept. * Polymorphic wrapper for the Stream concept. * Investigate handler-invocation hooking. * Develop a performance test framework. * Rework reactor implementations to support handler-based custom memory allocation. * New reactor implementations, e.g. /dev/poll and Solaris 10 ports. * Use the Windows API call ConnectEx where available for true asynchronous socket connect operations. * Add better support for line-based protocols. This might take the form of an {async_}read_line function or equivalent. * Add UNIX domain socket support. * Documentation, documentation, documentation. I hope that's everything!

5 10

[SoC-2006] Associative containers...
by Rene Rivera 04 May '06

04 May '06

To Whomever added the " Associative containers specialized for low memory footprint and fast lookup" item to the project list <http://tinyurl.com/lprgp>. I curious which containers in particular you have in mind? I'm asking because I'm working on such a container, both for myself and for Boost, bjam, and binutils/ld/bfd. The particular container is the radix tree <http://en.wikipedia.org/wiki/Radix_tree>. -- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - grafikrobot/yahoo

3 2

[serialization] Serialisation/deserialisation of floating-point values
by Paul Giaccone 03 May '06

03 May '06

I'm having problems with deserialising floating-point (double) values that are written to an XML file. I'm reading the values back in and comparing them to what I saved to ensure that my file has been written correctly. However, some of the values differ in about the seventeenth significant figure (or thereabouts). I thought Boost serialization used some numerical limit to make sure that values are serialised exactly to full precision, so what is happening here? Example: Value in original object, written to file: 0.0019075645054089487 Value actually stored in file (by examination of XML file): 0.0019075645054089487 [identical to value written to file] Value after deserialisation: 0.0019075645054089489 It looks like there is a difference in the least-significant bit, as examining the memory for these two values gives: Original value: b4 83 9b ca e7 40 5f 3f Deserialised value: b5 83 9b ca e7 40 5f 3f (where the least-significant byte is on the left) Note the difference in the first bytes. I'm using Boost 1.33.1 with Visual Studio 7.1.3088 in debug mode. Paul

14 59

[MPL] bug in quote.hpp?
by Jaakko Järvi 03 May '06

03 May '06

Hi, Starting from line 61: template< typename T, bool has_type_ > struct quote_impl // GCC has a problem with metafunction forwarding when T is a // specialization of a template called 'type'. # if BOOST_WORKAROUND(__GNUC__, BOOST_TESTED_AT(4)) \ && BOOST_WORKAROUND(__GNUC_MINOR__, BOOST_TESTED_AT(0)) \ && BOOST_WORKAROUND(__GNUC_PATCHLEVEL__ BOOST_TESTED_AT(2)) There seems to be a comma missing from the last line, before BOOST_TESTED_AT(2), is that right?

2 2

Graph library concept check problem
by Matthias Troyer 03 May '06

03 May '06

Hi, Trying to compile our ALPS project sources (http://alps.comp- phys.org) with the CVS HEAD I encountered a number of problems with the BGL concept checking. In particular on MacOS X, using the Apple g+ + 4.0.1, I encounter problems in depth_first_search.hpp with: template <class Visitor, class Graph> class DFSVisitorConcept { public: void constraints() { function_requires< CopyConstructibleConcept<Visitor> >(); vis.initialize_vertex(u, g); vis.start_vertex(u, g); vis.discover_vertex(u, g); vis.examine_edge(e, g); vis.tree_edge(e, g); vis.back_edge(e, g); vis.forward_or_cross_edge(e, g); vis.finish_vertex(u, g); } private: Visitor vis; Graph g; typename graph_traits<Graph>::vertex_descriptor u; typename graph_traits<Graph>::edge_descriptor e; }; Somehow this tries to instantiate a default constructor for the visitor and the graph. Since some graph types do not have a default constructor and no default constructor should actually be required, I want to ask whether this is a bug in the compiler or in the concept checks? Matthias

3 4

[numeric/conversion] Patch for BCB2006
by Nicola Musatti 02 May '06

02 May '06

Hallo, The following patch appears to be necessary to get BCB2006 to pass the conversion tests. If nobody has any objections (Fernando?) , I'll commit it in a few days. Cheers, Nicola Musatti ******Index: cast.hpp =================================================================== RCS file: /cvsroot/boost/boost/boost/numeric/conversion/cast.hpp,v retrieving revision 1.8 diff -d -u -r1.8 cast.hpp --- cast.hpp 27 Jun 2005 15:05:12 -0000 1.8 +++ cast.hpp 27 Apr 2006 20:43:24 -0000 @@ -22,7 +22,7 @@ #include <boost/detail/workaround.hpp> -#if BOOST_WORKAROUND(BOOST_MSVC, < 1300) || BOOST_WORKAROUND(__BORLANDC__, == 0x564) +#if BOOST_WORKAROUND(BOOST_MSVC, < 1300) || BOOST_WORKAROUND(__BORLANDC__, BOOST_TESTED_AT(0x582)) # include<boost/numeric/conversion/detail/old_numeric_cast.hpp>

2 3

[SoC][Act]Concurrency library
by Matt Calabrese 01 May '06

01 May '06

For the past month or so I've been working on a concurrency library which brings active types, atomic types, and parallel loop constructs and algorithms to C++. I am familiar with some of the concurrency projects already underway, though I'm sure my library would be able to coexist. I plan on proposing through SoC sometime later this week. Over the past two days I've also begun writing up some documentation using QuickBook to give a general idea as to the design of the library. Though I do have a lot of features already implemented, there is still a very large amount of work to be done prior to completion. Some feedback before I make the formal proposal would be nice, and it'd also be great to see if anyone is interested. Keep in mind that the documentation is only a rough outing and so there are a few unconnected links, but I do include a brief introduction and some basic usage examples including code snippets. The library is reference to as Surge.Act as that is the namespace I use for my personal libraries, though it will of course be changed to Boost.Act if it were to be accepted. http://members.gamedev.net/Rivorus/surge/html/index.html For those familiar with the concept of futures and are confused as to why they are not mentioned, I will be adding all of that information to the rationale over the upcoming few days. I also apologize for not having better code examples. -- -Matt Calabrese

4 7

Re: [boost] [Boost Review] Do property_tree and program_options overlap?"
by bwood 01 May '06

01 May '06

Gennadiy Rozental wrote: >> You've been touting multi_index here as well. > >Not at all. If you follow my recent concern with multi_index design I >actually believe it's lacking. I read those and was sympathetic to the terminology arguments you made. L?pez Mu?oz believes it is too late to make those sort of changes. I disagree with him on that, but it isn't clear enough yet to me what would be better terminology. I didn't follow the other matter you raised in those posts. Please correct me if I'm wrong, but it seems B.MI's serialization forces you to receive a B.MI instance with the same number of indices as was sent. While on one level it is impressive that B.MI preserves the relative order of equal elements, it has a cost. If you have a B.MI with 3 indices in a server and only need one or two indices in a client, you have some work to do. In order to accomplish this I think you would have to receive a B.MI with the same indices as was sent and then write code that copies the contents of that B.MI to either a simpler B.MI or an STL container. Does the preservation of the relative order of elements matter more than this sort of flexibility? Brian www.webEbenezer.net

2 1

[filesystem] Some questions about string use
by Ion Gaztañaga 01 May '06

01 May '06

Hi to all, While revising a bit the boost::filesystem error/exception use for looking a model for interprocess library and after reading the N1934 "Filesystem Library Proposal for TR2 (Revision 2)" I'm a bit concerned about the heavy use of std::string/string_type in the library. The following functions return some kind of path as value type: // observers const string_type string() const; const string_type file_string() const; const string_type directory_string() const; const external_string_type external_file_string() const; const external_string_type external_directory_string() const; string_type root_name() const; string_type root_directory() const; basic_path root_path() const; basic_path relative_path() const; string_type leaf() const; basic_path branch_path() const; I know that syntactically this is nicer and we can have RVO: string_type str = path.string(); But when iterating through directories I find that that returning a temporary object that must allocate/copy/deallocate hurts my performance paranoia. Even with move semantics we have a an overhead: std::vector<std::path> paths; //fill with paths std::path::iterator beg = paths.begin(), end = paths.end(); for(; beg != end; ++it){ std::path::string_type str = it->root_name();//temporary created str += "append some data"; std::cout << str; } Couldn't be better (although uglier) to take a reference to a string that will be filled? void fill_root_name(string_type &root_name) const; ... //////////////////////////////////////////////////////////////////// std::vector<std::path> paths; //fill with paths std::path::string_type root_name; root_name.reserve(PATH_LENGTH); std::path::iterator beg = paths.begin(), end = paths.end(); for(; beg != end; ++it){ it->fill_root_name(root_name); str += "append some data"; std::cout << str; } This way we only allocate memory if we don't have enough place in our string. We can also reserve it beforehand to speed up code. Apart from this I see that path::iterator has a string member. dereference will return a reference to that member but an iterator is supposed to be a "lightweight" pointer-like abstraction, which is value-copied between functions. A string member, in my opinion, converts an iterator in a heavy class (that depends on the string length, but an small string optimization of 16 bytes is not going to help much). Now that filesystem is proposed for the standard I would like to ask boosters (and Beman, of course) if they find these performance concerns serious enough. Regards, Ion

10 29

Re: [boost] [SoC] Specialized containers with Boost.MultiIndex
by JOAQUIN LOPEZ MU?Z 01 May '06

01 May '06

----- Mensaje original ----- De: Matias Capeletto <matias.capeletto(a)gmail.com> Fecha: Sábado, Abril 29, 2006 4:54 am Asunto: Re: [boost] [SoC] Specialized containers with Boost.MultiIndex > Hi, all, especially Joaquin... > > I am planning to submit an application to SoC. I will love to make > realthe specialized containers you mention and try to include some > useful others. [...] > Is the list the place to talk this about of? Hello Matías, good to know a Spanish-speaking fellow, we're only a handful round here. The way SoC is organized, I think the procedure involves your sending an application thru SoC online tool starting May 1st. Yours, along with all other applications related to Boost, will then be routed by Google to Boost mentors, which will do then the selection and P2P discussion with applicants. In your case, since you're interested in the B.MI proposal, I understand it's gonna be I who will be evaluating your application. I don't really know what the application format and required information are (that depends on the SoC online tool, I guess) but I can tell you what I'd like to see in those submissions under my evaluation responsibility: * As much CV data as possible, academic performance, programming-related courses, etc. * Programming experience: University assignments, open source projects, freelance activities, whatever. If you can point to some code of yours, so much the better. * As per the concrete proposal, expand on the idea described on the Boost wiki along the following directions: * A detailed description of your activity, specially if you plan to add some extra bonuses to the original idea (some other specialized container, for instance). Do not make a wishlist, but rather provide a realistic statement of goals you feel confident to achieve within the time schedule imposed by SoC. * Environmental details: IDEs you'll be using, how you'll do the docs, etc. * A roadmap of the project, or some approximation to it. In general, take into account there can be other applicants for the same project, so not only must you be convincing the mentors you're fit for the project, you must make it it clear you're the fittest :) Good luck with your application, I'll be seeing you around during these weeks, come back here if you have tecnhical doubts while designing your application. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

3 3