
While trying to find a reasonable interface for a socket class to support asynchronous I/O I was examining various implementations. Scott Woods had thought loudly about an asynchronous library which could be the platform for a network library and maybe others. I like this idea but look at this mess: 1) Signal-Driven I/O SIGIO is sent to a process whenever something happens with the socket. As there are too many things happening with a TCP socket the signal is generated too often and "signal-driven I/O is next to useless with a TCP socket" (W. Richard Stevens in his network bible). As there is just one signal handler per signal it shouldn't be used in a library anyway. 2) Threads There are different solutions possible eg. every thread calling a blocking function or one thread blocking while the other threads handle requests. However these are just variations of the blocking I/O model and multiplexing with the disadvantage of increased complexity because of threads. 3) AIO AIO is part of the Single UNIX Specification and an official standard for asynchronous input and output. The AIO API is based on file descriptors and defines a few functions inluding aio_read() and aio_write(). I/O is asynchronous because the process is notified when an input or output operation is completed - there is either a signal sent to the process or a callback function is executed (as a thread). Further more it is possible to poll a file descriptor if the process doesn't want to wait for the signal or the callback function. Now to the disadvantages: Linux 2.6 doesn't support AIO with sockets (see http://lse.sourceforge.net/io/aio.html). There are some AIO patches for Linux 2.4 but nothing official. Even if AIO support was complete there are no aio_connect() and aio_accept() functions. I wonder if connections have to be built up synchronously (I guess so as AIO wasn't designed only for sockets anyway). 4) I/O completion ports This I/O model is based on a concept called port. A port is similar to a multiplexor and waits for several input or output operations to complete. There can be several ports each multiplexing on several file descriptors. Windows XP, Windows 2000 Pro and Windows NT 3.5 support I/O completion ports. However the application has to call a blocking function for a port to be notified when an I/O operation is complete. 4) Event completion framework The event completion framework behaves like I/O completion ports. However the process can be notified of different types of events not only of the completion of I/O operations. Solaris 10 supports these events: AIO, timer, file descriptor, user-defined and waking up threads (see http://developers.sun.com/solaris/articles/event_completion.html). FreeBSD, OpenBSD and NetBSD support: read, write, AIO, file descriptor, process IDs (exit, fork etc. of other processes), signal, timer and network (see http://www.freebsd.org/cgi/man.cgi?query=kqueue&apropos=0&sektion=0&...). The framework of *BSD is called kqueue and not compatible to Solaris. I don't know what other operating systems like Mac OS X support. There is kqueue() is their man pages (see http://developer.apple.com/documentation/Darwin/Reference/ManPages/index.htm...) but I don't know if it is supported. AIO doesn't seem to be supported. Microsoft operating systems provide WaitForMultipleObjects() since Windows 95 which supports these objects: change notification, console input, event, job, memory resource notification, mutex, process, semaphore, thread and waitable timer (see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/bas...). Then I had a look at ACE as I wondered how they support asynchronous I/O with their Proactor class on different operating systems: They don't. They use this preprocessor directive in the beginning of Proactor.h: #if ((defined (ACE_WIN32) && !defined (ACE_HAS_WINCE)) || (defined (ACE_HAS_AIO_CALLS))) // This only works on Win32 platforms and on Unix platforms supporting POSIX aio calls. Someone wanted to port a Windows applications to Linux which was using the Proactor class. The ACE people told him he should say good-bye to Proactor (see http://groups.yahoo.com/group/ace-users/message/38548). Last but not least Linux 2.5 introduced a new API called epoll (see http://www.linuxmanpages.com/man4/epoll.4.php). It doesn't support asynchronous I/O but tries to get rid of all the disadvantages select() and poll() have. The Linux world seems to pursue the traditional multiplexing approach trying to improve it. Now who has an idea how the asynchronous C++ library should look like? ;) Boris

Hi Boris, Looks like some good research! I don't have lots of time just now, but let me try to give you my take on this problem.
Now who has an idea how the asynchronous C++ library should look like? ;)
I can only say this: how async I/O is accomplished will depend on platform, and sometimes that is a run-time result (ex: Win 9x vs. NT). All the various techniques you've mentioned have non-portable aspects to them (signals are *nix only, completion ports are NT only, AIO is a mixed bag, etc.), and as such a simple encapsulation of any of those mechanisms will also be either non-portable our unnecesarrily inefficient as it maps one set of semantics to another. I think we should start by asking a different question: can we write a portable asynchronous delivery mechanism? Answer: yes. We would have to define what is a "message" (my choice is a function<void ()> object), how is it queued and how is the queue drained. There are other picky details, but that's the start. Given such a mechanism, each type of object can provide a simple interface for calling a function<> on completion. As I said in previous posts: this completion callback should be free-threaded but a simple means to supply an auto-queue-callback as a callback should be provided. The reason that each type of object has to provide the async I/O implementation is that the object type is important to how one does async I/O on some platforms (ex Windows). We can present base classes with virtuals or just isomorphic methods, but this issue is unavoidable. Async file I/O on Windows (NT anyway since 9x cannot do it) is _not_ the same as async socket I/O. Under the covers, there may be great deals of common code that is mostly portable. For example, "select() + thread pool" can be used to implement async I/O on Windows, *nix and Mac OSX. It may not be optimal on Windows NT, or any particular flavor of *nix where poll() may be available, but it will work. Anyway, all of these things are implementation issues. A good starting point is a design of an async network I/O interface, along with what async delivery in general looks like. Neither of these should be overly effected by the various OS facilities available to implement them. Ccannot resist here: sockets are also just a mechansim that might not be everywhere you want to go <g>. Seriously, though, on Mac OS9 and older, there were various non-socket API's to do networking. I can only imagine what else may be out there. Best regards, Don __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

Don G wrote:
I think we should start by asking a different question: can we write a portable asynchronous delivery mechanism? Answer: yes. We would have to define what is a "message" (my choice is a function<void ()> object), how is it queued and how is the queue drained. There are other picky details, but that's the start.
Following a discussion started by Aaron W. LaFramboise about extensibility I'd like to suggest not to start with the highest level of abstraction, or at least to start from both ends. I think for many programmers it would be highly desirable to have a robust C++ API to deal with the various system APIs, even if they are not fully portable. Once a set of such platform-specific modules exist, the high level API you are referring to could be made to choose at compile-time (traits ?) to which low level library it maps. Thus, once I know the platform I'm coding for (i.e. the level of abstraction I need) I may or may not plug right into the underlaying low-level API to get access to the rich features each platform-independent API has to trade against portability. Regards, Stefan

Boris wrote:
3) AIO AIO is part of the Single UNIX Specification and an official standard for asynchronous input and output. The AIO API is based on file descriptors and defines a few functions inluding aio_read() and aio_write(). I/O is asynchronous because the process is notified when an input or output operation is completed - there is either a signal sent to the process or a callback function is executed (as a thread). Further more it is possible to poll a file descriptor if the process doesn't want to wait for the signal or the callback function. Now to the disadvantages: Linux 2.6 doesn't support AIO with sockets (see http://lse.sourceforge.net/io/aio.html).
Is this current? I have been looking a little, and I'm not sure, but from some of the posts I see I think this may be fixed.

Neal Becker wrote:
3) AIO AIO is part of the Single UNIX Specification and an official standard for asynchronous input and output. The AIO API is based on file descriptors and defines a few functions inluding aio_read() and aio_write(). I/O is asynchronous because the process is notified when an input or output operation is completed - there is either a signal sent to the process or a callback function is executed (as a thread). Further more it is possible to poll a file descriptor if the process doesn't want to wait for the signal or the callback function. Now to the disadvantages: Linux 2.6 doesn't support AIO with sockets (see http://lse.sourceforge.net/io/aio.html).
Is this current? I have been looking a little, and I'm not sure, but from some of the posts I see I think this may be fixed.
http://lse.sourceforge.net/io/aio.html is said to be the official AIO webpage for Linux. So I guess it was correct when Linux 2.6 got official. There are a lot of aio patches but even the guys on the mailing list linux-aio don't seem to know the current status: http://marc.theaimsgroup.com/?l=linux-aio&m=107970040531423&w=2 :) Boris

The MacOS X kernel's (XNU) BSD component has the kqueues and aio facilities. On Mar 17, 2005, at 7:08 AM, Boris wrote:
<snip> I don't know what other operating systems like Mac OS X support. There is kqueue() is their man pages (see http://developer.apple.com/documentation/Darwin/Reference/ManPages/ index.html) but I don't know if it is supported. AIO doesn't seem to be supported. <snip> Boris
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Kon Lovett wrote:
The MacOS X kernel's (XNU) BSD component has the kqueues and aio facilities.
Thank you! Meanwhile I found some sample code at Apple's website which uses kqueues. Do you know if http://vip.cs.utsa.edu/usp/programs/macupdate.html is correct about aio support in Mac OS X since 10.3? Boris
[...]

Glad to be of some use here. Per http://vip.cs.utsa.edu/usp/programs/macupdate.html: 1. ? (My book copy old & in storage. SIgh.) 2. aio.h is in /usr/include/sys 3. SIGRTMAX is not defined in sys/signal.h so the alias IS rqrd. 4. The union sigval is "correct" so the alias IS NOT not rqrd. The above are true for MacOS X 10.3.8 (the latest release). I cannot comment on any earlier release, sorry. Kon On Mar 17, 2005, at 12:56 PM, Boris wrote:
Kon Lovett wrote:
The MacOS X kernel's (XNU) BSD component has the kqueues and aio facilities.
Thank you! Meanwhile I found some sample code at Apple's website which uses kqueues. Do you know if http://vip.cs.utsa.edu/usp/programs/macupdate.html is correct about aio support in Mac OS X since 10.3?
Boris
[...]
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

Boris wrote:
While trying to find a reasonable interface for a socket class to support asynchronous I/O I was examining various implementations. Scott Woods had thought loudly about an asynchronous library which could be the platform for a network library and maybe others. I like this idea but look at this mess:
This (now slightly dated) page is a good summary of the unix situation: http://www.kegel.com/c10k.html Matt

----- Original Message ----- From: "Boris" <boris@gtemail.net> To: <boost@lists.boost.org>
1) Signal-Driven I/O SIGIO is sent to a process whenever something happens with the socket. As there are too many things happening with a TCP socket the signal is generated too often and "signal-driven I/O is next to useless with a TCP socket" (W. Richard Stevens in his network bible). As there is just one signal handler per signal it shouldn't be used in a library anyway.
[snip lovely research]
Now who has an idea how the asynchronous C++ library should look like? ;)
Good background and a little daunting. Happy to see your new subject; the change highlights the fact that issues around async sockets/networking have much broader scope. I think its appropriate to mention that typical software development is not async in nature, i.e. the picture is perhaps worse as there is a plethora of issues arising from the variety of OS async mechanisms and the global body of software developers does its best to express everything in procedural/OO terms. These two trains have to collide somewhere. Maybe I can describe this another way. It would seem logical (to me :-) to target an async I/O library that integrated cleanly with A. Hubers FSM work, with the view of implementing socket management with a Huber FSM. This follows from the belief that async software == state machine software. But what portion of Boosters would find an FSM-based "Boost socket" palatable? There is clearly a learning curve involved around Andrea's work and it has been contentious enough, without throwing sockets into the brew. This might all sound a bit gloomy but really its just creating the challenge; "can we write a portable asynchronous delivery mechanism?" (good one Don). Cheers.

In .NET the asynchronous call occurs on a delegate ( Boost equivalent is a boost::function<> ) running on a separate thread. The delegate is passed a structure which allows the asynchronous callback to complete. Thus to asynchronously accept a connection, one calls aSocket.BeginAccept passing the delegate as well as user-defined data, including the original socket, in a structure, and when the delegate on the thread accepts a connection, it calls aSocket.EndAccept(theStructure) to get back a socket for the connection. All .NET asynchronous socket calls work similiarly, and the use of threads and delegates seems to me a strong combination and works very easily. Obviously Boost has a thread library, although I have not used it, and it has a callback mechanism in boost::function<> which I have used extensively, so designing an asynchronous I/O library around these two concepts seems like a good design to me. "Boris" <boris@gtemail.net> wrote in message news:d1c6ks$btp$1@sea.gmane.org...
While trying to find a reasonable interface for a socket class to support asynchronous I/O I was examining various implementations. Scott Woods had thought loudly about an asynchronous library which could be the platform for a network library and maybe others. I like this idea but look at this mess:
1) Signal-Driven I/O SIGIO is sent to a process whenever something happens with the socket. As there are too many things happening with a TCP socket the signal is generated too often and "signal-driven I/O is next to useless with a TCP socket" (W. Richard Stevens in his network bible). As there is just one signal handler per signal it shouldn't be used in a library anyway.
2) Threads There are different solutions possible eg. every thread calling a blocking function or one thread blocking while the other threads handle requests. However these are just variations of the blocking I/O model and multiplexing with the disadvantage of increased complexity because of threads.
3) AIO AIO is part of the Single UNIX Specification and an official standard for asynchronous input and output. The AIO API is based on file descriptors and defines a few functions inluding aio_read() and aio_write(). I/O is asynchronous because the process is notified when an input or output operation is completed - there is either a signal sent to the process or a callback function is executed (as a thread). Further more it is possible to poll a file descriptor if the process doesn't want to wait for the signal or the callback function. Now to the disadvantages: Linux 2.6 doesn't support AIO with sockets (see http://lse.sourceforge.net/io/aio.html). There are some AIO patches for Linux 2.4 but nothing official. Even if AIO support was complete there are no aio_connect() and aio_accept() functions. I wonder if connections have to be built up synchronously (I guess so as AIO wasn't designed only for sockets anyway).
4) I/O completion ports This I/O model is based on a concept called port. A port is similar to a multiplexor and waits for several input or output operations to complete. There can be several ports each multiplexing on several file descriptors. Windows XP, Windows 2000 Pro and Windows NT 3.5 support I/O completion ports. However the application has to call a blocking function for a port to be notified when an I/O operation is complete.
4) Event completion framework The event completion framework behaves like I/O completion ports. However the process can be notified of different types of events not only of the completion of I/O operations. Solaris 10 supports these events: AIO, timer, file descriptor, user-defined and waking up threads (see http://developers.sun.com/solaris/articles/event_completion.html). FreeBSD, OpenBSD and NetBSD support: read, write, AIO, file descriptor, process IDs (exit, fork etc. of other processes), signal, timer and network (see
http://www.freebsd.org/cgi/man.cgi?query=kqueue&apropos=0&sektion=0&...).
The framework of *BSD is called kqueue and not compatible to Solaris.
I don't know what other operating systems like Mac OS X support. There is kqueue() is their man pages (see
http://developer.apple.com/documentation/Darwin/Reference/ManPages/index.htm...)
but I don't know if it is supported. AIO doesn't seem to be supported.
Microsoft operating systems provide WaitForMultipleObjects() since Windows 95 which supports these objects: change notification, console input, event, job, memory resource notification, mutex, process, semaphore, thread and waitable timer (see
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/bas...).
Then I had a look at ACE as I wondered how they support asynchronous I/O with their Proactor class on different operating systems: They don't. They use this preprocessor directive in the beginning of Proactor.h: #if ((defined (ACE_WIN32) && !defined (ACE_HAS_WINCE)) || (defined (ACE_HAS_AIO_CALLS))) // This only works on Win32 platforms and on Unix platforms supporting
POSIX
aio calls.
Someone wanted to port a Windows applications to Linux which was using the Proactor class. The ACE people told him he should say good-bye to Proactor (see http://groups.yahoo.com/group/ace-users/message/38548).
Last but not least Linux 2.5 introduced a new API called epoll (see http://www.linuxmanpages.com/man4/epoll.4.php). It doesn't support asynchronous I/O but tries to get rid of all the disadvantages select() and poll() have. The Linux world seems to pursue the traditional multiplexing approach trying to improve it.
Now who has an idea how the asynchronous C++ library should look like? ;)

On Fri, 18 Mar 2005 18:03:50 -0500, Edward Diener <eddielee@tropicsoft.com> wrote:
Obviously Boost has a thread library, although I have not used it, and it has a callback mechanism in boost::function<> which I have used extensively, so designing an asynchronous I/O library around these two concepts seems like a good design to me.
I'd say coupling asynchronous I/O to multi-threading would be a mistake. It is at its most useful IMHO in single-threaded cases. -- Caleb Epstein caleb dot epstein at gmail dot com

Caleb Epstein wrote:
On Fri, 18 Mar 2005 18:03:50 -0500, Edward Diener <eddielee@tropicsoft.com> wrote:
Obviously Boost has a thread library, although I have not used it, and it has a callback mechanism in boost::function<> which I have used extensively, so designing an asynchronous I/O library around these two concepts seems like a good design to me.
I'd say coupling asynchronous I/O to multi-threading would be a mistake. It is at its most useful IMHO in single-threaded cases.
I just want to add this link which has a very good overview about how asynchronicity works in .NET: http://blogs.msdn.com/cbrumme/archive/2003/05/06/51385.aspx It explains the four ways to do asynchronous operations in .NET: 1) callback 2) polling for completion 3) waiting for completion 4) calling the asynchronous operation a second time and then wait (somehow comparable to waiting for completion) Boris
participants (9)
-
Boris
-
Caleb Epstein
-
Don G
-
Edward Diener
-
Kon Lovett
-
Matthew Vogt
-
Neal Becker
-
Scott Woods
-
Stefan Seefeld