Hi Chris,
One of my design goals is that library users shouldn't generally need to use any thread locking primitives at all. If you have a single thread calling io_service::run(), and always use io_service::post() to post work into the active object's thread, then no locking is required.
Did you have explicit thread locking when you tried a single io_service with a pool of threads? That could certainly impact performance negatively.
I recently switched to a pool of worker threads using a single threaded io_service. If you remember, I posted several times about SSL wasn't working reliably. Well, the single threaded io_service seems to have fixed that problem (I do believe the SSL portion of ASIO has thread issues). However, I could never figure out how to avoid thread locking. The documentation says that asio::buffer() does not make a copy of the data, instead it makes copies of the pointer to the buffer. Because of this, I had to copy my heap data to be sent over the socket into a deque before calling post(). After the handler is called (the socket write has finished), I then remove the first entry in the deque and submit the next entry. This requires thread locking on the deque. I should also point out that because of our design, 2 or more worker threads could be doing work that will require data to be sent out over the same socket at exactly the same time. I had to lock access to the socket since the docs say it isn't thread safe. Is there a better design that I'm missing? Our system isn't a simple design where client makes a request and server responds. Our design is more like a client makes a request, that request changes state in the system, the state change triggers multiple responses to all logged in clients. It's possible other threads are also communicating to the same client with other data, therefore the need to lock socket access. Thanks, Scott