[interprocess][process] Synchronizing libraries and efforts (+ bugs)
Hi to all, I recently needed s library to launch processes and I took a look to the recently announced Process 0.2 (http://lists.boost.org/Archives/boost/2008/07/139322.php). I just needed the library for Windows, and I think the library is interesting although still lacking in some basic aspects (explained later in this post). A process-launching library is a very nice utility for Interprocess and I was thinking of adding this possibility for a future version (Boost 1.37 or 1.38) of the library. If Boost.Process is going forward, it would be glad to collaborate and develop a library that can satisfy all our needs and plays nicely with Interprocess. My main requests for a process-launching library are: 1) Ability to launch a process and a blocking or non-blocking wait operation to obtain its result (Boost.Process already has this). 2) Ability to redirect child process stdin/stdout/stderr to /dev/null, to share the stdin/stdout of the parent or to redirect it to a file. And in the latter case, append or overwrite options (Boost.Process implements some of this options). I slightly modified Boost.Process to implement a rought version of file redirection. 3) Ability to modify the environment variables of the child (Boost.Process already has this). 4) Blocking and non-blocking redirections of stdout and helper functions to simplify its usage (Boost.Process does not have this). For example: bool send_stdio_and_get_stdout (const void* stdin, std::size_t stdin_size, std::vector<char> &stdout); or similar would avoid many programming errors because at any moment stdin can block and we must extract stdout so that the child process can continue processing stdin. 5) I don't know if streams are the right way to communicate with processes, because they don't offer non-blocking support. 6) I think that process-launching interface should be similar to the soon to be standardized C++ futures. So I'm interested in a process-launching library and I offer my collaboration in case Boost.Process development needs help. Regards, Ion PS: Some possible bugs found on Boost.Process 0.2 version (Windows): -> main thread handle returned by CreateProcess in PROCESS_INFORMATION structure the must be closed and I can't find this in the code. -> child::wait function returns sporadic errors when multiple children are launched and waited. I changed the code from: do { ::GetExitCodeProcess(handle_, &code); ::Sleep(500); } while (code == STILL_ACTIVE); ::WaitForSingleObject(handle_, 0); ::CloseHandle(handle_); with ::WaitForSingleObject(handle_, INFINITE); ::GetExitCodeProcess(handle_, &code); ::CloseHandle(handle_); ->environment::set does not correctly overwrite existing variables): void set(const std::string &var, const std::string &value) { insert(value_type(var, value); } should be void set(const std::string &var, const std::string &value) { this->operator[](var) = value; } -> Using named pipes has been problematic because it returns errors when launching multiple processes aggressively: Error 231 - All Pipes are Busy. This does not happen with anonymous pipes and I think it has some relationship with the guid code. By the way, I think a uuid library is too much to create a unique identifier to be shared with the child a simple atomic count would to the job in my opinion. That would reduce Boost.Process dependencies. -> systembuf::close() should call sync() to push the last charaters into the pipe, othewise, characters are lost if the user does not flush the stream. -> I've changed the "stream_info::usefile" case in win32_start to support redirecting to files, changing "CREATE_NEW" parameter of CreateFileA with CREATE_ALWAYS. -> And now, the most important one ;-) Please, avoid tabs! Read boost development guidelines: http://www.boost.org/development/requirements.html#Tabs
On Sun, 31 Aug 2008 18:23:30 +0200, Ion Gaztañaga <igaztanaga@gmail.com> wrote: Hi Ion,
I recently needed s library to launch processes and I took a look to the recently announced Process 0.2 (http://lists.boost.org/Archives/boost/2008/07/139322.php).
I'm currently working on Boost.Process 0.3. After having released version 0.2 I had learnt that there was another version of Boost.Process somewhere in Subversion which was more recent than version 0.1 which I had used to create 0.2. With Boost.Process 0.3 all changes in all snapshots will be merged again.
[...]A process-launching library is a very nice utility for Interprocess and I was thinking of adding this possibility for a future version (Boost 1.37 or 1.38) of the library. If Boost.Process is going forward, it would be glad to collaborate and develop a library that can satisfy all our needs and plays nicely with Interprocess.
Sure! I had picked up Boost.Process myself which was created in a Google Summer of Project in 2006 by someone else.
[...]2) Ability to redirect child process stdin/stdout/stderr to /dev/null, to share the stdin/stdout of the parent or to redirect it to a file. And in the latter case, append or overwrite options (Boost.Process implements some of this options). I slightly modified Boost.Process to implement a rought version of file redirection.
This is already supported in Boost.Process 0.3.
[...]5) I don't know if streams are the right way to communicate with processes, because they don't offer non-blocking support.
I had added a method to access a file handle in version 0.2 I think. That way it's possible to use asynchronous I/O (you need Boost.Asio 1.1 at least; I think this version supports I/O objects for file descriptors the first time). There is no example yet but asynchronous I/O works as far as I can tell.
6) I think that process-launching interface should be similar to the soon to be standardized C++ futures.
I'm not sure to what you are referring (do you have a link?). However the Launcher concept is replaced by a Context concept in 0.3. This was a change in Subversion by whoever was doing the changes there (with 0.3 I'm really trying to get one version of Boost.Process again without making any other changes).
So I'm interested in a process-launching library and I offer my collaboration in case Boost.Process development needs help.
Can you wait two weeks? :) I should be done with version 0.3 until mid September. I'm nearly done going through the source code and can send you the .hpp files if you want? I still need to update the examples and documentation though.
PS: Some possible bugs found on Boost.Process 0.2 version (Windows):
Thanks, I'll go through the list later. There might be bugs in the POSIX code, too, because I never tested the code. But as I said priority #1 is currently merging all the snapshots again. :) Boris
[...]
Boris wrote:
On Sun, 31 Aug 2008 18:23:30 +0200, Ion Gaztañaga <igaztanaga@gmail.com> wrote:
Hi Ion,
I recently needed s library to launch processes and I took a look to the recently announced Process 0.2 (http://lists.boost.org/Archives/boost/2008/07/139322.php).
I'm currently working on Boost.Process 0.3. After having released version 0.2 I had learnt that there was another version of Boost.Process somewhere in Subversion which was more recent than version 0.1 which I had used to create 0.2. With Boost.Process 0.3 all changes in all snapshots will be merged again.
Nice to hear it.
[...]A process-launching library is a very nice utility for Interprocess and I was thinking of adding this possibility for a future version (Boost 1.37 or 1.38) of the library. If Boost.Process is going forward, it would be glad to collaborate and develop a library that can satisfy all our needs and plays nicely with Interprocess.
Sure! I had picked up Boost.Process myself which was created in a Google Summer of Project in 2006 by someone else.
Great!
[...]2) Ability to redirect child process stdin/stdout/stderr to /dev/null, to share the stdin/stdout of the parent or to redirect it to a file. And in the latter case, append or overwrite options (Boost.Process implements some of this options). I slightly modified Boost.Process to implement a rought version of file redirection.
This is already supported in Boost.Process 0.3.
Ok, glad to hear it.
[...]5) I don't know if streams are the right way to communicate with processes, because they don't offer non-blocking support.
I had added a method to access a file handle in version 0.2 I think. That way it's possible to use asynchronous I/O (you need Boost.Asio 1.1 at least; I think this version supports I/O objects for file descriptors the first time). There is no example yet but asynchronous I/O works as far as I can tell.
Ok. I think non-blocking support should be built in the Process library. I understand async I/O should be tied with ASIO. I think Boost.Iostreams has some kind of non-blocking IO, but I'm not sure. Anyway, we would just need to add to systembuf new functions implementing non-blocking IO. Something basic like write/read functions. Then we can build something better above this with Iostreams: I wouldn't like Boost.Process to play nice with Iostreams, but also the option of not using Boost.Iostreams at all. Another issue, maybe we should create different pipes depending on the request (at least in Windows): -> Async I/O: named pipes -> Blocking & non-blocking I/O: anonymous pipes (they need fewer resources according to MSDN).
6) I think that process-launching interface should be similar to the soon to be standardized C++ futures.
I'm not sure to what you are referring (do you have a link?). However the Launcher concept is replaced by a Context concept in 0.3. This was a change in Subversion by whoever was doing the changes there (with 0.3 I'm really trying to get one version of Boost.Process again without making any other changes).
See N2627: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2008/n2627.html You have a prototype written by Anthony Williams in Boost.Vault/Concurrent Programming. Obviously, I don't think we need to return an arbitrary value for processes. My first impression is that "child" (or whatever is called now ;-) ) should be similar to unique_future<int> in some basic operations so that any programmer that knows futures can start launching processes pretty soon. I don't think we can apply the rest of thread classes to processes (I don't think we can t propagate exceptions and returning arbitrary values would require some serious work with shared memory, that I prefer to avoid).
So I'm interested in a process-launching library and I offer my collaboration in case Boost.Process development needs help.
Can you wait two weeks? :) I should be done with version 0.3 until mid September. I'm nearly done going through the source code and can send you the .hpp files if you want? I still need to update the examples and documentation though.
I can wait ;-) Since you are working hard on the library, I think it's better to wait until you consider the library mature enough. Feel free to contact me if you have any question.
PS: Some possible bugs found on Boost.Process 0.2 version (Windows):
Thanks, I'll go through the list later. There might be bugs in the POSIX code, too, because I never tested the code. But as I said priority #1 is currently merging all the snapshots again. :)
Yes, that's the most important task right now and thanks for pushing the library. I think it will cover a basic hole in the C++ community.
Boris
Regards, Ion
On Sun, 31 Aug 2008 23:20:10 +0200, Ion Gaztañaga <igaztanaga@gmail.com> wrote:
[...]I understand async I/O should be tied with ASIO. I think Boost.Iostreams
To give you an idea how asynchronous I/O with Boost.Process and Boost.Asio currently looks like: boost::process::child child; // somehow launched #if defined(_WIN32) boost::asio::windows::stream_handle out(io_service, child.get_stdin().handle().release()); boost::asio::windows::stream_handle in(io_service, child.get_stdout().handle().release()); #else boost::asio::posix::stream_descriptor out(io_service, child.get_stdin().handle().release()); boost::asio::posix::stream_descriptor in(io_service, child.get_stdout().handle().release()); #endif in.async_read_some(...);
[...]I can wait ;-) Since you are working hard on the library, I think it's better to wait until you consider the library mature enough. Feel free to contact me if you have any question.
OK, I'll tell you once I'm done (as I said I expect until mid September). The code is nearly done. But with all the changes which I had to merge I definitely need to update the documentation and samples. By the way, I would be very interested in a communication channel between a parent and a child process based on Boost.Interprocess. I need asynchronous I/O though. Before I picked up Boost.Process I had created a Boost.Asio extension boost::asio::shared_memory which can be used like boost::asio::socket. The only (but big :) problem I haven't solved yet is how to handle more than one communication channel. As read/write operations in Boost.Interprocess need to be synchronized the calls might block. A parent process with many children will then have to use at least one thread per child. What would be required is a demultiplexer which would work like select() for file descriptors. Then I think asynchronous I/O could be done with Boost.Interprocess. The question is though what would that demultiplexer look like? You are not working on supporting asynchronous I/O in Boost.Interprocess? ;) Boris
[...]
Boris wrote:
On Sun, 31 Aug 2008 23:20:10 +0200, Ion Gaztañaga <igaztanaga@gmail.com> wrote:
[...]I understand async I/O should be tied with ASIO. I think Boost.Iostreams
To give you an idea how asynchronous I/O with Boost.Process and Boost.Asio currently looks like:
boost::process::child child; // somehow launched #if defined(_WIN32) boost::asio::windows::stream_handle out(io_service, child.get_stdin().handle().release()); boost::asio::windows::stream_handle in(io_service, child.get_stdout().handle().release()); #else boost::asio::posix::stream_descriptor out(io_service, child.get_stdin().handle().release()); boost::asio::posix::stream_descriptor in(io_service, child.get_stdout().handle().release()); #endif in.async_read_some(...);
I don't know ASIO but it's a shame that it does not offer a portable interface for async I/O.
[...]I can wait ;-) Since you are working hard on the library, I think it's better to wait until you consider the library mature enough. Feel free to contact me if you have any question.
OK, I'll tell you once I'm done (as I said I expect until mid September). The code is nearly done. But with all the changes which I had to merge I definitely need to update the documentation and samples.
Ok.
By the way, I would be very interested in a communication channel between a parent and a child process based on Boost.Interprocess. I need asynchronous I/O though. Before I picked up Boost.Process I had created a Boost.Asio extension boost::asio::shared_memory which can be used like boost::asio::socket. The only (but big :) problem I haven't solved yet is how to handle more than one communication channel. As read/write operations in Boost.Interprocess need to be synchronized the calls might block. A parent process with many children will then have to use at least one thread per child. What would be required is a demultiplexer which would work like select() for file descriptors. Then I think asynchronous I/O could be done with Boost.Interprocess. The question is though what would that demultiplexer look like? You are not working on supporting asynchronous I/O in Boost.Interprocess? ;)
I don't have much idea about async I/O but we can try to write something once we know what we need. I might be completly wrong but boost::interprocess::shared_memory_object is based on a file descriptor (shm_open in unix and a plan file on windows) so we can just obtain the handle and see if select() and WaitForMultipleObject() work on them. After all, write() should work on UNIX with a shared memory object descritor. Regards, Ion
On Mon, 01 Sep 2008 17:52:52 +0200, Ion Gaztañaga <igaztanaga@gmail.com> wrote:
Boris wrote:
On Sun, 31 Aug 2008 23:20:10 +0200, Ion Gaztañaga <igaztanaga@gmail.com> wrote:
[...]I understand async I/O should be tied with ASIO. I think Boost.Iostreams To give you an idea how asynchronous I/O with Boost.Process and Boost.Asio currently looks like: boost::process::child child; // somehow launched #if defined(_WIN32) boost::asio::windows::stream_handle out(io_service, child.get_stdin().handle().release()); boost::asio::windows::stream_handle in(io_service, child.get_stdout().handle().release()); #else boost::asio::posix::stream_descriptor out(io_service, child.get_stdin().handle().release()); boost::asio::posix::stream_descriptor in(io_service, child.get_stdout().handle().release()); #endif in.async_read_some(...);
I don't know ASIO but it's a shame that it does not offer a portable interface for async I/O.
It's not really Boost.Asio's fault here. The library is very portable but provides two platform-specific I/O objects which I use here. For better portability it would be Boost.Process' job to provide one I/O object instead of forcing developers to pass the underlying file descriptor or HANDLE to Boost.Asio's platform-specific I/O objects. :)
[...]I don't have much idea about async I/O but we can try to write something once we know what we need. I might be completly wrong but boost::interprocess::shared_memory_object is based on a file descriptor (shm_open in unix and a plan file on windows) so we can just obtain the handle and see if select() and WaitForMultipleObject() work on them. After all, write() should work on UNIX with a shared memory object descritor.
Hm, sounds like a good idea. Is there any method I could call to access the file descriptor and handle? Then I could try and use the same Boost.Asio I/O objects as above to check if the idea basically works. Boris
Boris wrote:
On Mon, 01 Sep 2008 17:52:52 +0200, Ion Gaztañaga <igaztanaga@gmail.com> wrote:
[...]I don't have much idea about async I/O but we can try to write something once we know what we need. I might be completly wrong but boost::interprocess::shared_memory_object is based on a file descriptor (shm_open in unix and a plan file on windows) so we can just obtain the handle and see if select() and WaitForMultipleObject() work on them. After all, write() should work on UNIX with a shared memory object descritor.
Hm, sounds like a good idea. Is there any method I could call to access the file descriptor and handle? Then I could try and use the same Boost.Asio I/O objects as above to check if the idea basically works.
Not yet ;-) But you can just take shared_memory_object class and modify it adding a function called get_os_handle(): class shared_memory_object { private: file_handle_t m_handle; mode_t m_mode; std::string m_filename; //... public: file_handle_t get_os_handle() const { return m_handle; } } file_handle_t is an int in UNIX and void * (HANDLE) in windows. I think that's all we need.
Boris
Regards, Ion
On Sun, 31 Aug 2008 18:23:30 +0200, Ion Gaztañaga <igaztanaga@gmail.com> wrote:
[...]-> main thread handle returned by CreateProcess in PROCESS_INFORMATION structure the must be closed and I can't find this in the code.
Changed (I've added a couple of CloseHandle() calls).
-> child::wait function returns sporadic errors when multiple children are launched and waited. I changed the code from: do { ::GetExitCodeProcess(handle_, &code); ::Sleep(500); } while (code == STILL_ACTIVE); ::WaitForSingleObject(handle_, 0); ::CloseHandle(handle_);
with
::WaitForSingleObject(handle_, INFINITE); ::GetExitCodeProcess(handle_, &code); ::CloseHandle(handle_);
Changed.
->environment::set does not correctly overwrite existing variables):
void set(const std::string &var, const std::string &value) { insert(value_type(var, value); }
should be
void set(const std::string &var, const std::string &value) { this->operator[](var) = value; }
No change required as in Boost.Process 0.3 environment is a typedef of std::map<std::string, std::string>.
-> Using named pipes has been problematic because it returns errors when launching multiple processes aggressively: Error 231 - All Pipes are Busy. This does not happen with anonymous pipes and I think it has some relationship with the guid code. By the way, I think a uuid library is too much to create a unique identifier to be shared with the child a simple atomic count would to the job in my opinion. That would reduce Boost.Process dependencies.
An atomic count might not be sufficient as it would be unique only per process. As I used Boost.Uuid only for convenience though I'll replace it with an atomic count plus a random number to drop the dependency.
-> systembuf::close() should call sync() to push the last charaters into the pipe, othewise, characters are lost if the user does not flush the stream.
Changed (in postream::close()).
-> I've changed the "stream_info::usefile" case in win32_start to support redirecting to files, changing "CREATE_NEW" parameter of CreateFileA with CREATE_ALWAYS.
Changed.
-> And now, the most important one ;-)
Please, avoid tabs! Read boost development guidelines:
Will change it. :) Boris
Boris wrote:
-> Using named pipes has been problematic because it returns errors when launching multiple processes aggressively: Error 231 - All Pipes are Busy. This does not happen with anonymous pipes and I think it has some relationship with the guid code. By the way, I think a uuid library is too much to create a unique identifier to be shared with the child a simple atomic count would to the job in my opinion. That would reduce Boost.Process dependencies.
An atomic count might not be sufficient as it would be unique only per process. As I used Boost.Uuid only for convenience though I'll replace it with an atomic count plus a random number to drop the dependency.
The PID of the parent plus an atomic count wouldn't be enough?
-> systembuf::close() should call sync() to push the last charaters into the pipe, othewise, characters are lost if the user does not flush the stream.
Changed (in postream::close()).
I think we must change this in systembuf, because a user might use the streambuf directly and it would be surprised to see that unlike std::filebuf, already written characters are missed. Regards, Ion
On Mon, 01 Sep 2008 17:44:56 +0200, Ion Gaztañaga <igaztanaga@gmail.com> wrote:
Boris wrote:
-> Using named pipes has been problematic because it returns errors when launching multiple processes aggressively: Error 231 - All Pipes are Busy. This does not happen with anonymous pipes and I think it has some relationship with the guid code. By the way, I think a uuid library is too much to create a unique identifier to be shared with the child a simple atomic count would to the job in my opinion. That would reduce Boost.Process dependencies. An atomic count might not be sufficient as it would be unique only per process. As I used Boost.Uuid only for convenience though I'll replace it with an atomic count plus a random number to drop the dependency.
The PID of the parent plus an atomic count wouldn't be enough?
I had the same idea first. But as the pipe class doesn't know the PID in the process class I thought a random number should be fine, too. However we can call of course getpid() and a Windows function if you prefer not to use a random number.
-> systembuf::close() should call sync() to push the last charaters into the pipe, othewise, characters are lost if the user does not flush the stream. Changed (in postream::close()).
I think we must change this in systembuf, because a user might use the streambuf directly and it would be surprised to see that unlike std::filebuf, already written characters are missed.
Ah, I thought it was a typo as there is no close() method in systembuf. As systembuf is derived from std::streambuf which doesn't provide close() either I was pretty sure you meant postream::close()? Boris
----- Original Message ----- From: "Boris" <boriss@web.de> To: <boost@lists.boost.org> Sent: Sunday, August 31, 2008 11:07 PM Subject: Re: [boost] [interprocess][process] Synchronizing libraries and efforts (+ bugs) On Sun, 31 Aug 2008 18:23:30 +0200, Ion Gaztañaga <igaztanaga@gmail.com> wrote:
-> Using named pipes has been problematic because it returns errors when launching multiple processes aggressively: Error 231 - All Pipes are Busy. This does not happen with anonymous pipes and I think it has some relationship with the guid code. By the way, I think a uuid library is too much to create a unique identifier to be shared with the child a simple atomic count would to the job in my opinion. That would reduce Boost.Process dependencies.
An atomic count might not be sufficient as it would be unique only per process. As I used Boost.Uuid only for convenience though I'll replace it with an atomic count plus a random number to drop the dependency. Hi, I have implemented a locally unique identifier that can be used interprocess (see. http://www.nabble.com/Request-interest-on-a-Locally-Unique-Identifier-librar...) Do you think that something like that could be useful for your needs? Best, Vicente
participants (3)
-
Boris -
Ion Gaztañaga -
vicente.botet