Re: [boost] [Boost-users] Brainstorming [WAS: Subject: Formal Review of Proposed Boost.Process library starts tomorrow]
On 11 February 2011 23:30, Boris Schaeling <boris@highscore.de> wrote:
On Fri, 11 Feb 2011 16:39:35 +0100, Jeff Flinn < TriumphSprint2000@hotmail.com> wrote:
[...]I think with the right abstractions the seemingly contradictory goals
can be accommodated. That's what I'm attempting to find. As stated at a previous boostcon: "... abstractions are discovered not invented..."
IMHO, previous version of process have not discovered the proper abstractions. I'm sure that the entirely different paradigms between windows and posix have contributed to the difficulties so far. It's difficult to be a master of the subject matter in both domains.
I think building upon existing boost filesystem and iostreams libraries and their abstractions that do successfully address platform differences will make the effort more successful. Additionally I think it's mandatory for a design to directly acknowledge the multi-phase constraints of the fork/exec paradigm. Also it's mandatory to break down the historical monolithic approaches into more tenable pieces.
Thanks for your comments, Jeff, Ilya and Nat! How would you (and others) feel about a Boost.Process library which only aims to replace std::system() with a more powerful function? You could configure the child process a bit (its streams, environment variables and a few more things) and that's all. This would be a less ambitious goal but obviously more easily to reach. I was under the impression that we want more? Maybe we still want more but settle realistically for this goal?
I've not caught up on all of the discussion about the pending review yet, but it seems to be hitting a few familiar issues this time around, unfortunately. I'd like Boost.Process to either mimic the Python subprocess module (as Nat mentioned) or to provide a few low-level pieces: 1. std::system replacement allowing me to do the call in an async way and capture the output. Ideally implemented in such a way that a "process pipeline" library could be implemented on top of Boost.Process. Pipelining input / output between different work items is a more general domain than fits in Boost.Process, IMHO. 2. Way to get process id in a platform independent way. 3. Way to look up location of processes and get a Boost.Filesystem path. 4. Version of "std::system" that works with Boost.Filesystem 5. Version of std::system that allows passing a custom set of environment variables 6. Expose a "native_handle_type" from the library, whatever that native handle type is. There seems to be a lot of justified disagreement about what the high-level view of the library should be, yet all but point 1 are almost trivially implemented in a cross-platform way and entirely orthogonal to the rest of any Boost.Process library. I'd like to see just points 2-6 addressed, reviewed and released before tackling the bigger picture. The disagreements over scope are a showstopper for any progress and threaten to lead Boost.Process astray for another another couple of years unless they are worked around, ie. by reducing scope, perhaps to the point that Boost.Process starts life as a basic utility library. Evolutionary rewrites and breaking changes are much better than nothing. After all, we're up to V2.x of Boost.Spirit and V3 of Boost.Filesystem and all versions have been immensely useful for us end users. Kind Regards, Darren
Am 12.02.2011 04:16, schrieb Darren Garvey:
On 11 February 2011 23:30, Boris Schaeling<boris@highscore.de> wrote:
On Fri, 11 Feb 2011 16:39:35 +0100, Jeff Flinn< TriumphSprint2000@hotmail.com> wrote:
[...]I think with the right abstractions the seemingly contradictory goals
can be accommodated. That's what I'm attempting to find. As stated at a previous boostcon: "... abstractions are discovered not invented..."
IMHO, previous version of process have not discovered the proper abstractions. I'm sure that the entirely different paradigms between windows and posix have contributed to the difficulties so far. It's difficult to be a master of the subject matter in both domains.
I think building upon existing boost filesystem and iostreams libraries and their abstractions that do successfully address platform differences will make the effort more successful. Additionally I think it's mandatory for a design to directly acknowledge the multi-phase constraints of the fork/exec paradigm. Also it's mandatory to break down the historical monolithic approaches into more tenable pieces.
Thanks for your comments, Jeff, Ilya and Nat! How would you (and others) feel about a Boost.Process library which only aims to replace std::system() with a more powerful function? You could configure the child process a bit (its streams, environment variables and a few more things) and that's all. This would be a less ambitious goal but obviously more easily to reach. I was under the impression that we want more? Maybe we still want more but settle realistically for this goal?
I've not caught up on all of the discussion about the pending review yet, but it seems to be hitting a few familiar issues this time around, unfortunately.
I'd like Boost.Process to either mimic the Python subprocess module (as Nat mentioned) or to provide a few low-level pieces:
1. std::system replacement allowing me to do the call in an async way and capture the output. Ideally implemented in such a way that a "process pipeline" library could be implemented on top of Boost.Process. Pipelining input / output between different work items is a more general domain than fits in Boost.Process, IMHO. 2. Way to get process id in a platform independent way. 3. Way to look up location of processes and get a Boost.Filesystem path. 4. Version of "std::system" that works with Boost.Filesystem 5. Version of std::system that allows passing a custom set of environment variables 6. Expose a "native_handle_type" from the library, whatever that native handle type is.
There seems to be a lot of justified disagreement about what the high-level view of the library should be, yet all but point 1 are almost trivially implemented in a cross-platform way and entirely orthogonal to the rest of any Boost.Process library. I'd like to see just points 2-6 addressed, reviewed and released before tackling the bigger picture.
The disagreements over scope are a showstopper for any progress and threaten to lead Boost.Process astray for another another couple of years unless they are worked around, ie. by reducing scope, perhaps to the point that Boost.Process starts life as a basic utility library.
Evolutionary rewrites and breaking changes are much better than nothing. After all, we're up to V2.x of Boost.Spirit and V3 of Boost.Filesystem and all versions have been immensely useful for us end users.
I agree - boost.process should provide only the sync. wait (async. wait should be left). Maybe boost.process or another library can provide the async. wait facility and the community has more time to discuss it.
On 02/11/2011 11:33 PM, Oliver Kowalke wrote:
[snip] I agree - boost.process should provide only the sync. wait (async. wait should be left). Maybe boost.process or another library can provide the async. wait facility and the community has more time to discuss it.
With only synchronous waiting, the library is fairly limited in utility. I would think that the sort of scripting tasks that synchronous waiting might be useful for would not likely be done in C++ to begin with.
Am 12.02.2011 19:51, schrieb Jeremy Maitin-Shepard:
On 02/11/2011 11:33 PM, Oliver Kowalke wrote:
[snip] I agree - boost.process should provide only the sync. wait (async. wait should be left). Maybe boost.process or another library can provide the async. wait facility and the community has more time to discuss it.
With only synchronous waiting, the library is fairly limited in utility. I would think that the sort of scripting tasks that synchronous waiting might be useful for would not likely be done in C++ to begin with.
you could implement async. wait with thread and future + sync. waiting (I know this solution has its limitations). the async. waiting shouldn't a show stopper for boost.process - as this thread shows more discussion is required for a careful design of async. waiting (at least on POSIX -> interaction with other signals). It can be added in the prospective development of boost.process.
On Sat, Feb 12, 2011 at 20:14, Oliver Kowalke <k-oli@gmx.de> wrote:
Am 12.02.2011 19:51, schrieb Jeremy Maitin-Shepard:
On 02/11/2011 11:33 PM, Oliver Kowalke wrote:
[snip] I agree - boost.process should provide only the sync. wait (async. wait should be left). Maybe boost.process or another library can provide the async. wait facility and the community has more time to discuss it.
With only synchronous waiting, the library is fairly limited in utility. I would think that the sort of scripting tasks that synchronous waiting might be useful for would not likely be done in C++ to begin with.
you could implement async. wait with thread and future + sync. waiting (I know this solution has its limitations). the async. waiting shouldn't a show stopper for boost.process - as this thread shows more discussion is required for a careful design of async. waiting (at least on POSIX -> interaction with other signals). It can be added in the prospective development of boost.process.
Hi, I'm observing the developpement of Boost.Process for a project for wich I (will later) need a main application to launch children processes, communicate with them AND know when they end, whatever the reason (sucessful, not successful and any kind of crash). As far as I know, from a previous discussion on this list, async_wait might be the only (cross-platform) way (without having to add another dependency than boost to the project...) to achive the last point, knowing when the child process ends. About the "thread and future + sync. waiting" way of doing it : would it work with any kind of child process end (including any kind of crash)? Or is it part of the limitations you're thinking about? About the initial question, I've always thought that Boost.Process would provide : 1. A cross-platform way to manipulate processes (child processes) including managing their "life-time". 2. Maybe some platform-specific utilites. 3. Easy Inter-process communication "would be cool" but as there is another library providing this feature, is not a show stopper for me. That don't include "efficiency" as personnally I wouldnt' assume that an abstraction of such manipulations would have a chance to be as fast as platform-specific API usage. Whatever the implementation (through platform-specific defines or plateform-specific cpp files), as far as I don't see platfrom specific-code in my own app and the abstract behaviour is the same on all platforms (if details changes I don't really care), that's fine. My other solution --if boost.process isn't stable enough for my need at the moment I'll use it directly in my app-- being to just isolate myself plateform-specific API calls, but I'm a bit worried about the child ending notification feature (because I'm not a specialist at all). That's the advice of an humble non-boost-dev-level boost-user, so it might be naive. Joël Lamotte
Am 14.02.2011 16:45, schrieb Klaim - Joël Lamotte:
As far as I know, from a previous discussion on this list, async_wait might be the only (cross-platform) way (without having to add another dependency than boost to the project...) to achive the last point, knowing when the child process ends.
Do you mean with async_wait the feature dealing with SIGCHLD and sigwait()? If so it is not cross-platform. As discussed in this thread waiting on signals asynchronously influences the application and other libs - see installing signal handlers. IMHO waiting on signals (SIGCHLD, SIGTERM, SIGSTOP, SIGUSR1, ...) should be implemented in a separate library.
About the "thread and future + sync. waiting" way of doing it : would it work with any kind of child process end (including any kind of crash)? Or is it part of the limitations you're thinking about?
Yes - waitpid( child_pid, ...) blocking in one separate thread - returning the result in a future. Depending on how many threads are created blocking in waitpid it may reduce the performance (at least at some amount of threads the scheduling overhead may become significant).
About the initial question, I've always thought that Boost.Process would provide : 1. A cross-platform way to manipulate processes (child processes) including managing their "life-time".
'Managing life-time' means stop, continue, kill children?
2. Maybe some platform-specific utilites. 3. Easy Inter-process communication "would be cool" but as there is another library providing this feature, is not a show stopper for me.
boost.asio/boost.interprocess
On Mon, Feb 14, 2011 at 17:14, Oliver Kowalke <k-oli@gmx.de> wrote:
About the initial question, I've always thought that Boost.Process would
provide : 1. A cross-platform way to manipulate processes (child processes) including managing their "life-time".
'Managing life-time' means stop, continue, kill children?
Yes.
2. Maybe some platform-specific utilites.
3. Easy Inter-process communication "would be cool" but as there is another library providing this feature, is not a show stopper for me.
boost.asio/boost.interprocess
Yes, in fact I'll use another high-level network library (for unrelated reasons) in addition to interprocess (because I tried with an old version of boost.process and already have the communication code, but boost.process was buggy few years ago).
Am 14.02.2011 19:19, schrieb Klaim - Joël Lamotte:
About the initial question, I've always thought that Boost.Process would
provide : 1. A cross-platform way to manipulate processes (child processes) including managing their "life-time".
'Managing life-time' means stop, continue, kill children?
Yes.
This was what I was requesting too but I got no answer from Boris. At least on POSIX we need a framework which deals with sending signals and async. handling of delivered signals (not only SIGCHLD). Because we have no conclusion how this should be handled by boost.process in this review - I suggest that this facility should be implemented by another library. boost.process could provide code for waiting synchronously on a child process (waitpid) - for this purpose the sync. wait could be done in a separate thread and the result will be transfered via a future == async. waiting on child process.
Oliver Kowalke wrote:
Am 14.02.2011 19:19, schrieb Klaim - Joël Lamotte:
About the initial question, I've always thought that Boost.Process would
provide : 1. A cross-platform way to manipulate processes (child processes) including managing their "life-time".
'Managing life-time' means stop, continue, kill children?
Yes.
This was what I was requesting too but I got no answer from Boris. At least on POSIX we need a framework which deals with sending signals and async. handling of delivered signals (not only SIGCHLD).
Because we have no conclusion how this should be handled by boost.process in this review - I suggest that this facility should be implemented by another library.
+1
boost.process could provide code for waiting synchronously on a child process (waitpid) - for this purpose the sync. wait could be done in a separate thread and the result will be transfered via a future == async. waiting on child process.
+1 Jeff
On 02/14/2011 08:14 AM, Oliver Kowalke wrote:
Am 14.02.2011 16:45, schrieb Klaim - Joël Lamotte:
As far as I know, from a previous discussion on this list, async_wait might be the only (cross-platform) way (without having to add another dependency than boost to the project...) to achive the last point, knowing when the child process ends.
Do you mean with async_wait the feature dealing with SIGCHLD and sigwait()? If so it is not cross-platform. As discussed in this thread waiting on signals asynchronously influences the application and other libs - see installing signal handlers. IMHO waiting on signals (SIGCHLD, SIGTERM, SIGSTOP, SIGUSR1, ...) should be implemented in a separate library.
I do not think that a general signal handling library, while potentially useful, is required for Boost.Process to do asynchronous waiting. The reason is that simply letting two independent pieces of code be notified of SIGCHLD is not sufficient, because (in order to be efficient) the SIGCHLD handler needs to then invoke waitpid(-1, ...), which affects the global state of the program. Instead, what is needed is basically a SIGCHLD library, which allows registering a callback to be invoked when a given PID changes state (exited, suspended, or continued). There should be an option to invoke the callback directly in the signal handler, and there should also be a way (possibly built on top of the first invocation method) to invoke a callback asynchronously using ASIO. Ideally, Boost.Process could use this SIGCHLD library by default, but also be able to work with an alternate SIGCHLD multiplexing library.
About the "thread and future + sync. waiting" way of doing it : would it work with any kind of child process end (including any kind of crash)? Or is it part of the limitations you're thinking about?
Yes - waitpid( child_pid, ...) blocking in one separate thread - returning the result in a future. Depending on how many threads are created blocking in waitpid it may reduce the performance (at least at some amount of threads the scheduling overhead may become significant).
In addition to the overhead of one thread per process (which is rather substantial overhead), there is the problem that all other code running in the same process is forced to use this same inefficient waiting mechanism, as opposed to a multiplexing interface as described above. For instance, glib has a similar SIGCHLD handling facility, and while I believe it would be possible to have Boost.Process make use of it, the inefficient waiting scheme would not inter-operate with the process launching facility in glib.
Am 14.02.2011 19:56, schrieb Jeremy Maitin-Shepard:
I do not think that a general signal handling library, while potentially useful, is required for Boost.Process to do asynchronous waiting. The reason is that simply letting two independent pieces of code be notified of SIGCHLD is not sufficient, because (in order to be efficient) the SIGCHLD handler needs to then invoke waitpid(-1, ...), which affects the global state of the program. Instead, what is needed is basically a SIGCHLD library, which allows registering a callback to be invoked when a given PID changes state (exited, suspended, or continued). There should be an option to invoke the callback directly in the signal handler, and there should also be a way (possibly built on top of the first invocation method) to invoke a callback asynchronously using ASIO. Ideally, Boost.Process could use this SIGCHLD library by default, but also be able to work with an alternate SIGCHLD multiplexing library.
This would require that the thread executing async. waiting of boost.process has to block all signals except SIGCHLD and the other thread handling the other signals must block SIGCHLD and unblock signals of interest (SIGTERM, SIGINT, ...). Other worker-threads have to block all signals. Wouldn't be the async. waiting of boost.process be a subset of the signalling library?
On 02/14/2011 11:43 AM, Oliver Kowalke wrote:
Am 14.02.2011 19:56, schrieb Jeremy Maitin-Shepard:
I do not think that a general signal handling library, while potentially useful, is required for Boost.Process to do asynchronous waiting. The reason is that simply letting two independent pieces of code be notified of SIGCHLD is not sufficient, because (in order to be efficient) the SIGCHLD handler needs to then invoke waitpid(-1, ...), which affects the global state of the program. Instead, what is needed is basically a SIGCHLD library, which allows registering a callback to be invoked when a given PID changes state (exited, suspended, or continued). There should be an option to invoke the callback directly in the signal handler, and there should also be a way (possibly built on top of the first invocation method) to invoke a callback asynchronously using ASIO. Ideally, Boost.Process could use this SIGCHLD library by default, but also be able to work with an alternate SIGCHLD multiplexing library.
This would require that the thread executing async. waiting of boost.process has to block all signals except SIGCHLD and the other thread handling the other signals must block SIGCHLD and unblock signals of interest (SIGTERM, SIGINT, ...). Other worker-threads have to block all signals. Wouldn't be the async. waiting of boost.process be a subset of the signalling library?
I don't think a dedicated thread is necessarily needed at all for SIGCHLD. Instead, at any point in the program that needs to fork(), use this sequence: block SIGCHLD in current thread wait until a lock is obtained on a signal-safe mutex (the SIGCHLD handler will also wait to lock the safe mutex; this cannot deadlock, since the mutex is only locked by threads that have blocked SIGCHLD) fork() add child pid to data structure containing the notification list for the SIGCHLD handler unlock mutex reset SIGCHLD blocked status SIGCHLD handler will repeatedly invoke waitpid(-1, &status, WNOHANG | WUNTRACED | WCONTINUED), and then invoke any registered handlers. These handlers need to be designed to be able to run from a signal handler in an arbitrary thread. If a handler needs to run outside of the signal handler context, something like ASIO can be notified to invoke the handler later.
Am 14.02.2011 21:14, schrieb Jeremy Maitin-Shepard:
SIGCHLD handler will repeatedly invoke waitpid(-1, &status, WNOHANG | WUNTRACED | WCONTINUED), and then invoke any registered handlers.
why polling - it waists CPU cycles? I expect that the thread waiting for child processes state change blocks (gets suspended) until it is notified by the kernel that something has happened. other worker-thread do other stuff like communicating with other process etc.
On 02/14/2011 12:50 PM, Oliver Kowalke wrote:
Am 14.02.2011 21:14, schrieb Jeremy Maitin-Shepard:
SIGCHLD handler will repeatedly invoke waitpid(-1, &status, WNOHANG | WUNTRACED | WCONTINUED), and then invoke any registered handlers.
why polling - it waists CPU cycles? I expect that the thread waiting for child processes state change blocks (gets suspended) until it is notified by the kernel that something has happened. other worker-thread do other stuff like communicating with other process etc.
It wouldn't poll indefinitely, as that would lock up the thread in which the signal handler was invoked. It is simply invoked repeatedly until the error ECHILD (meaning no more unwaited-for children) is returned. The notification is in the form of SIGCHLD, but that just means one or more children need to be waited on.
On 02/14/2011 01:00 PM, Jeremy Maitin-Shepard wrote:
On 02/14/2011 12:50 PM, Oliver Kowalke wrote:
Am 14.02.2011 21:14, schrieb Jeremy Maitin-Shepard:
SIGCHLD handler will repeatedly invoke waitpid(-1, &status, WNOHANG | WUNTRACED | WCONTINUED), and then invoke any registered handlers.
why polling - it waists CPU cycles? I expect that the thread waiting for child processes state change blocks (gets suspended) until it is notified by the kernel that something has happened. other worker-thread do other stuff like communicating with other process etc.
It wouldn't poll indefinitely, as that would lock up the thread in which the signal handler was invoked. It is simply invoked repeatedly until the error ECHILD (meaning no more unwaited-for children) is returned. The notification is in the form of SIGCHLD, but that just means one or more children need to be waited on.
Sorry, it is invoked repeatedly until waitpid returns 0.
Am 14.02.2011 22:02, schrieb Jeremy Maitin-Shepard:
On 02/14/2011 01:00 PM, Jeremy Maitin-Shepard wrote:
On 02/14/2011 12:50 PM, Oliver Kowalke wrote:
Am 14.02.2011 21:14, schrieb Jeremy Maitin-Shepard:
SIGCHLD handler will repeatedly invoke waitpid(-1, &status, WNOHANG | WUNTRACED | WCONTINUED), and then invoke any registered handlers.
why polling - it waists CPU cycles? I expect that the thread waiting for child processes state change blocks (gets suspended) until it is notified by the kernel that something has happened. other worker-thread do other stuff like communicating with other process etc.
It wouldn't poll indefinitely, as that would lock up the thread in which the signal handler was invoked. It is simply invoked repeatedly until the error ECHILD (meaning no more unwaited-for children) is returned. The notification is in the form of SIGCHLD, but that just means one or more children need to be waited on.
maybe you misunderstood my concerns. as I wrote in my previous posts - we have two possibilities in a multi-threaded env.: 1.) for each child process a thread is created calling waitpid() with the pid of the forked process only and returning the child state via a future 2.) only one thread is created dealing with the status of child processes - it calls sigwait() with SIGCHLD unblocked and waitpid(-1, & status, WNOHANG | WUNTRACED | WCONTINUED) until waitpid() returns ECHILD I've some concerns if boost.process would implement the second point because it is a subset of the functionality of a boost library dealing with delivered signals (as SIGUSR1, etc.).
On 02/14/2011 01:26 PM, Oliver Kowalke wrote:
Am 14.02.2011 22:02, schrieb Jeremy Maitin-Shepard:
On 02/14/2011 01:00 PM, Jeremy Maitin-Shepard wrote:
On 02/14/2011 12:50 PM, Oliver Kowalke wrote:
Am 14.02.2011 21:14, schrieb Jeremy Maitin-Shepard:
SIGCHLD handler will repeatedly invoke waitpid(-1, &status, WNOHANG | WUNTRACED | WCONTINUED), and then invoke any registered handlers.
why polling - it waists CPU cycles? I expect that the thread waiting for child processes state change blocks (gets suspended) until it is notified by the kernel that something has happened. other worker-thread do other stuff like communicating with other process etc.
It wouldn't poll indefinitely, as that would lock up the thread in which the signal handler was invoked. It is simply invoked repeatedly until the error ECHILD (meaning no more unwaited-for children) is returned. The notification is in the form of SIGCHLD, but that just means one or more children need to be waited on.
maybe you misunderstood my concerns. as I wrote in my previous posts - we have two possibilities in a multi-threaded env.:
1.) for each child process a thread is created calling waitpid() with the pid of the forked process only and returning the child state via a future
2.) only one thread is created dealing with the status of child processes - it calls sigwait() with SIGCHLD unblocked and waitpid(-1, & status, WNOHANG | WUNTRACED | WCONTINUED) until waitpid() returns ECHILD
I've some concerns if boost.process would implement the second point because it is a subset of the functionality of a boost library dealing with delivered signals (as SIGUSR1, etc.).
There is a third possibility, which is what I was referring to in my previous message: no additional threads are created to handle SIGCHLD and instead SIGCHLD is handled in any thread that does not block it (the user is expected to leave it unblocked in at least one thread). This would have the lowest overhead, I believe, and therefore be the best option to use, and it also would not require any coordination as far as signal handling with the rest of the program (but of course would require coordination as far as waitpid), except that it will manage SIGCHLD (but monopolizing SIGCHLD and calling waitpid go hand in hand).
Am 14.02.2011 22:51, schrieb Jeremy Maitin-Shepard:
maybe you misunderstood my concerns. as I wrote in my previous posts - we have two possibilities in a multi-threaded env.:
1.) for each child process a thread is created calling waitpid() with the pid of the forked process only and returning the child state via a future
2.) only one thread is created dealing with the status of child processes - it calls sigwait() with SIGCHLD unblocked and waitpid(-1, & status, WNOHANG | WUNTRACED | WCONTINUED) until waitpid() returns ECHILD
I've some concerns if boost.process would implement the second point because it is a subset of the functionality of a boost library dealing with delivered signals (as SIGUSR1, etc.).
There is a third possibility, which is what I was referring to in my previous message: no additional threads are created to handle SIGCHLD and instead SIGCHLD is handled in any thread that does not block it (the user is expected to leave it unblocked in at least one thread).
This would have the lowest overhead, I believe, and therefore be the best option to use, and it also would not require any coordination as far as signal handling with the rest of the program (but of course would require coordination as far as waitpid), except that it will manage SIGCHLD (but monopolizing SIGCHLD and calling waitpid go hand in hand).
The SIGCHLD signal would be delivered to only one of the threads and you don't know to which one. you would have to install in all threads the same code.
On 02/14/2011 05:07 PM, Oliver Kowalke wrote:
Am 14.02.2011 22:51, schrieb Jeremy Maitin-Shepard:
maybe you misunderstood my concerns. as I wrote in my previous posts - we have two possibilities in a multi-threaded env.:
1.) for each child process a thread is created calling waitpid() with the pid of the forked process only and returning the child state via a future
2.) only one thread is created dealing with the status of child processes - it calls sigwait() with SIGCHLD unblocked and waitpid(-1, & status, WNOHANG | WUNTRACED | WCONTINUED) until waitpid() returns ECHILD
I've some concerns if boost.process would implement the second point because it is a subset of the functionality of a boost library dealing with delivered signals (as SIGUSR1, etc.).
There is a third possibility, which is what I was referring to in my previous message: no additional threads are created to handle SIGCHLD and instead SIGCHLD is handled in any thread that does not block it (the user is expected to leave it unblocked in at least one thread).
This would have the lowest overhead, I believe, and therefore be the best option to use, and it also would not require any coordination as far as signal handling with the rest of the program (but of course would require coordination as far as waitpid), except that it will manage SIGCHLD (but monopolizing SIGCHLD and calling waitpid go hand in hand).
The SIGCHLD signal would be delivered to only one of the threads and you don't know to which one. you would have to install in all threads the same code.
Yes, it would be delivered to an arbitrary thread, but as signal handler settings are shared between threads, it would only be necessary to call signal/sigaction once.
Am 15.02.2011 02:44, schrieb Jeremy Maitin-Shepard:
Yes, it would be delivered to an arbitrary thread, but as signal handler settings are shared between threads, it would only be necessary to call signal/sigaction once.
Why do I need multiple threads doing the same? One thread handling signals is sufficient. Delivering the signals ansyc. with sigaction() etc. is dangerous in a multi-threaded app because if the code locks a mutex and then handles a signal delivered asynchronously with let the mutex be locked during the signal handling and may let the mutex be locked forever if the thread exits. This is not what you want at least how long the mutex remains locked should not depend on asnyc. signal handlers.
On 02/14/2011 06:12 PM, Oliver Kowalke wrote:
Am 15.02.2011 02:44, schrieb Jeremy Maitin-Shepard:
Yes, it would be delivered to an arbitrary thread, but as signal handler settings are shared between threads, it would only be necessary to call signal/sigaction once.
Why do I need multiple threads doing the same? One thread handling signals is sufficient. Delivering the signals ansyc. with sigaction() etc. is dangerous in a multi-threaded app because if the code locks a mutex and then handles a signal delivered asynchronously with let the mutex be locked during the signal handling and may let the mutex be locked forever if the thread exits.
Naturally the signal handler shouldn't terminate the thread. An important case is when there is just a single thread in total, in which case using a dedicated thread for signal handling would be excessive.
This is not what you want at least how long the mutex remains locked should not depend on asnyc. signal handlers.
Arguably then the signal handler could record that the signal was delivered rather than actually call waitpid. In any case, waitpid won't block, so I expect the signal handler would take on the order of a millisecond (this would really have to be tested, though, to know). Also, the user could still provide a dedicated thread by blocking SIGCHLD in every thread but one, without any special support in boost.process.
Arguably then the signal handler could record that the signal was delivered rather than actually call waitpid. In any case, waitpid won't block, so I expect the signal handler would take on the order of a millisecond (this would really have to be tested, though, to know).
Also, the user could still provide a dedicated thread by blocking SIGCHLD in every thread but one, without any special support in boost.process.
If you call waitpid(-1,...) in aloop inside a signal handler, then you get the pids of the children. What do you do next? Keep in mind that the functions you can call must be obstruction-free/async-safe. How do you tell the other threads your pid you got from waitpid()? -- GMX DSL Doppel-Flat ab 19,99 Euro/mtl.! Jetzt mit gratis Handy-Flat! http://portal.gmx.net/de/go/dsl
On 02/14/2011 11:12 PM, Oliver Kowalke wrote:
Arguably then the signal handler could record that the signal was delivered rather than actually call waitpid. In any case, waitpid won't block, so I expect the signal handler would take on the order of a millisecond (this would really have to be tested, though, to know).
Also, the user could still provide a dedicated thread by blocking SIGCHLD in every thread but one, without any special support in boost.process.
If you call waitpid(-1,...) in aloop inside a signal handler, then you get the pids of the children. What do you do next? Keep in mind that the functions you can call must be obstruction-free/async-safe. How do you tell the other threads your pid you got from waitpid()?
The status for each child could be written to a designated pre-allocated data structure, and the thread could be notified by writing a single byte in non-blocking mode to a designated pipe. If the write fails because the pipe buffer is full, it isn't a problem, because it means the reader end has yet to receive some previous notifications and will therefore still see the new status.
Am 15.02.2011 19:10, schrieb Jeremy Maitin-Shepard:
On 02/14/2011 11:12 PM, Oliver Kowalke wrote:
Arguably then the signal handler could record that the signal was delivered rather than actually call waitpid. In any case, waitpid won't block, so I expect the signal handler would take on the order of a millisecond (this would really have to be tested, though, to know).
Also, the user could still provide a dedicated thread by blocking SIGCHLD in every thread but one, without any special support in boost.process.
If you call waitpid(-1,...) in aloop inside a signal handler, then you get the pids of the children. What do you do next? Keep in mind that the functions you can call must be obstruction-free/async-safe. How do you tell the other threads your pid you got from waitpid()?
The status for each child could be written to a designated pre-allocated data structure, and the thread could be notified by writing a single byte in non-blocking mode to a designated pipe. If the write fails because the pipe buffer is full, it isn't a problem, because it means the reader end has yet to receive some previous notifications and will therefore still see the new status.
Seams not reasonable for me - too much overhead. The only possible way would be using a atomic variable which will be set by the signals handler that a SIGCHLD was delivered and the other threads running through a event-loop will detect the change of the atomic variable and then one thread would loop over waitpid( -1,...).
On 02/15/2011 10:19 AM, Oliver Kowalke wrote:
Am 15.02.2011 19:10, schrieb Jeremy Maitin-Shepard:
On 02/14/2011 11:12 PM, Oliver Kowalke wrote:
Arguably then the signal handler could record that the signal was delivered rather than actually call waitpid. In any case, waitpid won't block, so I expect the signal handler would take on the order of a millisecond (this would really have to be tested, though, to know).
Also, the user could still provide a dedicated thread by blocking SIGCHLD in every thread but one, without any special support in boost.process.
If you call waitpid(-1,...) in aloop inside a signal handler, then you get the pids of the children. What do you do next? Keep in mind that the functions you can call must be obstruction-free/async-safe. How do you tell the other threads your pid you got from waitpid()?
The status for each child could be written to a designated pre-allocated data structure, and the thread could be notified by writing a single byte in non-blocking mode to a designated pipe. If the write fails because the pipe buffer is full, it isn't a problem, because it means the reader end has yet to receive some previous notifications and will therefore still see the new status.
Seams not reasonable for me - too much overhead. The only possible way would be using a atomic variable which will be set by the signals handler that a SIGCHLD was delivered and the other threads running through a event-loop will detect the change of the atomic variable and then one thread would loop over waitpid( -1,...).
The only way to wake up the event loop, as far as I know, is to change the status of a file descriptor. (The signal itself might cause a wake-up, but I don't think there is any race-free way to rely on that.) I suppose your claim is that if all the signal handler is doing is simply notifying asio that a signal arrived, then it might as well be done using a generic signal handling library. That is valid, but it is still an implementation detail, and therefore isn't really a concern. Doing it in the signal handler has the advantage of keeping the SIGCHLD handling relatively independent from asio. For instance, it would then be possible to support a combination of synchronous and asynchronous waiting, and also perhaps allow interoperability with other libraries that want to manage processes. It would require testing to know, but I'm not convinced that doing this work in the signal handler would add significant latency, e.g. more than would be introduced by random kernel interrupt handlers, and if latency is very important in certain threads, SIGCHLD (and all other signals) could simply be blocked in those threads. The documentation could warn about this.
participants (5)
-
Darren Garvey -
Jeff Flinn -
Jeremy Maitin-Shepard -
Klaim - Joël Lamotte -
Oliver Kowalke