On Tue, Feb 8, 2011 at 10:28 AM, Mateusz Loskot <mateusz@loskot.net> wrote:

Why the review is discussed on the boost-users mailing list?

Well, imho, it is very related because one may think about basing one's design on the library. Or one may simply be interested in a more portable approach and might want to play with the library in its current form. Best / Asif

On Thu, 10 Feb 2011 15:41:25 +0100, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote: Hi Jeff,

[...]Problems: Still had to create wrapper function and classes and use ifdef's to handle platform dependencies in client code. Stung multiple times when passing a relative path when setting the working directory on unix systems. Different semantics regarding success/failure of CreateProcess/exec. Lack of synchronicity when launching. Lack of launch error reporting. Replication of existing boost library functionality, and functionality that better belongs to other libraries. Lots of additional 'stuff' in the header implementing create_child. Pay for process handles even when not used. Overly complex io specification. Customization points lack access to many of the required Windows CreateProcess arguments. Implementation is obfuscated by all of the ifdef'd platform dependencies within a single function. Lack of separation of concerns in create_child The monolithic code lacks ability to properly unit test components.

I started with a fresh slate over the last couple of weeks and have an initial high level design of the process launch part that alleviates many of the above problems. It's key aspect is that the stock facilities themselves are built using the same customization points that client code can use to extend the library. Initial high level docs can be found at: goo.gl/NvRAH and a rough initial implementation is in the attached zip file. This is still very preliminary, but I thought it important to add to the discussion now.

thanks, it's always interesting to see new approaches. Looking at your documentation and code I'd say that you basically turned the context class and create_child() function into an executor concept and class? I think I can see how this helps to break up the monolithic code in create_child(). However I would be interested to see how this works in practice. I wonder how much I would need to learn about the concepts of the library and how much code I would need to write to customize the executor that I wouldn't simply call CreateProcess() myself. :) I'm also not sure whether your approach solves the problems you listed above. For example if you introduce platform-specific classes you have to use #ifdefs even more? The very first versions of Boost.Process actually contained generic, POSIX and Windows classes. Unless you used the generic version it felt like using two different libraries. The idea of using one class for everyone plus extension points is to minimize the code you have to write and maintain if you would like to use a certain platform-specific feature. What do you mean with "Pay for process handles even when not used"? You can discard the return value of create_child(). If you want the process handle to be discarded even earlier I wonder whether you aim for a different goal. You want as much flexibility with Boost.Process as you have when using system functions directly? I'm also interested in what you mean with "Overly complex io specification". Is this about stream behaviors? Thanks, Boris

Ilya Sokolov wrote:

On Thu, 10 Feb 2011 19:41:25 +0500, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:

...

I've used the process library as a replacement for Qt's QProcess(which interfers with the main event loop in our application on Mac OSX). We were able to make boost process work across Windows and Mac OSX, but did not have that sense of satisfaction at the end that usually comes with using most boost libraries.

I do not think the library should be accepted into boost for the following reasons.

Problems: [Ilya Sokolov: I've reordered some items]

...

Still had to create wrapper function and classes and use ifdef's to handle platform dependencies in client code. Stung multiple times when passing a relative path when setting the working directory on unix systems. Customization points lack access to many of the required Windows CreateProcess arguments. Overly complex io specification.

Could you tell us where? Please show also how your library would solve it.

The myriad of stream behaviour, stream ends, and methods of specification to me do not best represent the problem domain. Thinking about this last night, I came up with the concept of a Ray, which is a connected pair of boost iostreams' Source and Sink concepts. Pipe is too general in that it conveys flow can be in either direction. So assuming unix still: "uses just one internal buffer for temporary storage of the flowing data, hence what goes in one direction overwrites whatever is a naive programmer tries to send backward" as stated in: http://goo.gl/RXY7p, makes Ray a more accurate term than pipe. Then a model of a Ray would be file_descriptor_ray, which comprises a boost iostream file_descriptor_sink and a file_descriptor_source. So having stdin, stdout, stderr Initializers with constructors or factory functions: stdin_from() stdin_from(const path&) stdin_from(const file_descriptor_source&) stdin_from(const file_descriptor_ray&) stdout_to() stdout_to(const path&) stdout_to(const file_descriptor_sink&) stdout_to(const file_descriptor_ray&) stdout_to() stderr_to(const path&) stderr_to(const file_descriptor_sink&) stderr_to(const file_descriptor_ray&) is to me conceptually much simpler than dealing with stream behaviour, stream end and context classes. It also minimizes copying and heap allocated handles. The default constructed initializers allow me to launch apps that don't have std io, such as GUI apps, without carrying that baggage around. The path overload opens the appropriate file_descriptor sink/source, keeping it open until initialization is complete. The file_descriptor_source / file_descriptor_sink overloads ensure at compile time that the handle has been opened as a source/sink as appropriate. The file_descriptor_ray overloads know whether to get a source/sink handle. It may be that file_descriptor_ray belongs to boost iostreams.

...

Lack of synchronicity when launching. Pay for process handles even when not used.

Not sure what that means.

Sorry, my thoughts didn't make it all the way out of my finger tips. I meant boost::process::child has a std::map<stream_id, handle> data member. Once the child process has been initialized/launched there is no need to know about any handles. Only the process id/handle is required when one needs to wait on the child process to complete. My view is that any stream definition info needing to be accessed after initialization does not belong in the child process class.

...

Different semantics regarding success/failure of CreateProcess/exec. Lack of launch error reporting.

That could be solved by using a separate pipe for error reporting. The only library where I've seen this technique is subprocess.py.

...

Replication of existing boost library functionality, and functionality that better belongs to other libraries.

Where and what functionality, exactly?

Boost filesystem IMO, is the proper currency when dealing with paths. It provides path decomposition which is done manually in operations.hpp. It also insulates it's clients from the whole unicode/utf8 issue(or at the least mitigates the issue) as described in several of the longest mailing list threads in my memory. Additionally, I'd like to see the find_executable_in_path from process/operations.hpp generalized into a facility in boost::filesystem which allows the iteration of a string of delimited directories, which can come from an environment variable. Besides PATH, I've needed search other environment variables like 'include', 'lib', etc. And as stated above using boost::iostreams encapsulates file_descriptor / handle management, reducing the needed code within the process library. I see postream/pistream as duplicating boost iostream facilities.

...

Lots of additional 'stuff' in the header implementing create_child. Implementation is obfuscated by all of the ifdef'd platform dependencies within a single function.

+1 if you meant operations.hpp

Yes, as well as child.hpp, context.hpp, handle.hpp, pid_type.hpp, pipe.hpp, self.hpp, stream_behavior.hpp, stream_id.hpp

...

Lack of separation of concerns in create_child The monolithic code lacks ability to properly unit test components.

I'm not sure that the 'separation of concerns' would help us to unit-test better.

How you can unit test all that's going on within create_child as it's not exposed?

...

I started with a fresh slate over the last couple of weeks and have an initial high level design of the process launch part that alleviates many of the above problems. It's key aspect is that the stock facilities themselves are built using the same customization points that client code can use to extend the library. Initial high level docs can be found at: goo.gl/NvRAH and a rough initial implementation is in the attached zip file. This is still very preliminary, but I thought it important to add to the discussion now.

Thanks, Jeff

Thank you for the review.

I look forward to boost process being the best that it can be. Thanks, Jeff

Boris Schaeling wrote:

On Fri, 11 Feb 2011 15:01:35 +0100, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:

...

[...]The myriad of stream behaviour, stream ends, and methods of specification to me do not best represent the problem domain. [...]So having stdin, stdout, stderr Initializers with constructors or factory functions:

stdin_from() stdin_from(const path&) stdin_from(const file_descriptor_source&) stdin_from(const file_descriptor_ray&)

stdout_to() stdout_to(const path&) stdout_to(const file_descriptor_sink&) stdout_to(const file_descriptor_ray&)

stdout_to() stderr_to(const path&) stderr_to(const file_descriptor_sink&) stderr_to(const file_descriptor_ray&)

is to me conceptually much simpler than dealing with stream behaviour, stream end and context classes. It also minimizes copying and heap allocated handles.

You basically replace boost::process::handle with various types? This makes redirecting to a file very simple indeed.

Some stream behaviors would be still required though - somehow you need to create a pipe for example? Stream behaviors would be called differently but they would still return two stream ends. These could now be objects of different types (so the return type is not necessarily called stream_ends either).

What I like about your idea is that it's not required anymore to use two objects (stream ends as they are currently called) if you really need only one (for example to redirect to a file). I also like that a stream behavior (or whatever it is called then) which creates for a example a pipe wouldn't need to return a parent and child end but can return a read and write end.

The Ray concept and it's model file_descriptor_ray as described in the paragraph that was clipped in your reply describes this. "Ray, which is a connected pair of boost iostreams' Source and Sink concepts. Pipe is too general in that it conveys flow can be in either direction. So assuming unix still: "uses just one internal buffer for temporary storage of the flowing data, hence what goes in one direction overwrites whatever is a naive programmer tries to send backward" as stated in: http://goo.gl/RXY7p, makes Ray a more accurate term than pipe. Then a model of a Ray would be file_descriptor_ray, which comprises a boost iostream file_descriptor_sink and a file_descriptor_source." 'stderr_to(const file_descriptor_ray&)' and it's siblings provide connection to a Ray(which is defined to be a directed pipe). If the parent is chaining children(hope this doesn't get flagged by law enforcement), it doesn't need to keep the ray around once it's done launching the children. The parent then should be able to extract the unconnected source/sink of a ray from which to read/write from/to. I'll try to post a file_descriptor_ray and corresponding initializer implementation later today.

How would the library support inheriting file descriptors > 2 on POSIX? With something like fd(int, const file_descriptor_source&) and fd(int, const file_descriptor_sink&)?

I've limited my focus to dealing with std io to begin with, and haven't given it the thought it requires yet. If required, I'm sure an initializer could be provided to do what's necessary. I don't remember the previous discussions on just how important it is to go beyond std io. I'll try to dig up a link, but I remember reading that pipes in general are the last choice for interprocess communication on some OS's at least.

...

[...]Sorry, my thoughts didn't make it all the way out of my finger tips. I meant boost::process::child has a std::map<stream_id, handle> data member. Once the child process has been initialized/launched there is no need to know about any handles. Only the process id/handle is required when one needs to wait on the child process to complete. My view is that any stream definition info needing to be accessed after initialization does not belong in the child process class.

The stream definitions are in the context class. The map in the child class is for stream ends to communicate with the child process. If you create for example a pipe and want the child process to write to stdout, the read end of the pipe is stored in this map.

I understand, and my view is that io entities should not be present in any form in a child process entity. The client code owns only those io entities it created to communicate with it's launched child processes, other io entities used to connect sibling children need only be temporaries. In fact often when launching gui apps as children, there is no stream io involved, inter process communication is done with boost::interprocess, asio, com,.... Thanks, Jeff

Ilya Sokolov wrote:

On Fri, 11 Feb 2011 19:01:35 +0500, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:

...

Ilya Sokolov wrote:

...

On Thu, 10 Feb 2011 19:41:25 +0500, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:

...

I've used the process library as a replacement for Qt's QProcess(which interfers with the main event loop in our application on Mac OSX). We were able to make boost process work across Windows and Mac OSX, but did not have that sense of satisfaction at the end that usually comes with using most boost libraries.

I do not think the library should be accepted into boost for the following reasons.

Problems: [Ilya Sokolov: I've reordered some items]

...

Still had to create wrapper function and classes and use ifdef's to handle platform dependencies in client code. Stung multiple times when passing a relative path when setting the working directory on unix systems. Customization points lack access to many of the required Windows CreateProcess arguments. Overly complex io specification. Could you tell us where? Please show also how your library would solve it.

The myriad of stream behaviour, stream ends, and methods of specification to me do not best represent the problem domain.

We disagree here. The only problem I see is absence of the 'Rationale' section in the docs.

...

Thinking about this last night, I came up with the concept of a Ray, which is a connected pair of boost iostreams' Source and Sink concepts. Pipe is too general in that it conveys flow can be in either direction.

The concept of pipe is known to almost every programmer.

So is the raw pointer concept, but most programmers see the constraints imposed by scoped/shared_ptr as being useful. Perhaps the term file_descriptor_directed_pipe is better.

...

So assuming unix still: "uses just one internal buffer for temporary storage of the flowing data, hence what goes in one direction overwrites whatever is a naive programmer tries to send backward"

I never saw such errors.

...

as stated in: http://goo.gl/RXY7p, makes Ray a more accurate term than pipe. Then a model of a Ray would be file_descriptor_ray, which comprises a boost iostream file_descriptor_sink and a file_descriptor_source.

So having stdin, stdout, stderr Initializers with constructors or factory functions:

stdin_from() stdin_from(const path&) stdin_from(const file_descriptor_source&) stdin_from(const file_descriptor_ray&)

stdout_to() stdout_to(const path&) stdout_to(const file_descriptor_sink&) stdout_to(const file_descriptor_ray&)

stdout_to() stderr_to(const path&) stderr_to(const file_descriptor_sink&) stderr_to(const file_descriptor_ray&)

is to me conceptually much simpler than dealing with stream behaviour, stream end and context classes.

The code above has the problem outlined here: http://article.gmane.org/gmane.comp.lib.boost.devel/207711

I'm starting to look at the posix io issues. There is nothing in the above interface that prevents handling the problem mentioned. Are you saying there is no amount of file descriptor shuffling to solve the problem?

...

It also minimizes copying and heap allocated handles.

CreateProcess() and fork()/exec() calls are much more greedy than any copying or heap allocation.

...

The default constructed initializers allow me to launch apps that don't have std io, such as GUI apps, without carrying that baggage around.

Current interface allows you the same.

The last I saw boost::process::null still creates and opens temporary files.

...

The path overload opens the appropriate file_descriptor sink/source, keeping it open until initialization is complete.

If all paths in your library is of type fs::path, then what is the type(s) of the arguments and environment of the child process?

In the attached zip, I've updated the windows executor implementation to use CreateProcessW. It uses wide string lpCommandLine as converted from the interface args and arg classes using boost lexical_cast<std::wstring>. I've not completed env var handling yet. The CreateProcess LPVOID lpEnvironment parameter can be either wide or narrow as long as dwCreationFlags includes CREATE_UNICODE_ENVIRONMENT flag. This further demonstrates the flexibility and openness of the executor architecture. There could be wide/narrow environment initializers because the initializers have access to set the startup state of the child process as appropriate. In fact it should be easy to parameterize the windows executor to be narrow or wide. The posix executor would remain string agnostic.

...

The file_descriptor_source / file_descriptor_sink overloads ensure at compile time that the handle has been opened as a source/sink as appropriate. The file_descriptor_ray overloads know whether to get a source/sink handle. It may be that file_descriptor_ray belongs to boost iostreams.

...

...

Lack of synchronicity when launching. Pay for process handles even when not used. Not sure what that means.

Sorry, my thoughts didn't make it all the way out of my finger tips. I meant boost::process::child has a std::map<stream_id, handle> data member. Once the child process has been initialized/launched there is no need to know about any handles. Only the process id/handle is required when one needs to wait on the child process to complete. My view is that any stream definition info needing to be accessed after initialization does not belong in the child process class.

It might be better to return something like boost::tuple<process, std::map<stream_id, handle> > from create_child().

Again, in my view of this problem domain, streams are used only in the launching of the child process. There is no need to pass anything related to streams back from 'create_child'! The parent already defines its communication entities with it's launched children. The following is taken from libs/process/example/create_child.cpp in the attached zip. namespace bp = boost::process; namespace fus = boost::fusion; typedef bp::io_initializer::sink_type sink_type; typedef boost::iostreams::stream<sink_type> sink_stream; typedef bp::file_descriptor_ray ray_type; sink_type s("C:\\TEMP\\log.txt"); sink_stream log(s, std::ios_base::out|std::ios_base::binary); ray_type ray1; sink_stream to_child(ray1.m_sink, std::ios_base::out|std::ios_base::binary); to_child << "testing_input_to_child " << std::endl; log << "- before exec:" << std::endl; try { bp::executor().exec(fus::make_vector ( bp::paths ("C:\\TEMP\\some.exe") , bp::args ("C:\\TEMP\\file.txt") , bp::throw_on_launch_error() , bp::std_out_to_path ("C:\\TEMP\\y.txt") , bp::std_err_to (s) , bp::std_in_from (ray1) )); boost::xtime xt; boost::xtime_get(&xt, boost::TIME_UTC); xt.sec += 1; boost::this_thread::sleep(xt); } catch(const std::exception& e) { std::string s = e.what(); std::cout << s; }

...

...

...

Different semantics regarding success/failure of CreateProcess/exec. Lack of launch error reporting. That could be solved by using a separate pipe for error reporting. The only library where I've seen this technique is subprocess.py.

...

Replication of existing boost library functionality, and functionality that better belongs to other libraries. Where and what functionality, exactly?

Boost filesystem IMO, is the proper currency when dealing with paths. It provides path decomposition which is done manually in operations.hpp. It also insulates it's clients from the whole unicode/utf8 issue(or at the least mitigates the issue) as described in several of the longest mailing list threads in my memory.

See above. IMHO, the only problem that fs::path solves is the lack of string type with known encoding. fs::path doesn't have any useful invariant and all of its methods could be implemented as free functions.

The lack of filesystem invariants is inconsequential here. Using filesystem path self documents the interface in addition to all of the facilities it provides. Whether all of path's methods could be implemented as free functions or as class methods, 'they' should be used rather than duplicating the code in process.

...

...

...

Lack of separation of concerns in create_child The monolithic code lacks ability to properly unit test components. I'm not sure that the 'separation of concerns' would help us to unit-test better.

How you can unit test all that's going on within create_child as it's not exposed?

I'd add more extension points to the bp::context class, something like boost::function<void()> createproces, boost::function<void()> fork and boost::function<void()> exec members. It would allow to call shellexecute(ex)() instead of CreateProcess() on windows and vfork() or spawn() instead of fork()/exec() on posix.

IMO, that seems a lot more complicated than supplying a mock executor and/or unit tests for the accessible nested classes supplying services for args, environtment, file_descriptor manipulation. Thanks, Jeff

Boris Schaeling wrote:

On Mon, 14 Feb 2011 15:29:57 +0100, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:

...

[...]In the attached zip, I've updated the windows executor implementation to [...]The posix executor would remain string agnostic.

Can you describe when I would like to use the Windows or POSIX code of the library?

Only when you need to.? I'm not sure what you're asking.

I ask as this code has obviously not the goal of being platform-independent. What's then the goal?

It does have the goal of being platform-independent and is portable for a large class of usage patterns. I've now filled in most of the interface on the POSIX side that I've shown for windows, and it compiles. I'll post it later today. I've compiled on MSVC8 and gcc 4.0.1 from XCode 3.1.2 on Mac OSX 10.5.8. This effort uncovered some missing boost iostreams functionality to easily get a file_descriptor_source/_sink from stdin/stdout/stderr that is worked around by #define fileno _fileno on windows in the process config file. That being said, I'm starting to see several areas of commonality that could be factored out into a few template classes.

- If I want to use system-specific features, I could access system APIs directly (as I think I have to use #ifdefs anyway?).

Yes you could access API's directly, In my view the utility of the executor/initializer framework makes process invocation easier, more robust and more testable. Our development group's strategy for platform dependencies is to avoid #ifdef's. Platform dependent code is abstracted and typically the dependent implementation is placed in a xxxPlatformImpl.cpp. The ide/build system then includes the appropriate file for the platform. Platform dependent headers are kept in separate directories with the appropriate path being added to the compile include paths command.

- If it should be easier to create child processes, the Windows or POSIX code is either a subset or a reinvention of the system API? Then I either have to call system APIs at some point again (if I need access to more features) or I have to learn a second API which does the same as the system API? Furthermore, the rationale for the design would need to depend on system APIs and not on generic code (for example a generic executor doesn't necessarily mean there should be a Windows and a POSIX executor).

- If I want to easily switch from generic to system-specific code and the library wants to help me: Where is the right trade-off between Windows and POSIX code which is as similar as possible to generic code and Windows and POSIX code which provides access to all platform-specific features a developer could be interested in?

An example of using the executor/initializer framework to support platform dependent behavior came up when our Windows/Mac gui app needed to run a third-party non-gui app that opens a curses based window for user interaction. The windows os also opens a console window by default which was confusing to our users. So we created a no_console initializer that set the appropriate visibility flags on windows and does nothing on Mac/POSIX. We placed the platform specific versions in separate files. But for exposition you could have this single header: no_console.hpp -------------- #pragma once #include <boost/process/initializer.hpp> struct no_console : public initializer { # if defined(BOOST_WINDOWS_API) void pre_create(executor& e) { e.m_startup_info.dwFlags |= STARTF_USESHOWWINDOW; e.m_startup_info.wShowWindow = SW_HIDE; } # endif }; The client code uses it like: bp::executor().exec(fus::make_vector ( bp::paths("some.exe") , bp::args ("some arg") , ... , no_console() , ... )); Hope that answers some of your questions. Thanks, Jeff

Boris Schaeling wrote:

On Wed, 16 Feb 2011 15:57:06 +0100, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:

...

[...]An example of using the executor/initializer framework to support platform dependent behavior came up when our Windows/Mac gui app needed to run a third-party non-gui app that opens a curses based window for user interaction. The windows os also opens a console window by default which was confusing to our users. So we created a no_console initializer that set the appropriate visibility flags on windows and does nothing on Mac/POSIX. We placed the platform specific versions in separate files. But for exposition you could have this single header:

Thanks, but I think you misunderstood me. What I've been trying to

Sorry, I am having a hard time understanding the questions.

understand and why I asked my questions is whether you think we need POSIX- and Windows-classes and -functions

Yes.

which would require a library user to use conditional compilation.

The former does NOT require the latter.

I thought that this is what you had proposed (eg. with a POSIX and Windows executor)? Or did I misunderstand?

In the example code I've previously posted there is NO conditional compilation in the user's code. There IS conditional compilation in the library extension code. I've attached the latest zip imple

I ask as a final decision whether we want library users to choose between a generic API, a Windows API and a POSIX API via conditional compilation could at least make future discussions easier. Even though I think that the majority wants one generic non-system-specific API in Boost.Process, outlining the reasons for or against it hopefully avoids going through all of this a few more times again in the future (especially as jumping around between these options would always require lots of changes in Boost.Process).

I'm not sure what exactly is being asked here. Are you asking(all reviewers) which of the following approaches are acceptable for providing library extensions? 1) use platform specific api to provide extensions providing access to ALL platform capabilities 2) use platform independent api to provide extensions limited to only those capabilities common to all platforms Thanks, Jeff

On Feb 16, 2011, at 7:11 PM, "Boris Schaeling" <boris@highscore.de> wrote:

What I've been trying to understand and why I asked my questions is whether you think we need POSIX- and Windows-classes and -functions which would require a library user to use conditional compilation.

I ask as a final decision whether we want library users to choose between a generic API, a Windows API and a POSIX API via conditional compilation could at least make future discussions easier.

Generic API at least to start. We already have platform-specific APIs; the highest value for Boost.Process will be support for portable code to execute a child process.

AMDG On 2/11/2011 6:01 AM, Jeff Flinn wrote:

So having stdin, stdout, stderr Initializers with constructors or factory functions:

stdin_from() stdin_from(const path&) stdin_from(const file_descriptor_source&) stdin_from(const file_descriptor_ray&)

stdout_to() stdout_to(const path&) stdout_to(const file_descriptor_sink&) stdout_to(const file_descriptor_ray&)

stdout_to() stderr_to(const path&) stderr_to(const file_descriptor_sink&) stderr_to(const file_descriptor_ray&)

is to me conceptually much simpler than dealing with stream behaviour, stream end and context classes. It also minimizes copying and heap allocated handles.

I have to say that I prefer the use of stream_id to having separate functions for each stream. In Christ, Steven Watanabe

On Feb 11, 2011, at 12:54 AM, "Ilya Sokolov" <ilyasokol@gmail.com> wrote:

On Fri, 11 Feb 2011 06:01:56 +0500, Boris Schaeling <boris@highscore.de

...

wrote:

...

As Boost users we want platform-independent code and convenient high-level interfaces.

That's my position, yes. If I want a program that will launch a child process and read its output on Windows, Mac and Linux, clearly "someone" must write three different implementations. I'd prefer to use a library that abstracts away the differences. Put differently, a library that still requires #ifdefs in my own code doesn't buy me that much over the native APIs. What I want is to write and test my parent-process code once, on one platform, with a reasonable level of confidence that I'm not leaving nasty surprises for testers on other platforms. The Python subprocess module gives me an abstraction layer adequate for my needs. I would be happy with a Boost.Process library at about that level.

...

As C++ developers we want efficient implementations

By the time we've asked the OS to load an executable file plus some number of dynamic libraries from disk and resolve references between them, a few more cycles in the requesting program hardly seem to matter. Someone intent on ignoring Knuth's warning about premature optimization can, of course, ignore the library and code directly to the various native APIs.

...

in my opinion the very different system interfaces are the main reason. If Boost.Process has platform-independent code only, developers will complain about the lack of flexibility.

Such library would be practically useless, so it's no-go.

Maybe I'm misreading what you said, but a library permitting a large suite of use cases that are meaningful cross-platform is just what I need. Again, I offer Python's subprocess module as proof by example that this is achievable.

...

If Boost.Process provides flexibility with Windows and POSIX APIs, developers will complain about #ifdefs and writing code twice.

I would call system functions directly, instead.

Provide a way to retrieve platform-specific handles from the high- level classes, like retrieving an OS socket from asio. This permits a hybrid approach if necessary.

...

If Boost.Process supports high-level concepts, developers will complain about inefficient implementations.

Not me. I'm far more frustrated by the present inability to write straightforward cross-platform C++ process-launch code uncluttered by #ifdefs.

...

If Boost.Process supports only what can be implemented cross- platform efficiently, the library is so small that it's practically useless.

No-go.

So remove the efficiency constraint.

...

And if Boost.Process supports a compromise like generic code with extension points, developers from both sides will complain about not generic and flexible enough.

It's ok ).

I'd rather have something in place that addresses the obvious use cases, and argue how to evolve it, than continue writing platform- dependent code every time I need to run a child process. That needs to be tested and debugged three distinct times, on three different computers, with three different IDEs, for each organization with that (common!) requirement. Hard to believe Boost still doesn't have this library.

On Feb 14, 2011, at 1:28 AM, "Ilya Sokolov" <ilyasokol@gmail.com> wrote:

On Fri, 11 Feb 2011 18:29:04 +0500, Nat Goodspeed wrote:

...

...

a library permitting a large suite of use cases that are meaningful cross-platform is just what I need. Again, I offer Python's subprocess module as proof by example that this is achievable.

subprocess.py is not a library that 'has platform-independent code only'. Look at preexec_fn, close_fds, startupinfo and creationflags parameters.

Shrug - I've never used those. They are not part of the large suite of cross-platform use cases I was referencing. Am I glad they exist? Yes, it shows someone was trying to be very thoughtful about the API design. I can't comment on how well those customization points address the cases they were intended to address. I have used subprocess in many situations without needing them, because typically I'm writing cross-platform scripts. In my opinion, a v1 Boost.Process library wouldn't strictly need support for that functionality to be tremendously useful. Put differently: for someone who must already write platform-dependent code, the various native APIs aren't that hard. But it's tedious and exasperating to have to write platform-dependent code for the subset of process-management operations that are not platform-dependent. To me, this is where a Boost.Process library would really shine.

On Fri, 11 Feb 2011 16:39:35 +0100, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:

[...]I think with the right abstractions the seemingly contradictory goals can be accommodated. That's what I'm attempting to find. As stated at a previous boostcon: "... abstractions are discovered not invented..."

IMHO, previous version of process have not discovered the proper abstractions. I'm sure that the entirely different paradigms between windows and posix have contributed to the difficulties so far. It's difficult to be a master of the subject matter in both domains.

I think building upon existing boost filesystem and iostreams libraries and their abstractions that do successfully address platform differences will make the effort more successful. Additionally I think it's mandatory for a design to directly acknowledge the multi-phase constraints of the fork/exec paradigm. Also it's mandatory to break down the historical monolithic approaches into more tenable pieces.

Thanks for your comments, Jeff, Ilya and Nat! How would you (and others) feel about a Boost.Process library which only aims to replace std::system() with a more powerful function? You could configure the child process a bit (its streams, environment variables and a few more things) and that's all. This would be a less ambitious goal but obviously more easily to reach. I was under the impression that we want more? Maybe we still want more but settle realistically for this goal? Boris

On 11 February 2011 23:30, Boris Schaeling <boris@highscore.de> wrote:

On Fri, 11 Feb 2011 16:39:35 +0100, Jeff Flinn < TriumphSprint2000@hotmail.com> wrote:

[...]I think with the right abstractions the seemingly contradictory goals

...

can be accommodated. That's what I'm attempting to find. As stated at a previous boostcon: "... abstractions are discovered not invented..."

IMHO, previous version of process have not discovered the proper abstractions. I'm sure that the entirely different paradigms between windows and posix have contributed to the difficulties so far. It's difficult to be a master of the subject matter in both domains.

I think building upon existing boost filesystem and iostreams libraries and their abstractions that do successfully address platform differences will make the effort more successful. Additionally I think it's mandatory for a design to directly acknowledge the multi-phase constraints of the fork/exec paradigm. Also it's mandatory to break down the historical monolithic approaches into more tenable pieces.

Thanks for your comments, Jeff, Ilya and Nat! How would you (and others) feel about a Boost.Process library which only aims to replace std::system() with a more powerful function? You could configure the child process a bit (its streams, environment variables and a few more things) and that's all. This would be a less ambitious goal but obviously more easily to reach. I was under the impression that we want more? Maybe we still want more but settle realistically for this goal?

I've not caught up on all of the discussion about the pending review yet, but it seems to be hitting a few familiar issues this time around, unfortunately. I'd like Boost.Process to either mimic the Python subprocess module (as Nat mentioned) or to provide a few low-level pieces: 1. std::system replacement allowing me to do the call in an async way and capture the output. Ideally implemented in such a way that a "process pipeline" library could be implemented on top of Boost.Process. Pipelining input / output between different work items is a more general domain than fits in Boost.Process, IMHO. 2. Way to get process id in a platform independent way. 3. Way to look up location of processes and get a Boost.Filesystem path. 4. Version of "std::system" that works with Boost.Filesystem 5. Version of std::system that allows passing a custom set of environment variables 6. Expose a "native_handle_type" from the library, whatever that native handle type is. There seems to be a lot of justified disagreement about what the high-level view of the library should be, yet all but point 1 are almost trivially implemented in a cross-platform way and entirely orthogonal to the rest of any Boost.Process library. I'd like to see just points 2-6 addressed, reviewed and released before tackling the bigger picture. The disagreements over scope are a showstopper for any progress and threaten to lead Boost.Process astray for another another couple of years unless they are worked around, ie. by reducing scope, perhaps to the point that Boost.Process starts life as a basic utility library. Evolutionary rewrites and breaking changes are much better than nothing. After all, we're up to V2.x of Boost.Spirit and V3 of Boost.Filesystem and all versions have been immensely useful for us end users. Kind Regards, Darren