Hi, Would there be any interest in wrappers for file handles / descriptors and FILE*? Sometimes one needs native handles or FILE* for interoperability or special features like memory mapped IO, but standard C++ doesn't provide convenient RAII friendly wrappers for these. The idea seems trivial yet quite useful. I'm aware of <boost/iostreams/device/file_descriptor.hpp> but it seems to be too high-level. I've put a simple start for FILE* at https://code.google.com/p/xbt/source/browse/trunk/xbt/misc/xbt/cfile.h What do you think? Also posted at https://groups.google.com/a/isocpp.org/forum/#!topic/std-proposals/Q4RdFSZgg... -- Olaf
On Wed, Nov 19, 2014 at 3:16 PM, Olaf van der Spek <ml@vdspek.org> wrote:
Hi,
Would there be any interest in wrappers for file handles / descriptors and FILE*?
Sometimes one needs native handles or FILE* for interoperability or special features like memory mapped IO, but standard C++ doesn't provide convenient RAII friendly wrappers for these. The idea seems trivial yet quite useful.
I'm aware of <boost/iostreams/device/file_descriptor.hpp> but it seems to be too high-level.
I've put a simple start for FILE* at https://code.google.com/p/xbt/source/browse/trunk/xbt/misc/xbt/cfile.h What do you think?
Also posted at https://groups.google.com/a/isocpp.org/forum/#!topic/std-proposals/Q4RdFSZgg...
shared_ptr/unique_ptr work fine for me with FILE*, one only has to write a deleter. A dedicated class would be needed for POSIX file descriptors though.
On Wed, Nov 19, 2014 at 12:33 PM, Andrey Semashev wrote:
On Wed, Nov 19, 2014 at 3:16 PM, Olaf van der Spek <ml@vdspek.org> wrote:
Hi,
Would there be any interest in wrappers for file handles / descriptors and FILE*?
Sometimes one needs native handles or FILE* for interoperability or special features like memory mapped IO, but standard C++ doesn't provide convenient RAII friendly wrappers for these. The idea seems trivial yet quite useful.
I'm aware of <boost/iostreams/device/file_descriptor.hpp> but it seems to be too high-level.
I've put a simple start for FILE* at https://code.google.com/p/xbt/source/browse/trunk/xbt/misc/xbt/cfile.h What do you think?
Also posted at https://groups.google.com/a/isocpp.org/forum/#!topic/std-proposals/Q4RdFSZgg...
shared_ptr/unique_ptr work fine for me with FILE*, one only has to write a deleter. A dedicated class would be needed for POSIX file descriptors though.
Peter Sommerlad and Andrew L. Sandoval's unique_resource RAII proposal uses both FILE* and POSIX file descriptors as examples. I think this is the latest revision: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4189.pdf Gareth ************************************************************************ The information contained in this message or any of its attachments may be confidential and is intended for the exclusive use of the addressee(s). Any disclosure, reproduction, distribution or other dissemination or use of this communication is strictly prohibited without the express permission of the sender. The views expressed in this email are those of the individual and not necessarily those of Sony or Sony affiliated companies. Sony email is for business use only. This email and any response may be monitored by Sony to be in compliance with Sony's global policies and standards
On Wed, Nov 19, 2014 at 1:47 PM, Sylvester-Bradley, Gareth <Gareth.Sylvester-Bradley@eu.sony.com> wrote:
shared_ptr/unique_ptr work fine for me with FILE*, one only has to write a deleter. A dedicated class would be needed for POSIX file descriptors though.
Peter Sommerlad and Andrew L. Sandoval's unique_resource RAII proposal uses both FILE* and POSIX file descriptors as examples. I think this is the latest revision: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4189.pdf
It does solve the RAII part but it doesn't provide a standard interface for native IO. -- Olaf
Not many responses... Does noone use native IO? On Thu, Nov 20, 2014 at 9:10 AM, Olaf van der Spek <ml@vdspek.org> wrote:
On Wed, Nov 19, 2014 at 1:47 PM, Sylvester-Bradley, Gareth <Gareth.Sylvester-Bradley@eu.sony.com> wrote:
shared_ptr/unique_ptr work fine for me with FILE*, one only has to write a deleter. A dedicated class would be needed for POSIX file descriptors though.
Peter Sommerlad and Andrew L. Sandoval's unique_resource RAII proposal uses both FILE* and POSIX file descriptors as examples. I think this is the latest revision: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4189.pdf
It does solve the RAII part but it doesn't provide a standard interface for native IO.
-- Olaf
-- Olaf
On Nov 26, 2014, at 11:10 AM, Olaf van der Spek <ml@vdspek.org> wrote: [Please don't top-post.]
On Thu, Nov 20, 2014 at 9:10 AM, Olaf van der Spek <ml@vdspek.org> wrote:
On Wed, Nov 19, 2014 at 1:47 PM, Sylvester-Bradley, Gareth <Gareth.Sylvester-Bradley@eu.sony.com> wrote:
shared_ptr/unique_ptr work fine for me with FILE*, one only has to write a deleter. A dedicated class would be needed for POSIX file descriptors though.
Peter Sommerlad and Andrew L. Sandoval's unique_resource RAII proposal uses both FILE* and POSIX file descriptors as examples. I think this is the latest revision: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4189.pdf
It does solve the RAII part but it doesn't provide a standard interface for native IO.
Not many responses... Does noone use native IO?
I use POSIX I/O calls frequently. I adapted Lisa Lippincott and Marshall Clow's work on Nitrogen to create a library called poseven, based on the same 'nucleus' core library. p7::fd_t is an enum type for a file descriptor, and n::owned<p7::fd_t> and n::shared<p7::fd_t> are smart resources that work like unique_ptr and shared_ptr (loosely speaking) respectively. Much of it is about a decade old, early in my exposure to generic programming, though just recently I added p7::coprocess. Let me know if you're interested in more info. <https://github.com/jjuran/metamage_1/tree/master/posix/poseven/poseven/types> <https://github.com/jjuran/metamage_1/tree/master/posix/poseven/poseven/functions> Josh
On Wed, Nov 26, 2014 at 9:28 PM, Josh Juran <jjuran@gmail.com> wrote:
I use POSIX I/O calls frequently. I adapted Lisa Lippincott and Marshall Clow's work on Nitrogen to create a library called poseven, based on the same 'nucleus' core library. p7::fd_t is an enum type for a file descriptor, and n::owned<p7::fd_t> and n::shared<p7::fd_t> are smart resources that work like unique_ptr and shared_ptr (loosely speaking) respectively. Much of it is about a decade old, early in my exposure to generic programming, though just recently I added p7::coprocess.
Interesting. Why are you using an enum to store the fd? Do you think it'd be useful to move parts of this functionality into Boost? -- Olaf
On Nov 29, 2014, at 6:16 AM, Olaf van der Spek <ml@vdspek.org> wrote:
On Wed, Nov 26, 2014 at 9:28 PM, Josh Juran <jjuran@gmail.com> wrote:
I use POSIX I/O calls frequently. I adapted Lisa Lippincott and Marshall Clow's work on Nitrogen to create a library called poseven, based on the same 'nucleus' core library. p7::fd_t is an enum type for a file descriptor, and n::owned<p7::fd_t> and n::shared<p7::fd_t> are smart resources that work like unique_ptr and shared_ptr (loosely speaking) respectively. Much of it is about a decade old, early in my exposure to generic programming, though just recently I added p7::coprocess.
Interesting. Why are you using an enum to store the fd?
For type-safety and efficiency. I don't use int, because that would allow any constant to be misinterpreted as a file descriptor. Nitrogen used to use complicated wrapper classes, but these had several defects: (a) they were slow to compile in the compiler used at the time (and which I still use for classic Mac OS development[1]), (b) the same compiler wouldn't place their values in registers, (c) their values couldn't be used in switch statements, and (d) creating constants for them was problematic, typically resulting in many `static const Foo x = Foo( ::x );` in headers. Using an enum addressed all of these issues (with (d) being at least a partial improvement). One of the downsides of C-style enums is non-portable memory layout. You can ensure an enum is large enough my declaring max_Foo = uint32_t( -1 ), but there's no guarantee that enum Boolean { False, True } occupies a single byte instead of, say, four. I'm new to C++11, but if I understand correctly, its enum classes address this very issue.
Do you think it'd be useful to move parts of this functionality into Boost?
Quite possibly. Poseven is useful if you want to add type-safety and exception-safety to a C-style POSIX codebase (or subset thereof) without rewriting it to use higher-level semantics like ASIO. In debug mode, it throws exceptions containing a backtrace. A key question is how to approach the overlap of poseven with existing Boost libraries like Filesystem and Thread. I think the first step toward adoption would be for me to familiarize myself with Boost.System. Josh [1] Classic Mac OS is relevant in that POSIX programs are supported through MacRelix. The latest compiler to fully support classic Mac OS is Metrowerks C++ 2.4.1 from CodeWarrior Pro 6.3, released in summer 2000.
On Wednesday 26 November 2014 20:10:31 Olaf van der Spek wrote:
On Thu, Nov 20, 2014 at 9:10 AM, Olaf van der Spek <ml@vdspek.org> wrote:
On Wed, Nov 19, 2014 at 1:47 PM, Sylvester-Bradley, Gareth
<Gareth.Sylvester-Bradley@eu.sony.com> wrote:
shared_ptr/unique_ptr work fine for me with FILE*, one only has to write a deleter. A dedicated class would be needed for POSIX file descriptors though.
Peter Sommerlad and Andrew L. Sandoval's unique_resource RAII proposal uses both FILE* and POSIX file descriptors as examples. I think this is the latest revision: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4189.pdf
It does solve the RAII part but it doesn't provide a standard interface for native IO.
Not many responses... Does noone use native IO?
I do, from time to time. But I always found RAII wrappers sufficient and used the native API to work with file descriptors. I usually need something low- level or non-portable anyway when I do this.
On November 26, 2014 3:57:06 PM EST, Andrey Semashev <andrey.semashev@gmail.com> wrote:
On Thu, Nov 20, 2014 at 9:10 AM, Olaf van der Spek <ml@vdspek.org> wrote:
On Wed, Nov 19, 2014 at 1:47 PM, Sylvester-Bradley, Gareth
<Gareth.Sylvester-Bradley@eu.sony.com> wrote:
shared_ptr/unique_ptr work fine for me with FILE*, one only has to write a deleter. A dedicated class would be needed for POSIX file descriptors though.
Peter Sommerlad and Andrew L. Sandoval's unique_resource RAII
uses both FILE* and POSIX file descriptors as examples. I think
On Wednesday 26 November 2014 20:10:31 Olaf van der Spek wrote: proposal this is
the latest revision: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4189.pdf
It does solve the RAII part but it doesn't provide a standard interface for native IO.
Does noone use native IO?
I do, from time to time. But I always found RAII wrappers sufficient and used the native API to work with file descriptors. I usually need something low-level or non-portable anyway when I do this.
I use filesystem or IOStreams when I want portability, and otherwise use RAII and native APIs when the code isn't portable. ___ Rob (Sent from my portable computation engine)
On Wed, Nov 26, 2014 at 10:21 PM, Rob Stewart <robertstewart@comcast.net> wrote:
I do, from time to time. But I always found RAII wrappers sufficient and used the native API to work with file descriptors. I usually need something low-level or non-portable anyway when I do this.
I use filesystem or IOStreams when I want portability, and otherwise use RAII and native APIs when the code isn't portable.
What RAII wrappers do you use in those cases? -- Olaf
On November 29, 2014 9:14:50 AM EST, Olaf van der Spek <ml@vdspek.org> wrote:
On Wed, Nov 26, 2014 at 10:21 PM, Rob Stewart <robertstewart@comcast.net> wrote:
I do, from time to time. But I always found RAII wrappers sufficient and used the native API to work with file descriptors. I usually need something low-level or non-portable anyway when I do this.
I use filesystem or IOStreams when I want portability, and otherwise use RAII and native APIs when the code isn't portable.
What RAII wrappers do you use in those cases?
In some cases, just my scope_guard class to invoke close(), for example, at end-of-scope. In other cases, I have wrappers around native handles to permit testing them and, of course, closing them. ___ Rob (Sent from my portable computation engine)
On Wed, Nov 26, 2014 at 4:21 PM, Rob Stewart <robertstewart@comcast.net> wrote:
On November 26, 2014 3:57:06 PM EST, Andrey Semashev <andrey.semashev@gmail.com> wrote:
On Wednesday 26 November 2014 20:10:31 Olaf van der Spek wrote:
On Thu, Nov 20, 2014 at 9:10 AM, Olaf van der Spek <ml@vdspek.org> wrote:
It does solve the RAII part but it doesn't provide a standard interface for native IO.
Does noone use native IO?
I do, from time to time. But I always found RAII wrappers sufficient and used the native API to work with file descriptors. I usually need something low-level or non-portable anyway when I do this.
I use filesystem or IOStreams when I want portability, and otherwise use RAII and native APIs when the code isn't portable.
Interestingly, I just stumbled across a bit of live legacy code that relies on __gnu_cxx::stdio_filebuf to allow constructing an istream or ostream from an open FILE*. This does not port to libc++. What surprised me was that I could find no support for FILE* in Boost.Iostreams. Writing a streambuf isn't rocket science, but I would think that this particular use case would be common enough to warrant library support. Am I overlooking something?
On December 15, 2014 6:25:44 PM EST, Nat Goodspeed <nat@lindenlab.com> wrote:
Interestingly, I just stumbled across a bit of live legacy code that relies on __gnu_cxx::stdio_filebuf to allow constructing an istream or ostream from an open FILE*. This does not port to libc++.
What surprised me was that I could find no support for FILE* in Boost.Iostreams. Writing a streambuf isn't rocket science, but I would think that this particular use case would be common enough to warrant library support.
Am I overlooking something?
FILE* is buffered and streambuf provides buffering. You can create a streambif that doesn't actually do any buffering, but it is contrary to the purpose of one. ___ Rob (Sent from my portable computation engine)
On Mon, Dec 15, 2014 at 9:29 PM, Rob Stewart <robertstewart@comcast.net> wrote:
On December 15, 2014 6:25:44 PM EST, Nat Goodspeed <nat@lindenlab.com> wrote:
Interestingly, I just stumbled across a bit of live legacy code that relies on __gnu_cxx::stdio_filebuf to allow constructing an istream or ostream from an open FILE*. This does not port to libc++.
What surprised me was that I could find no support for FILE* in Boost.Iostreams. Writing a streambuf isn't rocket science, but I would think that this particular use case would be common enough to warrant library support.
FILE* is buffered and streambuf provides buffering. You can create a streambif that doesn't actually do any buffering, but it is contrary to the purpose of one.
Very true. I surmise that the purpose of this functionality is to provide an adapter between code that opens a FILE* and code that wants to write to a std::ostream (or read from a std::istream). Another approach would be to dig into the provider of the FILE* and rewrite its own file handling in terms of fstream. That may or may not be more work than writing a streambuf adapter from scratch. It would certainly be more work than simply using a FILE*-based streambuf from a library, which is what I was hoping to find.
On 11/26/2014 08:10 PM, Olaf van der Spek wrote:
Not many responses... Does noone use native IO?
The reason I have not responded is because I would much rather have an asynchronous file I/O API (along the lines of Asio) with synchronous convenience functions. But this is a major task.
On Sat, Nov 29, 2014 at 2:59 PM, Bjorn Reese <breese@mail1.stofanet.dk> wrote:
On 11/26/2014 08:10 PM, Olaf van der Spek wrote:
Not many responses... Does noone use native IO?
The reason I have not responded is because I would much rather have an asynchronous file I/O API (along the lines of Asio) with synchronous convenience functions. But this is a major task.
I don't understand this kind of reasoning. It's not an exclusive or deal is it? -- Olaf
On 11/29/2014 03:13 PM, Olaf van der Spek wrote:
I don't understand this kind of reasoning. It's not an exclusive or deal is it?
While it is not mutually exclusive, it is easier to build a sync API on top of an async API than the other way around -- see the Half-Sync/ Half-Async design pattern in POSA2. You could design the sync API first, but it should not enforce an awkward design for the async API, and how are you going to ensure that without a reasonable idea about the latter? For example, the async API would need some kind of execution context (same as or similar to asio::io_service), and that should be reflected in the sync API. Regarding your wrapper class, I suggest that you make better use of C++ types. For instance, use filesystem::path (whether std::experimental or boost) for file names, std::io_base::openmode for the mode parameter, define a native_handle_type type instead of exposing FILE directly, and use a native_handle() member function to return it instead of using the conversion operator (this pattern is used in Asio and Networking TS.)
On Sat, Nov 29, 2014 at 4:12 PM, Bjorn Reese <breese@mail1.stofanet.dk> wrote:
On 11/29/2014 03:13 PM, Olaf van der Spek wrote:
I don't understand this kind of reasoning. It's not an exclusive or deal is it?
While it is not mutually exclusive, it is easier to build a sync API on top of an async API than the other way around -- see the Half-Sync/ Half-Async design pattern in POSA2.
You could design the sync API first, but it should not enforce an awkward design for the async API, and how are you going to ensure that without a reasonable idea about the latter? For example, the async API would need some kind of execution context (same as or similar to asio::io_service), and that should be reflected in the sync API.
Makes sense, though IMO the async part shouldn't negatively affect the sync part.
Regarding your wrapper class, I suggest that you make better use of C++ types. For instance, use filesystem::path (whether std::experimental or boost) for file names,
I kinda wanted to avoid the dependency on fs just for that, but I guess it does make sense.
std::io_base::openmode for the mode parameter, define a native_handle_type type instead of exposing FILE directly, and
Do you mean a real class (unowned handle) or just a typedef?
use a native_handle() member function to return it instead of using the conversion operator (this pattern is used in Asio and Networking TS.)
-- Olaf
On 12/05/2014 12:29 PM, Olaf van der Spek wrote:
std::io_base::openmode for the mode parameter, define a native_handle_type type instead of exposing FILE directly, and
Do you mean a real class (unowned handle) or just a typedef?
Just a typedef. I am thinking about something similar to: http://www.boost.org/doc/html/boost_asio/reference/basic_stream_socket/nativ...
On Fri, Dec 5, 2014 at 12:48 PM, Bjorn Reese <breese@mail1.stofanet.dk> wrote:
On 12/05/2014 12:29 PM, Olaf van der Spek wrote:
std::io_base::openmode for the mode parameter, define a native_handle_type type instead of exposing FILE directly, and
Do you mean a real class (unowned handle) or just a typedef?
Just a typedef. I am thinking about something similar to:
http://www.boost.org/doc/html/boost_asio/reference/basic_stream_socket/nativ...
Then functions having native handle parameters (so just int on POSIX) would have type safety issues. I think a real class makes more sense, like std::thread::id -- Olaf
On 29 Nov 2014 at 14:59, Bjorn Reese wrote:
On 11/26/2014 08:10 PM, Olaf van der Spek wrote:
Not many responses... Does noone use native IO?
The other problem with anything Boost.Iostreams based is that Iostreams has a severely flawed design in not supporting async i/o. This makes Iostreams useless for almost any task I've needed it for in recent years. It also doesn't help that it lacks a maintainer, and has many annoying bugs (I recently had to use Iostreams due to Boost.Process 0.5 forcing me, and I lost over a week of my time fighting Iostreams to behave when combined with ASIO. It was quite frankly a disaster).
The reason I have not responded is because I would much rather have an asynchronous file I/O API (along the lines of Asio) with synchronous convenience functions. But this is a major task.
As Bjorn only knows too well but Olaf may not, proposed Boost.AFIO (https://boostgsoc13.github.io/boost.afio/doc/html/afio.html) provides portable asynchronous file i/o implementation library extending ASIO. It however does not use the ASIO handler callback idiom as that is (in my opinion, though TBH I am right on this) unsuited for file i/o where _ordering_ of operations is paramount. The ability to easily program ordering makes an enormous difference to ease of use and tweaking algorithms for performance on cold and warm cache file systems. Instead AFIO uses a functional result passing design currently based on plain old futures. Benchmarks for the v1.2 engine are more than plenty for even a top end SSD (https://boostgsoc13.github.io/boost.afio/doc/html/afio/FAQ/closure_pe rformance.html) but latencies are in my opinion not good enough (https://boostgsoc13.github.io/boost.afio/doc/html/afio/FAQ/closure_la tency.html), so the work plan is as follows: v1.3 engine: Port Boost.AFIO to proposed Boost.BindLib, thus making AFIO standalone capable. This part is nearly done, it's mainly all dotting i's and crossing t's to go, mainly on older compilers like GCC 4.6. The red items on the CI dashboard (https://boostgsoc13.github.io/boost.afio/) are those todo fixme support items apart from anything before VS2013 support for which I am dropping. v1.4 engine: Replace spinlocked unordered_map with proposed Boost concurrent_unordered_map. This should yield a substantial performance increase due to hugely improved concurrency and a large decrease in latencies due to eliminating much use of malloc. v1.5 engine: Replace async_io_op with a custom future type based on the non-allocating constexpr continuations based basic_future I proposed on this list some time ago. This should completely eliminate malloc from the AFIO core engine, and I would assume we shall see perfectly optimal performance and latencies in AFIO. A "free" bonus is that AFIO becomes monadically programmable using either proposed Boost.Hana or proposed Boost.Expected, or indeed any monadic C++ programming library of your choice (if it provides basic_future support). This ought to make "simple" use of AFIO much more amenable and its current batch based API is a big gnarly for occasional users. If approved to present at C++ Now, I shall present on all this there for those interested, and I currently intend to submit a talk or two to CppCon. Niall -- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
participants (8)
-
Andrey Semashev -
Bjorn Reese -
Josh Juran -
Nat Goodspeed -
Niall Douglas -
Olaf van der Spek -
Rob Stewart -
Sylvester-Bradley, Gareth