March 2019 - Boost - lists.preview.boost.org

Networking TS + Beast, NEW Tutorials, Read this to learn std::net !!!
by Vinnie Falco 17 Mar '19

17 Mar '19

Fellow C++, Boost, and WG21 Enthusiasts, lend me your ear! I write to inform you about exciting developments in C++ networking. First, a bit of background. Networking comes in three flavors: * Networking TS <https://cplusplus.github.io/networking-ts/draft.pdf> * Boost.Asio <https://www.boost.org/doc/libs/1_69_0/doc/html/boost_asio.html> * Standalone Asio <https://github.com/chriskohlhoff/asio> These three are for the most part identical, except that Asio flavors have additional features like ssl::stream and signal_set which are not in the TS, but will very likely appear in a future update or version. We've had Asio for over a decade now, but there is a shortage of experts. Some people believe this shortage is because Asio in particular (and thus, Networking TS since they have identical interfaces) is "difficult to use." I believe it is wrong to blame Asio for this. Concurrent programs in general are hard to write. This is applicable: "Unfortunately, today's reality is that only thoughtful experts can write explicitly concurrent programs that are correct and efficient. This is because today's programming models for concurrency are subtle, intricate, and fraught with pitfalls that easily (and frequently) result in unforeseen races (i.e., program corruption) deadlocks (i.e., program lockup) and performance cliffs (e.g., priority inversion, convoying, and sometimes complete loss of parallelism and/or even worse performance than a single-threaded program). And even when a correct and efficient concurrent program is written, it takes great care to maintain — it's usually brittle and difficult to maintain correctly because current programming models set a very high bar of expertise required to reason reliably about the operation of concurrent programs, so that apparently innocuous changes to a working concurrent program can (and commonly do, in practice) render it entirely or intermittently nonworking in unintended and unexpected ways. Because getting it right and keeping it right is so difficult, at many major software companies there is a veritable priesthood of gurus who write and maintain the core concurrent code." - Herb Sutter, "The Trouble with Locks", <http://www.drdobbs.com/cpp/the-trouble-with-locks/184401930> Although this was written in 2005 it is still relevant today. It is understandable that Asio will be the first target of anger and frustration when writing concurrent programs, since it is on the "front line" so to speak. There has also been a distinct shortage of *good* tutorials and examples for Asio. Articles or blog posts which teach you step by step, explaining everything, and giving example code which demonstrates best practices. Boost.Beast is my low-level HTTP/WebSocket library which builds on Boost.Asio: <https://github.com/boostorg/beast> In the original release of Beast, the documentation stated "prior understanding of Boost.Asio is required." However, field experience has shown that users ignore that requirement and attempt to write complex, concurrent programs as their first-time introduction to both Beast and Asio. Based on feedback from committee members, and to serve users better, the scope of Beast has been enlarged to include first-time users of networking. The upcoming Boost 1.70 release reflects this new scope and I am excited to announce some very nice things which you can access today. First of all, Beast documentation and examples no longer use the "boost::asio" namespace, they the namespace alias "net::". While this is cosmetic, it reinforces the notion when inspecting code that it is equally applicable to Boost.Asio, Asio, and Networking TS (identifiers which are not in the TS, such as signal_set, are still qualified with boost::asio). A new documentation page explains the three flavors of networking: <https://www.boost.org/doc/libs/master/libs/beast/doc/html/beast/using_io.ht…> This is also explained in my 2018 CppCon presentation: <https://youtu.be/7FQwAjELMek?t=444> I have added a "Networking Refresher", a complete overview of networking from soup to nuts. No prior knowledge or understanding of networking is required, everything is explained in detail so if you want to learn this is the place to start. I also kept it short, but it is loaded with hyperlinks for further learning: <https://www.boost.org/doc/libs/master/libs/beast/doc/html/beast/using_io/as…> There was a recent paper in Kona, P1269R0 ("Three Years with the Networking TS") about difficulty of implementing timeouts. To address this, Beast now has a stream class which implements configurable timeouts for you, and callers no longer have to fiddle with timers manually anymore. Everything "Just Works." It achieves the P1269R0 author's goal of having timeouts "built-in to asynchronous operations", but in a way that fits in with the design of the TS: <https://www.boost.org/doc/libs/master/libs/beast/doc/html/beast/using_io/ti…> I feel that this `beast::basic_stream` serves as an existence proof that the current design of Networking TS is sound - the TS offers a flexible toolbox which lets you build your own framework the way that you want it, without making odd choices for you. We are still discovering ways of leveraging it to maximum use. The beast::websocket::stream also has built-in timeouts, but they are enhanced to support "idle pings" (keeping client connections alive) and everything is fully configurable: <https://www.boost.org/doc/libs/master/libs/beast/doc/html/beast/using_webso…> All you need to do to get sensible, suggested websocket timeouts is add one line of code after creating your stream: ws.set_option(websocket::stream_base::timeout::suggested( beast::role_type::server)); To address the cumbersome boilerplate of writing composed operations (specifically the need to propagate the associated allocator and associated executor, and to avoid invoking the completion handler from within the initiating function when the operation would complete immediately) Beast adds two new utility base classes, with plentiful documentation and examples throughout: <https://www.boost.org/doc/libs/master/libs/beast/doc/html/beast/using_io/wr…> There are two well-rounded examples which show you step by step how to write these things in a safe way: <https://www.boost.org/doc/libs/master/libs/beast/doc/html/beast/using_io/wr…> <https://www.boost.org/doc/libs/master/libs/beast/doc/html/beast/using_io/wr…> I have a big, new open source project which implements a server, that uses the `system_context`, taking full advantage of native Windows and Mac OS system-level execution context features. To support this use case and industry feedback, the examples in Beast now default to being always thread-safe. All examples use a "strand", and leverage P1322R0 ("Networking TS enhancement to enable custom I/O executors"). Yes, this paper which was approved in Kona, is now implemented in both Boost.Beast, and Boost.Asio, including all of the Beast examples, so if you pick up Boost 1.70 (or the master branches from github) you can start playing with this as soon as you're done reading this message!! <http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1322r0.html> We have an active #beast channel in the C++ Slack (https://slack.cpp.al) where experienced users can help, no question is too small! I hope you will join me and the rest of the Beast and Asio community in exploring what the very powerful Networking TS and Asio libraries have to offer, and build something great together! Regards P.S. Don't forget to star the repository! <https://github.com/boostorg/beast>

4 7

I am interested to participate in BOOST GSOC2019 project
by raghuchandra kanakamedala 17 Mar '19

17 Mar '19

I am raghuchandra studying 3rd year B.Tech at IIIT Hyderabad.It would be great to contribute to Boost.GIL: generic image library.I have ideas for both the projects.I had this course Digital Image Processing also done a project "Road detection from satellite images" based the paper attached.my implementation link on git https://github.com/savera2020/dip_project/.I have good skills in C++ programming. so can you provide the mail id of mentor so that we can go more deeper about the project.please consider my request to contribute to project.without a doubt i will give my best. Thanking You.

1 0

How should I teach a C++ college class?
by Joshua Marshall 17 Mar '19

17 Mar '19

I need some general input here. I'd like to pitch and run a semester long C++ course to students who have at least 1 year of programming experience. I'd like to have this open to open ended input. The biggest technical restrictions are that the University systems are running RHEL 7.4 (7.5?) with software collections devtoolset-8.

3 3

Re: [boost] [Boost-users] Networking TS + Beast, NEW Tutorials, Read this to learn std::net !!!
by Vinnie Falco 15 Mar '19

15 Mar '19

On Fri, Mar 15, 2019 at 7:38 AM Stian Zeljko Vrba <vrba(a)quine.no> wrote: > I have an idea of how to accomplish the same in C++, and > it's not pleasant -- worker thread, promise/future, blocking > queue and CancelSynchronousIO. Cannot even use > std::async because CancelSynchronousIO needs a > target thread ID. Yes, yes! I see now...your ideas on how to implement it in C++ convince me more than anything that the documentation is the problem. There's no need for any of that stuff, the code you provided could be implemented in C++ using asio without too much fuss, and there are actually several ways to do it all of which are succinct and will work wonderfully.

1 0

Re: [boost] [Boost-users] Networking TS + Beast, NEW Tutorials, Read this to learn std::net !!!
by Vinnie Falco 15 Mar '19

15 Mar '19

On Fri, Mar 15, 2019 at 7:19 AM Stian Zeljko Vrba <vrba(a)quine.no> wrote: > > I believe it is wrong to blame Asio for this. > > I disagree. Recently, I've been coding parallel, including networked & > asynchronous, programs in C# and Java, and the experience has been a JOY. You > get threads, executors, synchronization primitives, tasks, cancellation, > monadic futures and concurrent (blocking and non-blocking) data structures > out of the box with the platform, plus a myriad of extensions as libraries. > As for executors, you don't need to be bothered with them unless you really > want to for some reason. > If you were to compare POSIX or BSD sockets to C# and Java, you would say exactly the same things. Asio (like Beast) is a low-level, portable abstraction of I/O. You can BUILD "threads", "executors", "synchronization primitives", "tasks", "cancellation", "monadic futures", and "concurrent data structures" with it. Does this mean OS-level sockets are the wrong abstraction? Of course not. The reason C# seems "ahead" of C++ is two-fold: 1. WG21 goofed up by delaying networking 2. C# is not developed through the International Standards Process These mistakes are being fixed though, and you will see library support in C++ catch up by leaps and bounds. And when we finally get these things (built on top of Asio/Networking TS, which has proven over and over to be the correct low-level abstraction) it will be better than C#, Java, or Rust. Compare the documentation for Vertx (vertx.io) or netty (both Java toolkits > for writing asynchronous reactive programs) with that of asio. > Yes I agree, there is a lack of documentation and especially tutorials for beginners (which I am trying to fix). This has nothing to do with whether Asio is the right portable, low-level abstraction (it is) but the lack of guidance from the author could lead you to believe incorrect things about the lib. Once I attempted to write an asio service, tried to understand the simple > example from the documentation, and I gave up. I used a thread, blocking > call and CancelSynchronousIO (and I consider myself fortunate to develop > for Windows only that has it). > The documentation on services is particularly scant. There is no explanation of what they are for, how to use them, best practices, or even "Frequently Asked Questions." I was able to figure it out though, but I am able to focus all of my time on it. > Asio _is_ a relatively nice wrapper around socket and serial port APIs, > but that's about it > Yes now you're getting close to an accurate description. More correct would be to say that Asio provides robust abstractions of buffers, dynamic buffers, synchronous I/O, and asynchronous I/O. Networking is a subset of that (but not the only part, see signals and serial ports, and soon file I/O). > IMO. On the other hand, I could have written the same wrappers around > native APIs in two days and not haul along what I consider the baggage of > technical debt that is asio in the codebase. > There is no API technical debt. That is to say, that the interfaces are modern and up-to-date. You could argue that particular implementations such as Boost.Asio have extra code to provide C++03 compatibility, but this has nothing to do with the soundness of the design. As these interfaces are part of a TS, there will be implementations in gcc and clang appearing shortly (if not already so). These will be written from scratch so there will be no support for older C++. > Conclusion, if any: many people just want/need thread per client and > synchronous IO for simplicity, but until asio/networking TS provide for > timeouts and cancellation of synchronous requests, it's a wrong tool for > those people, me included. > If you define struct sync_socket { net::io_context ioc_; net::ip::tcp::socket_; net::steady_timer timer_; ... template<class Buffers> std::size_t write_some (Buffers const& b); }; You can easily implement timeouts in `write_some` thusly: template<class Buffers> std::size_t sync_socket:: write_some (Buffers const& b, error_code& ec) { std::size_t n; timer_.expires_after(seconds(30)); timer_.async_wait( [&](error_code ec) { socket_.cancel(ec): }); socket_.async_write_some(b, [&](error_code ec_, std::size_t n_) { ec = ec_; n = n_; }); ioc_.run(); return n; } Now, you can use `sync_socket` with all the usual sync write operations like `net::write`, and get timeouts. Thanks

1 0

Re: [boost] [Boost-users] Networking TS + Beast, NEW Tutorials, Read this to learn std::net !!!
by Vinnie Falco 15 Mar '19

15 Mar '19

On Fri, Mar 15, 2019 at 2:45 AM Cristian Morales Vega <cristian(a)samknows.com> wrote: > I have not used the synchronous API at all. But section 3.1 of P1269R0 > seems to make a good point about the lack of timeouts in synchronous > calls, doesn't it? > beast::basic_stream makes using timeouts in the async API way simpler, > but is the sync API really basically unusable because of the lack of > timeouts thing? Synchronous APIs are perfectly fine. Asio defines SyncReadStream and SyncWriteStream named requirements, allowing for generic algorithms to be implemented. Not everything is a socket (for example, a serial port). A synchronous stream represents the reliable, in-order delivery of bytes using functions that block, nothing more and nothing less. For example, serial file I/O can be represented using a synchronous stream. If the synchronous stream requirements mandated timeouts, it would eliminate a broad set of implementations (including sockets on some platforms).

1 0

Status of Disjoint Sets ?
by Brian Kuhl 15 Mar '19

15 Mar '19

Disjoint Sets is in the last few releases, has it's own test suite and github archive, but isn't mentioned on the documentation page? An oversight, or intentional? https://www.boost.org/doc/libs/1_69_0/ I see a defect that says it should move into Graph, is that part of the puzzle? https://github.com/boostorg/disjoint_sets/issues Brian

2 1

Networking TS + Beast, NEW Tutorials, Read this to learn std::net !!!
by Mike 15 Mar '19

15 Mar '19

> All in all nice intro, unfortunately it requires users to understand > templates quite well, but honestly not sure if it is avoidable, with a > gazillion customization points in ASIO. Anyone aware of a wrapper library around ASIO that doesn't offer all those customization points and maybe not the full functionality/ efficiency but is easier to use? Am 15.03.19, 01:09, Ivan Matek via Boost <boost(a)lists.boost.org> schrieb: Here are comments from somebody who "knows C++" but does not know a lot about NW and has read only the 1st <[1]https://www.boost.org/doc/libs/master/libs/beast/doc/html/beast/ using_io/asio_refresher.html> page. // get iterators representing the range of characters in the buffer auto begin = range::begin(buffer.data()); auto end = range::end(buffer.data()); Quite a weird pattern, where buffer does not have member begin, end. Maybe a sentence or two describing the rationale and complexity would be nice. I know it is O(1), but it is weird that begin/end are called on pointer. std::size_t read_lineI am tired, but is this function O(n^2)? cb += bytes_transferred; // adjust the pointer and sizeI would maybe switch to write instead of write_some to keep the code shorter, and mention write_some as another option in text. [](error_code ec, std::size_t bytes_transferred) For learning purposes I would capture something in lambda by value and note that reference capture would be buggy. using type = std::allocator<void>; Personal preference: not a fan of typedefs for relatively short types used once. All in all nice intro, unfortunately it requires users to understand templates quite well, but honestly not sure if it is avoidable, with a gazillion customization points in ASIO. _______________________________________________ Unsubscribe & other changes: [2]http://lists.boost.org/mailman/listinfo.cgi/boost References 1. https://www.boost.org/doc/libs/master/libs/beast/doc/html/beast/using_io/as… 2. http://lists.boost.org/mailman/listinfo.cgi/boost

1 0

Coordinating Boost Releases with compiler releases
by Marshall Clow 14 Mar '19

14 Mar '19

In the thread "MSVC 2019 and 1.70 beta", there has been a discussion about what to do about support for MSVC 2019, which is due to be released right about the same time that Boost 1.70. One person suggested that we delay the boost release until after the MSVC release (I think the suggestion was "May"). This general question has been discussed before (several times), since tools get released fairly often - and so does Boost. In general, I am not in favor of delaying boost releases in order to add support for 'about to be released' development tools. There are two main reasons for this: * There's always another compiler / tool about to be released. Clang 8 is imminent - currently at RC5. GCC 9.1 is scheduled to be released in early May. * Boost releases happen every four months. August is not really that far off. People who want support for tool XXXX can use "the trunk", getting it either from git or from bintray. [ Everyone knows about the tarballs at bintray, right? https://dl.bintray.com/boostorg/master/ ] Also, if we were to delay the boost release until May, then what would happen to the August release? The release process for 1.71.0 is scheduled to start at the end of June. -- Marshall

5 7

[filesystems]
by Florian Lindner 14 Mar '19

14 Mar '19

Hello, I have this innocent piece of code: namespace fs = boost::filesystem; auto path = getFilename(); // returns a string fs::create_directories(fs::path(path).parent_path()); std::ofstream ofs(path + "~"); ofs << info; ofs.close(); fs::rename(path + "~", path); which causes the exception: boost::filesystem::rename: No such file or directory: "../9f/061b4f7a5e529c964659226eedd4e5~", "../9f/061b4f7a5e529c964659226eedd4e5" However, I have no idea how that could happen. I use the rename, so that a reading process never sees an empty file, but only no file or filed with info. Is there any race involved between ofs.close() and fs:rename()? The code was executed on a distributed network filesystem (lustre). Any ideas anyone? Thanks, Florian

5 9