- Boost - lists.preview.boost.org

[thread, async] Mini design review requested for a generic extensible Boost monadic continuations framework
by Niall Douglas 09 Mar '14

09 Mar '14

Dear Boost, I've been working on a Boost generalisation framework for N3857 "Improvements to std::future<T> and Related APIs" (Jan 2014). It's also an extension of Vicente's N3865 "More Improvements to std::future<T>" (Jan 2014), though slightly incompatible with it. I would appreciate the Boost community's feedback on what is essentially pure exploratory prototype code before I commit any major design flaws in a proper implementation. The proposed design is specifically intended to allow proposed Boost.AFIO's async_io_op future-like object to seamlessly interoperate with any other future-like object e.g. std::future<T> or boost::future<T>. The design is intended to be extensible to any other third party future-like object, even including ones whose implementation is completely external (e.g. Qt). I would like to thank some people whose off-list discussions helped inform this design: to Bjorn Reese who has done so much to help me knock the kinks out of AFIO; to Vicente J. Botet Escriba for engaging with me pitching this proposal to him (he disagrees with it by the way); to Artur Laksberg and Niklas Gustafsson for continuing to reply to my criticisms and thoughts of the papers which precede N3857. What N3857 proposes for reference: * future<T> and shared_future<T> gain a .then(R(prec_future_type<T>)) method. Callable will be called with the signalling future after the future signals. Return type is a future<R>. * A new when_all(...) can be used to compose a future which signals when all input futures are signalled. Inputs can be futures of heterogeneous types, in which case a future<tuple<...>> is returned. * A new when_any(...) composes a future which signals when any of its input futures signals. Inputs can be futures of heterogeneous types, in which case a future<tuple<...>> is returned. To avoid the linear scan on return, for a homogeneous types a when_any_swapped(...) swaps the last item in the return value with the ready item. Note there is no way in N3857 to avoid a when_any() linear scan when input types are heterogenous (something I feel is a deficiency, all you need to return is an index!). * A new executor base class, with subclass thread_pool are added. Executors allow adding a std::function<void()> for asynchronous execution. These are detailed by N3785 "Executors and schedulers, revision 3" (Oct 2013). I haven't mentioned in detail here, but N3857 specifies overloads for .then(executor, ...) to send some continuation to some executor. You will note the below framework makes it possible to use tag dispatch instead which looks much cleaner and is easier to write, nevertheless this explicit form is intended to be preserved. I won't mention executor support again for brevity. Where I find issue with a naïve implementation of N3857 (and Vicente's N3865 mentions some of these same issues): * N3857 only understands std::future<> and std::shared_future<>. That means the following code will not compile: std::future<int> &a; boost::future<int> &b; boost::afio::async_io_op &c; std::future<std::tuple<int, int, std::shared_ptr<async_io_handle>>> f=when_all(a, b, c); This is unhelpful for interoperation. It doesn't just affect Boost: imagine mashing up Qt async objects and C++17 objects for example, or even WinRT Task<T> objects and C++17 objects. Most C++ (and C) libraries of any complexity supply their own async notification objects, and this problem of writing never ending boilerplate to get some async object from library X to work with library Y is very tedious considering the compiler can do it for you. * As I mentioned earlier, when_any() is suboptimal when called with heterogenous types. I don't particularly like the when_any_swapped() hack either - a better solution would solve when_any() of all kinds optimally. * The .then(R(prec_future_type<T>)) is not tremendously useful in practice because it's too limited. Witness the very common use pattern of this form: auto ret=openedfile .then(write({ "He", "ll", "o ", "Wo", "rl", "d\n" }, 0)) .then(sync()); .then(write({ "He", "ll", "o ", "Wo", "rl", "d\n" }, 12)) .then(sync()); This sequence of code is clearly a *transaction* i.e. the programmer intends the set of continuations as a contained group of operations, and indeed some async i/o providers may guarantee atomic rollback (hint: forthcoming TripleGit does exactly this). A big additional problem is the lack of error handling logic: if say the second write() fails, you may wish to undo the first write, or do something custom. The problem with N3857 is that only the state of the first sync() is passed into the second write, and it may be the case that in generic code sync() needs to always sync irrespective of its preceding operation and therefore might pass through a garbage state. N3857 provides no easy method for a later .then() to inspect preceding .then()'s and do something about them (e.g. rewinding until it finds a valid open file handle rather than error states), other than by declaring special wrapper lambdas to pass through state. I personally think that is ugly. * Vicente mirrors my point above in N3865 by pointing out that having .then(R(prec_future_type<T>)) push the error handling into the callable makes your callables unnecessarily boilerplate. N3865 proposes adding .next(R(prec_future_type<T>)) for when a preceding operation succeeds and .recover(R(exception_ptr)) for when a preceding operation fails, with .fallback_to(value) as a way of default setting the return from an errored future. I think Vicente's argument here is a good one: but it's not generic enough in my opinion. I think continuations should be able to _filter_ outputs from preceding continuations in a really generic and customisable way. What I'd really like is this: continuations::thenable_get_placeholder<-1> last; auto ret=openedfile .then(write({ "He", "ll", "o ", "Wo", "rl", "d\n" }, 0)) .then(if_error(last, [](future<...> f){ do something with last; })) .then(sync()); .then(write({ "He", "ll", "o ", "Wo", "rl", "d\n" }, 12)) .then(if_error(last, [](thenable<...> t){ do something more with everything; })) .then(sync()); Which treats the continuation as a functional transformation. In other words, Vicente's .recover(R(exception_ptr)) becomes tag dispatched instead of explicitly specified, so it's very easy for the programmer to transform state according to state. What I am proposing instead: a generic extensible Boost monadic continuations framework which via metaprogramming lets the compiler assemble the right code from N3857 like syntax. A quick summary of the proposed generic extensible Boost monadic continuations framework follows: * Future-like behaviours such as .valid(), .then(), .get(), .wait() and .share() have functional mixin classes of the form (simplified for brevity, but you get the picture): namespace boost { namespace continuations { // Indicates the derived type provides a future compatible valid() template<class I> struct validable : public I { bool valid() const; }; // Indicates the derived type provides a future compatible then() template<class I, class... Types> struct thenable : public I { template<class F> thenable<decltype(F(...))>, Types..., I> then(F &&f); }; // Indicates the derived type provides a future compatible get() template<class T, class I> struct gettable: public I { T get(); }; // Indicates the derived type provides a future compatible wait() template<class I> struct waitable: public I { void wait() const; template<class Rep, class Period> std::future_status wait_for(const std::chrono::duration<Rep,Period> &timeout_duration) const; template<class Clock, class Duration> std::future_status wait_until(const std::chrono::time_point<Clock,Duration> &timeout_time) const; }; } } To mark up a type as being future-like, one might use: namespace boost { namespace continuations { template<class T> using monad = thenable<waitable<gettable<T, validable<std::future<T>>>>>; // Do all the partial specialisations of the internal machinery // for this type to hook it into the metaprogramming } } // Voila, boost::continuations::monad<T> is now an interoperable // form of std::future<T> * .then() can now take callables of the following forms: 1. R() and R(prec_future_type<T>) for compatibility with N3857. 2. R(..., continuations::thenable_get_placeholder<idx>, ...) inserts an item from the continuation chain into the call. thenable_get_placeholder can be fed a negative number to wrap from the bottom, so thenable_get_placeholder<-1> is the most recently preceding operation. Therefore R(prec_future_type<T>) from above is actually implemented as R(continuations::thenable_get_placeholder<-1>). This can be used to insert error processing filters as illustrated earlier which can view a preceding chunk of chain at once, and act. 3. R(thenable<...> &&) which gives a continuation access to the entire continuation chain, just for those people who really need it. You saw this in the example above. 4. Any specially treated prototype depending on what the metaprogramming has extended for some future_type. For example, if one does async_io_op.then(void(const boost::system::error_code& error, size_t bytes_transferred)) - which is the ASIO callback spec - then AFIO would emulate the ASIO callback calling convention for the completion of some i/o. This only happens if and only if the item being .then() is an async_io_op i.e. such custom extensions are type specific. * future_type<T>.then(R()) now returns a thenable<future_type<R>, future_type<T>> which auto decays to a future_type<R> on demand for compatibility with N3857. Similarly a future_type<T>.then(R()).then(S()) would return a thenable<future_type<S>, future_type<T>, future_type<R>> and so on. You can convert a thenable<T, ...> into a std::tuple<..., T> using an overload of the make_tuple() function. * when_all() and when_any() gain overloads for thenable<...>. These simply convert the tuple taken out of a thenable chain into a future with the results - AFIO will implement this generically using its closure engine, but a more intelligent solution might decompose all inputs into HANDLEs and do an asynchronous WaitForMultipleObjects() for example. Note this is a BREAKING CHANGE from N3857: If you do a when_all() under N3857 on a .then() chain, you will get a future to the last then() return, whereas under this proposal you will get a future to all the returns from all the then()s. This breaking change may still actually compile and work as expected - however when_any() actually now has completely different behaviour: when_any() under N3857 will be the same effect as when_all() on a .then() chain under N3857, whereas under this proposal when_any() will signal when any of the then() chain signals (I would assume the first item). I personally doubt this breaking change would affect any real world code, but it's worth noting. Note the following: 1. The proposed continuations framework, despite being monadic, is separate from the mooted Boost.Functional/Monads framework. This continuations framework could be implemented using such a Boost.Functional/Monads framework, but I suspect there wouldn't be much gain except interoperability with other monadic patterns. Continuation monads are a very limited specialisation of general monads, especially under the tight constraints of the N3857 requirements, so reuse of any Boost.Functional/Monads I suspect would be minimal. 2. The proposed framework is 100% compatible with N3857 apart from the when_all()/when_any() breakages which I would assume shouldn't turn up in real world code, and it should be viewed as a Boost-specific extension which may get standardised later if it proves popular. Here is the very rough prototype code: https://github.com/BoostGSoC/boost.afio/blob/monadic_continuations/lib s/afio/test/tests/monadic_continuations_test.cpp. It compiles using VS2013 only. It probably doesn't work, but it's simply there to demonstrate the viability of the concept and to help those confused by my unclear explanation above. Thoughts regarding the design much appreciated! My thanks in advance. Niall -- Currently unemployed and looking for work in Ireland. Work Portfolio: http://careers.stackoverflow.com/nialldouglas/

2 5

Segment Tree Implementation for GSoC 2014
by Kevin Zuniga Garate 09 Mar '14

09 Mar '14

Hello everyone, I'm Kevin and I'd like to contribute to Boost in this SoC. I enjoy coding in C++ and participate a lot in competitive programming sites such as TopCoder, Codeforces, etc. A data structure I have learn from participating in those contests is segment tree. A segment tree is a data structure for storing data and querying and updating it in intervals, I haven't found a segment tree library in C++, so I thought it may be a good data structure for Boost. Do you think that would be a good addition for Boost? If not, what other data structure would you like to see implemented? Greetings. -- Kevin

4 9

Multi-methods and type switch
by Camille GILLOT 08 Mar '14

08 Mar '14

Hello, SoC2014 page on trac mentions interest for an implementation of type-switch for C++. For now, I have implemented a toy library for open multiple dispatch. It is based on compressed dispatch tables. It still just supports single inheritance and needs to attach integer tags to the dispatched class hierarchy. Is there any interest for further development of this library ? What would be the typical use cases ? Note : Although I opened with a reference to SoC2014, I do not intend to write a GSoC proposal for this work. Best regards, Camille Gillot

3 5

[tuple] any interest in a unique_tuple<...>?
by Hui Li 08 Mar '14

08 Mar '14

Testing the water, see if anyone might find this useful. Any comments/suggestions are absolutely welcome and appreciated! unique_tuple<T...>, where types in <T...> may themselves be other unqiue_tupe<...>'s, contains exactly one copy of each of the "finally nested" element types in <T...>, regardless of how many times a type appears in the type parameter pack, or how deeply it is nested inside one or more unique_tuple's in <T...>. unique_tuple<T...> differs from std::tuple<T...> and boost::fusion::set<T...> etc in that nested unique_tuple's are effectively "flattened", and elements contained are unique no matter how complex the nesting gets, or how many times or how deeply nested the element type is in <T...>. unique_tuple<T...> in a way emulates virtual inheritance, but at compile time. The implementation does NOT use virtual inheritance. implementation is available at: https://www.dropbox.com/sh/f7ur7j6jetabovj/MuhhhNWU0K/unique_tuple.h and some demonstration of uses at (tested with "clang-3.5 -std=c++1y" on mac osx/macport) https://www.dropbox.com/sh/f7ur7j6jetabovj/cy_cvpPIYc/main.cpp In the following example, t1,t2,t3 all contain exactly one A, and exactly one B: struct A{}; struct B{}; unique_tuple<A,B,A,B> t1(A{},B{}); unique_tuple<A,unique_tuple<B>,unique_tuple<A,B>> t2{B{},A{}}; // order doesn't matter in constructor // B is default constructed, empty unique_tuple (possibly nested) doesn't matter unique_tuple<A,unique_tuple<unique_tuple<>>,unique_tuple<A,B,unique_tuple<A,B,unique_tuple<A,B>>>> t3{A{}}; // unique_tuple<A,B> t4(A{},B{},A{}); // doesn't compile because more than one A is supplied // unique_tuple<A,B> t5(A{},B{},1.0); // doesn't compile because double doesn't belong There are helper classes and functions, e.g.: A& a = get<A>(t0); auto x = make_unique_tuple(A{},B{},1,1.0); // unique_tuple<A,B,int,double> static_assert( is_unique_tuple<decltype(t0)>::value, "t0 is a unique_tuple type" ); static_assert( is_element_of<A,decltype(t1)>::value, "A is an element in t1" ); static_assert( is_subset_of<decltype(t1),decltype(x)>::value, "<A,B> is a subset of <A,B,int,double>" ); static_assert( unique_tuple_size<decltype(t3)>::value == 2, "there are 2 elements in t3: A and B" ); Equality comparason exists for two unique_tuple<...>s that contain the same element types, e.g., t0 == t2 && t1 != t3; Mixed operations with std::tuple are available if std::get<typename>(std::tuple<...>) exists (c++14), which effectively requires that the element types are all unique in the std::tuple<...>. synopsis: template < typename... T > struct unique_tuple { template < typename... U > explicit constexpr unique_tuple(U&&...); template < typename... U > explicit constexpr unique_tuple(const std::tuple<U...>&); template < typename... U > explicit constexpr unique_tuple(std::tuple<U...>&&); // allocator extended constructors template < typename Alloc, typename... U > unique_tuple(std::allocator_arg_t, const Alloc& a, U&&...); template < typename Alloc, typename... U > unique_tuple(std::allocator_arg_t, const Alloc& a, const unique_tuple<U...>&); template < typename Alloc, typename... U > unique_tuple(std::allocator_arg_t, const Alloc& a, unique_tuple<U...>&&); template < typename Alloc, typename... U > unique_tuple(std::allocator_arg_t, const Alloc& a, const std::tuple<U...>&); template < typename Alloc, typename... U > unique_tuple(std::allocator_arg_t, const Alloc& a, std::tuple<U...>&&); // assign template < typename... U > unique_tuple& operator=(const unique_tuple<U...>&); template < typename... U > unique_tuple& operator=(unique_tuple<U...>&&); template < typename... U > unique_tuple& operator=(const std::tuple<U...>&); template < typename... U > unique_tuple& operator=(std::tuple<U...>&&); // swap template < typename... U > void swap(unique_tuple<U...>&); template < typename... U > void swap(std::tuple<U...>&); }; // helpers template < typename... T > unique_tuple<typename std::decay<T>::type...> make_unique_tuple(T&&... t); template < typename... T > struct unique_tuple_size<unique_tuple<T...>>; // number of unique elements template < typename T > struct is_unique_tuple; template < typename T, typename... U > struct is_element_of<T,unique_tuple<U...>>; template < typename... T, typename... U > struct is_subset_of<unique_tuple<T...>,unique_tuple<U...>>; // element access template < typename T, typename... U > T& get<T>(unique_tuple<U...>&); template < typename T, typename... U > const T& get<T>(const unique_tuple<U...>&); template < typename T, typename... U > T&& get<T>(unique_tuple<U...>&&); // equality compare template < typename... T, typename... U > bool operator==(const unique_tuple<T...>&, const unique_tuple<U...>&); template < typename... T, typename... U > bool operator==(const unique_tuple<T...>&, const std::tuple<U...>&); template < typename... T, typename... U > bool operator==(const std::tuple<T...>&, const unique_tuple<U...>&); template < typename... T, typename... U > bool operator!=(const unique_tuple<T...>&, const unique_tuple<U...>&); template < typename... T, typename... U > bool operator!=(const unique_tuple<T...>&, const std::tuple<U...>&); template < typename... T, typename... U > bool operator!=(const std::tuple<T...>&, const unique_tuple<U...>&); // swap template < typename... T, typename... U > void swap(unique_tuple<T...>&, unique_tuple<U...>&); template < typename... T, typename... U > void swap(unique_tuple<T...>&, std::tuple<U...>&); template < typename... T, typename... U > void swap(std::tuple<T...>&, unique_tuple<U...>&);

2 2

Re: [boost] testing potential GSOC students
by Ganesh Prasad 08 Mar '14

08 Mar '14

( (c) TM (R) Will Smith) --> (The man in BLACK ) ... The actor one ??? I love that guy. Not sure if I would go competitive against him. :D

1 0

Re: [boost] testing potential GSOC students
by Ganesh Prasad 07 Mar '14

07 Mar '14

As this year the main focus of UBLAS will be compile time optimizations, and I wish to submit a proposal for this project, I would like to show the compile time library Gns::META_LISCPP , written by me. In reference to *Boost Digest, Vol 4288, Issue 3, Message :3 ,* I am sending this library by me for evaluation. Though I have not implemented a minimal Vector library over it yet , still it demonstrates, I optimally guess, my understandings of the generic meta programming paradigm. And I guess, I might come up with some vector processing features for it soon. Here is the github url to the repository : *https://github.com/GnsP/GNS--META_LISCPP/ <https://github.com/GnsP/GNS--META_LISCPP/>* Looking forward to your responses. Best Wishes Ganesh Prasad

3 2

Re: [boost] [Re] [Re] JSON Parsing Library for GSoC 14
by Sebastian Theophil 07 Mar '14

07 Mar '14

There are many libraries that parse JSON to maps and vectors. IIRC Microsoft’s REST SDK includes a JSON library as well (http://casablanca.codeplex.com) There are at least 15 other C++ libraries like that and we don’t need to write another one. But there are only two libraries AFAIK that parse JSON to C++ structs using serialization adaptors: - https://github.com/Loki-Astari/ThorsSerializer - http://fossil.wanderinghorse.net/repos/nosjob/index.cgi/index I consider the ability to read and write C++ structs essential for any library admitted to boost. Shouldn't such a library be an extension to boost.serialization? Boost.Serialization already defines a way for each data structure to define which data should be written/read. The Boost.Serialization docs say that the library even offers methods & macros to output a class member together with its member variable name as a name value pair. Exactly what a JSON library would need. There has been a discussion about last year’s GSoC mentioning further desirable properties of a JSON library: http://boost.2283326.n4.nabble.com/JSON-Parser-GSoC-2013-td4645117.html Regards Sebastian -- Dr. Sebastian Theophil | stheophil(a)think-cell.com Senior Software Engineer We are looking for C++ Developers: http://www.think-cell.com/career think-cell Software GmbH | Chausseestr. 8/E | 10115 Berlin | Germany http://www.think-cell.com | phone +49 30 666473-10 | US phone +1 800 891 8091 Amtsgericht Berlin-Charlottenburg, HRB 85229 | European Union VAT Id DE813474306 Directors: Dr. Markus Hannebauer, Dr. Arno Schödl

3 4

[GSoC] pipelines questions
by Thaler Benedek 06 Mar '14

06 Mar '14

Hi, I'm interested in the Boost.Pipelines proposal of GSoC 2014. [0] I'd like to ask the stakeholder (Vicente J. Botet Escriba ?) the following: * Is this supposed to be built from scratch or rather to be some kind of improvement over the already existing google implementation? [1] * If the former is true, should be n3534 [2] considered as a strict requirement or as a vague suggestion? * If the later is true, which features should be the main distinctive ones? I think about things like using Boost libraries (e.g: Boost.Atomic) instead of custom solutions, cross platform support, comply to Boost coding standard, etc. Are there any areas of improvement identified in the reference implementation? * I understand it might take quite a long time for a proposal to make it to Boost, therefore it's not a requirement. Is there any objective milestone that a successful project should reach anyway? Thanks, Benedek [0]: https://svn.boost.org/trac/boost/wiki/SoC2014#Boost.Pipelines [1]: https://code.google.com/p/google-concurrency-library/source/browse/include/… [2]: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3534.html#Proposal

3 4

[uuid] sequential_generator / comb_generator / utc_generator / timestamp_generator
by Adam Walling 06 Mar '14

06 Mar '14

For personal use, I've created a few generators that create sequential and time-based uuids. However there does not seem to be a single standard for these kinds of algorithms. While tidying up this code, I've run into some obstacles with generalization of the logic. Although some applications like SQL Server will use GUIDs and lexicographically compare based on the least significant bytes of Data4 for boost::uuids' purposes we can simply assume a block of 16 bytes. The generic approach is to combine (basic_comb_generator) a number of bytes from a sequential source, filling the rest of the bytes with random data. What concerns me is the choice of source and the number of bytes to use. And additionally the value which specifies the variant / version. The usual choices seem to be 8 or 6 bytes. The source is also usually UTC Unix time. However the original uuid version used nanosecond intervals and the MAC address. Am I worrying too much about extensibility? As it is now my basic_comb_generator has a template param for a source, a number of bytes of the source to use, and a UniformRandomNumberGenerator to fill the rest. Since resolution may be a second if using Unix time they may not be consistently sequential but generally so, hence my hesitation to name the generator typedef as sequential. Of course it is possible to use even higher resolution. I suppose I am gauging interest in this feature, as well as asking if there are any thoughts or suggestions on how others would likely use this, and alternative or standard algorithms defined. Once I have some feedback I intend to complete the code and bounce it back on here for any further comments before lobbying to include anything within the library. It seems like commonly needed functionality at least! Nilsson article here for SQL Server: http://www.informit.com/articles/article.aspx?p=25862&seqNum=7 CouchDB's uuid generation algorithms: https://wiki.apache.org/couchdb/HttpGetUuids

2 1

[signals2] Test failure in C++11 (trivial fix for incorrect usage of boost::optional)
by Ben Pope 06 Mar '14

06 Mar '14

I have the following errors: "clang++-3.4" -c -x c++ -std=c++11 -stdlib=libc++ -O0 -g -fno-inline -Wall -g -DBOOST_ALL_NO_LIB=1 -DBOOST_TEST_NO_AUTO_LINK=1 -I".." -o "/home/ben/development/boost/test/build/results/boost/bin.v2/libs/signals2/test/signal_test.test/clang-linux-3.4~c11_libc++/debug/link-static/signal_test.o" "../libs/signals2/test/signal_test.cpp" ../libs/signals2/test/signal_test.cpp:32:16: error: invalid operands to binary expression ('boost::optional<int>' and 'int') if(max == false) max = *first; ~~~ ^ ~~~~~ "clang++-3.4" -c -x c++ -std=c++11 -stdlib=libc++ -O0 -g -fno-inline -Wall -g -DBOOST_ALL_NO_LIB=1 -DBOOST_TEST_NO_AUTO_LINK=1 -I".." -o "/home/ben/development/boost/test/build/results/boost/bin.v2/libs/signals2/test/track_test.test/clang-linux-3.4~c11_libc++/debug/link-static/track_test.o" "../libs/signals2/test/track_test.cpp" ../libs/signals2/test/track_test.cpp:36:14: error: invalid operands to binary expression ('boost::optional<int>' and 'int') if(max == false) Richard smith explains the problem here: http://llvm.org/bugs/show_bug.cgi?id=18180#4 "OK, I see what's happening -- in Clang 3.3 and GCC 4.8, 'false' is being treated as a null pointer constant. boost::optional implicitly converts to an "unspecified-bool-type" that happens to be a pointer-to-member type, and thus can be compared against a null pointer constant via the built-in operator!=(pointer-to-member, pointer-to-member) overload. In C++11 onwards, 'false' isn't a null pointer constant any more, so this comparison is no longer valid. Clang 3.3 and GCC 4.8 don't implement that rule; Clang 3.4 does." The trivial fix is to change the if statements at: ../libs/signals2/test/signal_test.cpp:32:16: ../libs/signals2/test/track_test.cpp:36:14: to if(!max) rather than if(max == false). Ben

7 19