September 2015 - Boost - lists.preview.boost.org

Fwd: [sort][container] Challenge for sorting experts, an improved stable sort
by Francisco José Tapia 23 Sep '15

23 Sep '15

Hi Jon, If the elements in the flat associative container are sorted, you need only to sort the elements to insert and after do a fusion of the two parts. If your container admit repeated elements, you can use the std::sort, if not, you must use an stable sort , and delete the repeated. You have several algorithms for to do a stable sort. In the library we are developing (https://github.com/fjtapia/sort_parallel ) you have a smart_merge_sort, which is a very fast stable sort and use only one half of the memory used by the data, is exception safe and use the move semantics I didn't see any implementation of the stable sort algorithms without additional memory, in any library GCC, TBB, IPP … I suppose because they are slow. If you have 100 elements and want to insert 30 elements, need to resize the vector to 130 elements, the last 30 are empty. If you have other vector with the data to insert, you can use smart_merge_sort, and use as auxiliary memory the last 15 elements of the main vector. After sort, delete the repeated elements. And with the two vectors you must do a half_merge fusion , and all is done. With this method you need only the vector for to sort the elements to insert. If you want I can write a small program doing this, using the code of the library.

2 2

[sort][container] Challenge for sorting experts, an improved stable sort
by Ion Gaztañaga 23 Sep '15

23 Sep '15

Hi, I'm trying to improve insertion times for flat associative containers. In range insertions one optimization is to insert the whole range in the end of the container and sort the whole vector instead of trying a loop. An additional optimization proposed in the mailing list is to have an option to create externally the internal vector type of the flat map/set, fill it and then try to move it into the flat container. That requires sorting the moved vector (and removing duplicates if needed). I could try to use std::sort/std::stable_sort but there are some problems: -> Non-portable for C++03 compilers and Boost.Move, and Boost.Container fully supports those platforms. -> Additional O(N) temporary memory for stable sorting. Usually this means a temporary N element array, although there are algorithm to use only N/2 temporary elements. With big containers, this is a lot of memory. So I started analyzing several papers and implementations to see if a custom implementation was possible, but I felt Boost.Sort could be a good place to have that implementation and the Boost community a good place to discuss design alternatives. My initial requirements: 1) Stable sort. A non-stable sort is faster and useful for unique keys containers, but stable sort can be used initially in all flat containers until the non-stable version is implemented. 2) Externally supplied auxiliary memory with optionally no auxiliary memory at all. It seems that there are newer algorithms like Grailsort (https://github.com/Mrrl/GrailSort) or WikiSort (https://github.com/BonzaiThePenguin/WikiSort) that claim O(1) memory, even no auxiliary buffer, and key comparison and data exchanges with complexity O(n*log(n)). However I don't think those implementation correctly handle the basic exception guarantee or non-trivially copyable types. Reason: I want to avoid allocating short-lived temporary memory: if it's available I want to pass the extra capacity that flat containers' internal vector holds for future insertions. 3) Customization points (without ADL, which IMHO is not customizable enough). The sort algorithm should use a SortTraits template parameter for basic operations like "move", "swap", "initialized_move". E.g: template<class T> struct sort_traits { static? void move(T &from, T &to); static? void swap(T &l, T &r); static? void uninitialized_move(T &from, void *to); static? void destroy(T &from, void *to); }; I don't know if this trait should be stateful or not (static members). I could imagine the usefulness of a stateful implementation of sort_traits that could store a reference to an allocator so that some special "construct" version is used in some operations. Or just a version that counts the number of each operation to ease debugging. In any case, a stateless version would be enough for most cases. Reason: In Boost.Container, the idea is to use boost::move + boost::adl_move_swap in the initial implementation, maybe some destructive move implementation in a second iteration if Boost.Move implements it. In any case those operations should be customizable by the user, independently from the type to be sorted. This also avoids unnecessary dependencies in Boost.Sort. -> Dependencies: the implementation should only rely on Boost.Core, Boost.Config, maybe Boost.TypeTraits and no heavy STL dependencies (optionally). Currently Boost.Sort depends on Serialization for BOOST_STATIC_WARNING, but I think this is easily solvable, maybe this static assert utility should go to the core/utility library. Reason: Sort is a low-level algorithm that should avoid pulling unneeded dependencies to other boost libraries. * * * I don't know if other Boost libraries could benefit from this new stable_sort implementation. Maybe it's too special and I should try to write it myself in Boost.Container. I would love to hear comments and suggestions from sort experts, this might be a common need that could enrich the Boost.Sort library. Best, Ion

4 9

[regression] Tests for 'develop' not showing
by Edward Diener 23 Sep '15

23 Sep '15

Trying http://www.boost.org/development/tests/develop/developer/summary.html I get: "Error extracting file: The specified zipfile was not found."

4 4

Re: [boost] Error while building boost
by Roshan 22 Sep '15

22 Sep '15

Hi, This list is specific to the Boost.Compute library. This mail is probably meant for the main Boost mailing list. Forwarding you there. P.S. I'm not sure if the attachment is being forwarded properly, please check. On Tue, Sep 22, 2015 at 12:24 PM, <valnci89(a)gmail.com> wrote: > Hi, > > > I'm trying to install boost on my machine but its not building properly. > Attached is the output generated on running ./b2 install. > > > Commands used: > > ./bootstrap.sh --prefix=/usr/include/boost > > ./b2 install > > > Machine Info: > > OS: RHEL ppc64le > > Boost version: boost_1_59_0 > > > Let me know if any other info is required. > > > Please advice me on this issue. > > > Thanks, > > Valnci > -- Warm regards Roshan

1 0

[Container] interest in a flat_unordered_map
by Treb Connell 21 Sep '15

21 Sep '15

I'm hoping to port a flat hash map that I've written to boost. Since boost has a flat_map I think it'd be good to have a flat_unordered_map as well, especially since there's an opportunity to be much more efficient. The one I would like to use is based on the hash map used in Lua, which is a mix of Coalesced and Robin Hood hashes. I don't think there's a name that describes it yet unfortunately, but I describe it near the end of this post. *Performance (primary reason for use):* It uses memory and cache very effectively and ends up being *very fast*. It's hard to benchmark real world situations, but in inserting, finding, erasing, and iterating over a dictionary it was up to six orders of magnitude faster than VS2013's unordered_map. This isn't a totally fair test because the cache efficiency really shines. *Standards compliance:* Differences from std::unordered_map are widely the same differences as flat_map listed here <http://www.boost.org/doc/libs/1_59_0/doc/html/container/non_standard_contai…>. Also bucket related methods no longer apply, so will be removed. *That said there's one major standards compliance issue that I'm struggling with:* The most optimal implementation of iterators for this container *does not have constant time traversal*. The implementation is a vector with elements scattered across it. To increment an iterator I continue to increment the internal vector iterator until I hit a new element. In practice elements should be scattered evenly, and will be close to one another, but in the worst case this is N-1. Additional memory or logic could be added to make this O(1), but I don't think that trade is worth it. In my applications of unordered_map I rarely iterate, and would not want to penalize insert, find, or erase to make iteration faster. I could make a new concept for this, pay the cost for making it truly O(1), or *since the common case is not O(N) would it be okay to consider these bidirectional iterators*? *Implementation details:* You can see Lua's implementation here <http://www.lua.org/source/5.3/ltable.c.html>. The applicable logic is in and below *luaH_newkey, *comments are useful. It's primarily a Coalesced hash table, but differs in collision behavior. When a new key collides with an old key, we check to see if the old key is in it's "*main position"*. The main position is the index that the key hashes to. If it's not in it's main position that means that it previously collided and was displaced. Since this is the new keys main position, we displace again the current key, and place ourselves there. This drastically decreases chain length, and improves the load factor that the hash map can optimally reach. Sorry about long post, can go into additional details upon request. Thanks, Treb

6 7

A compile-time-counter code snippet for boost
by Igor V. Chesnokov 21 Sep '15

21 Sep '15

2 1

[serialization] bug in static_warning.hpp
by Alain Miniussi 21 Sep '15

21 Sep '15

Hi, In static_warning.hpp we have the following code (which, if I trust its name, should trigger a *warning*): #define BOOST_SERIALIZATION_BSW(B, L) \ typedef boost::serialization::BOOST_SERIALIZATION_SS< \ sizeof( boost::serialization::static_warning_test< B, L > ) \ > BOOST_JOIN(STATIC_WARNING_LINE, L) BOOST_STATIC_ASSERT_UNUSED_ATTRIBUTE; #define BOOST_STATIC_WARNING(B) BOOST_SERIALIZATION_BSW(B, __LINE__) Now, since, since there is no BOOST_STATIC_ASSERT_UNUSED_ATTRIBUTE defined in boost (there used to be one in static_assert.hpp) this thing will generate a typedef of the form: typedef boost::serialization::BOOST_SERIALIZATION_SS< \ sizeof( boost::serialization::static_warning_test< B, L > ) \ > $STATIC_WARNING_LINE$L BOOST_STATIC_ASSERT_UNUSED_ATTRIBUTE; which is wrong. The MPI test are failing on that. Maybe BOOST_STATIC_ASSERT_UNUSED_ATTRIBUTE should be replaced with BOOST_ATTRIBUTE_UNUSED, but I am not sure what that macro is trying to achieve. Thks Alain

1 1

[locale] char conversion issue
by Alain Miniussi 21 Sep '15

21 Sep '15

Hi, My build is failing on message.cpp with: ../../../libs/locale/src/shared/message.cpp(439): error: invalid type conversion: "const wchar_t *" to "const char *" static_cast<char const *>(p), ^ detected during: [...] caused by code like: ================= template<typename CharType> struct hash_function { size_t operator()(message_key<CharType> const &msg) const { pj_winberger_hash::state_type state = pj_winberger_hash::initial_state; CharType const *p = msg.context(); if(*p != 0) { CharType const *e = p; while(*e) e++; state = pj_winberger_hash::update_state(state, static_cast<char const *>(p), static_cast<char const *>(e)); state = pj_winberger_hash::update_state(state,'\4'); } =================== Which looks like a legitimate error since that conversion does not seems t be covered by static_cast. This is on bot master and develop branch. The compiler is intel 16: [alainm@gurney build]$ icpc --version icpc (ICC) 16.0.0 20150815 Copyright (C) 1985-2015 Intel Corporation. All rights reserved. [alainm@gurney build]$ Regards Alain

2 1

[Compute] array support
by Fabian Bösch 19 Sep '15

19 Sep '15

Hi, I've been using Boost.Compute for some tests in the last couple of days. So far, it has proven very useful and I came to like it quite a bit. One thing I would add to it, though, is support for arrays when adapting your own classes for OpenCl through the Macro BOOST_COMPUTE_ADAPT_STRUCT. A simple overload would do the trick in my opinion: namespace boost { namespace compute { namespace detail { template<class Struct, class T> inline std::string adapt_struct_insert_member(T Struct::*, const char *name) { std::stringstream s; s << " " << type_name<T>() << " " << name << ";\n"; return s.str(); } template<class Struct, class T, int N> inline std::string adapt_struct_insert_member(T (Struct::*)[N], const char *name) { std::stringstream s; s << " " << type_name<T>() << " " << name << "[" << N << "]" << ";\n"; return s.str(); } } // end detail namespace } // end compute namespace } // end boost namespace Hope something along this lines can be added, thanks, Fabian

2 1

[Coroutine] Back to the caller
by Mikhail Strelnikov 18 Sep '15

18 Sep '15

Hi, I have an issue after switching from Boost.Coroutine to Boost.Coroutine2 - nested coroutine switches to its creator - not to the caller. Here is the code - http://codepad.org/WGNs0N7F There are two coroutines (1st coroutine starts, makes some progress, then spawns 2nd coroutine. But 1st coroutine knows nothing about 2nd and does not communicate with it - main thread does). With Boost.Coroutine it works as expected and outputs: g1g1g1g1g2g1g2g1g2g1g2g1g2g1g2g1 leaving coroutine 1 g2g2g2g2 leaving coroutine 2 Here you can see that 1st coroutine exits but 2nd keeps working normally. And control from coroutines is returned back to the caller (g - caller, 1 and 2 - coroutines). But with Boost.Coroutine2 it outputs: g1g1g1g1g21g1g21g1g1g21 leaving coroutine 1 assertion failed : (false) && ("pull_coroutine is complete") Here you can see that when caller (g) calls 2nd coroutine, it does not yield back to caller, but instead switches to 1st coroutine ("g21" in output).Then it terminates on assert "pull_coroutine is complete". Is it a bug of Boost.Coroutine2? If not, how can I change this code to work equally for Boost.Coroutine and Boost.Coroutine2? Thanks! P.S. What is the reason of boost::coroutines2::coroutine<void>::pull_type not having default constructor? It is still possible to have an "empty" pull_type, but it is painful process.

2 6

Interest in algorithm for arithmetic mean of integers
by Matt Calabrese 17 Sep '15

17 Sep '15

AFAICT the only function for calculating arithmetic mean in boost is in the accumulators library and it's not really appropriate for integral types. In practice, properly calculating the mean for a range (especially a range that's only an input range but not a forward range) is surprisingly tricky to do in a way that avoids overflow and provides an exact result. I have a couple of solutions, including one that works for input ranges. Is this something that people would find useful? Is there a similar algorithm already implemented in boost that I'm not aware of? The approach I've taken yields an exact result as a mixed number and should work without overflow (assuming only that the range's size is able to be represented by its difference type). I believe the implementation that works for input ranges could theoretically be adapted to work with Boost.Accumulators, though I'm admittedly not too familiar with the library. -- -Matt Calabrese

5 10

integrating NaCl support into boost
by Konstantin Ivlev 17 Sep '15

17 Sep '15

Hi boost community, some introdution: NaCl (Google NativeClient, https://developer.chrome.com/native-client) - open-source technology authored by Google Inc. providing way to run native code inside web-browser in portable and secure manner. NaCl provides posix-like environment, and supports newlib and glibc standard libraries. currently NaCl is supported by Google Chrome web-browser, and range of applications and software libraries are ported to NaCl (include zlib, quake, etc.). how about integrating NaCl support patches into the boost main-line (trunk/master), I am talking about these NaCl ports patches from Google: https://chromium.googlesource.com/external/naclports/+/master/ports/boost (I actually have a bit more correct patches that allow to compile boost 1.59 for NaCl platforms, while Google patches are still for old version 1.55). some details about patches and which boost areas are covered by patches: - boost/config - common setup for NaCl - boost/cstdint - small change to expose boost::uintptr_t/boost::intptr_t for NaCl, similarly to how it was done to other compilers - buitin.jam and gcc.jam - introduced new os-name for NaCl, in order to properly setup threading libraries (NaCl has pthread but no rt, just like BSD) - boost/system/errorcode - oddball version of strerror_r (just like cygwin), also ENOTRECOVERABLE & EOWNERDEAD errno codes are not supported by PNaCl - boost/log/light_rw_mutex - not supported on NaCl note that boost build is incomplete - certain libraries are not (yet) ported to NaCl, because of platform-specific stuff like file/network I/O : - context - coroutine - coruutine2 - log - filesystem - test - timer - wave not sure if it's OK for start, or all boost libraries should be ported together in order to process further. I would be grateful to hear suggestion about how to process with it, e.g. which actions should I take to make it really happen. I personally suppose making boost to officially support more platforms will make boost even more popular and allow more developers to use it and port their applications to other platforms. I look forward to hearing from you soon. yours sincerely, Konstantin

4 5

Windows Phone and Visual Studio 2015
by Vladimir Prus 17 Sep '15

17 Sep '15

Folks, did anybody had success building Boost for Windows Phone using Visual Studio 2015 (aka 14.0)? We have Boost.Build code to do so, but it tries to find vcvarsphoneall.bat script, and some other similarly named scripts, that I don't seem to have anywhere, despite been able to build WP app from the IDE. Has anybody solved this? Thanks, Volodya

1 1

[predef] BOOST_ARCH_BITS_* feature request
by Antony Polukhin 16 Sep '15

16 Sep '15

Hi, It looks like for low-level programming following macro could be really useful: BOOST_ARCH_BITS_64 BOOST_ARCH_BITS_32 BOOST_ARCH_BITS_16 ? Predef also missing BOOST_ARCH_ARM_64, BOOST_ARCH_PPC_64. Boost.DLL really needs those macro for detecting bit's dependent elf/macho/pe structures layout. Is it possible to add them? -- Best regards, Antony Polukhin

4 4

Copyright-less licence references
by Gennadiy Rozental 16 Sep '15

16 Sep '15

Hi, I suggest we should drop "Copyright Joe Coder 20xx" from all our files. My reasons: * git/svn history provides much better copyright attribution. Each developer has a copyright on lines authored. As it is these copyright disclaimers are very frequently wrong. * I am never clear what this line really mean. Copyright Joe Coder 2010 - does this mean Joe has copyright up to 2010, starting 2010 or only in 2010? Should one keep updating these disclaimers every January? * For libraries which are maintained by multiple developers and/or which were moved from one maintainer to another maintainer - these disclaimers either wrong or pain to maintain. * these are just a clutter with no real value IMO we should just keep licence reference and that's it: // Distributed under the Boost Software License, Version 1.0. // (See accompanying file LICENSE_1_0.txt or copy at // http://www.boost.org/LICENSE_1_0.txt) Gennadiy

12 38

Determining interest in container with efficient random access insert and erase
by Chris Clearwater 16 Sep '15

16 Sep '15

Greetings, I have written a library with the intention of submitting it to Boost.Container. My container implements O(log n) random access insert/erase and has a strong focus on performance while supporting the interfaces of the standard library sequence types. An example application would be the character buffer of a text editor. Useful links: - Source code: https://github.com/det/segmented_tree_seq - Documentation: http://det.github.io/segmented_tree_seq/ - Benchmarks: https://github.com/det/segmented_tree_seq/tree/benchmarks Any feedback is greatly appreciated.

14 28

BOOST_PP array extension proposal
by Antoine Leblanc 14 Sep '15

14 Sep '15

Hello to all of you! A few years ago, I've developped a small extension to BOOST_PP that adds a few functions to it. Namely: * ARRAY_CAT, which concatenates two BOOST_PP arrays; * ARRAY_LOWER_BOUND, which finds the index of the lower bound of an item in a given array of numerical values; * ARRAY_SORT, which sorts the given array of numerical values; * ARRAY_SORT_U, which sorts the given array and remove duplicates. I'm not sure it'd be worth adding to BOOST_PP; but if you think it might be, I'd be happy to submit a merge request. The code is rather small and can be browsed at https://github.com/nicuveo/TOOLS_PP Any suggestion or input is welcome. Thanks for your time! -- Antoine Leblanc

6 34

1.59 release notes don't mention new property_tree::json_parser
by Jonathan Wakely 14 Sep '15

14 Sep '15

The commit logs say: TL;DR: The new parser fixes long-standing bugs and has full Unicode support, but removes non-standard extensions of the old parser, which could break code: - String concatenation: the old parser concatenated adjacent string literals like C does. - Comments: the old parser supported C and C++-style comments. JSON doesn't allow comments. Removing support for comments did break something. Unfortunately there was no sign of this in the release notes, so I had to debug it the hard way (in code that swallowed the exception from the parser - gah!) and only found the commit message once I already knew what the problem was. </rant>

4 5

[type_traits] New version in master
by John Maddock 14 Sep '15

14 Sep '15

Just a quick heads up that the new version of type_traits is now in master, if there are any issues please do shout, John.

5 7

[type_traits] New version in master
by Glen Fernandes 14 Sep '15

14 Sep '15

Hi John, Any particular reason why 'is_trivially_default_constructible' was not provided as just the functionally-equivalent of 'has_trivial_default_constructor'? Glen

2 1

b2 headers failing
by Marcel Raad 13 Sep '15

13 Sep '15

Hi, when I try "b2 headers" on a fresh checkout of the develop branch, I get the following errors: C:/boost/tools/build/src/build\feature.jam:326: in validate-feature from module feature error: unknown feature "<segmented-stacks>" C:/boost/tools/build/src/build\feature.jam:360: in expand-subfeatures-aux from module feature C:/boost/tools/build/src/build\feature.jam:425: in feature.expand-subfeatures from module feature C:/boost/tools/build/src/build\property.jam:508: in property.translate from module property C:/boost/tools/build/src/build\property-set.jam:432: in property-set.create-from-user-input from module property-set C:/boost/tools/build/src/build\property-set.jam:484: in property-set.refine-from-user-input from module property-set C:/boost/tools/build/src/build\project.jam:700: in class(a)project-attributes.set from module object(project-attributes)@369 C:/boost/tools/build/src/build\project.jam:1099: in project from module project-rules libs\coroutine2\build\Jamfile.v2:26: in modules.load from module Jamfile<C:\boost\libs\coroutine2\build> C:/boost/tools/build/src/build\project.jam:325: in load-jamfile from module project C:/boost/tools/build/src/build\project.jam:64: in load from module project C:/boost/tools/build/src/build\project.jam:89: in load-used-projects from module project C:/boost/tools/build/src/build\project.jam:75: in load from module project C:/boost/tools/build/src/build\project.jam:145: in project.find from module project C:/boost/tools/build/src\build-system.jam:535: in load from module build-system C:\boost\tools\build\src/kernel\modules.jam:295: in import from module modules C:\boost\tools\build\src/kernel/bootstrap.jam:139: in boost-build from module C:\boost\boost-build.jam:17: in module scope from module Does anyone know what's wrong here? Thanks, Marcel -- View this message in context: http://boost.2283326.n4.nabble.com/b2-headers-failing-tp4680039.html Sent from the Boost - Dev mailing list archive at Nabble.com.

1 1

[AFIO] Formal review
by Jeremy Maitin-Shepard 12 Sep '15

12 Sep '15

> 1. Should Boost.AFIO be accepted into Boost? Please state all conditions for > acceptance explicity. In its current state, Boost.AFIO should NOT be accepted into Boost. However, I am confident that Niall will be able to deliver a future version of AFIO that should be accepted into Boost after another review. There is a great need for more powerful and efficient file handling functionality in C++ than is provided by iostreams and Boost.Filesystem/Filesystem TS. I appreciate the enormous complexity of trying to define a multi-platform filesystem interface, and it is clear that Niall has done a ton of very valuable research on this topic. > 2. What is your evaluation of the design? A key strength of the design is that it provides a common interface for asynchronous operations regardless of whether they are internally implemented using Windows IOCP support, async IO on FreeBSD, or regular POSIX operations via a thread pool. Another key strength is that it exposes handle-based (i.e. *at), rather than path-based APIs where possible in order to minimize race conditions, which tend to be difficult or impossible to avoid completely with file system operations. I do have a number of concerns about several aspects of the design, however: Batch API: - It is not clear what, if any, purpose the batch API serves. The significant added complexity of using it would have to be justified by a significant performance advantage, but it is not at all clear that there is even any performance advantage at all. - I suggest removing it unless there is a strong argument otherwise. File handle API: - The file handle abstraction in AFIO is very heavy weight, involving a shared_ptr to a structure containing a whole bunch of fields in addition to the native file handle, including a mutex, numerous flags, the path used to open it, a pointer to a handle to the parent directory, a chrono time_point specifying the time the file was opened, the total number of bytes read, written, and written since last fsync using this file handle. - Because file handles are at the core of AFIO, it is critical that users precisely understand what this abstraction involves, and specifically how it differs from a native file handle. - I'd be much more comfortable with a much leaner abstraction, like a unique_handle type that just contains the bare native handle, and uses unique ownership semantics. The fat handles impose both computational/memory overhead as well as mental overhead on the programmer. I understand these extra fields are probably necessary to provide various functionality, particularly emulation of behavior provided by other operating systems, but it would be nicer to be able to opt-in to such functionality and only pay for it when needed. - At the very least, if these fat file handles are kept, a very strong argument needs to be given as a rationale. (Perhaps you have found the overhead to be negligible relative to the cost of the native file handle.) - It should be possible to construct an AFIO handle from a native file descriptor. - On POSIX, AFIO should support supplying platform-specific open flags, like O_NOATIME. - I'd much prefer native_handle() to return the actual appropriate native handle type for the platform, e.g. int on POSIX, rather than a void *. I'm guessing this is done in order to support alternative dispatchers, that aren't necessarily based on OS files at all, in the future, but perhaps a better way of doing this that doesn't require casting the void* to an int, can be found. Future-based API: - I agree with Thomas Heller regarding the Future-based API: - The implicit future.then built into the API should be eliminated. Instead, API functions should take a file handle directly. - future<T> should not implicitly store a file handle. The API functions that open a file can return a future to a file_handle. API functions that don't open a new file should just return a future to the actual result. Inconsistency in how operations are defined: - Some operations are methods of the dispatcher, some are methods of the file handle, and others are free functions. - I suggest that all file system operations be free functions, and take a dispatcher as an argument. Thread-local current dispatcher: - Some operations involve an explicitly specified dispatcher, others seem to obtain it from the file handle, and others default to a thread-local dispatcher that has been set. - Given that it is extremely likely for user code to execute from multiple threads given the nature of AFIO, relying on a thread-local current dispatcher seems very error prone. - I suggest that the thread-local current dispatcher be eliminated, and that the dispatcher always be specified explicitly to any API call that requires one. Directory handle cache: - A cache of open handles to directories is kept by AFIO by default. This is particularly problematic because (as is documented) it makes reading directory entries thread-unsafe by default. If this is to be kept, a very strong rationale needs to be given, and the behavior must be changed somehow to ensure that directory reading is reliable by default. Enumerate API: - The future-based API of async_enumerate seems to be unnecessarily complicated, due to the need to specify maxitems and deal with making repeated calls, and all of the copying and memory allocation required to return a std::vector of directory_entry. I suggest instead something like passing in a std::function<bool (expected<directory_entry> const &next)> callback (possibly replacing expected by the new name for your monad type). An empty next value means end of directory has been reached. Returning true means continue listing the directory, returning false means stop listing the directory. Miscellaneous comments: - Operations including rmdir, rmfile, symlink, rmsymlink (as well as async variants) that normally would not necessarily even operate on file handles, actually return a handle_ptr. Perhaps if on some platforms a handle has to be opened, then there could be a way for the user to specify that a handle is desired. Otherwise, it would make sense to just return void or std::future<void> equivalent for async variants. - The documentation notes that on MS Windows, DOS short names are not supported. I don't have much familiarity with Windows, but if programs generally accept DOS short names when a filename is required, then AFIO should probably support these names as well. - The documentation notes: "Don't do directory enumerations from an x86 binary on a x64 Windows system. This is due to a bug in Microsoft's WOW64 syscall translation layer which Microsoft have decided is wontfix. " Given that a major purpose of AFIO is to provide a portable abstraction that works around OS limitations, AFIO needs to support directory enumeration even in this case (e.g. by detecting WOW64 and using a synchronous call via a threadpool in that case). - It is noted that to do an asynchronous stat you have to use enumerate with an appropriate glob specified, but constructing such a glob may be non-trivial if the path contains characters that are interpreted specially by the glob syntax, and impossible to do in a race-free manner. - As a general matter, it is important to provide leanest possible API that offers the most flexibility to users, in order to avoid the case that users choose not to use the library despite it offering a lot of functionality that is useful to them, because of excessive overhead or insufficiently powerful interfaces. I often forgo use of Boost Filesystem and C++ iostreams for this reason. 3. What is your evaluation of the implementation? - I understand that Niall already intends to rewrite much of the implementation, making much of the current implementation irrelevant. - As has been noted by other reviewers, there are a bunch of #if 0 blocks, many undocumented. Some seem to contain work in progress, which might better be placed in a separate branch. Some contain debugging code, in which case it might be better to replace #if 0 with #ifdef BOOST_AFIO_DEBUG_XXX where XXX specifies the particular feature being debugged. That way it is self-documenting. Some seem to contain alternate implementations. Ideally these would also be guarded by a descriptive macro, and a unit test case be defined that tests it as well to make sure it is still meaningful. Otherwise #if 0 code tends to bitrot, which admittedly is not such a great problem if it is just for debugging anyway, but I'd say it is less than optimal. - The implementation might be better split up into multiple independent files. There are various independent facilities, like secded_ecc, that could be placed in a separate file. - Although this tends to be a problem with most Boost libraries, it would be nice if the compile times could be reduced, particularly when not in header-only mode. Some libraries like Boost Spirit have an interface inherently based on template metaprogramming, such that long compile times are pretty much unavoidable, but the Boost AFIO interface seems to be mostly non-template based, which suggests that it may be possible to get pretty good compile times. > 4. What is your evaluation of the documentation? - The extensive documentation about race guarantees is extremely useful. - I appreciate the in-depth examples. - Much more design rationale is needed. - The term "filing system" is used in place of the more common "file system", leading me to wonder if there is a difference in meaning. - Every instance where there is a platform-specific #ifdef in an example should be evaluated, so that you can figure out the appropriate platform-independent abstraction that could be added to AFIO in order to move that #ifdef into the library and out of user code. - There is an attempt to document differences in behavior on different platforms, but this is not done sufficiently. For example, it is stated that async_write requires that the destination region does not go off the end of the file, but on POSIX pwrite supports this just fine as a way to extend the file. I assume that this restriction applies to Windows only. - The semantics of the Batch API are not precisely documented, nor is any rationale given for it. For instance, I understand that the operations themselves cannot happen atomically, since that is not supported by operating systems, though this is not clearly stated. It is stated: "Once a batch of input ops has been verified at the point of entry as not errored, you are guaranteed that the batch is atomically scheduled as a whole, unless a failure to allocate memory occurs." Does this means that if the *_req object involves futures, it waits for all of them to be ready before launching any of the operations in the batch? Is that the purpose of the Batch API, to do a wait_all on all of the relevant futures, then to launch each operation? This issue goes away though if the Batch API is eliminated, as seems to make sense. - It would be useful to document for each operation which underlying system calls it invokes on each platform. - Many operations other than open (e.g. rmfile) take file_flags, but it is not specified which flags are supported. - References to historical versions of AFIO (prior to the review) should be removed from the main parts of the documentation, since they only distract from the useful information. This information could be kept in a different section at the end for the extra curious. - Formatting of the "description" in the Boost Quickbook format function documentation has some problems: the lack of line breaks between the general description, other notes, and the common note about the use of NT kernel paths make it difficult to read. > 5. What is your evaluation of the potential usefulness of the library? The library is potentially extremely useful. C++ and Boost iostreams and Boost/TS Filesystem provide only very limited functionality. The C library libuv is a close competitor but seems to offer less functionality and a less convenient API. For achieving similar functionality, the status quo in C++ is to rely on platform-specific C APIs that are much more cumbersome to use. > 6. Did you try to use the library? With what compiler? Did you have any problems? I tried some very simple usage of the library with GCC 5.2.0 on Linux. There were a few warnings (variable set but not used) with -Wall that would be best eliminated. - The error messages are not ideal: for instance, in the case that there was an error opening a file, this is the result: Dynamic exception type: boost::system::system_error std::exception::what: File '/home/jbms/tmp/afio-test/test.txt' not found [Host OS Error: No such file or directory (2) in '/home/jbms/tmp/boost.afio/include/boost/afio/v2/detail/impl/afio.ipp': async_io_handle_posix:805]: No such file or directory. Op was scheduled at: 1. 0x42fb4f: ./test (+0x2fb4f) : No such file or directory Some of this is only useful when debugging AFIO, some only useful when debugging the application, and there is a lot of redundancy, for instance the actual error "No such file or directory" is shown in 3 places. Ideally there would be a way to get a simple message along the lines of "Error opening file '/home/jbms/tmp/afio-test/test.txt': No such file or directory" that could be shown directly to the user. I tried to #define BOOST_AFIO_EXCEPTION_DISABLESOURCEINFO to see if that would make the error message better, but instead that seemed to trigger a bug: Dynamic exception type: std::logic_error std::exception::what: basic_string::_S_construct null not valid - A separate issue is that attempting to read a range that is not present in a file, something that the library should handle elegantly, leads to the program aborting immediately with an error message, rather than allowing the error to be handled: FATAL EXCEPTION: Failed to read all buffers 1. 0x40f0f4: ./test (+0xf0f4) terminate called after throwing an instance of 'std::runtime_error' what(): Failed to read all buffers zsh: abort (core dumped) ./test > 7. How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? I gave the library a fairly in-depth study, spending at least 8 hours in total. I looked at the documentation fairly thoroughly, looked through parts of the implementation, and tried running a couple small programs that use the library. > 8. Are you knowledgeable about the problem domain? I have a fair amount of experience with asynchronous programming in C++, and I have pretty extensive knowledge of the file system APIs on Linux. As a final note, I'd like to thank Niall for all of the work he has done to get the library this far. A powerful, elegant, cross-platform interface for file system operations is very much needed in C++, I really do want to see (a future version of) this library in Boost and use it myself.

11 32

Boost.Fiber mini-review September 4-13
by Nat Goodspeed 11 Sep '15

11 Sep '15

Hi all, The mini-review of Boost.Fiber by Oliver Kowalke begins today, Friday September 4th, and closes Sunday September 13th. It was reviewed in January 2014; the verdict at that time was "not in its present form." Since then Oliver has substantially improved documentation, performance, library customization and the underlying implementation, and is bringing the library back for mini-review. The substance of the library API remains the same, which is why a mini-review is appropriate. The Fiber library now requires a C++14-conforming compiler. I will monitor reviews and discussion on both the boost-users@ list and the boost@ developers' list. Please include at least "fiber" and "review" in your mail subject, e.g. by replying to this message. (Please reply to only ONE list, however.) Thank you for your interest and your feedback! ----------------------------------------------------- About the library: Boost.Fiber provides a framework for micro-/userland-threads (fibers) scheduled cooperatively. The API contains classes and functions to manage and synchronize fibers similar to Boost.Thread. Each fiber has its own stack. A fiber can save the current execution state, including all registers and CPU flags, the instruction pointer, and the stack pointer and later restore this state. The idea is to have multiple execution paths running on a single thread using a sort of cooperative scheduling (versus threads, which are preemptively scheduled). The running fiber decides explicitly when it should yield to allow another fiber to run (context switching). Boost.Fiber internally uses execution_context from Boost.Context; the classes in this library manage, schedule and, when needed, synchronize those contexts. A context switch between threads usually costs thousands of CPU cycles on x86, compared to a fiber switch with a few hundred cycles. A fiber can only run on a single thread at any point in time. docs: http://olk.github.io/libs/fiber/doc/html/index.html git: https://github.com/olk/boost-fiber --------------------------------------------------- Please always state in your review whether you think the library should be accepted as a Boost library! Additionally please consider giving feedback on the following general topics: - What is your evaluation of the design? - What is your evaluation of the implementation? - What is your evaluation of the documentation? - What is your evaluation of the potential usefulness of the library? - Did you try to use the library? With what compiler? Did you have any problems? - How much effort did you put into your evaluation? A glance? A quick reading? In-depth study? - Are you knowledgeable about the problem domain? Nat Goodspeed Boost.Fiber Review Manager ________________________________

11 108

[fiber] on schedulers and wakeups
by Giovanni Deretta 11 Sep '15

11 Sep '15

Hi Oliver, I'm looking at the boost.fiber scheduling customization options. I have a few comments. My first concern is that scheduler is a global (per thread) property. This makes sense, but it limits its usability in libraries, unless the library completely owns a thread. It would be nice if schedulers where schedulable entities themselves, so that they could be nested (more at the end). Also the description of the scheduler interface does not specify any thread safety requirements. I assume that at least awakened must be thread safe as the scheduling might be caused by a signal coming from another thread. Any requirements should be specified properly. This leads to two additional points. First of all, there should be a way to retrieve the scheduler associated with a fiber: I haven't looked at the source, but the association must exist internally, otherwise cross thread scheduling wouldn't work. Second, there does not seem to be a way to allow signaled fibers to run in the context of the signaling thread. This seems an important optimization.I understand this is an explicit decision as currently it is not possible to portably migrate fibers, but the option should be left to the user if he knows it is safe in their setup. Possibly require 'awakened(fiber*x)' to call x->get_scheduler()->awakened(x) if it does not support running the fiber. My preference is to add a queryable boolean flag to each schedulable entity that states whether it is allowed to move from its scheduler (this makes a difference when you have multiple nested schedulers). Among other things this would also prevent work stealing. The flag would by default be set to prevent migration of course. Now, what do nested schedulers give you? In addition to composability, you can have the equivalent of an asio strand without explicit mutual exclusion. Let say you have a bunch of fibers that all access the same resource; you do not care where they run, as long as they never run concurrently. By binding all of them to the same scheduler, the guarantee is implicit. It should be possible to implement nested schedulers by having each fiber set the current scheduler pointer if it differs and store the previous one. On a scheduling event, if the current scheduler runs out of runnables, the signal/yield code would restore the previous scheduler and try again. Of course handling priorities and non strict fifo scheduling might be more complex. Note that the multi scheduler logic need to be implemented by Boost.Fiber itself. Finally, it might be possible (but I haven't tested this) to make fibers migrable on platforms where it normally isn't safe by allocating an actual thread for each one. The thread would be stopped as soon as it has been created, but a thread control block plush TLS entries would have been allocated. The existing context switch would need to be augmented with the ability to swap the thread context pointer. HTH, -- gpd

4 23

Critical. Help required. Boost.DLL Jamfile breaks testing with multiple toolsets
by Antony Polukhin 10 Sep '15

10 Sep '15

Hi, There's a big problem with Boost.DLL's Jamfile which I do not know how to fix. Introduction Tests of Boost.DLL create multiple *.so files and install them to a specified TEST_DIR path. Tests consume that TEST_DIR path and search for *.so in there: path-constant TEST_DIR : $(BOOST_ROOT)/bin.v2/libs/dll/test ; project : source-location . : requirements ; { # our test lib for shared library tests lib test_library : test_library.cpp : <link>shared ; lib getting_started_library : ../example/getting_started_library.cpp : <link>shared ; install install-bin : test_library getting_started_library : <location>$(TEST_DIR) ; test-suite boostdll : [ run shared_library_load_test.cpp : $(TEST_DIR) : : <dependency>install-bin ] [ run shared_library_search_symbol_test.cpp : $(TEST_DIR) : : $(RDYNAMIC) <dependency>install-bin ] ; } The Problem If multiple toolsets specified, then `install-bin` target gets broken: cd libs/dll/test ../../../b2 toolset=gcc,clang error: Tried to build the target twice, with property sets having error: these incompatible properties: error: error: - <cxxflags>-Wall -Wextra -pedantic -Wno-long-long <library>object(file-target)@3829 <library>object(file-target)@3862 <toolset-gcc:version>4.8 <toolset>gcc <xdll-path>/home/antoshkka/boost_maintain/boost/bin.v2/libs/filesystem/build/gcc-4.8/debug/threading-multi <xdll-path>/home/antoshkka/boost_maintain/boost/bin.v2/libs/system/build/gcc-4.8/debug/threading-multi error: - <library>object(file-target)@7264 <library>object(file-target)@7296 <toolset-clang:platform>linux <toolset-clang:version>3.4 <toolset>clang <xdll-path>/home/antoshkka/boost_maintain/boost/bin.v2/libs/filesystem/build/clang-linux-3.4/debug/threading-multi <xdll-path>/home/antoshkka/boost_maintain/boost/bin.v2/libs/system/build/clang-linux-3.4/debug/threading-multi Correct fix would be to embed into TEST_DIR the names of toolset and all other configuration flags. path-constant TEST_DIR : $(BOOST_ROOT)/bin.v2/libs/dll/test/$(FLAGS)/$(TOOLSET) ; The Question How to get the $(FLAGS) and $(TOOLSET) variables? If that requires writing rules or implies some other Boost.Build magic I'd really appreciate a pull request: any nontrivial modification to Jamfile takes months to apply and test because of my poor knowledge of Boost.Build. Jamfile: https://github.com/boostorg/dll/blob/develop/test/Jamfile.v2 Boost.DLL: https://github.com/boostorg/dll This issue is critical, because it breaks regression testing (see "[boost] [android][regression] bjam failed") -- Best regards, Antony Polukhin

5 9