October 2018 - Boost - lists.preview.boost.org

[histogram] Variance
by Bjorn Reese 02 Oct '18

02 Oct '18

The variance of individual bins can be obtained when using the adaptive_storage (via h.at(i).variance().) I am trying to understand the overhead of this feature. If I interpret the code correctly, there is a space overhead because each counter has to keep track of both the count and the sum of squares. The computational overhead is that the sum of squares has to be calculated for each insertion. Is this correct? If so, is there any way to use the adaptive storage policy without variance? Furthermore, why does variance() return the sum of squares? Should this not be divided by the sample size?

5 8

[asio] Order of asynchronous messages
by Florian Lindner 01 Oct '18

01 Oct '18

Hello, in a distributed application (only local are network) we are experiencing issues which *may* stem from the order of asychronous messages sent over sockets. We create a send buffer: auto buffer = std::make_shared<std::vector<double>>(); and fill it: buffer.push_back(some_data). This buffer is then given to our communication routine request = aSend(*buffer, recipient); to keep the send buffer alive we place it in a class attribute: bufferedRequests.emplace_back(request, buffer); The aSend method from above works like (asio is boost asio, of course) PtrRequest aSend(std::vector<double> const & itemsToSend, int recipient) { PtrRequest request(new SocketRequest); asio::async_write(*_sockets[recipient], asio::buffer(itemsToSend), [request](boost::system::error_code const &, std::size_t) { std::static_pointer_cast<SocketRequest>(request)->complete(); }); return request; } This way, we can check for completion using request->test(), which is set from the callback request->complete(). For checking of completed requests we have a loop, which is called reqularly void checkBufferedRequests(bool blocking) { do { for (auto it = bufferedRequests.begin(); it != bufferedRequests.end();) { if (it->first->test()) it = bufferedRequests.erase(it); else ++it; } if (bufferedRequests.empty()) return; if (blocking) std::this_thread::yield(); // give up our time slice, so communication way work } while (blocking); } which operates on bufferedRequests, that is a list<pair<PtrRequest, PtrBuffer>>. We use that, so that the sending peer is not blocked by a slow receiving peer. The sockets are created like tcp::acceptor acceptor(*_ioService); { tcp::endpoint endpoint(tcp::v4(), _portNumber); acceptor.open(endpoint.protocol()); acceptor.set_option(tcp::acceptor::reuse_address(_reuseAddress)); acceptor.bind(endpoint); acceptor.listen(); } Now we are having problems with mangled data that are unexplainable right now. - Is the order of requests guaranteed to be preserved? I think so, based that asio works on TCP. - What happens if there is another read or write performed on the socket. I think that there is no way to guarantee that this does not interfere with the order of the asynchronous send or queued requests. Last but not least: Do you see any potential problems with that code? Thanks a lot! Florian

3 2

[container][interprocess][intrusive][move] Removing support for old compilers
by Ion Gaztañaga 01 Oct '18

01 Oct '18

Hi, My libraries have supported quite old compilers, starting from Visual 2003 and GCC 3.4 and I'm asking for feedback before I start dropping old compilers. The number of supported plus locally+regression tested versions has increased a lot these years and that's a lot of work with reduced benefits. I run locally for: - From Mingw GCC 3.4 to GCC 8.1, all released minor versions (combining C++03-C++11-C++14-C++14). - From MSVC 7.1 to MSVC 14.1, all released versions (default language standard). I also run boost regression tests for many of those compilers (see "igaztanaga-*" runners in https://www.boost.org/development/tests/master/developer/summary.html). The number of supported and tested compilers is now unmanageable for me as many of those have specific warnings/bugs that I try to fix. So I was thinking about reducing that list a bit dropping support for the oldest ones and replacing them with recent GCC/Clang from WSL (Windows Subsystem for Linux). My first idea is to drop anything older than 10 years, which means that the minimum will be MSVC 9.0 and GCC 4.3 (both released in 2008). I've reviewed some popular long support Linux distributions: -> RHEL 6 uses GCC 4.4. -> SUSE Linux Enterprise Server 11 Service Pack 4 (SP4) uses GCC 4.3 Microsoft no longer supports Visual 2008 (support ended on April 2018) but I can continue compiling and testing it at least for another year. Note that my libraries will continue support for -std=C++03, but I will not find workarounds or fixes for old GCC and MSVC versions. Thoughts, problems, comments? Best, Ion

6 9

[histogram] Formal review
by Bjorn Reese 01 Oct '18

01 Oct '18

Boost.Histogram contains useful and well-designed features, like excess (over/underflow) bins, and axis transforms to name a few. As this is a review, I am going to spent most time below on critical scrutiny. I. DESIGN --------- Iterators are const. This prevents us from bootstrapping the histogram with a prior distribution using STL algorithms like std::fill(), std::generate(), or std::sample(). The adaptive_storage does not work well with STL algorithms, because weight_counter is not an arithmetic type. See the Implementation section below for more detail. I therefore propose that the default storage policy is changed to something without weight_counter. The adaptive_storage has two responsibilities: data compaction and weights. Would it be possible to split this into two separate storage policies? I have no real use for arrival variance. I am not too fond of using operator() for insertion. The code looks like a function call with side-effect. I prefer an explicitly named member function. The axis::ouflow_type is oddly named. I suggest that this is renamed to something like axis::excess_type. II. IMPLEMENTATION ------------------ The implementation is generally of high quality. However, I did encounter the three problem areas listed below. (A) Integration with STL algorithms can be improved. Here are some examples: First, std::distance() does not work on a histogram with array_storage. This means that other STL algorithms, like std::any_of(), fails to compile. I solved this problem by copying the distance_to() function from the axis iterator to the histogram iterator. Second, std::max_element (and brethren) cannot be used on a histogram. Consider the following example: auto element = std::max_element(h.begin(), h.end()); Compilation fails for adaptive_storage because weight_counter has no less-than operator. Furthermore, compilation fails for array_storage because iterator_over<H>::operator= fails. It attempts to re-assign a reference, but accidentally triggers a copy-constructor instead. Letting the iterator store a pointer instead of a reference solves the problem. Apropos, iterator_over<H>::operator= does not return *this. Consider adding the -Werror=return-type compiler flag and a unit-test to that calls this operator. Third, std::inner_product fails to compile for adaptive_storage because weight_counter does not have a binary operator*. The inner product of two histograms is useful for calculating the cosine similarity, which can be used to compare two distributions. std::inner_product works for array_storage though. (B) Indexing and size use different types (int versus std::size_t.) I assume that this is because of the underflow bin, which is indexed by -1. I am not certain whether or not this is a real problem, but it does cause some oddities when stressed to the limit. For example, if we need a histogram that counts each unsigned int, then we get an "lower < upper" exception during construction: using range_type = unsigned int; auto h = make_static_histogram_with( array_storage<int>(), axis::integer<>(0, std::numeric_limits<range_type>::max())); It works if we reduce the end by 2, but then we are not collecting all values (unless we shift the range left by 1 and misuse the excess bins for the two missing values.) A possible solution could be to replace the -1 and N excess indices with an "enum struct excess { lower, upper }" and let operator[] et al use overloading on this enum. (C) Sometimes the compilation errors are nonsensical. For example: // Forgot to use make_static_histogram_with() auto h = make_static_histogram(array_storage<int>, axis::regular<>(10, 0, 1)); triggers an incomprehensible static_assert plus an error about a missing .shape() function. III. DOCUMENTATION ------------------ User guide is clear and pedagogical. I would like to see more examples that uses STL algorithms, such as calculating the CDF using std::partial_sum(), or calculating the cosine similarity of two histograms using std::inner_product(). Most examples use std::cout plus a comment to document the results of operations. Consider using assert() instead. Consider using a tabular layout similar to that of the C++ standard (or cppreference.com for that matter) on the Concepts page. Reference documentation is rather meager and shows implementation details. The documentation contains both a "Reference" and a "References" chapter which are completely different. Consider renaming the latter to something like "External references" , "Literature references", or "Bibliography". IV. MISC -------- I have spent around 15 hours on the review, mainly writing small examples that uses STL algorithms on Boost.Histogram. I am well-versed in the topic. I work with statistical distributions for real-time analysis of data, although I mainly dabble around in one dimension. I have written a library for online/streaming statistical algorithms. V. VERDICT ---------- While Boost.Histogram is a small niche library, it has a wide range of practical applications to warrant the inclusion into Boost. Boost.Histogram should be ACCEPTED into Boost, provided: 1. Reference documentation is finalized. I furthermore strongly recommend that the default storage policy is changed to something without weight_counter because of the various problems with STL algorithms. This recommendation is not a prerequisite for acceptance.

2 3