Accumulators: Using data vector from Interprocess
Hi, I have large data to be processed hence planning to use memory mapped file backed vector from boost::interprocess. Could I use this vector directly in accumulators and perform computations on data in the boost::interprocess container without having to copy it all into the contain in accumulator_set? Also, is there a way to remove an entry once I add it to an accumulator_set? I want to emulate a sliding window calculation. Ex: If my window size is 100, when I add the 101st entry, I want to remove the 1st entry so that the size is always 100. -dhruva -- Contents reflect my personal views only!
dhruva wrote:
Hi, I have large data to be processed hence planning to use memory mapped file backed vector from boost::interprocess. Could I use this vector directly in accumulators and perform computations on data in the boost::interprocess container without having to copy it all into the contain in accumulator_set?
Also, is there a way to remove an entry once I add it to an accumulator_set? I want to emulate a sliding window calculation. Ex: If my window size is 100, when I add the 101st entry, I want to remove the 1st entry so that the size is always 100.
-dhruva
You can use an accumulator with any container that allows you to walk through the contained values, so it will work with this one. Most accumulators don't store any of the data points, and the ones that do store any, don't store all. So, in general you will not be making a copy of all the data. For the moving average on a widow, the accumulators library doesn't currently include an accumulator that does this so a custom accumulator has to be defined. However, you might look in the sandbox and see if one is there. I don't recall who did it, but I remember a discussion about making one of these in the past. I think there is an available version somewhere. John
John Phillips wrote:
dhruva wrote:
Hi, I have large data to be processed hence planning to use memory mapped file backed vector from boost::interprocess. Could I use this vector directly in accumulators and perform computations on data in the boost::interprocess container without having to copy it all into the contain in accumulator_set?
Also, is there a way to remove an entry once I add it to an accumulator_set? I want to emulate a sliding window calculation. Ex: If my window size is 100, when I add the 101st entry, I want to remove the 1st entry so that the size is always 100.
You can use an accumulator with any container that allows you to walk through the contained values, so it will work with this one. Most accumulators don't store any of the data points, and the ones that do store any, don't store all. So, in general you will not be making a copy of all the data.
Correct, thanks John.
For the moving average on a widow, the accumulators library doesn't currently include an accumulator that does this so a custom accumulator has to be defined. However, you might look in the sandbox and see if one is there. I don't recall who did it, but I remember a discussion about making one of these in the past. I think there is an available version somewhere.
Also correct. A moving average accumulator is an often requested feature. I knocked one together as a proof-of-concept during the accumulators review, but never polished it. You can find it in this message: http://lists.boost.org/Archives/boost/2007/07/124979.php HTH, -- Eric Niebler BoostPro Computing http://www.boostpro.com
Eric Niebler wrote:
A moving average accumulator is an often requested feature. I knocked one together as a proof-of-concept during the accumulators review, but never polished it. You can find it in this message:
Whoops, that's a rolling average algorithm for the time series library.
The implementation for the accumulators library is considerably simpler.
I just knocked this together, but it seems to work.
///////////////////////////////////////////////////////////////////////////////
// Copyright 2008 Eric Niebler. Distributed under the Boost
// Software License, Version 1.0. (See accompanying file
// LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
#include
Hello,
The help and support on this list is unmatched by any of the paid
support you get from most companies. Thank you all for such detailed
responses.
For someone like be getting into boost, this really helps.
On Wed, Dec 24, 2008 at 12:49 AM, Eric Niebler
Eric Niebler wrote:
A moving average accumulator is an often requested feature. I knocked one together as a proof-of-concept during the accumulators review, but never polished it. You can find it in this message:
Whoops, that's a rolling average algorithm for the time series library. The implementation for the accumulators library is considerably simpler. I just knocked this together, but it seems to work.
/////////////////////////////////////////////////////////////////////////////// // Copyright 2008 Eric Niebler. Distributed under the Boost // Software License, Version 1.0. (See accompanying file // LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
#include
BOOST_PARAMETER_NESTED_KEYWORD(tag, rolling_average_window_size, window_size)
In this example, the window size is know and fixed (I agree it was
based on my requirement that I had posted in the original post).
However, my window is a time window. Something like weekly sliding
window average. Since the data collection frequency could vary in my
case, I will not be able to use a circular list. I used a deque where
I add at the end and pop off from the top till the difference between
the last time stamp (most recently added) to first time stamp is NOT
greater than the window size.
It is here that I intend to create the deque using interprocess. I
will have deque will have a pair of time stamp and value
(pair
participants (3)
-
dhruva
-
Eric Niebler
-
John Phillips