[container] varray aka static_vector

As Adam mentioned in an earlier thread we have renamed static_vector to varray to better represent how it works like an array with size without requiring the values to be default constructible, and to better distinguish it from hybrid_vector. varray location: https://svn.boost.org/svn/boost/sandbox/varray/ Documentation: https://svn.boost.org/svn/boost/sandbox/varray/doc/html/index.html One open question is if the 3rd strategy parameter should be public or in the detail namespace. Based on some of the feedback we moved the strategy from container to container_detail, and made the public version varray<typename T, std::size_t N>. However, we wanted to get others' thoughts on this change. Additionally, should primitives be default initialized with the contents of memory like an array or value initialized to 0 like a standard container in the constructor, resize(), and emplace_back()? Generally, built-in types are initialized when the constructor is called explicitly. For example: int a; // a has the value of underlying memory int b = int(); // b has a value of 0 Right now the default behavior of our container is more like a vector than an array when creating values using the default ctor. Default constructable values are explicitly initialized in the varray ctor, resize(), emplace() and emplace_back() functions, even when they are trivially constructable. We find this behavior more intuitive and therefore we have made it the default. If the performance needs to be improved slightly the behavior may be changed or disabled in container_detail::varray_traits<V, C, S>. Here is an example demonstrating the current default behavior: vector<int> vec(3); varray<int, 3> varr(3); array<int, 3> arr; vec[0] == varr[0] and vec[0] != arr[0] Any feedback is very much appreciated. Cheers! Andrew Hundt Adam Wulkiewicz

2013/2/11 Andrew Hundt <athundt@gmail.com>
As Adam mentioned in an earlier thread we have renamed static_vector to varray to better represent how it works like an array with size without requiring the values to be default constructible, and to better distinguish it from hybrid_vector.
varray location: https://svn.boost.org/svn/boost/sandbox/varray/
Documentation: https://svn.boost.org/svn/boost/sandbox/varray/doc/html/index.html
Many thanks for this effort. I'm already using your class (the previous static_vector at the moment).
One open question is if the 3rd strategy parameter should be public or in the detail namespace. Based on some of the feedback we moved the strategy from container to container_detail, and made the public version varray<typename T, std::size_t N>. However, we wanted to get others' thoughts on this change.
As I wrote in the previous thread, I'd rather the strategy was public. I can imagine using varray in a few places in the same project with different strategies: 1) In most places I'd probably use the default strategy. 2) At the time of optimizations I might want to disable all checking (asserts) only for one specific use. In practice we don't create separate debug/release builds, since we need the debug version to be fast enough anyway, and in some places this means disabling asserts. 3) In some cases temporary cases I might want logic errors to be thrown, so that the crash of an early version of one module doesn't crash the whole system.
Additionally, should primitives be default initialized with the contents of memory like an array or value initialized to 0 like a standard container in the constructor, resize(), and emplace_back()?
Now that I think of it, I wish there was a way to push an element to the back of varray without zero-initializing it (for PODs). But I think emplace_back should be consistent with other containers, so maybe some new member function can be added for this? I'd much like those new member functions to be added to other boost containers too. In particular, I have a use case of a vector, in which memory is reserved earlier, and then POD elements are pushed to the back in a time-critical function. Being able to do some uninitialized_push_back or uninitialized_resize, and then setting contents of elements in place would be handy.
Generally, built-in types are initialized when the constructor is called explicitly. For example:
int a; // a has the value of underlying memory int b = int(); // b has a value of 0
Right now the default behavior of our container is more like a vector than an array when creating values using the default ctor. Default constructable values are explicitly initialized in the varray ctor, resize(), emplace() and emplace_back() functions, even when they are trivially constructable. We find this behavior more intuitive and therefore we have made it the default. If the performance needs to be improved slightly the behavior may be changed or disabled in container_detail::varray_traits<V, C, S>.
I assume I can't change the Strategy or emplace_back()/resize() behavior for just one particular varray object, can I? I mean: varray<int,5> a; varray<int,5> b; I'd like to change the Strategy only for b, can I do that? Regards, Kris

Krzysztof Czainski wrote:
2013/2/11 Andrew Hundt <athundt@gmail.com>
Generally, built-in types are initialized when the constructor is called explicitly. For example:
int a; // a has the value of underlying memory int b = int(); // b has a value of 0
Right now the default behavior of our container is more like a vector than an array when creating values using the default ctor. Default constructable values are explicitly initialized in the varray ctor, resize(), emplace() and emplace_back() functions, even when they are trivially constructable. We find this behavior more intuitive and therefore we have made it the default. If the performance needs to be improved slightly the behavior may be changed or disabled in container_detail::varray_traits<V, C, S>.
I assume I can't change the Strategy or emplace_back()/resize() behavior for just one particular varray object, can I?
I mean:
varray<int,5> a; varray<int,5> b;
I'd like to change the Strategy only for b, can I do that?
You may do it by partially specialize varray_traits for your strategy. Currently the varray allowing to pass user-defined strategy is implemented in the container_detail namespace. So it would look more or less like this: namespace bc = boost::container; // define my strategy template <typename V> struct my_strategy : public bc::container_detail::strategy::def<V> {}; // specialize traits for my strategy namespace boost { namespace container { namespace container_detail { template <typename V, typename C> struct varray_traits<V, C, my_strategy> : public varray_traits<V, C, strategy::def<V> > { typedef boost::true_type disable_trivial_init; }; }}} /*...*/ // use non-default version of varray bc::container_detail::varray<int, 5, my_strategy<int> > b; Future alternative: Currently settings like disable_trivial_init are hidden deeply inside details. However they might be a part of default Strategy concept. Assuming that varray was moved to the boost::container namespace the example would be shorter: // define my strategy template <typename V> struct my_strategy : public bc::strategy::def<V> { typedef boost::true_type disable_trivial_init; }; /*...*/ // use non-default version of varray bc::varray<int, 5, my_strategy<int> > b; Regards, Adam

2013/2/11 Adam Wulkiewicz <adam.wulkiewicz@gmail.com>
Krzysztof Czainski wrote:
I assume I can't change the Strategy or emplace_back()/resize() behavior
for just one particular varray object, can I?
I mean:
varray<int,5> a; varray<int,5> b;
I'd like to change the Strategy only for b, can I do that?
You may do it by partially specialize varray_traits for your strategy. Currently the varray allowing to pass user-defined strategy is implemented in the container_detail namespace. So it would look more or less like this:
namespace bc = boost::container;
// define my strategy template <typename V> struct my_strategy : public bc::container_detail::**strategy::def<V> {};
// specialize traits for my strategy namespace boost { namespace container { namespace container_detail {
template <typename V, typename C> struct varray_traits<V, C, my_strategy> : public varray_traits<V, C, strategy::def<V> > { typedef boost::true_type disable_trivial_init; };
}}}
/*...*/
// use non-default version of varray bc::container_detail::varray<**int, 5, my_strategy<int> > b;
Aha, so I just use the hidden in container_detail version of varray with 3 template parameters. Future alternative:
Currently settings like disable_trivial_init are hidden deeply inside details. However they might be a part of default Strategy concept. Assuming that varray was moved to the boost::container namespace the example would be shorter:
// define my strategy template <typename V> struct my_strategy : public bc::strategy::def<V> { typedef boost::true_type disable_trivial_init; };
/*...*/
// use non-default version of varray bc::varray<int, 5, my_strategy<int> > b;
Ok, then +1 for this option ;-) Thank you, Adam, for elaborating on this topic. Cheers, Kris

On 11 February 2013 14:32, Krzysztof Czainski <1czajnik@gmail.com> wrote:
Now that I think of it, I wish there was a way to push an element to the back of varray without zero-initializing it (for PODs). But I think emplace_back should be consistent with other containers, so maybe some new member function can be added for this?
FYI: You can with the standard containers by using an allocator with a different method for construct. -- Nevin ":-)" Liber <mailto:nevin@eviloverlord.com> (847) 691-1404

Nevin Liber wrote:
On 11 February 2013 14:32, Krzysztof Czainski <1czajnik@gmail.com> wrote:
Now that I think of it, I wish there was a way to push an element to the back of varray without zero-initializing it (for PODs). But I think emplace_back should be consistent with other containers, so maybe some new member function can be added for this?
FYI: You can with the standard containers by using an allocator with a different method for construct.
If the library is written in C++11 or like Boost.Container emulates some mechanisms, in-place construction in allocator_traits in this case. Regards, Adam

2013/2/12 Adam Wulkiewicz <adam.wulkiewicz@gmail.com>
Nevin Liber wrote:
On 11 February 2013 14:32, Krzysztof Czainski <1czajnik@gmail.com> wrote:
Now that I think of it, I wish there was a way to push an element to the
back of varray without zero-initializing it (for PODs). But I think emplace_back should be consistent with other containers, so maybe some new member function can be added for this?
FYI: You can with the standard containers by using an allocator with a different method for construct.
If the library is written in C++11 or like Boost.Container emulates some mechanisms, in-place construction in allocator_traits in this case.
Thank you, Adam and Nevin for these insights, I'll definitely try one of the above allocator::construct tricks when it turns out I need to optimize my code further. However, I'd rather there was a new member function for "noinit" pushing/resizing. Just like the emplace* family of functions was added, while the push* family was left alone ;-) Cheers, Kris

On 12 February 2013 09:10, Krzysztof Czainski <1czajnik@gmail.com> wrote:
However, I'd rather there was a new member function for "noinit" pushing/resizing. Just like the emplace* family of functions was added, while the push* family was left alone ;-)
Do you also envision another method for emplace construction indicating list initialization (aka uniform initialization, ::new((void *)p) U{std::forward<Args>(args)...}) vs. direct initialization (::new((void *)p) U(std::forward<Args>(args)...))? -- Nevin ":-)" Liber <mailto:nevin@eviloverlord.com> (847) 691-1404

2013/2/12 Nevin Liber <nevin@eviloverlord.com>
On 12 February 2013 09:10, Krzysztof Czainski <1czajnik@gmail.com> wrote:
However, I'd rather there was a new member function for "noinit" pushing/resizing. Just like the emplace* family of functions was added, while the push* family was left alone ;-)
Do you also envision another method for emplace construction indicating list initialization (aka uniform initialization, ::new((void *)p) U{std::forward<Args>(args)...}) vs. direct initialization (::new((void *)p) U(std::forward<Args>(args)...))?
I am not familiar with C++11 initialization syntax details, I only read about that, and I'm stuck with C++03 for now. Are you suggesting, that this would mean adding a lot of (too many?) member functions? Anyway, personally I'm only interested in features, that can be added for C++03 ;-) And noinit_push*/noinit_resize (enabled_if< is_pod<value_type> >) are things I have use cases for. Cheers, Kris

On Tue, Feb 12, 2013 at 3:02 PM, Krzysztof Czainski <1czajnik@gmail.com>wrote:
2013/2/12 Nevin Liber <nevin@eviloverlord.com>
On 12 February 2013 09:10, Krzysztof Czainski <1czajnik@gmail.com> wrote:
However, I'd rather there was a new member function for "noinit" pushing/resizing. Just like the emplace* family of functions was added, while the push* family was left alone ;-)
Anyway, personally I'm only interested in features, that can be added for C++03 ;-) And noinit_push*/noinit_resize (enabled_if< is_pod<value_type> >) are things I have use cases for.
Could you explain a little more about why emplace_back is not sufficient? I believe it includes rvalue reference emulation for C++03. Cheers! Andrew Hundt

2013/2/12 Andrew Hundt <athundt@gmail.com>
On Tue, Feb 12, 2013 at 3:02 PM, Krzysztof Czainski <1czajnik@gmail.com
wrote:
2013/2/12 Nevin Liber <nevin@eviloverlord.com>
On 12 February 2013 09:10, Krzysztof Czainski <1czajnik@gmail.com> wrote:
However, I'd rather there was a new member function for "noinit" pushing/resizing. Just like the emplace* family of functions was added, while the push* family was left alone ;-)
Anyway, personally I'm only interested in features, that can be added for C++03 ;-) And noinit_push*/noinit_resize (enabled_if< is_pod<value_type> ) are things I have use cases for.
Could you explain a little more about why emplace_back is not sufficient? I believe it includes rvalue reference emulation for C++03.
Sure: struct X { int a; double b; } typedef boost::container::vector<X> V; // or varray<X,5> or some other container ;-) V v; v.emplace_back(); // unnecessarily zero-initializes memory v.back().a = 1; v.back().b = 3.14; If emplace_back() didn't zero-initialize PODs, it would be inconsistent with std containers, and therefore surprising, so I suggest a new function for this. Same for resize(). Or perhaps if not a new member function, maybe an overload for a noinit tag, used like so: v.emplace_back( noinit ); With noinit defined in the library something like: struct noinit_t {}; noinit_t const noinit = {}; Regards, Kris

On 12 February 2013 16:22, Krzysztof Czainski <1czajnik@gmail.com> wrote:
If emplace_back() didn't zero-initialize PODs, it would be inconsistent with std containers,
Please be precise; it is only inconsistent with the standard containers when using the default argument for the allocator (aka std::allocator). The standard containers, at least in C++11, have no embedded knowledge on how to construct/destroy their elements; they merely delegate that functionality off to the allocator. -- Nevin ":-)" Liber <mailto:nevin@eviloverlord.com> (847) 691-1404

On 12 February 2013 14:02, Krzysztof Czainski <1czajnik@gmail.com> wrote:
2013/2/12 Nevin Liber <nevin@eviloverlord.com>
I am not familiar with C++11 initialization syntax details, I only read about that, and I'm stuck with C++03 for now. Are you suggesting, that this would mean adding a lot of (too many?) member functions?
There are some corner cases where list initialization behaves differently than direct initialization. Take the following example: template<typename T> struct ListInitializationAllocator { typedef T value_type; typedef typename std::aligned_storage<sizeof(T), alignof(T)>::type storage_type; T* allocate(size_t n) { return static_cast<T*>(static_cast<void*>(::new storage_type[n])); } void deallocate(T* p, size_t) { ::delete [] static_cast<storage_type*>(static_cast<void*>(p)); } template<typename... Args> void construct(T* c, Args&&... args) { ::new (static_cast<void*>(c)) T{ std::forward<Args>(args)... }; } }; vector<vector<int>> vd; vd.emplace_back(2); assert(2 == vd.back().size()); vector<vector<int>, ListInitializationAllocator<vector<int>>> vl; vl.emplace_back(2); assert(1 == vl.back().size()); The element in vd is a vector<int> of 2 elements which are value-initialized to 0. The element in vl is a vector<int> of 1 element copy-initialized to 2.
Anyway, personally I'm only interested in features, that can be added for C++03 ;-)
Boost needs to be more concerned than that.
And noinit_push*/noinit_resize (enabled_if< is_pod<value_type> >) are things I have use cases for.
That is the difference between designing a library for yourself and a library for the community at large. This use case might not be important enough to implement, but it should at least be considered. -- Nevin ":-)" Liber <mailto:nevin@eviloverlord.com> (847) 691-1404

2013/2/13 Nevin Liber <nevin@eviloverlord.com>
On 12 February 2013 14:02, Krzysztof Czainski <1czajnik@gmail.com> wrote:
I am not familiar with C++11 initialization syntax details, I only read about that, and I'm stuck with C++03 for now. Are you suggesting, that this would mean adding a lot of (too many?) member functions?
There are some corner cases where list initialization behaves differently than direct initialization.
Take the following example:
template<typename T> struct ListInitializationAllocator { typedef T value_type; typedef typename std::aligned_storage<sizeof(T), alignof(T)>::type storage_type;
T* allocate(size_t n) { return static_cast<T*>(static_cast<void*>(::new storage_type[n])); } void deallocate(T* p, size_t) { ::delete [] static_cast<storage_type*>(static_cast<void*>(p)); }
template<typename... Args> void construct(T* c, Args&&... args) { ::new (static_cast<void*>(c)) T{ std::forward<Args>(args)... }; } };
vector<vector<int>> vd; vd.emplace_back(2); assert(2 == vd.back().size());
vector<vector<int>, ListInitializationAllocator<vector<int>>> vl; vl.emplace_back(2); assert(1 == vl.back().size());
The element in vd is a vector<int> of 2 elements which are value-initialized to 0. The element in vl is a vector<int> of 1 element copy-initialized to 2.
Thank you, Nevin, for the above example. I think I see your point now. Furthermore, it gives me an idea: On 12 February 2013 23:17, Krzysztof Czainski <1czajnik@gmail.com> wrote:
Or perhaps if not a new member function, maybe an overload for a noinit tag, used like so:
v.emplace_back( noinit );
With noinit defined in the library something like: struct noinit_t {}; noinit_t const noinit = {};
Since the emplace* functions just forward construction args to the Allocator, the above noinit overloads can be implemented by just supplying a user Allocator: #include <boost/container/vector.hpp> #include <boost/type_traits/is_pod.hpp> #include <boost/utility/enable_if.hpp> #include <iostream> using namespace std; struct noinit_t {}; noinit_t const noinit = {}; template < class T > struct MyAllocator : std::allocator<T> { using std::allocator<T>::construct; // for v.emplace_back(noinit); void construct( T* c, noinit_t/*, typename boost::enable_if< boost::is_pod<T> >::type* = 0*/ ) {} }; int main() { #if !defined(BOOST_NO_CXX11_VARIADIC_TEMPLATES) namespace cont = std; #else namespace cont = boost::container; #endif cont::vector< int, MyAllocator<int> > vi; vi.emplace_back(5); vi.pop_back(); vi.emplace_back(noinit); assert( 0 != vi.back() ); } Does this make sense? It works with mingw-4.7.2, but unfortunately it doesn't compile with -std=c++11, any ideas why? I also tried the same trick with listinit_t and list initialization (code attached), as Nevin suggests above, but that also doesn't compile. I don't think this can be done for resize(), can it? Cheers, Kris

On 13 February 2013 07:20, Krzysztof Czainski <1czajnik@gmail.com> wrote:
Thank you, Nevin, for the above example. I think I see your point now. Furthermore, it gives me an idea: template < class T > struct MyAllocator : std::allocator<T> { using std::allocator<T>::construct;
// for v.emplace_back(noinit); void construct( T* c, noinit_t/*, typename boost::enable_if< boost::is_pod<T> >::type* = 0*/ ) {}
};
I wouldn't even bother with the enable_if; just use the default initialization placement new syntax when no arguments are passed, as in: // direct initialization template<typename... Args> void construct(T* c, Args&&... args) { ::new (static_cast<void*>(c)) T(std::forward<Args>(args)... ); } // default initialization for zero arguments void construct(T* c) { ::new (static_cast<void*>(c)) T; } Note: there is a bit more work that has to be done for allocators than just deriving from std::allocator<A>; for instance, rebind won't do the right thing. I don't think this can be done for resize(), can it?
Works for resize too under C++11. -- Nevin ":-)" Liber <mailto:nevin@eviloverlord.com> (847) 691-1404

2013/2/13 Nevin Liber <nevin@eviloverlord.com>
On 13 February 2013 07:20, Krzysztof Czainski <1czajnik@gmail.com> wrote:
Thank you, Nevin, for the above example. I think I see your point now. Furthermore, it gives me an idea: template < class T > struct MyAllocator : std::allocator<T> { using std::allocator<T>::construct;
// for v.emplace_back(noinit); void construct( T* c, noinit_t/*, typename boost::enable_if< boost::is_pod<T> >::type* = 0*/ ) {}
};
I wouldn't even bother with the enable_if; just use the default initialization placement new syntax when no arguments are passed, as in:
// direct initialization template<typename... Args> void construct(T* c, Args&&... args) { ::new (static_cast<void*>(c)) T(std::forward<Args>(args)... ); }
// default initialization for zero arguments void construct(T* c) { ::new (static_cast<void*>(c)) T; }
Oh yeah, that will just do the right thing in all cases, thanks ;-) But I think you missed the noinit_t trick above, didn't you?
Note: there is a bit more work that has to be done for allocators than just deriving from std::allocator<A>; for instance, rebind won't do the right thing.
Oh, thanks, I'll try if that fixes my compiling errors and post results later.
I don't think this can be done for resize(), can it?
Works for resize too under C++11.
I mean the noinit_t trick - can that be done for resize just by writing an Allocator? I don't see how. Regards, Kris

On 13 February 2013 10:51, Krzysztof Czainski <1czajnik@gmail.com> wrote:
I mean the noinit_t trick - can that be done for resize just by writing an Allocator? I don't see how.
No, as you basically need two different placement new syntaxes to make it work. I'm not sure how much the ability to control default vs. value initialization on a per element basis (as opposed to a per container basis, which allocators give you) really comes up in practice. One can also apply the same reasoning to direct vs. list initialization. Do we really need all these knobs, or are we just adding them because we can? -- Nevin ":-)" Liber <mailto:nevin@eviloverlord.com> (847) 691-1404

2013/2/13 Nevin Liber <nevin@eviloverlord.com>
On 13 February 2013 10:51, Krzysztof Czainski <1czajnik@gmail.com> wrote:
I mean the noinit_t trick - can that be done for resize just by writing an Allocator? I don't see how.
No, as you basically need two different placement new syntaxes to make it work.
I'm not sure how much the ability to control default vs. value initialization on a per element basis (as opposed to a per container basis, which allocators give you) really comes up in practice.
My main concern isn't, that I'll ever need both in the same container (but maybe I will). The main point is I'd rather know, that emplace_back() and resize() always zero-initialize without having to check the details of the Allocator used. And so, I'd rather see a different function call for non-initializing operations. One can also apply the same reasoning to direct vs. list initialization.
I think the same arguments apply to list initialization, but I only think about it as a mind exercise, since I don't have any use-cases for it ;-)
Do we really need all these knobs, or are we just adding them because we can?
I see a need for noinit versions, and with list initialization - yes it's just for fun ;-) Regards, Kris

2013/2/13 Krzysztof Czainski <1czajnik@gmail.com>
2013/2/13 Nevin Liber <nevin@eviloverlord.com>
On 13 February 2013 07:20, Krzysztof Czainski <1czajnik@gmail.com> wrote:
Thank you, Nevin, for the above example. I think I see your point now. Furthermore, it gives me an idea
[...]
Note: there is a bit more work that has to be done for allocators than
just deriving from std::allocator<A>; for instance, rebind won't do the
right thing.
Oh, thanks, I'll try if that fixes my compiling errors and post results later.
I confirm, adding rebind<> fixed my example - it now works both with and without -std=c++11 (code attached).
I don't think this can be done for resize(), can it?
Works for resize too under C++11.
I mean the noinit_t trick - can that be done for resize just by writing an Allocator? I don't see how.
And so the last above question remains ;-) I'll elaborate on that: struct noinit_t {}; noinit_t const noinit = {}; template < class T > struct MyAllocator : std::allocator<T> { using std::allocator<T>::construct; // for v.emplace_back(noinit); void construct( T* c, noinit_t ) { ::new (static_cast<void*>(c)) T; // note: T; instead of T(); } template < class U > struct rebind { typedef MyAllocator<U> other; }; }; int main() { namespace cont = boost::container; cont::vector< int, MyAllocator<int> > vi; vi.emplace_back(5); vi.pop_back(); vi.emplace_back(noinit); assert( 0 != vi.back() ); vi.pop_back(); vi.resize( 1, noinit ); // !!! assert( 0 != vi.back() ); } vi.resize( 1, noinit ) does't compile - and it shouldn't, but my question is, can I get the desired effect: leave default resize as is, and have an overload that doesn't zero-initialize PODs? Cheers, Kris

El 12/02/2013 16:10, Krzysztof Czainski escribió:
Thank you, Adam and Nevin for these insights, I'll definitely try one of the above allocator::construct tricks when it turns out I need to optimize my code further.
Some months ago I read an interesting proposal to control value initialization vs. default initialization in containers: https://groups.google.com/a/isocpp.org/forum/#!topic/std-proposals/m9KFTXtHJ... Maybe it's an interesting feature to add to Boost.Container Best, Ion
participants (5)
-
Adam Wulkiewicz
-
Andrew Hundt
-
Ion Gaztañaga
-
Krzysztof Czainski
-
Nevin Liber