[Serialization] STL Containers and Non-Default Constructors

I have recently changed from a hand-rolled serializer to Boost's and while it's mostly been fantastic, there's one problem that I can't seem to shake - I cannot get it to serialize non-default constructible types. It fails on trying to generate code that requires the usage of a default constructor. In non-associative types (list/deque/vector), the loader seems to be calling the container's resize function with a default value_type argument, resulting in the attempted creation of the object through the default constructor. In associative types, the loader attempts to create a default constructed value_type, which results in an std::pair being defaultly constructed with one of the pieces being the non-default constructible type. The code below fails during compilation with the error "'A' : no appropriate default constructor available". It was built with MSVC++2013. #include <list> #include <fstream> #include <boost\archive\binary_iarchive.hpp> #include <boost\archive\binary_oarchive.hpp> #include <boost\serialization\access.hpp> #include <boost\serialization\list.hpp> class A { private: template<class Archive> void serialize(Archive &archive, unsigned int version) { archive & myInt; } friend boost::serialization::access; public: int myInt; A(int myInt) : myInt(myInt) {} A(const A &arg) : myInt(arg.myInt) {} A& operator=(const A &arg) { myInt = arg.myInt; return *this; } bool operator==(const A &arg) const { return myInt == arg.myInt; } }; namespace boost { namespace serialization { template<class Archive> inline void load_construct_data(Archive &archive, A *a, unsigned int file_version) { ::new(a)A(0); } } } int main() { { std::ofstream stream("file"); boost::archive::binary_oarchive archive(stream); std::list cont; cont.push_back(A(1)); archive << cont; } { std::ifstream stream("file"); boost::archive::binary_iarchive archive(stream); std::list cont; archive >> cont; } } My question is: is this behavior known and what is the best solution to this for an end-user? The tutorial for STL container serialization mentions using load/save_construct_data to get around this problem, yet the attempted code generation above do not take that solution into account so compilation will fail even if you overload those templates. The only solutions I can think of to circumvent this problem would be to require a public (the code ultimately fails in std code, which I guess you may be able to friend but would be quite strange) default constructor, or to hand-roll STL container serialization through non-intrusive serialize/save/load overloads and skip the provided overloads entirely. -- View this message in context: http://boost.2283326.n4.nabble.com/Serialization-STL-Containers-and-Non-Defa... Sent from the Boost - Users mailing list archive at Nabble.com.

On 05/20/2015 08:17 AM, Trevor Thoele wrote:
The only solutions I can think of to circumvent this problem would be to require a public (the code ultimately fails in std code, which I guess you may be able to friend but would be quite strange) default constructor, or to hand-roll STL container serialization through non-intrusive serialize/save/load overloads and skip the provided overloads entirely.
I think boost/serialization/list.hpp checks for the default constructor in the wrong way. If you use std::vector instead of std::list, then it compiles.

On 5/23/15 10:54 AM, Bjorn Reese wrote:
I think boost/serialization/list.hpp checks for the default constructor in the wrong way. If you use std::vector instead of std::list, then it compiles.
I've recently spent some time on this and I believe that I've now fixed it. I had some difficulty with "is_default_constructible". a) seems that type traits aren't correct on latest version of clang - though I'm not sure about this. b) if a type has is default constructor but it's marked private - (some people do this!) it shows up as not having a default constructor. c) if there is not default constructor it shows up as not having a default constructor. The above seems obvious in retrospect - but it's takes a while to sort out when the problem is "doesn't handle private constructor" etc. Hopefully I've got it sorted out. It should be faster in some cases as well. And there maybe an opportunity to make it faster in more cases - but I haven't time to spend more of on it. Robert Ramey

Bjorn Reese wrote
I think boost/serialization/list.hpp checks for the default constructor in the wrong way. If you use std::vector instead of std::list, then it compiles.
Unfortunately this isn't exactly true. I had it using an std::vector before using std::list, which I ultimately sent and it fails as well. I have checked std::list, std::vector, std::forward_list, std::unordered_map, std::map, and std::unordered_set. Robert Ramey wrote
I've recently spent some time on this and I believe that I've now fixed it. I had some difficulty with "is_default_constructible".
Unfortunately this isn't where the problem is coming from, at least not on MSVC++2013. The problem with std::vector and std::list (and the other sequential, non-associative containers) is that regardless of is_default_contructible<T> with T being non-default constructible, the compiler will generate code that calls T::T() and will have a compiler error because of it. Specifically, you call std::vector::resize, std::list::resize and std::forward_list::resize with the second argument being default which results in the generation of code calling T::T(). Inside boost/serialization/list.hpp and the load function: if(detail::is_default_constructible<U>()){ t.resize(count); /////////////////////////////////////////////////////// <- here typename std::list<U, Allocator>::iterator hint; hint = t.begin(); while(count-- > 0){ ar >> boost::serialization::make_nvp("item", *hint++); } } The problem is when U is non-default constructible. That if will never stop the code generation under it even if is_default_constructible<U> will always be false. std::list::resize has the signature "void resize (size_type n, value_type val = value_type())"; this is where the compiler error originates from, at least for std::list, std::vector and std::forward_list. The best solution I thought of would be to change the basic if to a tag dispatch method, or to utilize SFINAE, to preclude the compiler from generating code calling U::U(). std::vector also has an additional layer above it using tag dispatch to select between unoptimized and optimized algorithms. The above will work for both of those algorithms (the problem is the same, IE they're both calling resize). The problem for the associative containers is a bit different, but (potentially?) easier to fix. Inside of boost\serialization\unordered_map.hpp: struct archive_input_unordered_map { inline void operator()( Archive &ar, Container &s, const unsigned int v ){ typedef typename Container::value_type type; detail::stack_construct<Archive, type> t(ar, v); //////////////////////////////////// <- here // borland fails silently w/o full namespace ar >> boost::serialization::make_nvp("item", t.reference()); std::pair<typename Container::const_iterator, bool> result = s.insert(t.reference()); // note: the following presumes that the map::value_type was NOT tracked // in the archive. This is the usual case, but here there is no way // to determine that. if(result.second){ ar.reset_object_address( & (result.first->second), & t.reference().second ); } } }; Type = std::unordered_map::value_type, which is an std::pair<Key, Value>. detail::stack_construct calls boost::serialization::load_construct_data_adl and because you're giving it an std::pair type, it'll call std::pair's load_contruct_data; std::pair does not have a load_construct_data, so it calls the default constructor - that default constructor in turn calls the default constructors for both elements it watches over, and in this case one of those types is non-default constructible. The solution I came up with to fix this was to not serialize the std::pair's and instead just serialize the two elements directly. This has fixed this problem and I am now able to serialize non-default constructible types in an std::unordered_map. The code below is the new loader: template<class Archive, class Container> struct archive_input_unordered_map { inline void operator()(Archive &ar, Container &s, const unsigned int v) { typedef typename Container::key_type First; typedef typename Container::mapped_type Second; // borland fails silently w/o full namespace detail::stack_construct<Archive, First> first(ar, v); ar >> boost::serialization::make_nvp("first", first.reference()); detail::stack_construct<Archive, Second> second(ar, v); ar >> boost::serialization::make_nvp("second", second.reference()); std::pair<typename Container::const_iterator, bool> result = s.emplace(std::move(first.reference()), std::move(second.reference())); // note: the following presumes that the map::value_type was NOT tracked // in the archive. This is the usual case, but here there is no way // to determine that. if(result.second) ar.reset_object_address(&(result.first->second), &second.reference()); } }; One would also have to change the saving code to reflect the new loading above. I'm not sure if that works for all cases. It might not be savory because of the elimination of the item nvp for XML. I also had to add in the std::move's to work with std::unique_ptr's, not sure if your usage of insert would differ from my usage of emplace to impact that in any way, though; obviously these would need more testing. -- View this message in context: http://boost.2283326.n4.nabble.com/Serialization-STL-Containers-and-Non-Defa... Sent from the Boost - Users mailing list archive at Nabble.com.

On 5/23/15 6:07 PM, Trevor Thoele wrote:
The problem with std::vector and std::list (and the other sequential, non-associative containers) is that regardless of is_default_contructible<T> with T being non-default constructible, the compiler will generate code that calls T::T() and will have a compiler error because of it.
I know that. The current version uploaded to devel and merged to release addresses this problem such that I don't think that it will occur any more. It's currently passing tests on all gcc, clang and msvc platforms. In order to do this I had to: define a custom version if is_default_constructable - I don't know if that is really necessary. but I seemed to have some issues with the default one. It didnt' fix the total problem so maybe in hindsight it wasn't necessary, but for now it's in there. I had to do use different algorithms depending on whether is_default_constructible resolves to true or false. And I enhanced tests to detect future failures. I believe that this issue is resolved. Feel free to inspect the current version and let me know if I got something wroing. I did detect some unexploited opportunities to improve peformance with C++11 and also some interface errors on very obscure functions. But for now I'm moving on to something else. Robert Ramey.
participants (3)
-
Bjorn Reese
-
Robert Ramey
-
Trevor Thoele