[Boost.MPI] question about serializing custom object
hello everybody, this is first post i have some questions after reading the Boost documentation of serialization and MPI, is it possible to serialize my custom objects with Boost.serialize and then send/receive them using plain openMPI as MPI_BYTE type?? the other question is on the tutorials i only found examples of serializing objects to archive and saving to file, but i didnt find any example where they serialize it to a binary form and send them through mpi, is there any example of this type on the documentation? best regards Cristobal
Cristobal Navarro wrote:
the other question is on the tutorials i only found examples of serializing objects to archive and saving to file, but i didnt find any example where they serialize it to a binary form and send them through mpi, is there any example of this type on the documentation?
no Feel free to contribute one. If I had nothing else to do, I would a) make a stream_buf which uses MPI b) use binary_iarchive and binary_oarchive with this new stream_buf c) open one program with binary_oarchive and the other with binary_iarchive d) do ar << on one end and ar >> at the other e) stand back and let'er rip. Robert Ramey
thanks for the quick response. i've never used Boost before, so need to understand some of its basics. but as soon as i get this working, ill will contribute my own case as an example for sure. no
Feel free to contribute one.
If I had nothing else to do, I would
a) make a stream_buf which uses MPI b) use binary_iarchive and binary_oarchive with this new stream_buf c) open one program with binary_oarchive and the other with binary_iarchive d) do ar << on one end and ar >> at the other e) stand back and let'er rip.
--> so this way it would be possible to mix plain MPi with Boost::Serialize right? thanks
Robert Ramey
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 26 Aug 2010, at 20:13, Cristobal Navarro wrote:
thanks for the quick response.
i've never used Boost before, so need to understand some of its basics. but as soon as i get this working, ill will contribute my own case as an example for sure.
no
Feel free to contribute one.
If I had nothing else to do, I would
a) make a stream_buf which uses MPI b) use binary_iarchive and binary_oarchive with this new stream_buf c) open one program with binary_oarchive and the other with binary_iarchive d) do ar << on one end and ar >> at the other e) stand back and let'er rip.
--> so this way it would be possible to mix plain MPi with Boost::Serialize right? thanks
Change the "would" into "is": it is possible to mix Boost.MPI with Boost.Serialization. Any objext that can be serialized can be sent by Boost.MPI. This was actually the basic idea behind Boost.MPI, to use Boost.Serialization to pack and unpack MPI buffers, or to create custom MPI datatypes. Matthias
the thing is that im using openMPI 1.4.2 and recently i added this Boost::serialization library which is phenomenal. i would like to keep like this, i mean try not to use Boost:MPI but still use its serialize features. On Thu, Aug 26, 2010 at 2:23 PM, Matthias Troyer <troyer@phys.ethz.ch>wrote:
On 26 Aug 2010, at 20:13, Cristobal Navarro wrote:
thanks for the quick response.
i've never used Boost before, so need to understand some of its basics. but as soon as i get this working, ill will contribute my own case as an example for sure.
no
Feel free to contribute one.
If I had nothing else to do, I would
a) make a stream_buf which uses MPI b) use binary_iarchive and binary_oarchive with this new stream_buf c) open one program with binary_oarchive and the other with binary_iarchive d) do ar << on one end and ar >> at the other e) stand back and let'er rip.
--> so this way it would be possible to mix plain MPi with Boost::Serialize right? thanks
Change the "would" into "is": it is possible to mix Boost.MPI with Boost.Serialization. Any objext that can be serialized can be sent by Boost.MPI. This was actually the basic idea behind Boost.MPI, to use Boost.Serialization to pack and unpack MPI buffers, or to create custom MPI datatypes.
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Sent from my iPhone On Aug 26, 2010, at 10:32 AM, Cristobal Navarro <axischire@gmail.com> wrote:
the thing is that im using openMPI 1.4.2 and recently i added this Boost::serialization library which is phenomenal. i would like to keep like this, i mean try not to use Boost:MPI but still use its serialize features.
If you do that, you may be giving up huge optimizations or signing up for a lot of tedious work defining and maintaining MPI datatypes. I wonder if our docs really make that clear?
well, actually nothing blocks me of using boost.mpi, but is boost.MPI compatible with openMPI 1.4.2? on the doc it recommends a very old version 1.0.x i think On Thu, Aug 26, 2010 at 3:25 PM, Dave Abrahams <dave@boostpro.com> wrote:
Sent from my iPhone
On Aug 26, 2010, at 10:32 AM, Cristobal Navarro <axischire@gmail.com> wrote:
the thing is that im using openMPI 1.4.2 and recently i added this Boost::serialization library which is phenomenal. i would like to keep like this, i mean try not to use Boost:MPI but still use its serialize features.
If you do that, you may be giving up huge optimizations or signing up for a lot of tedious work defining and maintaining MPI datatypes. I wonder if our docs really make that clear? _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 26 Aug 2010, at 21:35, Cristobal Navarro wrote:
well, actually nothing blocks me of using boost.mpi, but is boost.MPI compatible with openMPI 1.4.2? on the doc it recommends a very old version 1.0.x i think
It is compatible with any MPI-standard compliant MPI implementation. The docs just state the minimal version we recommend. Matthias
On 26 Aug 2010, at 21:25, Dave Abrahams wrote:
Sent from my iPhone
On Aug 26, 2010, at 10:32 AM, Cristobal Navarro <axischire@gmail.com> wrote:
the thing is that im using openMPI 1.4.2 and recently i added this Boost::serialization library which is phenomenal. i would like to keep like this, i mean try not to use Boost:MPI but still use its serialize features.
If you do that, you may be giving up huge optimizations or signing up for a lot of tedious work defining and maintaining MPI datatypes. I wonder if our docs really make that clear?
He would lose optimization features only by not using the skeleton&contents mechanism. For plain send and receive it is equivalent, but just a waste of time because he reimplements send and recv in an identical way to Boost.MPI Matthias
Change the "would" into "is": it is possible to mix Boost.MPI with Boost.Serialization. Any objext that can be serialized can be sent by Boost.MPI. This was actually the basic idea behind Boost.MPI, to use Boost.Serialization to pack and unpack MPI buffers, or to create custom MPI datatypes.
On thing that I never understood is why you need MPI data types at all. If one is serializing (packing) to a binary array, and sending that, what hae MPI data types have to do with it. If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible. So I'm just not seeing this. Robert Ramey
i made some progress while we were mailing between us. first of all, im trying to see if my object can be serialized / deserialized in the common way without going into MPI yet. objects are Lattices, they have Nodes and Edges. nothing more. this is how i've setup serialization for all three types of classes: #NODE class Node{ private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version){ ar & sp; ar & id; ar & ext; ar & x; ar & y; ar & z; ar & eMap; } public: bool sp; //special flag int id, ext; float x, y, z; multimap<int,int> eMap; //plus some methods which i didnt put }; #CLASS EDGE class Edge{ private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version){ ar & sp; ar & dead; ar & n1; ar & n2; ar & id; } public: int n1; int n2; int id; bool sp; bool dead; //plus some methods, constructors, etc }; #finally, CLASS LATTICE class Lattice{ private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version){ ar & numNodes; //int ar & numEdges; //int ar & nodes; //map ar & edges; //map //ar & key; //! had to comment this attribute, stringstream couldn't be serialized ar & keyLists; //list < list> > ar & corrupt;//bool ar & rec;//bool ar & nop;//int } public: int numNodes; int numEdges; map<int, Edge> edges; map<int, Node> nodes; stringstream key; list< list<int> > keyLists; bool corrupt; bool rec; int nop; //plus constructors, methods, etc } it compiles and links ok, and when i test serialization / deserialization with the following code..: printf("serializing to binlat.x...\n"); //!file mode /* std::ofstream ofs("binlat.x", ios::binary); boost::archive::binary_oarchive oa(ofs); */ oa << *(this->lat); ofs.close(); lat->print(); //check what lattice im serializing //!Deserialization - read file binlat.x and reconstruct printf("reconstructing serialized binlat.x...\n"); Lattice nl; std::ifstream ifs("binlat.x", ios::binary); boost::archive::binary_iarchive ia(ifs); //!deserialize object ia >> nl; ifs.close(); nl.print(); i get bad errors when deserializing, complaining about a bool and char. serializing to binlat.x... Lattice::print().....START Nodes=2 Edges= 1 Node[0] sp=1 ext=1 Linked to edge 2 (Node 3 or 0 ) Node[3] sp=1 ext=2 Linked to edge 2 (Node 3 or 0 ) Lattice::print().....END [enter] reconstructing swerialized binlat.x... [enter] plattice: /usr/local/include/boost/archive/basic_binary_iprimitive.hpp:98: void boost::archive::basic_binary_iprimitive<Archive, Elem, Tr>::load(bool&) [with Archive = boost::archive::binary_iarchive, Elem = char, Tr = std::char_traits<char>]: Assertion `0 == i || 1 == i' failed. [lenovo00:29384] *** Process received signal *** [lenovo00:29384] Signal: Aborted (6) [lenovo00:29384] Signal code: (-6) [lenovo00:29384] [ 0] [0xfea410] [lenovo00:29384] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0xd41a82] [lenovo00:29384] [ 2] /lib/tls/i686/cmov/libc.so.6(__assert_fail+0xf8) [0xd37718] [lenovo00:29384] [ 3] .. .. *** End of error message *** i started debugging, and realized that the problem is completely gone when i remove "edges" from the serialization members of "Lattice" object. its weird, because "edge" is a map of <int, Edge> and much simplier than "nodes" which does work. obviously i cannot continue without edges, so i need to fix it somehow, but i dont understand the error related to edges, maybe is something im not seeing. dont get distracted by the messy code, i had to edit it a little to focus on the case. regards Cristobal On Thu, Aug 26, 2010 at 4:35 PM, Robert Ramey <ramey@rrsd.com> wrote:
Change the "would" into "is": it is possible to mix Boost.MPI with Boost.Serialization. Any objext that can be serialized can be sent by Boost.MPI. This was actually the basic idea behind Boost.MPI, to use Boost.Serialization to pack and unpack MPI buffers, or to create custom MPI datatypes. On thing that I never understood is why you need MPI data types at all. If one is serializing (packing) to a binary array, and sending that, what hae MPI data types have to do with it.
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible. So I'm just not seeing this.
Robert Ramey
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 26 Aug 2010, at 22:03, Cristobal Navarro wrote:
i made some progress while we were mailing between us.
first of all, im trying to see if my object can be serialized / deserialized in the common way without going into MPI yet. objects are Lattices, they have Nodes and Edges. nothing more.
this is how i've setup serialization for all three types of classes:
...
i get bad errors when deserializing, complaining about a bool and char. serializing to binlat.x... Lattice::print().....START Nodes=2 Edges= 1 Node[0] sp=1 ext=1 Linked to edge 2 (Node 3 or 0 ) Node[3] sp=1 ext=2 Linked to edge 2 (Node 3 or 0 ) Lattice::print().....END [enter] reconstructing swerialized binlat.x... [enter] plattice: /usr/local/include/boost/archive/basic_binary_iprimitive.hpp:98: void boost::archive::basic_binary_iprimitive<Archive, Elem, Tr>::load(bool&) [with Archive = boost::archive::binary_iarchive, Elem = char, Tr = std::char_traits<char>]: Assertion `0 == i || 1 == i' failed. [lenovo00:29384] *** Process received signal *** [lenovo00:29384] Signal: Aborted (6) [lenovo00:29384] Signal code: (-6) [lenovo00:29384] [ 0] [0xfea410] [lenovo00:29384] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0xd41a82] [lenovo00:29384] [ 2] /lib/tls/i686/cmov/libc.so.6(__assert_fail+0xf8) [0xd37718] [lenovo00:29384] [ 3] .. .. *** End of error message ***
i started debugging, and realized that the problem is completely gone when i remove "edges" from the serialization members of "Lattice" object. its weird, because "edge" is a map of <int, Edge> and much simplier than "nodes" which does work. obviously i cannot continue without edges, so i need to fix it somehow, but i dont understand the error related to edges, maybe is something im not seeing.
Robert, that question is best looked into by you Matthias
Cristobal Navarro wrote:
i made some progress while we were mailing between us.
first of all, im trying to see if my object can be serialized / deserialized in the common way without going into MPI yet.
good idea - one thing at a time.
objects are Lattices, they have Nodes and Edges. nothing more.
it compiles and links ok, and when i test serialization / deserialization with the following code..:
printf("serializing to binlat.x...\n"); //!file mode /* std::ofstream ofs("binlat.x", ios::binary); boost::archive::binary_oarchive oa(ofs); */ oa << *(this->lat); ofs.close(); lat->print(); //check what lattice im serializing //!Deserialization - read file binlat.x and reconstruct printf("reconstructing serialized binlat.x...\n"); Lattice nl; std::ifstream ifs("binlat.x", ios::binary); boost::archive::binary_iarchive ia(ifs); //!deserialize object ia >> nl; ifs.close(); nl.print();
Hmmm- what surrounds this?
oa << *(this->lat);
doesn't look right to me. coincidentally, this same construct is used on another thread anyay try { const class Lattice lat ... ... ar << lat } that might work better. Robert Ramey
i get bad errors when deserializing, complaining about a bool and char. serializing to binlat.x... Lattice::print().....START Nodes=2 Edges= 1 Node[0] sp=1 ext=1 Linked to edge 2 (Node 3 or 0 ) Node[3] sp=1 ext=2 Linked to edge 2 (Node 3 or 0 ) Lattice::print().....END [enter] reconstructing swerialized binlat.x... [enter] plattice: /usr/local/include/boost/archive/basic_binary_iprimitive.hpp:98: void boost::archive::basic_binary_iprimitive<Archive, Elem, Tr>::load(bool&) [with Archive = boost::archive::binary_iarchive, Elem = char, Tr = std::char_traits<char>]: Assertion `0 == i || 1 == i' failed. [lenovo00:29384] *** Process received signal *** [lenovo00:29384] Signal: Aborted (6) [lenovo00:29384] Signal code: (-6) [lenovo00:29384] [ 0] [0xfea410] [lenovo00:29384] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0xd41a82] [lenovo00:29384] [ 2] /lib/tls/i686/cmov/libc.so.6(__assert_fail+0xf8) [0xd37718] [lenovo00:29384] [ 3] .. .. *** End of error message ***
i started debugging, and realized that the problem is completely gone when i remove "edges" from the serialization members of "Lattice" object. its weird, because "edge" is a map of <int, Edge> and much simplier than "nodes" which does work. obviously i cannot continue without edges, so i need to fix it somehow, but i dont understand the error related to edges, maybe is something im not seeing.
dont get distracted by the messy code, i had to edit it a little to focus on the case. regards Cristobal
On Thu, Aug 26, 2010 at 4:35 PM, Robert Ramey <ramey@rrsd.com> wrote:
Change the "would" into "is": it is possible to mix Boost.MPI with Boost.Serialization. Any objext that can be serialized can be sent by Boost.MPI. This was actually the basic idea behind Boost.MPI, to use Boost.Serialization to pack and unpack MPI buffers, or to create custom MPI datatypes.
On thing that I never understood is why you need MPI data types at all. If one is serializing (packing) to a binary array, and sending that, what hae MPI data types have to do with it.
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible. So I'm just not seeing this.
Robert Ramey
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Hmmm- what surrounds this?
-> there is more code part of an algorithm, but nothing important for the example embedded.
try
{ const class Lattice lat ... ... ar << lat }
that might work better.
Robert Ramey
thanks Robert, let me tell you about lat just in case i missed to point something lat was created this way Lattice *lat = new Lattice("filename"); i thought that pointers passed to archive wasnt possible, thats why i did "ao << *(this->lat);", and it actually worked when i removed edges. however i also tried what yo suggested, and it also works without "edges", but otherwise i get the same error. this is updated code if i use class pointers printf("serializing to binlat.x...\n"); //!file mode /* std::ofstream ofs("binlat.x", ios::binary); boost::archive::binary_oarchive oa(ofs); */ oa << lat; //lat was created earlier at some point of the program "Lattice *lat = new Lattice("filename")" ofs.close(); lat->print(); //check what lattice im serializing //!Deserialization - read file binlat.x and reconstruct printf("reconstructing serialized binlat.x...\n"); Lattice *nl; std::ifstream ifs("binlat.x", ios::binary); boost::archive::binary_iarchive ia(ifs); //!deserialize object ia >> nl; ifs.close(); nl->print(); i had to change the receiving lattice to a class pointer too. in the end, it also throws the same error when "edges" map is included on the serialization. i will keep trying different ideas to find the problem, in the meanwhile any help is welcome. by considering that when i remove "edges" from the serialization of Lattice object it works, i suspect the problem comes from "edges" map or maybe from the "Edge" class at a particular atribute.
i get bad errors when deserializing, complaining about a bool and char. serializing to binlat.x... Lattice::print().....START Nodes=2 Edges= 1 Node[0] sp=1 ext=1 Linked to edge 2 (Node 3 or 0 ) Node[3] sp=1 ext=2 Linked to edge 2 (Node 3 or 0 ) Lattice::print().....END [enter] reconstructing swerialized binlat.x... [enter] plattice: /usr/local/include/boost/archive/basic_binary_iprimitive.hpp:98: void boost::archive::basic_binary_iprimitive<Archive, Elem, Tr>::load(bool&) [with Archive = boost::archive::binary_iarchive, Elem = char, Tr = std::char_traits<char>]: Assertion `0 == i || 1 == i' failed. [lenovo00:29384] *** Process received signal *** [lenovo00:29384] Signal: Aborted (6) [lenovo00:29384] Signal code: (-6) [lenovo00:29384] [ 0] [0xfea410] [lenovo00:29384] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0xd41a82] [lenovo00:29384] [ 2] /lib/tls/i686/cmov/libc.so.6(__assert_fail+0xf8) [0xd37718] [lenovo00:29384] [ 3] .. .. *** End of error message ***
i started debugging, and realized that the problem is completely gone when i remove "edges" from the serialization members of "Lattice" object. its weird, because "edge" is a map of <int, Edge> and much simplier than "nodes" which does work. obviously i cannot continue without edges, so i need to fix it somehow, but i dont understand the error related to edges, maybe is something im not seeing.
dont get distracted by the messy code, i had to edit it a little to focus on the case. regards Cristobal
On Thu, Aug 26, 2010 at 4:35 PM, Robert Ramey <ramey@rrsd.com> wrote:
Change the "would" into "is": it is possible to mix Boost.MPI with Boost.Serialization. Any objext that can be serialized can be sent by Boost.MPI. This was actually the basic idea behind Boost.MPI, to use Boost.Serialization to pack and unpack MPI buffers, or to create custom MPI datatypes.
On thing that I never understood is why you need MPI data types at all. If one is serializing (packing) to a binary array, and sending that, what hae MPI data types have to do with it.
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible. So I'm just not seeing this.
Robert Ramey
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
ok now it worked!! i kept the object pointer method, started to enable/disable diferent serialization atributes of class "Edge" and i found something inexplicable. This is the original "Edge Class" when program crashed. #CLASS EDGE class Edge{ private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version){ ar & sp; ar & dead; ar & n1; ar & n2; ar & id; } public: int n1; int n2; int id; bool sp; bool dead; //plus some methods, constructors, etc }; now this is "Edge" Class with "bool dead" removed. #EDGE 2.0 not dead class Edge{ private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version){ ar & n1; ar & n2; ar & id; ar & sp; } public: int n1; int n2; int id; bool sp; }; i dont know why removing that bool "dead", program worked, what influence did that have on Boost? On Thu, Aug 26, 2010 at 6:59 PM, Cristobal Navarro <axischire@gmail.com>wrote:
Hmmm- what surrounds this?
-> there is more code part of an algorithm, but nothing important for the example embedded.
try
{ const class Lattice lat ... ... ar << lat }
that might work better.
Robert Ramey
thanks Robert, let me tell you about lat just in case i missed to point something lat was created this way Lattice *lat = new Lattice("filename"); i thought that pointers passed to archive wasnt possible, thats why i did "ao << *(this->lat);", and it actually worked when i removed edges. however i also tried what yo suggested, and it also works without "edges", but otherwise i get the same error. this is updated code if i use class pointers
printf("serializing to binlat.x...\n"); //!file mode /* std::ofstream ofs("binlat.x", ios::binary); boost::archive::binary_oarchive oa(ofs); */ oa << lat; //lat was created earlier at some point of the program "Lattice *lat = new Lattice("filename")" ofs.close(); lat->print(); //check what lattice im serializing //!Deserialization - read file binlat.x and reconstruct printf("reconstructing serialized binlat.x...\n"); Lattice *nl; std::ifstream ifs("binlat.x", ios::binary); boost::archive::binary_iarchive ia(ifs); //!deserialize object ia >> nl; ifs.close(); nl->print();
i had to change the receiving lattice to a class pointer too. in the end, it also throws the same error when "edges" map is included on the serialization. i will keep trying different ideas to find the problem, in the meanwhile any help is welcome. by considering that when i remove "edges" from the serialization of Lattice object it works, i suspect the problem comes from "edges" map or maybe from the "Edge" class at a particular atribute.
i get bad errors when deserializing, complaining about a bool and char. serializing to binlat.x... Lattice::print().....START Nodes=2 Edges= 1 Node[0] sp=1 ext=1 Linked to edge 2 (Node 3 or 0 ) Node[3] sp=1 ext=2 Linked to edge 2 (Node 3 or 0 ) Lattice::print().....END [enter] reconstructing swerialized binlat.x... [enter] plattice: /usr/local/include/boost/archive/basic_binary_iprimitive.hpp:98: void boost::archive::basic_binary_iprimitive<Archive, Elem, Tr>::load(bool&) [with Archive = boost::archive::binary_iarchive, Elem = char, Tr = std::char_traits<char>]: Assertion `0 == i || 1 == i' failed. [lenovo00:29384] *** Process received signal *** [lenovo00:29384] Signal: Aborted (6) [lenovo00:29384] Signal code: (-6) [lenovo00:29384] [ 0] [0xfea410] [lenovo00:29384] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0xd41a82] [lenovo00:29384] [ 2] /lib/tls/i686/cmov/libc.so.6(__assert_fail+0xf8) [0xd37718] [lenovo00:29384] [ 3] .. .. *** End of error message ***
i started debugging, and realized that the problem is completely gone when i remove "edges" from the serialization members of "Lattice" object. its weird, because "edge" is a map of <int, Edge> and much simplier than "nodes" which does work. obviously i cannot continue without edges, so i need to fix it somehow, but i dont understand the error related to edges, maybe is something im not seeing.
dont get distracted by the messy code, i had to edit it a little to focus on the case. regards Cristobal
On Thu, Aug 26, 2010 at 4:35 PM, Robert Ramey <ramey@rrsd.com> wrote:
Change the "would" into "is": it is possible to mix Boost.MPI with Boost.Serialization. Any objext that can be serialized can be sent by Boost.MPI. This was actually the basic idea behind Boost.MPI, to use Boost.Serialization to pack and unpack MPI buffers, or to create custom MPI datatypes.
On thing that I never understood is why you need MPI data types at all. If one is serializing (packing) to a binary array, and sending that, what hae MPI data types have to do with it.
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible. So I'm just not seeing this.
Robert Ramey
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 26 Aug 2010, at 22:35, Robert Ramey wrote:
Change the "would" into "is": it is possible to mix Boost.MPI with Boost.Serialization. Any objext that can be serialized can be sent by Boost.MPI. This was actually the basic idea behind Boost.MPI, to use Boost.Serialization to pack and unpack MPI buffers, or to create custom MPI datatypes. On thing that I never understood is why you need MPI data types at all. If one is serializing (packing) to a binary array, and sending that, what hae MPI data types have to do with it.
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible.
No, it does not presume this.
So I'm just not seeing this.
BoostPro Computing, http://boostpro.com Sent from coveted but awkward mobile device -- On Aug 26, 2010, at 12:35 PM, "Robert Ramey" <ramey@rrsd.com> wrote:
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible.
You're mistaken.
So I'm just not seeing this.
Start reading here: http://www.boost.org/doc/libs/1_39_0/doc/html/mpi/tutorial.html#mpi.skeleton... and all will be revealed
Dave Abrahams wrote:
BoostPro Computing, http://boostpro.com Sent from coveted but awkward mobile device
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible.
You're mistaken.
So I'm just not seeing this.
Start reading here: http://www.boost.org/doc/libs/1_39_0/doc/html/mpi/tutorial.html#mpi.skeleton... and all will be revealed
lol - I've read that several times. I just never found it to be very revealing. The word skeleton seemed pretty suggestive. It's still not clear to me how such a think can work between heterogeneous machines. For example, if I have an array of 2 byte integers and they each need to get transformed one by one into a 4 byte integer because that's closest MPI data type, I don't see how the fact that the "shape" doesn't change helps you. So when I read the documentation I didn't find an obvious answer to this question. Admitadly my attention span is pretty short when I'm looking at something just because I'm curious rather than really needing a solution to my problem. So maybe its OK if one has more time to spend on it. Robert Ramey
hello Robert, now that serialization is working, i would like to try what you suggested on the first messages, mixing plain MPI with serialization is not that bad, if its possible could you guide me a little more hints on the streambuf structure. at the moment what i can do is save objects into binary_oarchive and read them with a binary_iarchive. what i dont know is where to put the binary_oarchive so it gets transfered through MPI send as bytes. that would be the only step remaining right? best regards Cristobal On Thu, Aug 26, 2010 at 8:05 PM, Robert Ramey <ramey@rrsd.com> wrote:
Dave Abrahams wrote:
BoostPro Computing, http://boostpro.com Sent from coveted but awkward mobile device
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible.
You're mistaken.
So I'm just not seeing this.
Start reading here:
http://www.boost.org/doc/libs/1_39_0/doc/html/mpi/tutorial.html#mpi.skeleton...
and all will be revealed
lol - I've read that several times. I just never found it to be very revealing. The word skeleton seemed pretty suggestive. It's still not clear to me how such a think can work between heterogeneous machines. For example, if I have an array of 2 byte integers and they each need to get transformed one by one into a 4 byte integer because that's closest MPI data type, I don't see how the fact that the "shape" doesn't change helps you. So when I read the documentation I didn't find an obvious answer to this question. Admitadly my attention span is pretty short when I'm looking at something just because I'm curious rather than really needing a solution to my problem. So maybe its OK if one has more time to spend on it.
Robert Ramey
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Cristobal Navarro wrote:
hello Robert,
now that serialization is working, i would like to try what you suggested on the first messages, mixing plain MPI with serialization is not that bad,
if its possible could you guide me a little more hints on the streambuf structure. at the moment what i can do is save objects into binary_oarchive and read them with a binary_iarchive.
what i dont know is where to put the binary_oarchive so it gets transfered through MPI send as bytes. that would be the only step remaining right? best regards Cristobal
You should first investigate Matthias suggestion. This will be equivalent or better than my suggestion and it's already done and ready go to. Robert Ramey
On Aug 27, 2010, at 8:25 AM, Robert Ramey wrote:
Cristobal Navarro wrote:
hello Robert,
now that serialization is working, i would like to try what you suggested on the first messages, mixing plain MPI with serialization is not that bad,
if its possible could you guide me a little more hints on the streambuf structure. at the moment what i can do is save objects into binary_oarchive and read them with a binary_iarchive.
what i dont know is where to put the binary_oarchive so it gets transfered through MPI send as bytes. that would be the only step remaining right? best regards Cristobal You should first investigate Matthias suggestion. This will be equivalent or better than my suggestion and it's already done and ready go to.
Use a packed_iarchive and a packed_oarchive and then look at the functions in point_to_point.cpp for instructions on how to send the buffer. Or, even ssimpler, just use send and recv of Boost.MPI since that does exactly what you want to do. Matthias
Hi, I am trying to build boost 1.42.0 on MAC but I am having some problem. I am using the following command line ./bjam --build-dir=/tmp/build-boost toolset=gcc stage At the end of build I get most of the ".a" version of lib but none of ".dylib"(dynamic library). I see multiple of the following error for different library, "...failed gcc.link.dll stage/lib/libboost_math_tr1.dylib...". I usually work on window and new to MAC, so could some one point me what's I am doing wrong? System Configuration: MAC OS X, Version 10.6.4. X Code version :3.2.3 Here is the partial build log, remove some of the lines to make sure more than one linking error is included. ******************************************************************* warning: Building Boost.Regex with the optional Unicode/ICU support disabled. note: Please refer to the Boost.Regex documentation for more information note: this is a strictly optional feature. warning: Graph library does not contain MPI-based parallel components. note: to enable them, add "using mpi ;" to your user-config.jam warning: skipping optional Message Passing Interface (MPI) library. note: to enable MPI support, add "using mpi ;" to user-config.jam. note: to suppress this message, pass "--without-mpi" to bjam. note: otherwise, you can safely ignore this message. ...patience... ...patience... ...patience... ...found 6641 targets... ...updating 720 targets... common.mkdir stage common.mkdir stage/lib gcc.compile.c++.pch /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/../src/tr1/pch.hpp.gch gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/assoc_laguerre.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/assoc_legendre.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_neumann.o gcc.link.dll stage/lib/libboost_math_tr1.dylib ld: unknown option: -R collect2: ld returned 1 exit status "g++" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config" -o "stage/lib/libboost_math_tr1.dylib" -Wl,-h -Wl,libboost_math_tr1.dylib -shared -Wl,--start-group "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/assoc_laguerre.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/assoc_legendre.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/beta.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/comp_ellint_1.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/comp_ellint_2.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/comp_ellint_3.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_bessel_i.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_bessel_j.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_bessel_k.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_neumann.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/ellint_1.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/ellint_2.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/ellint_3.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/expint.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/hermite.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/laguerre.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/legendre.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/riemann_zeta.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_bessel.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_legendre.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_neumann.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -Wl,--strip-all ...failed gcc.link.dll stage/lib/libboost_math_tr1.dylib... ggcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_legendref.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_neumannf.o gcc.link.dll stage/lib/libboost_math_tr1f.dylib ld: unknown option: -R collect2: ld returned 1 exit status "g++" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config" -o "stage/lib/libboost_math_tr1f.dylib" -Wl,-h -Wl,libboost_math_tr1f.dylib -shared -Wl,--start-group "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/assoc_laguerref.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/assoc_legendref.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/betaf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/comp_ellint_1f.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/comp_ellint_2f.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/comp_ellint_3f.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_bessel_if.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_bessel_jf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_bessel_kf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_neumannf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/ellint_1f.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/ellint_2f.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/ellint_3f.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/expintf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/hermitef.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/laguerref.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/legendref.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/riemann_zetaf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_besself.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_legendref.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_neumannf.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -Wl,--strip-all ...failed gcc.link.dll stage/lib/libboost_math_tr1f.dylib... gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/assoc_laguerrel.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_legendrel.o ld: unknown option: -R collect2: ld returned 1 exit status "g++" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config" -o "stage/lib/libboost_math_tr1l.dylib" -Wl,-h -Wl,libboost_math_tr1l.dylib -shared -Wl,--start-group "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/assoc_laguerrel.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/assoc_legendrel.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/betal.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/comp_ellint_1l.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/comp_ellint_2l.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/comp_ellint_3l.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_bessel_il.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_bessel_jl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_bessel_kl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cyl_neumannl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/ellint_1l.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/ellint_2l.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/ellint_3l.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/expintl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/hermitel.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/laguerrel.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/legendrel.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/riemann_zetal.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_bessell.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_legendrel.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/sph_neumannl.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -Wl,--strip-all ...failed gcc.link.dll stage/lib/libboost_math_tr1l.dylib... gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/trunc.o gcc.link.dll stage/lib/libboost_math_c99.dylib ld: unknown option: -R collect2: ld returned 1 exit status "g++" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config" -o "stage/lib/libboost_math_c99.dylib" -Wl,-h -Wl,libboost_math_c99.dylib -shared -Wl,--start-group "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/acosh.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/asinh.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/atanh.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cbrt.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/copysign.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/erfc.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/erf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/expm1.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/fmax.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/fmin.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/fpclassify.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/hypot.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/lgamma.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/llround.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/log1p.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/lround.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/nextafter.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/nexttoward.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/round.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/tgamma.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/trunc.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -Wl,--strip-all ...failed gcc.link.dll stage/lib/libboost_math_c99.dylib... gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/truncf.o gcc.link.dll stage/lib/libboost_math_c99f.dylib ld: unknown option: -R collect2: ld returned 1 exit status "g++" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config" -o "stage/lib/libboost_math_c99f.dylib" -Wl,-h -Wl,libboost_math_c99f.dylib -shared -Wl,--start-group "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/acoshf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/asinhf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/atanhf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cbrtf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/copysignf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/erfcf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/erff.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/expm1f.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/fmaxf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/fminf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/fpclassifyf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/hypotf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/lgammaf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/llroundf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/log1pf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/lroundf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/nextafterf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/nexttowardf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/roundf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/tgammaf.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/truncf.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -Wl,--strip-all ...failed gcc.link.dll stage/lib/libboost_math_c99f.dylib... ggcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/tgammal.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/truncl.o gcc.link.dll stage/lib/libboost_math_c99l.dylib ld: unknown option: -R collect2: ld returned 1 exit status "g++" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config" -o "stage/lib/libboost_math_c99l.dylib" -Wl,-h -Wl,libboost_math_c99l.dylib -shared -Wl,--start-group "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/acoshl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/asinhl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/atanhl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/cbrtl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/copysignl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/erfcl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/erfl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/expm1l.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/fmaxl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/fminl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/fpclassifyl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/hypotl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/lgammal.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/llroundl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/log1pl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/lroundl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/nextafterl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/nexttowardl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/roundl.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/tgammal.o" "/tmp/build-boost/boost/bin.v2/libs/math/build/gcc-4.2.1/release/threading-multi/truncl.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -Wl,--strip-all ...failed gcc.link.dll stage/lib/libboost_math_c99l.dylib... gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/split.o gcc.link.dll stage/lib/libboost_program_options.dylib ld: unknown option: -R collect2: ld returned 1 exit status "g++" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config" -o "stage/lib/libboost_program_options.dylib" -Wl,-h -Wl,libboost_program_options.dylib -shared -Wl,--start-group "/tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/cmdline.o" "/tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/config_file.o" "/tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/options_description.o" "/tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/parsers.o" "/tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/variables_map.o" "/tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/value_semantic.o" "/tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/positional_options.o" "/tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/utf8_codecvt_facet.o" "/tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/convert.o" "/tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/winmain.o" "/tmp/build-boost/boost/bin.v2/libs/program_options/build/gcc-4.2.1/release/threading-multi/split.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -Wl,--strip-all ...failed gcc.link.dll stage/lib/libboost_program_options.dylib... gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/slice.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/converter/from_python.o libs/python/src/converter/from_python.cpp: In function boost::python::converter::rvalue_from_python_stage1_data boost::python::converter::rvalue_from_python_stage1(PyObject*, const boost::python::converter::registration&): libs/python/src/converter/from_python.cpp:42: warning: data.boost::python::converter::rvalue_from_python_stage1_data::construct may be used uninitialized in this function libs/python/src/converter/from_python.cpp: In function void* boost::python::converter::rvalue_result_from_python(PyObject*, boost::python::converter::rvalue_from_python_stage1_data&): libs/python/src/converter/from_python.cpp:42: warning: data.boost::python::converter::rvalue_from_python_stage1_data::construct may be used uninitialized in this function libs/python/src/converter/from_python.cpp:42: note: data.boost::python::converter::rvalue_from_python_stage1_data::construct was declared here gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/converter/registry.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/converter/type_id.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/enum.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/class.o libs/python/src/object/class.cpp: In function int boost::python::property_init(PyObject*, PyObject*, PyObject*): libs/python/src/object/class.cpp:79: warning: deprecated conversion from string constant to char* libs/python/src/object/class.cpp:79: warning: deprecated conversion from string constant to char* libs/python/src/object/class.cpp:79: warning: deprecated conversion from string constant to char* libs/python/src/object/class.cpp:79: warning: deprecated conversion from string constant to char* gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/function.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/converter/builtin_converters.o libs/python/src/converter/builtin_converters.cpp:369: warning: boost::python::converter::<unnamed>::py_unicode_as_string_unaryfunc defined but not used gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/import.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/exec.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/function_doc_signature.o gcc.link.dll /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/libboost_python.dylib ld: unknown option: -R collect2: ld returned 1 exit status "g++" -L"/System/Library/Frameworks/Python.framework/Versions/2.6/lib" -L"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config" -o "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/libboost_python.dylib" -Wl,-h -Wl,libboost_python.dylib -shared -Wl,--start-group "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/numeric.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/list.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/long.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/dict.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/tuple.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/str.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/slice.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/converter/from_python.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/converter/registry.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/converter/type_id.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/enum.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/class.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/function.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/inheritance.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/life_support.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/pickle_support.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/errors.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/module.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/converter/builtin_converters.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/converter/arg_to_python_base.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/iterator.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/stl_iterator.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object_protocol.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object_operators.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/wrapper.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/import.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/exec.o" "/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/object/function_doc_signature.o" -Wl,-Bstatic -Wl,-Bdynamic -lpython2.6 -Wl,--end-group -Wl,--strip-all ...failed gcc.link.dll /tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi/libboost_python.dylib... ...skipped <pstage/lib>libboost_python.dylib for lack of <p/tmp/build-boost/boost/bin.v2/libs/python/build/gcc-4.2.1/release/threading-multi>libboost_python.dylib... gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_archive.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_iarchive.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/archive_exception.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_grammar.o In file included from ./boost/archive/impl/basic_xml_grammar.hpp:59, from libs/serialization/src/xml_grammar.cpp:16: ./boost/spirit/core/non_terminal/rule.hpp:18:4: warning: #warning "This header is deprecated. Please use: boost/spirit/include/classic_rule.hpp" In file included from libs/serialization/src/basic_xml_grammar.ipp:25, from libs/serialization/src/xml_grammar.cpp:63: ./boost/spirit/core/composite/operators.hpp:18:4: warning: #warning "This header is deprecated. Please use: boost/spirit/include/classic_operators.hpp" In file included from libs/serialization/src/basic_xml_grammar.ipp:26, from libs/serialization/src/xml_grammar.cpp:63: ./boost/spirit/core/composite/actions.hpp:18:4: warning: #warning "This header is deprecated. Please use: boost/spirit/include/classic_actions.hpp" In file included from libs/serialization/src/basic_xml_grammar.ipp:27, from libs/serialization/src/xml_grammar.cpp:63: ./boost/spirit/core/primitives/numerics.hpp:18:4: warning: #warning "This header is deprecated. Please use: boost/spirit/include/classic_numerics.hpp" gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_iarchive.o In file included from ./boost/archive/impl/basic_xml_grammar.hpp:59, from ./boost/archive/impl/xml_iarchive_impl.ipp:41, from libs/serialization/src/xml_iarchive.cpp:32: ./boost/spirit/core/non_terminal/rule.hpp:18:4: warning: #warning "This header is deprecated. Please use: boost/spirit/include/classic_rule.hpp" gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_oarchive.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_archive_exception.o ...on 200th target... gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/shared_ptr_helper.o gcc.link.dll stage/lib/libboost_serialization.dylib ld: unknown option: -R collect2: ld returned 1 exit status "g++" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config" -o "stage/lib/libboost_serialization.dylib" -Wl,-h -Wl,libboost_serialization.dylib -shared -Wl,--start-group "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_archive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_iarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_iserializer.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_oarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_oserializer.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_pointer_iserializer.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_pointer_oserializer.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_serializer_map.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_text_iprimitive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_text_oprimitive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_xml_archive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/binary_iarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/binary_oarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/extended_type_info.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/extended_type_info_typeid.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/extended_type_info_no_rtti.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/polymorphic_iarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/polymorphic_oarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/stl_port.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/text_iarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/text_oarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/void_cast.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/archive_exception.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_grammar.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_iarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_oarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_archive_exception.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/shared_ptr_helper.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -Wl,--strip-all ...failed gcc.link.dll stage/lib/libboost_serialization.dylib... gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_text_wiprimitive.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_text_woprimitive.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/text_wiarchive.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/text_woarchive.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_wgrammar.o In file included from ./boost/archive/impl/basic_xml_grammar.hpp:59, from libs/serialization/src/xml_wgrammar.cpp:18: ./boost/spirit/core/non_terminal/rule.hpp:18:4: warning: #warning "This header is deprecated. Please use: boost/spirit/include/classic_rule.hpp" In file included from libs/serialization/src/basic_xml_grammar.ipp:25, from libs/serialization/src/xml_wgrammar.cpp:146: ./boost/spirit/core/composite/operators.hpp:18:4: warning: #warning "This header is deprecated. Please use: boost/spirit/include/classic_operators.hpp" In file included from libs/serialization/src/basic_xml_grammar.ipp:26, from libs/serialization/src/xml_wgrammar.cpp:146: ./boost/spirit/core/composite/actions.hpp:18:4: warning: #warning "This header is deprecated. Please use: boost/spirit/include/classic_actions.hpp" In file included from libs/serialization/src/basic_xml_grammar.ipp:27, from libs/serialization/src/xml_wgrammar.cpp:146: ./boost/spirit/core/primitives/numerics.hpp:18:4: warning: #warning "This header is deprecated. Please use: boost/spirit/include/classic_numerics.hpp" gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_wiarchive.o In file included from ./boost/archive/impl/basic_xml_grammar.hpp:59, from ./boost/archive/impl/xml_wiarchive_impl.ipp:45, from libs/serialization/src/xml_wiarchive.cpp:37: ./boost/spirit/core/non_terminal/rule.hpp:18:4: warning: #warning "This header is deprecated. Please use: boost/spirit/include/classic_rule.hpp" gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_woarchive.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/codecvt_null.o gcc.link.dll /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/libboost_serialization.dylib ld: unknown option: -h collect2: ld returned 1 exit status "g++" -o "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/libboost_serialization.dylib" -Wl,-h -Wl,libboost_serialization.dylib -shared -Wl,--start-group "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_archive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_iarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_iserializer.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_oarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_oserializer.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_pointer_iserializer.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_pointer_oserializer.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_serializer_map.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_text_iprimitive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_text_oprimitive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/basic_xml_archive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/binary_iarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/binary_oarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/extended_type_info.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/extended_type_info_typeid.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/extended_type_info_no_rtti.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/polymorphic_iarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/polymorphic_oarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/stl_port.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/text_iarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/text_oarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/void_cast.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/archive_exception.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_grammar.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_iarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_oarchive.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/xml_archive_exception.o" "/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/shared_ptr_helper.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -Wl,--strip-all ...failed gcc.link.dll /tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi/libboost_serialization.dylib... ...skipped <pstage/lib>libboost_wserialization.dylib for lack of <p/tmp/build-boost/boost/bin.v2/libs/serialization/build/gcc-4.2.1/release/threading-multi>libboost_serialization.dylib... gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/signals/build/gcc-4.2.1/release/threading-multi/signal_base.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/signals/build/gcc-4.2.1/release/threading-multi/slot.o gcc.link.dll stage/lib/libboost_signals.dylib ld: unknown option: -R collect2: ld returned 1 exit status "g++" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib" -Wl,-R -Wl,"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config" -o "stage/lib/libboost_signals.dylib" -Wl,-h -Wl,libboost_signals.dylib -shared -Wl,--start-group "/tmp/build-boost/boost/bin.v2/libs/signals/build/gcc-4.2.1/release/threading-multi/trackable.o" "/tmp/build-boost/boost/bin.v2/libs/signals/build/gcc-4.2.1/release/threading-multi/connection.o" "/tmp/build-boost/boost/bin.v2/libs/signals/build/gcc-4.2.1/release/threading-multi/named_slot_map.o" "/tmp/build-boost/boost/bin.v2/libs/signals/build/gcc-4.2.1/release/threading-multi/signal_base.o" "/tmp/build-boost/boost/bin.v2/libs/signals/build/gcc-4.2.1/release/threading-multi/slot.o" -Wl,-Bstatic -Wl,-Bdynamic -Wl,--end-group -Wl,--strip-all ...failed gcc.link.dll stage/lib/libboost_signals.dylib... gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/test/build/gcc-4.2.1/release/threading-multi/execution_monitor.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/test/build/gcc-4.2.1/release/threading-multi/debug.o gcc.compile.c++ /tmp/build-boost/boost/bin.v2/libs/test/build/gcc-4.2.1/release/threading-multi/cpp_main.o gcc.link.dll stage/lib/libboost_prg_exec_monitor.dylib ld: unknown option: -R collect2: ld returned 1 exit status Akhilesh Kumar (408) 352-4673
At Fri, 27 Aug 2010 14:55:28 -0700, Kumar, Akhilesh wrote:
Hi,
I am trying to build boost 1.42.0 on MAC but I am having some problem. I am using the following command line ./bjam --build-dir=/tmp/build-boost toolset=gcc stage
At the end of build I get most of the ".a" version of lib but none of ".dylib"(dynamic library). I see multiple of the following error for different library, "...failed gcc.link.dll stage/lib/libboost_math_tr1.dylib...".
I usually work on window and new to MAC, so could some one point me what's I am doing wrong?
If you *promise* not to shoot the messenger... I'll tell you that I think you need "toolset=darwin" rather than "toolset=gcc" -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On Aug 27, 2010, at 2:05 AM, Robert Ramey wrote:
Dave Abrahams wrote:
BoostPro Computing, http://boostpro.com Sent from coveted but awkward mobile device
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible.
You're mistaken.
So I'm just not seeing this.
Start reading here: http://www.boost.org/doc/libs/1_39_0/doc/html/mpi/tutorial.html#mpi.skeleton... and all will be revealed
lol - I've read that several times. I just never found it to be very revealing. The word skeleton seemed pretty suggestive. It's still not clear to me how such a think can work between heterogeneous machines. For example, if I have an array of 2 byte integers and they each need to get transformed one by one into a 4 byte integer because that's closest MPI data type, I don't see how the fact that the "shape" doesn't change helps you.
MPI will take care of that for you.
So when I read the documentation I didn't find an obvious answer to this question. Admitadly my attention span is pretty short when I'm looking at something just because I'm curious rather than really needing a solution to my problem. So maybe its OK if one has more time to spend on it.
Robert Ramey
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
At Thu, 26 Aug 2010 16:05:41 -0800, Robert Ramey wrote:
Dave Abrahams wrote:
BoostPro Computing, http://boostpro.com Sent from coveted but awkward mobile device
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible.
You're mistaken.
So I'm just not seeing this.
Start reading here: http://www.boost.org/doc/libs/1_39_0/doc/html/mpi/tutorial.html#mpi.skeleton... and all will be revealed
lol - I've read that several times.
I always wonder, when you write that, whether you're physically laughing out loud. That's OK; don't spoil the mystique ;-)
I just never found it to be very revealing. The word skeleton seemed pretty suggestive. It's still not clear to me how such a think can work between heterogeneous machines. For example, if I have an array of 2 byte integers and they each need to get transformed one by one into a 4 byte integer because that's closest MPI data type,
I think you don't understand what MPI datatypes do. They are not like an API that you have to conform to.
I don't see how the fact that the "shape" doesn't change helps you.
Well, let me try to explain all this; I think it is important stuff and not generally as well-understood as it should be. And since Boost.MPI and Boost.Serialization are so closely related, I think it's especially important that *you* underestand. I hope Matthias will correct me if I get anything wrong. The first thing to understand is at the hardware level. Network cards have a fixed-size buffer. Sending anything over the network involves getting it into the buffer. If the buffer fills up, packets are waiting to go out, and you can't put anything further in the buffer until that happens. One mission of MPI (the non-boost variety) is to provide a portable high-level API for sending out these messages. Therefore, MPI deals with the low-level stuff and has/needs direct access to the network buffer *but code written on top of MPI does not* <== note emphasis. <<<=== No, really, note it! Now let's assume a heterogeneous environment. In that case, you can't just “blit the bytes;” somebody needs to figure out how to encode data for transmission and decode it upon receipt so that it has the same meaninig on both ends. In particular, bytes may need to be permuted to account for endian-ness. Let's further assume a system that transmits in little-endian, so before sending from a big-endian machine one needs to swap bytes (all the other possible schemes have the same consequences---by choosing one we can work with specific examples). The code doing the permutation has to know about the data structure, rather than operating on the data as raw bytes only. For example, if the data structure is a sequence of 32-bit integers, you need to reverse each group of 4 consecutive bytes in-place. If it's a sequence of 16-bit integers, you need to reverse pairs of bytes in-place, and if it's a sequence of chars, you don't need to do anything. Now suppose MPI only knew about byte sequences --- sort of like files only know about byte sequences. What would we do? We'd use something like Boost.Serialization to handle the translation. We'd serialize our source data into a portable representation that could be passed to MPI, which would then copy the bytes into the network buffer as needed. That's two copies of every byte. One copy to serialize, and another copy into the network buffer. However, we have MPI datatypes and type maps, which describe the structure of data to be transmitted. Take a quick look at http://ww2.cs.mu.oz.au/498/notes/node20.html, and note that an MPI type map is a sequence of (type, byteoffset) pairs. If there's padding in your data structure, the type map captures that fact so padding bytes aren't sent. So we can tell MPI about the data structure and let MPI put the serialized representation *directly* into the network buffer. That's one copy of every byte. Of course, making this efficient depends on sending the same data structure multiple times. MPI type maps are on the order of the same size as the data structure they describe, so creating one might be roughly the same cost as a copy. So using the same "shape" over and over again is important. Unfortunately, an MPI type map is a pain to create, and is even more painful to maintain as data structures change. So, typically, type maps get created for a very few structs (essentially) and more complex structures are typically sent with a series of MPI calls (some of which use type maps). The genius of Boost.MPI is in three realizations: 1. You can represent all the values in an arbitrarily complex non-contiguous data structure with a single type map. It's like a struct that extends from the data's minimum address to its maximum, probably with *lots* of padding. 2. When you serialize a complex data structure, the archive sees the type and address of every datum. 3. You can treat addresses as byte offsets (from address zero). So Boost.MPI has an Archive type one that creates an MPI type map by treating addresses as offsets and translating fundamental C++ types into MPI datatypes. This involves no actual data copying. Then it uses the type map to send the "giant struct beginning at address zero." This avoids an expensive intermediate serialization phase before MPI actually gets its paws on your data. HTH, and I hope I got all that right :-) -- Dave Abrahams BoostPro Computing http://www.boostpro.com
David Abrahams wrote:
At Thu, 26 Aug 2010 16:05:41 -0800, Robert Ramey wrote:
Dave Abrahams wrote:
BoostPro Computing, http://boostpro.com Sent from coveted but awkward mobile device
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible.
You're mistaken.
So I'm just not seeing this.
Start reading here: http://www.boost.org/doc/libs/1_39_0/doc/html/mpi/tutorial.html#mpi.skeleton... and all will be revealed
lol - I've read that several times.
I always wonder, when you write that, whether you're physically laughing out loud. That's OK; don't spoil the mystique ;-)
I actually do think I snicker a little bit. Sort of one "snick". This above is exactly the type of thing which provokes this reaction. Reading your comment, one has to conclude that a) you think I haven't read it but am expressing an opinion anyway b) that you think that the documentation is clear and complete c) that I'm somehow responsable for not finding/reading/understanding it. You make this kind of presumption all the time. It doesn't bother me but it always provokes at least one "snick"
I just never found it to be very revealing. The word skeleton seemed pretty suggestive. It's still not clear to me how such a think can work between heterogeneous machines. For example, if I have an array of 2 byte integers and they each need to get transformed one by one into a 4 byte integer because that's closest MPI data type,
I think you don't understand what MPI datatypes do.
This is true. I suppose that's one reason why the documentation made no sense to me. They just looked like special types to make identify primitives accross differing architectures. <snip> Interesting information which one should consider adding to the MPI documentation. </snip>
And since Boost.MPI and Boost.Serialization are so closely related, I think it's especially important that *you* underestand.
I disagree. Boost MPI depends upon Boost.Serialization but not the other way around. I shouldn't have to understand Boost MPI just as I can't be expected to understand all the usages to which boost serialization might be applied. Trying to do this, aside from the time involved, might well be counter productive in that it can trick one into coupling things which would otherwise not be. For all these reasons I've refrained from investing a lot of time in understanding MPI as it relates to serialization. I'm happy with Mathias efforts and commitment to supporting his library and don't want to muck up the works. I only really have few observations/suggestions at this point. a) It would be helpful if there were a way to test the serialization of the archive classes in boost MPI without having MPI installed. If this is not possible, it would seem to me that the serialization is intertwined with the data transport - rather than be separated as it is in the stream/streambuf io/stream design. This would look like a design flaw to me. b) user experience seems to show that archive construction/destruction is a significant performance issue when a new archive is made for each data transmission. On the otherhand, one has to do this since the current archive implemenation track addresses of serialized obects so the same archive can't e use send the same structure (maybe with changed data) multiple times. Given that MPI has a focus on performance, I wonder if this has been considered. I looked a the documentation, code and examples and it wasn't obvious to me how this is question was addressed - if at all. c) I think the above information regarding how MPI and serialization fit together in boost MPI would be a worth addition go the MPI documentation. AND it's already written ! You should know that all your efforts to educate me are not wasted. a) Sometimes I pass your advice along to other people. b) Motivated by our recent discussions on the subject as well as some other issues has clarified my ideas on how the archive concept should be refined to be more helpful to those creating their own archives. I think there will be improvements in this area in part due to your observations. Robert Ramey
At Fri, 27 Aug 2010 09:44:39 -0800, Robert Ramey wrote:
David Abrahams wrote:
At Thu, 26 Aug 2010 16:05:41 -0800, Robert Ramey wrote:
Dave Abrahams wrote:
BoostPro Computing, http://boostpro.com Sent from coveted but awkward mobile device
If one were using heterogenious machines, I could understand the usage of MPI types. But as I understand it, the MPI serialization presumes that the machines are binary compatible.
You're mistaken.
So I'm just not seeing this.
Start reading here: http://www.boost.org/doc/libs/1_39_0/doc/html/mpi/tutorial.html#mpi.skeleton... and all will be revealed
lol - I've read that several times.
I always wonder, when you write that, whether you're physically laughing out loud. That's OK; don't spoil the mystique ;-)
I actually do think I snicker a little bit. Sort of one "snick".
This above is exactly the type of thing which provokes this reaction. Reading your comment, one has to conclude that
a) you think I haven't read it but am expressing an opinion anyway b) that you think that the documentation is clear and complete c) that I'm somehow responsable for not finding/reading/understanding it.
You make this kind of presumption all the time.
It's a weakness of mine, I admit, for which I apologize. But I wasn't doing that in this case. I was a bit terse, because as I noted above the message was sent from an awkward mobile device. Actually, a) I did think you hadn't read it, but I was trying to be helpful by pointing at the part of the doc that relates to your question. I thought you were saying "I don't get it; please help me understand," not that you were "expressing an opinion." b) I haven't read all the documentation. I did presume it was complete, but not that it was clear, hence http://groups.google.com/group/boost-list/msg/d14a38f3c55613b6 c) I don't see how anything I wrote could lead to that conclusion.
It doesn't bother me but it always provokes at least one "snick"
I just never found it to be very revealing. The word skeleton seemed pretty suggestive. It's still not clear to me how such a think can work between heterogeneous machines. For example, if I have an array of 2 byte integers and they each need to get transformed one by one into a 4 byte integer because that's closest MPI data type,
I think you don't understand what MPI datatypes do.
This is true. I suppose that's one reason why the documentation made no sense to me. They just looked like special types to make identify primitives accross differing architectures.
<snip> Interesting information which one should consider adding to the MPI documentation. </snip>
Actually, I wish you had given me some detailed feedback. I was thinking of writing it up in such a way that the Boost.MPI docs could point at it, but I can't tell whether it connected for you or not.
And since Boost.MPI and Boost.Serialization are so closely related, I think it's especially important that *you* underestand.
I disagree. Boost MPI depends upon Boost.Serialization but not the other way around.
I didn't say they were interdependent, just that they were closely-related. But anyway, I think you're being shortsighted: The success of tools built on top of Boost.Serialization is an important indicator of the correctness and genericity of its design. Boost.Serialization *is* dependent on Boost.MPI for a portion of its userbase, and to the extent that you are interested in supporting that portion of the userbase and enabling anything like Boost.MPI to exist, the things can/should do to Boost.Serialization depend on the requirements of Boost.MPI.
I shouldn't have to understand Boost MPI just as I can't be expected to understand all the usages to which boost serialization might be applied.
Welcome to generic library design! :-) You have to decide what your application domain(s) is/are. Do you want to serve people who are trying to save/load their desktop applications' files? Do you want to serve people who want to save/load XML? Do you want to serve people who want to checkpoint the state of long-running calculations? Do you want to serve people who are trying to implement RPC? etc...
Trying to do this, aside from the time involved, might well be counter productive in that it can trick one into coupling things which would otherwise not be. For all these reasons I've refrained from investing a lot of time in understanding MPI as it relates to serialization. I'm happy with Mathias efforts and commitment to supporting his library and don't want to muck up the works.
You can't be a generic serialization library without reference to real applications. Maybe you still haven't really decided whether serving Boost.MPI and its users is something you want to do. I think that might account for a good deal of the recurring friction we experience. It would be a good idea to settle on an answer to that question, along with the question of what other applications you're willing to support.
I only really have few observations/suggestions at this point.
a) It would be helpful if there were a way to test the serialization of the archive classes in boost MPI without having MPI installed. If this is not possible, it would seem to me that the serialization is intertwined with the data transport - rather than be separated as it is in the stream/streambuf io/stream design. This would look like a design flaw to me.
Helpful to whom? I agree this is all a good idea, but I'm not sure how this relates to anything else we're discussing.
b) user experience seems to show that archive construction/destruction is a significant performance issue when a new archive is made for each data transmission. On the other hand, one has to do this since the current archive implemenation track addresses of serialized obects so the same archive can't be use send the same structure (maybe with changed data) multiple times.
I can't understand how you reach that conclusion. In my long explanation I thought I made it clear that Boost.MPI and its database get a great deal of its performance advantage from exactly that: sending the same structure multiple times with the same MPI type map.
Given that MPI has a focus on performance, I wonder if this has been considered. I looked a the documentation, code and examples and it wasn't obvious to me how this is question was addressed - if at all.
We've obviously misunderstood one another somewhere along the way. It would be good if we could get that cleared up.
c) I think the above information regarding how MPI and serialization fit together in boost MPI would be a worth addition go the MPI documentation. AND it's already written !
It has been my intention to get it out in a more public place than this mailing list. Again, detailed feedback would be helpful. It can't already be perfect, or the misunderstanding cited in b) wouldn't have arisen.
You should know that all your efforts to educate me are not wasted.
Glad to hear it. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
David Abrahams wrote:
And since Boost.MPI and Boost.Serialization are so closely related, I think it's especially important that *you* underestand.
I disagree. Boost MPI depends upon Boost.Serialization but not the other way around.
I didn't say they were interdependent, just that they were closely-related. But anyway, I think you're being shortsighted: The success of tools built on top of Boost.Serialization is an important indicator of the correctness and genericity of its design.
Boost.Serialization *is* dependent on Boost.MPI for a portion of its userbase,
<off topic> what portion of the user base? (This is a rhetorical question please don't spend any time answering it.) It's always bothered me that we have no per library data/feedback on the interest/usage on a library by library basis. I've spoken about this before and I know you agree with me about it. But until this get's addressed I think we just have to set these questions aside and recognize that the answers are anyone's guess. <snip>
You can't be a generic serialization library without reference to real applications. Maybe you still haven't really decided whether serving Boost.MPI and its users is something you want to do.
I think that
might account for a good deal of the recurring friction we experience. It would be a good idea to settle on an answer to that question, along with the question of what other applications you're willing to support.
I am acutely aware of this aspect of "generic library design". Every post on this list reminds me of it.
I only really have few observations/suggestions at this point.
a) It would be helpful if there were a way to test the serialization of the archive classes in boost MPI without having MPI installed. If this is not possible, it would seem to me that the serialization is intertwined with the data transport - rather than be separated as it is in the stream/streambuf io/stream design. This would look like a design flaw to me.
Helpful to whom? I agree this is all a good idea, but I'm not sure how this relates to anything else we're discussing.
It would permit me to use the MPI serialization as a use case to verify that I hadn't broken anything. I know what your going to say - concepts - but that is going to be a while before that happens.
b) user experience seems to show that archive construction/destruction is a significant performance issue when a new archive is made for each data transmission. On the other hand, one has to do this since the current archive implemenation track addresses of serialized obects so the same archive can't be use send the same structure (maybe with changed data) multiple times.
I can't understand how you reach that conclusion. In my long explanation I thought I made it clear that Boost.MPI and its database get a great deal of its performance advantage from exactly that: sending the same structure multiple times with the same MPI type map.
hmm - what about tracked objects? I suggested turning off tracking and Matthias told me that it was needed to send pointers. But for this to work an archive has to be reconstructed everytime to re-initialize the tracking lists. I don't want to spend a lot of time on this as that means I have to delve into the MPI serialization and I don't want to do this.
Given that MPI has a focus on performance, I wonder if this has been considered. I looked a the documentation, code and examples and it wasn't obvious to me how this is question was addressed - if at all.
We've obviously misunderstood one another somewhere along the way. It would be good if we could get that cleared up.
I think it would take too much time for me to understand. And if I did come to understand it, I might have something to say about it. If I said it, then it might start a thread which would cost even more time which I don't have right now. Robert Ramey
On 27 Aug 2010, at 22:26, Robert Ramey wrote:
David Abrahams wrote:
b) user experience seems to show that archive construction/destruction is a significant performance issue when a new archive is made for each data transmission. On the other hand, one has to do this since the current archive implemenation track addresses of serialized obects so the same archive can't be use send the same structure (maybe with changed data) multiple times.
I can't understand how you reach that conclusion. In my long explanation I thought I made it clear that Boost.MPI and its database get a great deal of its performance advantage from exactly that: sending the same structure multiple times with the same MPI type map.
hmm - what about tracked objects? I suggested turning off tracking and Matthias told me that it was needed to send pointers. But for this to work an archive has to be reconstructed everytime to re-initialize the tracking lists. I don't want to spend a lot of time on this as that means I have to delve into the MPI serialization and I don't want to do this.
The typical use case is that the pointer structure is built once, and then only the values of arrays and other data members needs to be updated by communication. The "skeleton" creates all the objects and sets the pointers, and is sent only once, and thereafter only the "contents" is transmitted. Matthias
Matthias Troyer wrote:
On 27 Aug 2010, at 22:26, Robert Ramey wrote:
hmm - what about tracked objects? I suggested turning off tracking and Matthias told me that it was needed to send pointers. But for this to work an archive has to be reconstructed everytime to re-initialize the tracking lists. I don't want to spend a lot of time on this as that means I have to delve into the MPI serialization and I don't want to do this.
The typical use case is that the pointer structure is built once, and then only the values of arrays and other data members needs to be updated by communication. The "skeleton" creates all the objects and sets the pointers, and is sent only once, and thereafter only the "contents" is transmitted.
So tracking is effectively turned off by this archive implemenation. Which makes sense to me from a practical standpoint. But it does make things harder to understand. To be fair, the other archive classes each have their own implemention quirks which are more or less ad hoc. Hopefully one day we can rationalize this to some extent.
Matthias
On 27 Aug 2010, at 23:53, Robert Ramey wrote:
Matthias Troyer wrote:
On 27 Aug 2010, at 22:26, Robert Ramey wrote:
hmm - what about tracked objects? I suggested turning off tracking and Matthias told me that it was needed to send pointers. But for this to work an archive has to be reconstructed everytime to re-initialize the tracking lists. I don't want to spend a lot of time on this as that means I have to delve into the MPI serialization and I don't want to do this.
The typical use case is that the pointer structure is built once, and then only the values of arrays and other data members needs to be updated by communication. The "skeleton" creates all the objects and sets the pointers, and is sent only once, and thereafter only the "contents" is transmitted.
So tracking is effectively turned off by this archive implemenation. Which makes sense to me from a practical standpoint. But it does make things harder to understand. To be fair, the other archive classes each have their own implemention quirks which are more or less ad hoc. Hopefully one day we can rationalize this to some extent.
What happens is that when using the skeleton&contents mechanism the serialized data is split into two parts: all the "structural" data (the skeleton), such as version, object id, class name .... is serialized into a buffer as the skeleton, and all the "normal" data members are used to create an MPI data type. You might argue that this second one actually turns racking off. The nicest thing, however, is that these two "archives" are both created only once. The MPI datatype created for the contents is then used many times, and all of work there is done by MPI and the network hardware without needed the serialization library anymore. Matthias
Matthias Troyer wrote:
On 27 Aug 2010, at 23:53, Robert Ramey wrote: What happens is that when using the skeleton&contents mechanism the serialized data is split into two parts: all the "structural" data (the skeleton), such as version, object id, class name .... is serialized into a buffer as the skeleton, and all the "normal" data members are used to create an MPI data type. You might argue that this second one actually turns racking off.
OK - this seems to make more sense now
The nicest thing, however, is that these two "archives" are both created only once. The MPI datatype created for the contents is then used many times, and all of work there is done by MPI and the network hardware without needed the serialization library anymore.
I might mention - though it's slightly off topic - that a similar idea occurred to me with the xml_archive so that when one created an xml_archive he would have the option of simultaneously creating a xml_schema which could be used by an xml_editor to create/validate changes to the archive. I never acted on this idea since I've always had other stuff that interested me more and I'm not convinced that there would be demand for it. I'm aware that some work has to be done to make the library more usable for those making archives. I'm sort of surprised at how few questions I get about this. I'm presuming that the number of users making archives is extremely small - at least in comparison to those using archives. I don't think it's because making an archive class is all that straight forward.
Matthias
On Fri, Aug 27, 2010 at 2:38 PM, Robert Ramey <ramey@rrsd.com> wrote:
I might mention - though it's slightly off topic - that a similar idea occurred to me with the xml_archive so that when one created an xml_archive he would have the option of simultaneously creating a xml_schema which could be used by an xml_editor to create/validate changes to the archive.
The problem is that depth-first pointer serialization as done by Boost.Serialization is really inappropriate for any kind of sensible XML schema creation. Things that really ought to be done with xml:ref end up being child elements, and you don't end up with a structure that's coherent for humans. One of our clients has already spent a great of money for us to develop XML serialization for their application, in part because they were dissatisfied with the results they could get using Boost.Serialization. So this is another area where you might ask yourself whether it's a use-case you really want to make the effort to support. It seems like an important application, but it could mean fundamental changes for you. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
Matthias Troyer wrote:
On 27 Aug 2010, at 23:53, Robert Ramey wrote:
Matthias Troyer wrote:
hmm - what about tracked objects? I suggested turning off tracking and Matthias told me that it was needed to send pointers. But for this to work an archive has to be reconstructed everytime to re-initialize the tracking lists. I don't want to spend a lot of time on this as that means I have to delve into the MPI serialization and I don't want to do this. The typical use case is that the pointer structure is built once, and
On 27 Aug 2010, at 22:26, Robert Ramey wrote: then only the values of arrays and other data members needs to be updated by communication. The "skeleton" creates all the objects and sets the pointers, and is sent only once, and thereafter only the "contents" is transmitted. So tracking is effectively turned off by this archive implemenation. Which makes sense to me from a practical standpoint. But it does make things harder to understand. To be fair, the other archive classes each have their own implemention quirks which are more or less ad hoc. Hopefully one day we can rationalize this to some extent.
What happens is that when using the skeleton&contents mechanism the serialized data is split into two parts: all the "structural" data (the skeleton), such as version, object id, class name .... is serialized into a buffer as the skeleton, and all the "normal" data members are used to create an MPI data type. You might argue that this second one actually turns racking off.
The nicest thing, however, is that these two "archives" are both created only once. The MPI datatype created for the contents is then used many times, and all of work there is done by MPI and the network hardware without needed the serialization library anymore.
So what happens if the some data member that is a base class pointer gets modified to point to some other instance of a derived class? Doesn't that change the 'structure'? Jeff
BoostPro Computing, http://boostpro.com Sent from coveted but awkward mobile device -- On Aug 28, 2010, at 6:15 AM, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:
So what happens if the some data member that is a base class pointer gets modified to point to some other instance of a derived class? Doesn't that change the 'structure'?
Yes of course. However, many MPI applications never need to do that. who-uses-oop-in-hpc-applications-anyway-ly y'rs Dave
Dave Abrahams wrote:
BoostPro Computing, http://boostpro.com Sent from coveted but awkward mobile device --
On Aug 28, 2010, at 6:15 AM, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:
So what happens if the some data member that is a base class pointer gets modified to point to some other instance of a derived class? Doesn't that change the 'structure'?
Yes of course. However, many MPI applications never need to do that.
No doubt many don't. I'm just trying to gain a better understanding of boost.mpi and provide some of that feedback you asked for else thread. :-) Does MPI detect those changes automatically resending the skeleton information as required? Or does the user code need to explicitly manage such situations? Or does this disallow usage of the skeleton approach altogether in these situations. Thanks, Jeff
On 28 Aug 2010, at 18:36, Jeff Flinn wrote:
Dave Abrahams wrote:
BoostPro Computing, http://boostpro.com Sent from coveted but awkward mobile device -- On Aug 28, 2010, at 6:15 AM, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:
So what happens if the some data member that is a base class pointer gets modified to point to some other instance of a derived class? Doesn't that change the 'structure'? Yes of course. However, many MPI applications never need to do that.
No doubt many don't. I'm just trying to gain a better understanding of boost.mpi and provide some of that feedback you asked for else thread. :-)
Does MPI detect those changes automatically resending the skeleton information as required? Or does the user code need to explicitly manage such situations? Or does this disallow usage of the skeleton approach altogether in these situations.
The user code needs to explicitly manage this, and if your code typically changes the structure, then I recommend to just use the normal send and recv mechanism which sends the whole object. Matthias
hello again i wrote my code for sending one object from rank 0 to rank 1 process sing boost.mpi because it is very simple. it worked when sending a string just like in the documentation, however when i sent the object i showed earlier i am getting errors. terminate called after throwing an instance of 'boost::archive::archive_ exception' what(): class version St8multimapIiiSt4lessIiESaISt4pairIKiiEEE [lenovo00:06116] *** Process received signal *** [lenovo00:06116] Signal: Aborted (6) [lenovo00:06116] Signal code: (-6) [lenovo00:06116] [ 0] [0x71f410] [lenovo00:06116] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0x74da82] [lenovo00:06116] [ 2] /usr/lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x14f) [0x4da52f] [lenovo00:06116] [ 3] /usr/lib/libstdc++.so.6(+0xbd465) [0x4d8465] [lenovo00:06116] [ 4] /usr/lib/libstdc++.so.6(+0xbd4a2) [0x4d84a2] [lenovo00:06116] [ 5] /usr/lib/libstdc++.so.6(+0xbd5e1) [0x4d85e1] [lenovo00:06116] [ 6] bin/release/plattice(_ZN5boost13serialization15throw_exceptionINS_7archive17archive_exceptionEEEvRKT_+0x3e) [0x80ee491] ... ... ... says something about multimap type error and other things. if i remove the multimap attribute from the serialization method i get another error, and if i comment almost everything and only keep 2 integers (numNodes numEdges) for testing... the receiving object has different values on those integers. this can be a problem of the internal serializing which uses packed_oarchive/iarchive instead of the binary_oarchive/iarchive i was testing earlier when it worked?? any indication is welcome if you need more debugging information i can provide Cristobal <http://www.youtube.com/neoideo> On Sat, Aug 28, 2010 at 3:11 PM, Matthias Troyer <troyer@phys.ethz.ch>wrote:
On 28 Aug 2010, at 18:36, Jeff Flinn wrote:
BoostPro Computing, http://boostpro.com Sent from coveted but awkward mobile device -- On Aug 28, 2010, at 6:15 AM, Jeff Flinn <TriumphSprint2000@hotmail.com> wrote:
So what happens if the some data member that is a base class pointer gets modified to point to some other instance of a derived class? Doesn't
Dave Abrahams wrote: that change the 'structure'?
Yes of course. However, many MPI applications never need to do that.
No doubt many don't. I'm just trying to gain a better understanding of boost.mpi and provide some of that feedback you asked for else thread. :-)
Does MPI detect those changes automatically resending the skeleton information as required? Or does the user code need to explicitly manage such situations? Or does this disallow usage of the skeleton approach altogether in these situations.
The user code needs to explicitly manage this, and if your code typically changes the structure, then I recommend to just use the normal send and recv mechanism which sends the whole object.
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On Aug 30, 2010, at 2:05 AM, Cristobal Navarro wrote:
hello again
i wrote my code for sending one object from rank 0 to rank 1 process sing boost.mpi because it is very simple. it worked when sending a string just like in the documentation,
however when i sent the object i showed earlier i am getting errors.
terminate called after throwing an instance of 'boost::archive::archive_ exception' what(): class version St8multimapIiiSt4lessIiESaISt4pairIKiiEEE [lenovo00:06116] *** Process received signal *** [lenovo00:06116] Signal: Aborted (6) [lenovo00:06116] Signal code: (-6) [lenovo00:06116] [ 0] [0x71f410] [lenovo00:06116] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0x74da82] [lenovo00:06116] [ 2] /usr/lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x14f) [0x4da52f] [lenovo00:06116] [ 3] /usr/lib/libstdc++.so.6(+0xbd465) [0x4d8465] [lenovo00:06116] [ 4] /usr/lib/libstdc++.so.6(+0xbd4a2) [0x4d84a2] [lenovo00:06116] [ 5] /usr/lib/libstdc++.so.6(+0xbd5e1) [0x4d85e1] [lenovo00:06116] [ 6] bin/release/plattice(_ZN5boost13serialization15throw_exceptionINS_7archive17archive_exceptionEEEvRKT_+0x3e) [0x80ee491] ... ... ... says something about multimap type error and other things. if i remove the multimap attribute from the serialization method i get another error, and if i comment almost everything and only keep 2 integers (numNodes numEdges) for testing... the receiving object has different values on those integers.
this can be a problem of the internal serializing which uses packed_oarchive/iarchive instead of the binary_oarchive/iarchive i was testing earlier when it worked?? any indication is welcome if you need more debugging information i can provide
Please post the code that just sends two integers and fails to receive them correctly Matthias
sure, this is master process, sends objects. #SENDER PROCESS (rank 0) printf("sending latticec to worker 1\n"); lat->print(); //to see what i am sending cin >> a; world.send(1, 1, lat); #RECEIVER PROCESS (rank 1) Lattice* latr; printf("worker%2d:: waiting master signal...\n", world.rank()); world.recv(0, 1, latr); printf("OK\n"); latr->print(); #this is the Lattice class, i have commented almost all atributes on the serialization method except 2 integers. class Lattice{ private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version){ ar & numNodes; ar & numEdges; //ar & nodes; //ar & edges; //ar & keyLists; //ar & corrupt; //ar & rec; //ar & nop; //ar & acumCoef; //ar & key; } public: int numNodes; int numEdges; map<int, Edge> edges; map<int, Node> nodes; string key; list< list<int> > keyLists; bool corrupt; bool rec; int nop; string acumCoef; } #when i send the object it has Nodes=2 Edges= 1 #when i receive it, i get Nodes=131072 Edges= 65536 On Mon, Aug 30, 2010 at 5:46 AM, Matthias Troyer <troyer@phys.ethz.ch>wrote:
On Aug 30, 2010, at 2:05 AM, Cristobal Navarro wrote:
hello again
i wrote my code for sending one object from rank 0 to rank 1 process sing boost.mpi because it is very simple. it worked when sending a string just like in the documentation,
however when i sent the object i showed earlier i am getting errors.
terminate called after throwing an instance of 'boost::archive::archive_ exception' what(): class version St8multimapIiiSt4lessIiESaISt4pairIKiiEEE [lenovo00:06116] *** Process received signal *** [lenovo00:06116] Signal: Aborted (6) [lenovo00:06116] Signal code: (-6) [lenovo00:06116] [ 0] [0x71f410] [lenovo00:06116] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0x74da82] [lenovo00:06116] [ 2]
/usr/lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x14f)
[0x4da52f] [lenovo00:06116] [ 3] /usr/lib/libstdc++.so.6(+0xbd465) [0x4d8465] [lenovo00:06116] [ 4] /usr/lib/libstdc++.so.6(+0xbd4a2) [0x4d84a2] [lenovo00:06116] [ 5] /usr/lib/libstdc++.so.6(+0xbd5e1) [0x4d85e1] [lenovo00:06116] [ 6]
[0x80ee491] ... ... ... says something about multimap type error and other things. if i remove the multimap attribute from the serialization method i get another error, and if i comment almost everything and only keep 2 integers (numNodes numEdges) for testing... the receiving object has different values on those integers.
this can be a problem of the internal serializing which uses packed_oarchive/iarchive instead of the binary_oarchive/iarchive i was testing earlier when it worked?? any indication is welcome if you need more debugging information i can
bin/release/plattice(_ZN5boost13serialization15throw_exceptionINS_7archive17archive_exceptionEEEvRKT_+0x3e) provide
Please post the code that just sends two integers and fails to receive them correctly
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
i found the problem, and found a solution, but would be better if i learn some explanations of you guys which know much more. problem was that i was sending the pointer of my object { Lattice* lat = new Lattice("filename"); world.send(1, 1, lat); } that generated the errors on the packing algorithms i suppose (correct me if im wrong) so instead of changing all my program i only sent the object pointed world.send(1, 1, *lat ); and on the receiver process i receive it with Lattice lat; world.recv(0, 1, lat); then i can transform it to pointer again to keep my algorithms unchanged, Lattice plat = ⪫ i wonder, is this a good approach to solve the error i was having?? at least does work and i can continue programming, which makes me happy :) any suggestions welcome and thanks everybody for all the help already viva boost On Mon, Aug 30, 2010 at 11:09 AM, Cristobal Navarro <axischire@gmail.com>wrote:
sure,
this is master process, sends objects. #SENDER PROCESS (rank 0) printf("sending latticec to worker 1\n"); lat->print(); //to see what i am sending cin >> a; world.send(1, 1, lat);
#RECEIVER PROCESS (rank 1) Lattice* latr; printf("worker%2d:: waiting master signal...\n", world.rank()); world.recv(0, 1, latr); printf("OK\n"); latr->print();
#this is the Lattice class, i have commented almost all atributes on the serialization method except 2 integers. class Lattice{
private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version){ ar & numNodes; ar & numEdges; //ar & nodes; //ar & edges; //ar & keyLists; //ar & corrupt; //ar & rec; //ar & nop; //ar & acumCoef; //ar & key; }
public: int numNodes; int numEdges; map<int, Edge> edges; map<int, Node> nodes; string key; list< list<int> > keyLists; bool corrupt; bool rec; int nop; string acumCoef; }
#when i send the object it has Nodes=2 Edges= 1
#when i receive it, i get Nodes=131072 Edges= 65536
On Mon, Aug 30, 2010 at 5:46 AM, Matthias Troyer <troyer@phys.ethz.ch>wrote:
On Aug 30, 2010, at 2:05 AM, Cristobal Navarro wrote:
hello again
i wrote my code for sending one object from rank 0 to rank 1 process sing boost.mpi because it is very simple. it worked when sending a string just like in the documentation,
however when i sent the object i showed earlier i am getting errors.
terminate called after throwing an instance of 'boost::archive::archive_ exception' what(): class version St8multimapIiiSt4lessIiESaISt4pairIKiiEEE [lenovo00:06116] *** Process received signal *** [lenovo00:06116] Signal: Aborted (6) [lenovo00:06116] Signal code: (-6) [lenovo00:06116] [ 0] [0x71f410] [lenovo00:06116] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0x74da82] [lenovo00:06116] [ 2]
/usr/lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x14f)
[0x4da52f] [lenovo00:06116] [ 3] /usr/lib/libstdc++.so.6(+0xbd465) [0x4d8465] [lenovo00:06116] [ 4] /usr/lib/libstdc++.so.6(+0xbd4a2) [0x4d84a2] [lenovo00:06116] [ 5] /usr/lib/libstdc++.so.6(+0xbd5e1) [0x4d85e1] [lenovo00:06116] [ 6]
[0x80ee491] ... ... ... says something about multimap type error and other things. if i remove the multimap attribute from the serialization method i get another error, and if i comment almost everything and only keep 2 integers (numNodes numEdges) for testing... the receiving object has different values on those integers.
this can be a problem of the internal serializing which uses packed_oarchive/iarchive instead of the binary_oarchive/iarchive i was testing earlier when it worked?? any indication is welcome if you need more debugging information i can
bin/release/plattice(_ZN5boost13serialization15throw_exceptionINS_7archive17archive_exceptionEEEvRKT_+0x3e) provide
Please post the code that just sends two integers and fails to receive them correctly
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
Cristobal Navarro wrote:
i found the problem, and found a solution, but would be better if i learn some explanations of you guys which know much more.
problem was that i was sending the pointer of my object { Lattice* lat = new Lattice("filename"); world.send(1, 1, lat); } that generated the errors on the packing algorithms i suppose (correct me if im wrong)
so instead of changing all my program i only sent the object pointed
world.send(1, 1, *lat );
and on the receiver process i receive it with
Lattice lat; world.recv(0, 1, lat);
then i can transform it to pointer again to keep my algorithms unchanged,
Lattice plat = ⪫
i wonder, is this a good approach to solve the error i was having?? at least does work and i can continue programming, which makes me happy :)
If it were me I would declare world.send(?, ?, const Lattice & lat); world.recv(0, 1, Lattice & lat); then: Lattice lat("filename"); world.send(1, 1, lat); and Lattice lat; world.recv(0, 1, lat); transparent, easy and guarenteed correct. Robert Ramey
yes i thought too, sadly at this point changing from Lattice* lat(filename"); to Lattice lat("filename"); means changing many lines of code of the algorithm asociated (which was initially a secuential program) anyways, this whole experience and problems debugging, resulted in a lot of learning. On Mon, Aug 30, 2010 at 5:53 PM, Robert Ramey <ramey@rrsd.com> wrote:
Cristobal Navarro wrote:
i found the problem, and found a solution, but would be better if i learn some explanations of you guys which know much more.
problem was that i was sending the pointer of my object { Lattice* lat = new Lattice("filename"); world.send(1, 1, lat); } that generated the errors on the packing algorithms i suppose (correct me if im wrong)
so instead of changing all my program i only sent the object pointed
world.send(1, 1, *lat );
and on the receiver process i receive it with
Lattice lat; world.recv(0, 1, lat);
then i can transform it to pointer again to keep my algorithms unchanged,
Lattice plat = ⪫
i wonder, is this a good approach to solve the error i was having?? at least does work and i can continue programming, which makes me happy :)
If it were me I would declare
world.send(?, ?, const Lattice & lat); world.recv(0, 1, Lattice & lat);
then: Lattice lat("filename");
world.send(1, 1, lat); and Lattice lat; world.recv(0, 1, lat);
transparent, easy and guarenteed correct.
Robert Ramey
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 30 Aug 2010, at 13:36, Cristobal Navarro wrote:
i found the problem, and found a solution, but would be better if i learn some explanations of you guys which know much more.
problem was that i was sending the pointer of my object { Lattice* lat = new Lattice("filename"); world.send(1, 1, lat); } that generated the errors on the packing algorithms i suppose (correct me if im wrong)
so instead of changing all my program i only sent the object pointed
world.send(1, 1, *lat );
and on the receiver process i receive it with
Lattice lat; world.recv(0, 1, lat);
then i can transform it to pointer again to keep my algorithms unchanged,
Lattice plat = ⪫
i wonder, is this a good approach to solve the error i was having?? at least does work and i can continue programming, which makes me happy :)
Hi, this problem should be fixed now on the trunk and in the forthcoming 1.45 release Matthias
On 30 Aug 2010, at 17:09, Cristobal Navarro wrote:
sure,
this is master process, sends objects. #SENDER PROCESS (rank 0) printf("sending latticec to worker 1\n"); lat->print(); //to see what i am sending cin >> a; world.send(1, 1, lat);
#RECEIVER PROCESS (rank 1) Lattice* latr; printf("worker%2d:: waiting master signal...\n", world.rank()); world.recv(0, 1, latr); printf("OK\n"); latr->print();
#this is the Lattice class, i have commented almost all atributes on the serialization method except 2 integers. class Lattice{
private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version){ ar & numNodes; ar & numEdges; //ar & nodes; //ar & edges; //ar & keyLists; //ar & corrupt; //ar & rec; //ar & nop; //ar & acumCoef; //ar & key; }
public: int numNodes; int numEdges; map<int, Edge> edges; map<int, Node> nodes; string key; list< list<int> > keyLists; bool corrupt; bool rec; int nop; string acumCoef; }
#when i send the object it has Nodes=2 Edges= 1
#when i receive it, i get Nodes=131072 Edges= 65536
Can you please post a full program and not just fragments of your code. With those fragments I cannot reproduce your problem. Matthias
On 30 Aug 2010, at 17:09, Cristobal Navarro wrote:
sure,
this is master process, sends objects. #SENDER PROCESS (rank 0) printf("sending latticec to worker 1\n"); lat->print(); //to see what i am sending cin >> a; world.send(1, 1, lat);
#RECEIVER PROCESS (rank 1) Lattice* latr; printf("worker%2d:: waiting master signal...\n", world.rank()); world.recv(0, 1, latr); printf("OK\n"); latr->print();
#this is the Lattice class, i have commented almost all atributes on the serialization method except 2 integers. class Lattice{
private: friend class boost::serialization::access; template<class Archive> void serialize(Archive & ar, const unsigned int version){ ar & numNodes; ar & numEdges; //ar & nodes; //ar & edges; //ar & keyLists; //ar & corrupt; //ar & rec; //ar & nop; //ar & acumCoef; //ar & key; }
public: int numNodes; int numEdges; map<int, Edge> edges; map<int, Node> nodes; string key; list< list<int> > keyLists; bool corrupt; bool rec; int nop; string acumCoef; }
#when i send the object it has Nodes=2 Edges= 1
#when i receive it, i get Nodes=131072 Edges= 65536
Can you please post a full program and not just fragments of your code. With those fragments I cannot reproduce your problem. Matthias
On 28 Aug 2010, at 16:15, Jeff Flinn wrote:
Matthias Troyer wrote:
What happens is that when using the skeleton&contents mechanism the serialized data is split into two parts: all the "structural" data (the skeleton), such as version, object id, class name .... is serialized into a buffer as the skeleton, and all the "normal" data members are used to create an MPI data type. You might argue that this second one actually turns racking off. The nicest thing, however, is that these two "archives" are both created only once. The MPI datatype created for the contents is then used many times, and all of work there is done by MPI and the network hardware without needed the serialization library anymore.
So what happens if the some data member that is a base class pointer gets modified to point to some other instance of a derived class? Doesn't that change the 'structure'?
Indeed pointers are not part of the "content" and if you change the structure you need to send the complete object, or first update it by sending the new skeleton. This is however a rare occurence in the typical MPI program. Matthias
On 27 Aug 2010, at 19:44, Robert Ramey wrote:
David Abrahams wrote:
At Thu, 26 Aug 2010 16:05:41 -0800, Robert Ramey wrote:
I just never found it to be very revealing. The word skeleton seemed pretty suggestive. It's still not clear to me how such a think can work between heterogeneous machines. For example, if I have an array of 2 byte integers and they each need to get transformed one by one into a 4 byte integer because that's closest MPI data type,
I think you don't understand what MPI datatypes do.
This is true. I suppose that's one reason why the documentation made no sense to me. They just looked like special types to make identify primitives accross differing architectures.
<snip> Interesting information which one should consider adding to the MPI documentation. </snip>
I disagree. We cannot and should not copy dozens of pages of MPI documentation. Boost.MPI is targeted at an audience that understands MPI but does not presume to teach MPI.
I only really have few observations/suggestions at this point.
a) It would be helpful if there were a way to test the serialization of the archive classes in boost MPI without having MPI installed. If this is not possible, it would seem to me that the serialization is intertwined with the data transport - rather than be separated as it is in the stream/streambuf io/stream design. This would look like a design flaw to me.
No, we separate transport from serialization, it is just that both transport and serialization require calls to the MPI library.
b) user experience seems to show that archive construction/destruction is a significant performance issue when a new archive is made for each data transmission. On the otherhand, one has to do this since the current archive implemenation track addresses of serialized obects so the same archive can't e use send the same structure (maybe with changed data) multiple times. Given that MPI has a focus on performance, I wonder if this has been considered. I looked a the documentation, code and examples and it wasn't obvious to me how this is question was addressed - if at all.
It has been considered and that is the concept behind skeleton&content and was a key reason to Boost.MPI: you set up a description of the data structures once using the "skeleton" and then can just blast data from object member to object member across the network without the use of any archive or buffer (except for any buffering in the network hardware).
You should know that all your efforts to educate me are not wasted.
a) Sometimes I pass your advice along to other people.
b) Motivated by our recent discussions on the subject as well as some other issues has clarified my ideas on how the archive concept should be refined to be more helpful to those creating their own archives. I think there will be improvements in this area in part due to your observations.
Thanks for your efforts Matthias
At Fri, 27 Aug 2010 21:05:38 +0200, Matthias Troyer wrote:
On 27 Aug 2010, at 19:44, Robert Ramey wrote:
David Abrahams wrote:
At Thu, 26 Aug 2010 16:05:41 -0800, Robert Ramey wrote:
I just never found it to be very revealing. The word skeleton seemed pretty suggestive. It's still not clear to me how such a think can work between heterogeneous machines. For example, if I have an array of 2 byte integers and they each need to get transformed one by one into a 4 byte integer because that's closest MPI data type,
I think you don't understand what MPI datatypes do.
This is true. I suppose that's one reason why the documentation made no sense to me. They just looked like special types to make identify primitives accross differing architectures.
<snip> Interesting information which one should consider adding to the MPI documentation. </snip>
I disagree. We cannot and should not copy dozens of pages of MPI documentation. Boost.MPI is targeted at an audience that understands MPI but does not presume to teach MPI.
Dozens of pages? Copy? Teach MPI? AFAICT, Robert is only suggesting adding 83 lines (~4K characters) to the docs. And frankly, even though I understand and have used MPI, none of what I wrote seems obvious to me, and frankly it seems important in explaining to people why they should use Boost.MPI rather than trying to use MPI directly. Frankly, I'd like to put that information in a blog post, if I got it right (did I?), so you wouldn't have to do more than point to it, even though IMO it's ill-advised not to explain some of this right in the MPI doc. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On 27 Aug 2010, at 21:25, David Abrahams wrote:
At Fri, 27 Aug 2010 21:05:38 +0200, Matthias Troyer wrote:
On 27 Aug 2010, at 19:44, Robert Ramey wrote:
David Abrahams wrote:
At Thu, 26 Aug 2010 16:05:41 -0800, Robert Ramey wrote:
I just never found it to be very revealing. The word skeleton seemed pretty suggestive. It's still not clear to me how such a think can work between heterogeneous machines. For example, if I have an array of 2 byte integers and they each need to get transformed one by one into a 4 byte integer because that's closest MPI data type,
I think you don't understand what MPI datatypes do.
This is true. I suppose that's one reason why the documentation made no sense to me. They just looked like special types to make identify primitives accross differing architectures.
<snip> Interesting information which one should consider adding to the MPI documentation. </snip>
I disagree. We cannot and should not copy dozens of pages of MPI documentation. Boost.MPI is targeted at an audience that understands MPI but does not presume to teach MPI.
Dozens of pages? Copy? Teach MPI?
AFAICT, Robert is only suggesting adding 83 lines (~4K characters) to the docs. And frankly, even though I understand and have used MPI, none of what I wrote seems obvious to me, and frankly it seems important in explaining to people why they should use Boost.MPI rather than trying to use MPI directly.
The question is where should we stop, where should we start, what should be explained. The MPI standard documents is several hundred pages long. If you want just a few lines discussing the advantages of Boost.MPI and of skeleton&content then this can be done. If you want us to explain all details of MPI datatypes, all details of the semantics of nonblocking operations (which caused other issues in the past), ....then it will soon be most of the MPI standard or an MPI text book.
Frankly, I'd like to put that information in a blog post, if I got it right (did I?), so you wouldn't have to do more than point to it, even though IMO it's ill-advised not to explain some of this right in the MPI doc.
The docs were written by Doug but he's probably too busy at the moment. Matthias
At Fri, 27 Aug 2010 21:55:36 +0200, Matthias Troyer wrote:
<snip> Interesting information which one should consider adding to the MPI documentation. </snip>
I disagree. We cannot and should not copy dozens of pages of MPI documentation. Boost.MPI is targeted at an audience that understands MPI but does not presume to teach MPI.
Dozens of pages? Copy? Teach MPI?
AFAICT, Robert is only suggesting adding 83 lines (~4K characters) to the docs. And frankly, even though I understand and have used MPI, none of what I wrote seems obvious to me, and frankly it seems important in explaining to people why they should use Boost.MPI rather than trying to use MPI directly.
The question is where should we stop, where should we start, what should be explained.
I don't think any of that is really in question. We should just add the stuff I wrote there.
The MPI standard documents is several hundred pages long. If you want just a few lines discussing the advantages of Boost.MPI and of skeleton&content then this can be done. If you want us to explain all details of MPI datatypes, all details of the semantics of nonblocking operations (which caused other issues in the past), ....then it will soon be most of the MPI standard or an MPI text book.
Robert only suggested adding the information that I posted, almost none of which is covered in the MPI docs. -- Dave Abrahams BoostPro Computing http://www.boostpro.com
I hereby note the energy expended on assertions that we don't have time to create real understanding. All of the things we've encountered here can be handled in the space of a few sentences, if we're willing. At Fri, 27 Aug 2010 21:55:36 +0200, Matthias Troyer wrote:
The question is where should we stop, where should we start, what should be explained. The MPI standard documents is several hundred pages long. If you want just a few lines discussing the advantages of Boost.MPI and of skeleton&content then this can be done. If you want us to explain all details of MPI datatypes, all details of the semantics of nonblocking operations (which caused other issues in the past), ....then it will soon be most of the MPI standard or an MPI text book.
At Fri, 27 Aug 2010 12:26:03 -0800, Robert Ramey wrote:
We've obviously misunderstood one another somewhere along the way. It would be good if we could get that cleared up.
I think it would take too much time for me to understand. And if I did come to understand it, I might have something to say about it. If I said it, then it might start a thread which would cost even more time which I don't have right now.
Cheers! -- Dave Abrahams BoostPro Computing http://www.boostpro.com
On 26 Aug 2010, at 20:56, Robert Ramey wrote:
Cristobal Navarro wrote:
the other question is on the tutorials i only found examples of serializing objects to archive and saving to file, but i didnt find any example where they serialize it to a binary form and send them through mpi, is there any example of this type on the documentation?
no
Actually the answer is yes. This is exactly what is done by Boost.MPI! When you send an object using Boost.MPI, it is serialized into an MPI buffer by the packed_oarchive. Then that buffer's content is sent via MPI and it is unpacked at the other end. All this is transparently done by the send and recv calls of Boost.MPI.
Feel free to contribute one.
No need since it already exists.
If I had nothing else to do, I would
a) make a stream_buf which uses MPI
b) use binary_iarchive and binary_oarchive with this new stream_buf
we avoid streambufs by directly packing using detail::mpi_[io]primitive. This allows us to make use of conversion facilities in MPI, rather than packing as native binary.
c) open one program with binary_oarchive and the other with binary_iarchive
we call them packed_oarchive and packed_iarchive
d) do ar << on one end and ar >> at the other
That is what the send and recv functions do, but in addition we have the MPI_Send and MPI_Recv calls that transfer the buffer. Matthias
Actually the answer is yes. This is exactly what is done by Boost.MPI!
When you send an object using Boost.MPI, it is serialized into an MPI buffer by the packed_oarchive. Then that buffer's content is sent via MPI and it is unpacked at the other end. All this is transparently done by the send and recv calls of Boost.MPI.
Matthias, so is possible to create this packed_oarchive for sender process,
and packed_oarchive for receiver process and use plain MPI instructions MPI_send MPI_recv. i think im getting the idea, i just need an example i could look at and understand the silly parts to connect everything thanks for all the help really
Feel free to contribute one.
No need since it already exists.
If I had nothing else to do, I would
a) make a stream_buf which uses MPI
b) use binary_iarchive and binary_oarchive with this new stream_buf
we avoid streambufs by directly packing using detail::mpi_[io]primitive. This allows us to make use of conversion facilities in MPI, rather than packing as native binary.
c) open one program with binary_oarchive and the other with binary_iarchive
we call them packed_oarchive and packed_iarchive
d) do ar << on one end and ar >> at the other
That is what the send and recv functions do, but in addition we have the MPI_Send and MPI_Recv calls that transfer the buffer.
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
On 26 Aug 2010, at 20:40, Cristobal Navarro wrote:
Actually the answer is yes. This is exactly what is done by Boost.MPI!
When you send an object using Boost.MPI, it is serialized into an MPI buffer by the packed_oarchive. Then that buffer's content is sent via MPI and it is unpacked at the other end. All this is transparently done by the send and recv calls of Boost.MPI.
Matthias, so is possible to create this packed_oarchive for sender process, and packed_oarchive for receiver process and use plain MPI instructions MPI_send MPI_recv.
Yes, that can be done. However you may consider just using Boost.MPI right away since that is all that is done: pack the archive and send it. Matthias
participants (7)
-
Cristobal Navarro -
Dave Abrahams -
David Abrahams
-
Jeff Flinn -
Kumar, Akhilesh -
Matthias Troyer -
Robert Ramey