Mersenne twister problem
Hello
I've built a meta-heuristics library in which I use the boost random
library. I am using several independent generators through the application.
The problem is in the core of my library. The tricky part is that I have
three different algorithms using the core, all of them using the class where
a segmentation fault happens, but only one of them crashes when it tries to
use the class, the others run as expected. Gdb points to the boost library,
as you can see in the snippet bellow. I couldn't find any information on
this issue, and I cannot see something wrong with my code (the relevant part
is bellow).
The parent class where generators are declared:
template< class SType, typename MemP >
class SearchFunctor{
(...)
public:
virtual SPtr operator()( SPtr s, int iter ) = 0;
SearchFunctor(){
(...)
//This is how I initialize teh generators, being MASTER SEED 666
generator1 = base_generator_type(MASTER_SEED*13);
uni_dist1 = boost::uniform_real<>(0,1);
uniEmp = new GeneratorI(generator1, uni_dist1);
generator2 = base_generator_type(MASTER_SEED*17);
uni_dist2 = boost::uniform_real<>(0,1);
uniRes = new GeneratorI(generator2, uni_dist2);
}
protected:
base_generator_type generator1, generator2;
boost::uniform_real<> uni_dist1, uni_dist2;
GeneratorI* uniEmp;
GeneratorI* uniRes;
};
The derived class using the generators:
template< class SType, typename MemP >
struct SwapSearch : public SearchFunctor
On 11 Sep 2009, at 11:19, ruya wrote:
Hello
I've built a meta-heuristics library in which I use the boost random library. I am using several independent generators through the application. The problem is in the core of my library. The tricky part is that I have three different algorithms using the core, all of them using the class where a segmentation fault happens, but only one of them crashes when it tries to use the class, the others run as expected. Gdb points to the boost library, as you can see in the snippet bellow. I couldn't find any information on this issue, and I cannot see something wrong with my code (the relevant part is bellow).
The parent class where generators are declared:
template< class SType, typename MemP > class SearchFunctor{ (...) public:
virtual SPtr operator()( SPtr s, int iter ) = 0;
SearchFunctor(){ (...) //This is how I initialize teh generators, being MASTER SEED 666 generator1 = base_generator_type(MASTER_SEED*13); uni_dist1 = boost::uniform_real<>(0,1); uniEmp = new GeneratorI(generator1, uni_dist1); generator2 = base_generator_type(MASTER_SEED*17); uni_dist2 = boost::uniform_real<>(0,1); uniRes = new GeneratorI(generator2, uni_dist2); } protected: base_generator_type generator1, generator2; boost::uniform_real<> uni_dist1, uni_dist2; GeneratorI* uniEmp; GeneratorI* uniRes; };
I would write this very differently, and can give you a better version that should have no problems, but could you please send me the definition of GeneratorI first? Matthias
Hi Matthias Troyer-2 wrote:
I would write this very differently, and can give you a better version that should have no problems, but could you please send me the definition of GeneratorI first?
Sorry about that, the missing typedefs at SearchFunctor:
typedef boost::mt19937 base_generator_type;
typedef boost::variate_generator
On Sep 14, 2009, at 9:40 AM, ruya wrote:
Hi
Matthias Troyer-2 wrote:
I would write this very differently, and can give you a better version that should have no problems, but could you please send me the definition of GeneratorI first?
Sorry about that, the missing typedefs at SearchFunctor:
typedef boost::mt19937 base_generator_type; typedef boost::variate_generator
> GeneratorI;
Another question: what does the destructor of your class look like? Can you send the full class? Matthias
SearchFunctor has a virtual destructor, the derivees have empty destructor.
MemHandler is just a helper does not store anything.
class SearchFunctor{
protected:
typedef SmartPtr<SType> SPtr;
typedef typename SType::Domain Domain;
typedef typename SType::DPtr DPtr;
typedef typename Domain::MType MType;
typedef typename Domain::MPtr MPtr;
typedef typename Domain::EPtr EPtr;
typedef typename Domain::RPtr RPtr;
typedef vector<typename SType::MPtr> MVector;
typedef typename SType::DataType DataType;
typedef boost::mt19937 base_generator_type;
typedef boost::variate_generator
On Sep 14, 2009, at 9:40 AM, ruya wrote:
Hi
Matthias Troyer-2 wrote:
I would write this very differently, and can give you a better version that should have no problems, but could you please send me the definition of GeneratorI first?
Sorry about that, the missing typedefs at SearchFunctor:
typedef boost::mt19937 base_generator_type; typedef boost::variate_generator
> GeneratorI; Another question: what does the destructor of your class look like? Can you send the full class?
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25450374.html Sent from the Boost - Users mailing list archive at Nabble.com.
On Sep 15, 2009, at 10:50 AM, ruya wrote:
SearchFunctor has a virtual destructor, the derivees have empty destructor. MemHandler is just a helper does not store anything.
Here is my guess as to what happens: SPtr points to a SearchFunctor that has been deallocated. The generators were never deleted and can still be used, but the underlying engine got destroyed. Your code contains memory leaks, and could be simplified a lot. Matthias
class SearchFunctor{ protected: typedef SmartPtr<SType> SPtr; typedef typename SType::Domain Domain; typedef typename SType::DPtr DPtr; typedef typename Domain::MType MType; typedef typename Domain::MPtr MPtr; typedef typename Domain::EPtr EPtr; typedef typename Domain::RPtr RPtr; typedef vector<typename SType::MPtr> MVector; typedef typename SType::DataType DataType; typedef boost::mt19937 base_generator_type; typedef boost::variate_generator
> GeneratorI; typedef PrivateMemP::ObjectType::_ACTIVE,MemP MemHandler; public: virtual SPtr operator()( SPtr s, int iter ) = 0;
SearchFunctor(){ memh = MemHandler(); generator1 = base_generator_type(MASTER_SEED+13); uni_dist1 = boost::uniform_real<>(0,1); uniEmp = new GeneratorI(generator1, uni_dist1); generator2 = base_generator_type(MASTER_SEED+17); uni_dist2 = boost::uniform_real<>(0,1); uniRes = new GeneratorI(generator2, uni_dist2); } protected: virtual ~SearchFunctor(){}
base_generator_type generator1, generator2; boost::uniform_real<> uni_dist1, uni_dist2; GeneratorI* uniEmp; GeneratorI* uniRes; MemHandler memh; };
Matthias Troyer-2 wrote:
On Sep 14, 2009, at 9:40 AM, ruya wrote:
Hi
Matthias Troyer-2 wrote:
I would write this very differently, and can give you a better version that should have no problems, but could you please send me the definition of GeneratorI first?
Sorry about that, the missing typedefs at SearchFunctor:
typedef boost::mt19937 base_generator_type; typedef boost::variate_generator
> GeneratorI; Another question: what does the destructor of your class look like? Can you send the full class?
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25450374.html Sent from the Boost - Users mailing list archive at Nabble.com.
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
SType is not a SearchFunctor. It is a template parameter of SearchFunctor. Look, I am not a professional coder or something. I could accept that my code has memory leaks, except that I monitored the memory usage and it is stable and quite small too. I have three other algorithms using the same SearchFunctors and the behavior is as expected and no memory leaks. As for the complication that my code appears to be to you, how would you simplify it? You argued before that you would do it diferently and avoid for sure the segmentation fault. Rui Matthias Troyer-2 wrote:
On Sep 15, 2009, at 10:50 AM, ruya wrote:
SearchFunctor has a virtual destructor, the derivees have empty destructor. MemHandler is just a helper does not store anything.
Here is my guess as to what happens: SPtr points to a SearchFunctor that has been deallocated. The generators were never deleted and can still be used, but the underlying engine got destroyed. Your code contains memory leaks, and could be simplified a lot.
Matthias
template< class SType, typename MemP > class SearchFunctor{ protected: typedef SmartPtr<SType> SPtr; typedef typename SType::Domain Domain; typedef typename SType::DPtr DPtr; typedef typename Domain::MType MType; typedef typename Domain::MPtr MPtr; typedef typename Domain::EPtr EPtr; typedef typename Domain::RPtr RPtr; typedef vector<typename SType::MPtr> MVector; typedef typename SType::DataType DataType; typedef boost::mt19937 base_generator_type; typedef boost::variate_generator
> GeneratorI; typedef PrivateMemP::ObjectType::_ACTIVE,MemP MemHandler; public: virtual SPtr operator()( SPtr s, int iter ) = 0;
SearchFunctor(){ memh = MemHandler(); generator1 = base_generator_type(MASTER_SEED+13); uni_dist1 = boost::uniform_real<>(0,1); uniEmp = new GeneratorI(generator1, uni_dist1); generator2 = base_generator_type(MASTER_SEED+17); uni_dist2 = boost::uniform_real<>(0,1); uniRes = new GeneratorI(generator2, uni_dist2); } protected: virtual ~SearchFunctor(){}
base_generator_type generator1, generator2; boost::uniform_real<> uni_dist1, uni_dist2; GeneratorI* uniEmp; GeneratorI* uniRes; MemHandler memh; };
Matthias Troyer-2 wrote:
On Sep 14, 2009, at 9:40 AM, ruya wrote:
Hi
Matthias Troyer-2 wrote:
I would write this very differently, and can give you a better version that should have no problems, but could you please send me the definition of GeneratorI first?
Sorry about that, the missing typedefs at SearchFunctor:
typedef boost::mt19937 base_generator_type; typedef boost::variate_generator
> GeneratorI; Another question: what does the destructor of your class look like? Can you send the full class?
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25450374.html Sent from the Boost - Users mailing list archive at Nabble.com.
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25468208.html Sent from the Boost - Users mailing list archive at Nabble.com.
On 16 Sep 2009, at 10:38, ruya wrote:
SType is not a SearchFunctor. It is a template parameter of SearchFunctor. Look, I am not a professional coder or something. I could accept that my code has memory leaks, except that I monitored the memory usage and it is stable and quite small too. I have three other algorithms using the same SearchFunctors and the behavior is as expected and no memory leaks. As for the complication that my code appears to be to you, how would you simplify it? You argued before that you would do it diferently and avoid for sure the segmentation fault.
SearchFunctor(){ memh = MemHandler(); generator1 = base_generator_type(MASTER_SEED+13); uni_dist1 = boost::uniform_real<>(0,1); uniEmp = new GeneratorI(generator1, uni_dist1); generator2 = base_generator_type(MASTER_SEED+17); uni_dist2 = boost::uniform_real<>(0,1); uniRes = new GeneratorI(generator2, uni_dist2); } protected: virtual ~SearchFunctor(){}
Just look at this code fragment: you allocate memory in the constructor using new and never release it using delete
Matthias Troyer-2 wrote:
On 16 Sep 2009, at 10:38, ruya wrote:
SType is not a SearchFunctor. It is a template parameter of SearchFunctor. Look, I am not a professional coder or something. I could accept that my code has memory leaks, except that I monitored the memory usage and it is stable and quite small too. I have three other algorithms using the same SearchFunctors and the behavior is as expected and no memory leaks. As for the complication that my code appears to be to you, how would you simplify it? You argued before that you would do it diferently and avoid for sure the segmentation fault.
SearchFunctor(){ memh = MemHandler(); generator1 = base_generator_type(MASTER_SEED+13); uni_dist1 = boost::uniform_real<>(0,1); uniEmp = new GeneratorI(generator1, uni_dist1); generator2 = base_generator_type(MASTER_SEED+17); uni_dist2 = boost::uniform_real<>(0,1); uniRes = new GeneratorI(generator2, uni_dist2); } protected: virtual ~SearchFunctor(){}
Just look at this code fragment: you allocate memory in the constructor using new and never release it using delete
I probably don't see this leak in the memory usage because only a limited
amount of SearchFunctors are created at the beginning and they live for as
long as the program runs. (I think this makes sense)
Anyhow, writing the proper destructors doesn't solve my problem, but now gdb
points to immediatly before the creation of any of the SearchFunctors. So,
to remember, I have three SearchFunctors (SFs). In three different
applications the three of them work as expected. In this particular app, the
SFs were successfully created, then two ran well, and one would give a seg
fault. Now, with the proper destructor in class SearchFunctor, the fault
comes at the time of creation. gdb points to the line where the following
functions is called:
functors = vector<SearchF>();
TL2FunV
On 17 Sep 2009, at 17:11, ruya wrote:
Matthias Troyer-2 wrote:
On 16 Sep 2009, at 10:38, ruya wrote:
SType is not a SearchFunctor. It is a template parameter of SearchFunctor. Look, I am not a professional coder or something. I could accept that my code has memory leaks, except that I monitored the memory usage and it is stable and quite small too. I have three other algorithms using the same SearchFunctors and the behavior is as expected and no memory leaks. As for the complication that my code appears to be to you, how would you simplify it? You argued before that you would do it diferently and avoid for sure the segmentation fault.
SearchFunctor(){ memh = MemHandler(); generator1 = base_generator_type(MASTER_SEED+13); uni_dist1 = boost::uniform_real<>(0,1); uniEmp = new GeneratorI(generator1, uni_dist1); generator2 = base_generator_type(MASTER_SEED+17); uni_dist2 = boost::uniform_real<>(0,1); uniRes = new GeneratorI(generator2, uni_dist2); } protected: virtual ~SearchFunctor(){}
Just look at this code fragment: you allocate memory in the constructor using new and never release it using delete
I probably don't see this leak in the memory usage because only a limited amount of SearchFunctors are created at the beginning and they live for as long as the program runs. (I think this makes sense) Anyhow, writing the proper destructors doesn't solve my problem, but now gdb points to immediatly before the creation of any of the SearchFunctors. So, to remember, I have three SearchFunctors (SFs). In three different applications the three of them work as expected. In this particular app, the SFs were successfully created, then two ran well, and one would give a seg fault. Now, with the proper destructor in class SearchFunctor, the fault comes at the time of creation. gdb points to the line where the following functions is called:
functors = vector<SearchF>(); TL2FunV
::FillVector( functors ); //gdb points fault here but does not go inside template< typename TList, class SearchFun > struct TL2FunV{ static void FillVector( std::vector< SearchFun >& v){ typename TList::Head h; v.push_back( SearchFun(h) ); TL2FunV
::FillVector( v ); } }; template< class SearchFun > struct TL2FunV< Loki::NullType, SearchFun>{ static void FillVector( std::vector< SearchFun >& v ){}; };
Rui
Since you use pointers inside your class you also need to write a copy
constructor and assignment operator creating new generators. I now
understand the reason for your crash:
- you create a SearchFunctor, this creates the engine as a member and
the generators through pointers
- you copy the SearchFunctor in the FillVector function. Using the
implicitly generated copy constructor this just copies the pointers
- after copying the original class gets destroyed - including the
random number engine.
- you call the generator, but the engine no longer exists ->
segmentation fault.
Whenever you allocate memory on a constructor you need to provide a
special destructor, copy constructor and assignment operator. Either
add those, or - much easier, just use the following class:
template< class SType, typename MemP >
class SearchFunctor{
protected:
typedef boost::mt19937 base_generator_type;
typedef boost::variate_generator
Hi The problem was the missing copy constructor, which I usually write by default but for some reason had it commented. I still don't understand why it didn't happened with the remaining algorithms but anyway, thank you for the help on the fault. Now, for the code refactoring, if the generator is static all SFs will share the same generator and the distributions will not be independent anymore, which is a requisite in this kind of algorithm. Regards Rui Lopes Matthias Troyer-2 wrote:
On 17 Sep 2009, at 17:11, ruya wrote:
Matthias Troyer-2 wrote:
On 16 Sep 2009, at 10:38, ruya wrote:
SType is not a SearchFunctor. It is a template parameter of SearchFunctor. Look, I am not a professional coder or something. I could accept that my code has memory leaks, except that I monitored the memory usage and it is stable and quite small too. I have three other algorithms using the same SearchFunctors and the behavior is as expected and no memory leaks. As for the complication that my code appears to be to you, how would you simplify it? You argued before that you would do it diferently and avoid for sure the segmentation fault.
SearchFunctor(){ memh = MemHandler(); generator1 = base_generator_type(MASTER_SEED+13); uni_dist1 = boost::uniform_real<>(0,1); uniEmp = new GeneratorI(generator1, uni_dist1); generator2 = base_generator_type(MASTER_SEED+17); uni_dist2 = boost::uniform_real<>(0,1); uniRes = new GeneratorI(generator2, uni_dist2); } protected: virtual ~SearchFunctor(){}
Just look at this code fragment: you allocate memory in the constructor using new and never release it using delete
I probably don't see this leak in the memory usage because only a limited amount of SearchFunctors are created at the beginning and they live for as long as the program runs. (I think this makes sense) Anyhow, writing the proper destructors doesn't solve my problem, but now gdb points to immediatly before the creation of any of the SearchFunctors. So, to remember, I have three SearchFunctors (SFs). In three different applications the three of them work as expected. In this particular app, the SFs were successfully created, then two ran well, and one would give a seg fault. Now, with the proper destructor in class SearchFunctor, the fault comes at the time of creation. gdb points to the line where the following functions is called:
functors = vector<SearchF>(); TL2FunV
::FillVector( functors ); //gdb points fault here but does not go inside template< typename TList, class SearchFun > struct TL2FunV{ static void FillVector( std::vector< SearchFun >& v){ typename TList::Head h; v.push_back( SearchFun(h) ); TL2FunV
::FillVector( v ); } }; template< class SearchFun > struct TL2FunV< Loki::NullType, SearchFun>{ static void FillVector( std::vector< SearchFun >& v ){}; };
Rui
Since you use pointers inside your class you also need to write a copy constructor and assignment operator creating new generators. I now understand the reason for your crash:
- you create a SearchFunctor, this creates the engine as a member and the generators through pointers - you copy the SearchFunctor in the FillVector function. Using the implicitly generated copy constructor this just copies the pointers - after copying the original class gets destroyed - including the random number engine. - you call the generator, but the engine no longer exists -> segmentation fault.
Whenever you allocate memory on a constructor you need to provide a special destructor, copy constructor and assignment operator. Either add those, or - much easier, just use the following class:
template< class SType, typename MemP > class SearchFunctor{ protected: typedef boost::mt19937 base_generator_type; typedef boost::variate_generator
> GeneratorI; public: SearchFunctor() : uniEmp(base_generator_type(MASTER_SEED+13),boost::uniform_real<> (0,1)) , uniRes(base_generator_type(MASTER_SEED+17),boost::uniform_real<> (0,1)) {}
protected: virtual ~SearchFunctor(){}
GeneratorI uniEmp; GeneratorI uniRes; }
Note that I removed the & after the generator type to avoid a problem with the copy constructor. Better yet, make the generator a static class member and use the &:
template< class SType, typename MemP > class SearchFunctor{ protected: typedef boost::mt19937 base_generator_type; typedef boost::variate_generator
> GeneratorI; public: SearchFunctor() : uniEmp(generator,boost::uniform_real<>(0,1)) , uniRes(generator,boost::uniform_real<>(0,1)) {}
protected: virtual ~SearchFunctor(){} GeneratorI uniEmp; GeneratorI uniRes; private: static base_generator_type generator; }
template< class SType, typename MemP > typename SearchFunctor< SType, MemP>::base_generator_type SearchFunctor< SType, MemP>::generator(MASTER_SEED);
Matthias
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
-- View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25505730.html Sent from the Boost - Users mailing list archive at Nabble.com.
On Sep 18, 2009, at 11:30 AM, ruya wrote:
Hi
The problem was the missing copy constructor, which I usually write by default but for some reason had it commented. I still don't understand why it didn't happened with the remaining algorithms but anyway, thank you for the help on the fault.
You also need to implement the assignment operator
Now, for the code refactoring, if the generator is static all SFs will share the same generator and the distributions will not be independent anymore, which is a requisite in this kind of algorithm.
Actually, as long as the generator is good they are uncorrelated. At least this is what is tested for. This is tested much better than correlations between different seeds. Matthias
Matthias Troyer-2 wrote:
You also need to implement the assignment operator
Actually, as long as the generator is good they are uncorrelated. At least this is what is tested for. This is tested much better than correlations between different seeds.
But isn't the assignment equal to calling the copy constructor by default? What I mean is: if you have obj1 = obj2 then obj1 is created by Object(obj2). As for the generators I was not aware of this. can you give me some link for a technical paper on this? Rui -- View this message in context: http://www.nabble.com/Mersenne-twister-problem-tp25382114p25510845.html Sent from the Boost - Users mailing list archive at Nabble.com.
On 18 Sep 2009, at 17:35, ruya wrote:
Matthias Troyer-2 wrote:
You also need to implement the assignment operator
Actually, as long as the generator is good they are uncorrelated. At least this is what is tested for. This is tested much better than correlations between different seeds.
But isn't the assignment equal to calling the copy constructor by default? What I mean is: if you have obj1 = obj2 then obj1 is created by Object(obj2).
If you have: Object obj1=obj2; then indeed the copy constructor is used, but if you have: Obj obj1; obj1=obj2; then the assignment constructor is called. The default is again a member-wise copy which is wrong in your example.
As for the generators I was not aware of this. can you give me some link for a technical paper on this?
All test suites test correlations within a sequence, but tests between different sequences with different seeds are rare. For lagged Fibonacci generators there are proofs that uncorrelated seed blocks can be generated, but not for other generators. Especially your using MASTER_SEED*13 and MASTER_SEED*17 is completely untested (and in the case MASTER_SEED=0 it would give identical results!) Matthias
participants (2)
-
Matthias Troyer
-
ruya