Eli Zakashansky escribió:

Joaquín hi,

First thanks for your response.

The iterators I use are not persistent (I am retrieving the iterators from the index each time I am adding or removing element to/from the container). And even though the crash happens during massive use of the container, it does not happens in the middle of multi_index_container op (I have an applications log where I can see when exactly where in the code the app. crashes). I thought that may be the multi_index_container shall be flushed to the disk (it will effect performance) but I didnt find any method to do it.

For more information I would like to tell you in general about the test I do: 1. at first I populate my multi_index_container with elements (~4000). 2. I invoke a process which removes elements from the container (and processing each element after its removal - this phase doesn't relate to any boost structure ). During stage # 2 I am killing the process via task manager.

Umm, maybe we're looking at the wrong area and this has nothing to do with persistence. Consider the following piece of pseudocode: multi_index_container<...> m; auto it=m.find(...); m.erase(it); auto it2=it1; // WRONG See the last line? Once you've deleted the element a given iterator points at, this iterator becomes invalid and the only thing you can do with it is assign it a new value --you can't copy it, use it as the right part of an assignment, etc. If safe mode is on, assertions like the one you're seeing will pop up. Maybe you're having such a problem in your code? Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

[Please don't top-post, see http://www.boost.org/community/policy.html#quoting ] Eli Zakashansky escribió:

Joaquín hi,

I understand perfectly what u tried to do (and also experienced such problems before), therefore I verified that after I obtain the index(idx = container->get<INDEX>();) it is not changes (only read methods such as end(), begin() etc. are invoked) until I try to remove an element from the container (idx.erase(iterator);), and on the first try I try to call erase the assertion pops up.

Well, where does iterator come from? Is there any chance that a) it's end() because the lookup condition was not met? b) it's invalid because you already deleted the element it was pointing to? c) it belongs to a different container? Have you swapped containers, maybe? Is this systematic, i.e. does it happen the first time that particular line of code is hit, or does it happen more or less at random? Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

Eli Zakashansky escribió:

Joaquín hi,

a) it's end() because the lookup condition was not met? - I am checking it before trying to erase. b) it's invalid because you already deleted the element it was pointing to? - No, I am looking for a record and then the first time I try to erase it, it happens. c) it belongs to a different container? Have you swapped containers, maybe? -No way.

I've checked it 1000 times, and this is the same code that is used in my regular run and I had never experienced such problems (in this phase of the flow, this piece of code). Only when testing after crash the assertion pops up in 100% of the cases.

Hi Eli, first of all please don't top-post, it's again this list posting rules, see http://www.boost.org/community/policy.html#quoting . I'm afraid I can't be of more help with the information you've got. From your description of the problem, the container can't possibily be corrupted because you know that crashes never happen in the middle of an op container. Also, iterators are not persisted and they seemingly work OK except when recovering after a crash. I don't see where the problem might be, though a problem certainly is there. Unless you can provide more info and/or some sample code I don't see what else to try from my position. From yours you might want to debug the program harder and/or set Boost.MultiIndex invariant-checking mode (warning: extremely time-consuming) to validate your hypothesis that crashes do not corrupt the container. Best regards, Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

Hello, After asking my question the first time (a few month ago) Andrew and Joaquín both tried to help me, but unfortunately I still didn't find problem's root cause / solution. Partially it might be because I did not supplied enough information therefore I wrote a short-simple-single threaded console app that might help you reproduce the problem / find my mistake (this of course not a production app therefore there are a lot of hard-coded constants etc.). The apps code contains 3 files " BoostTestingApp.cpp", " DataContainer.h", and "DataContainer.cpp" which could be found below. But first the instructions: To create the "damaged" data file use the following input: 2 (insert) -1 (maximum) 5 (empty) - during this operation kill proc manually from the program manager (yes I am using windows) which simulates the crush. To reproduce the problem (assertion window pops up) use the following inputs (restart the app using debug after creating the damaged file) : Input #1: 4 (delete all) Input #2: 3 (delete) 500000 (this particular record) The program : BoostTestingApp.cpp ------------------ #include "stdafx.h" #include "DataContainer.h" #include <iostream> #include using namespace std; int ID; void insert(CDataContainer* dataContainer) { cout<<"how many rows"<>rows; rows = (rows==-1)?1300000:rows; bool success = true; for (int i=0;iInsert(data); ID++; } if (success) { cout<<"command finished successfully"<>key.id; if (dataContainer->GetAndDelete(key,&entry,true)) cout<<"command finished successfully"<m_dataTable->size()!=0) { CDataContainer::MainIDIndex& idx= dataContainer->m_dataTable->get(); CDataContainer::MainIDIndex::const_iterator it = idx.begin(); dataContainer->m_dataTable->erase(it); } cout<<"command finished successfully"<Initialize(); int option; int max_size =0; bool loop = true; while (loop) { cout<<"options:"<>option; switch(option) { case 1: cout<<"current size : "<<(int)dataContainer->m_dataTable->size()<m_dataTable->clear(); cout<<"finished"< #include #include #include #include #include #include #include "Windows.h" #include "stdafx.h" using boost::multi_index_container; using namespace boost::multi_index; typedef struct { int id;//key1 int key2; int key3; int key4; int key5; char text[20]; } DataBasicStructure; struct DataKey { DataKey(){} DataKey(int pid):id(pid){} int id; }; struct DataEntry{ DataBasicStructure data; int ID()const { return data.id; } int Key2()const { return data.key2; } int Key3()const { return data.key3; } int Key4()const { return data.key4; } int Key5()const { return data.key5; } char Text() const { return data.text[0]; } }; class ContainerLocker; typedef indexed_by< ordered_unique, ordered_non_unique, ordered_non_unique, ordered_non_unique, ordered_non_unique> ContainerIndexSpecifierList; typedef boost::interprocess::allocator ContainerAllocator; typedef multi_index_container data_container; #define MAIN_ID 0 #define KEY_2 1 #define KEY_3 2 #define KEY_4 3 #define KEY_5 4 typedef nth_index_iterator::type Data_Iterator; class CDataContainer { public: typedef nth_index::type MainIDIndex; typedef nth_index::type Key2Index; typedef nth_index::type Key3Index; typedef nth_index::type Key4Index; typedef nth_index::type Key5Index; bool m_isInitialized; long DataTableMaxEntries; boost::interprocess::managed_mapped_file *m_containerMapFile; data_container *m_dataTable; bool Init(const char *fileName); void BackUpFile( const char * fileName ); class IteratorLocation { friend class CDataContainer; public: bool regetRange; IteratorLocation() { regetRange = true; } std::pair m_MainIDIterator; std::pair m_key2Iterator; std::pair m_key3Iterator; std::pair m_key4Iterator; std::pair m_key5Iterator; }; BOOL Initialize(); bool Insert( const DataBasicStructure &data); bool GetAndDelete(DataKey key, DataEntry *data, bool del); }; DataContainer.cpp ----------------- #include "DataContainer.h" bool CDataContainer::Init( IN const char *fileName) { bool returnStatus = true; int count = 0; int dataRequiredSize = DataTableMaxEntries*sizeof(DataEntry); dataRequiredSize += ((int)(0.1*dataRequiredSize) > 100*1024*1024) ? (int)(0.1*dataRequiredSize) : 100*1024*1024; dataRequiredSize *= 3; try { m_containerMapFile = new boost::interprocess::managed_mapped_file(boost::interprocess::open_only,fileName); bool suc = true; suc &= (m_containerMapFile != NULL); suc &= (m_containerMapFile->get_size()== dataRequiredSize); suc &= m_containerMapFile->check_sanity(); if (suc) { bool initSucceed = true; m_dataTable = m_containerMapFile->find_or_construct(fileName)( data_container::ctor_args_list(), data_container::allocator_type(m_containerMapFile->get_segment_manager())); if (NULL!=m_dataTable) { if (m_dataTable->size() > DataTableMaxEntries) m_containerMapFile = NULL; } initSucceed = (NULL!=m_containerMapFile)?true:false; if (!initSucceed) m_containerMapFile = NULL; } else m_containerMapFile = NULL; } catch (...) { m_containerMapFile = NULL; } try { if (m_containerMapFile == NULL) { std::string backupFile(fileName); backupFile.append("_bak"); remove(backupFile.c_str()); rename(fileName, backupFile.c_str()); m_containerMapFile = new boost::interprocess::managed_mapped_file( boost::interprocess::create_only,fileName, dataRequiredSize); m_dataTable = m_containerMapFile->construct(fileName)( data_container::ctor_args_list(), data_container::allocator_type(m_containerMapFile->get_segment_manager())); } m_isInitialized = true; } catch (...) { m_isInitialized = false; returnStatus = false; } return returnStatus; } void CDataContainer::BackUpFile( const char * fileName ) { std::string backupFile(fileName); backupFile.append("_bak"); remove(backupFile.c_str()); rename(fileName, backupFile.c_str()); } bool CDataContainer::Insert( IN const DataBasicStructure &data) { bool rv = true; if ((int)m_dataTable->size() < DataTableMaxEntries) { DataEntry entry; entry.data = data; try { m_dataTable->insert(entry); } catch (...) { rv = false; } } else { rv = false; } return rv; } bool CDataContainer::GetAndDelete( DataKey key, DataEntry *data,bool del) { bool rc = true; MainIDIndex& idx=m_dataTable->get(); MainIDIndex::iterator it = idx.find(key.id); if (it == idx.end()) rc = false; else { *data = *it; if (del) { idx.erase(it); } } return rc; } BOOL CDataContainer::Initialize() { BOOL result = TRUE; DataTableMaxEntries = 1300000; if (Init("boost_Testing") != true) return FALSE; return result; } Thank you very much, Eli.

Joaquin hi, Thanks a lot for you quick response (again!). It is hard to understand what is the problem. What I think that the difference between you and I might be the fact that I used static linking (with boost lib files) . In case you fail to run the program again try to use a VC project which can be downloaded at https://rcpt.yousendit.com/808014672/0cd55316ab7da75c1c45acf265f9df31 as baseline (don't forget changing the include folders path). Thanks a lot, Eli Zakashansky

Joaquin hi, I've opened your project in VS2005, added the required lib files (libboost_date_time-vc80-mt-s-1_41.lib and libboost_date_time-vc80-mt-sgd-1_41.lib) which I compiled especially for this test, and besides the additional directories I've changed the property RuntimeLibrary which I set to be "0" [Project properties -> Configuration properties -> Code Generation Runtime Library <= Multi-threaded(/MT)]. I was surprised twice: 1. I've executed the program with no difficulties (didn't reproduced what you've mentioned about endless loops). 2. I didn't reproduced the initial problem which I've complained about (the assertions). Which makes me think that this a result of the VC9 compilation. I've did those tests with several boost versions (1.35,1.39,1.41) and reproduced the problem in 100% of the cases, but always compiled with VC9 compiler, changing the compiler is not in option in my case therefore I must find a solution to this problem. I'm glad that now I might be more focused about where the problem is, but have no clue in how to solve it. Any new ideas ? Eli Zakashansky. ________________________________________ Your project is for MSVC 9.0, alas I've got MSVC 8.0 here. I've rebuilt with static linking and the problem persists. Maybe you can take a look yourself, please find attached the MSVC 8.0 solution, which hopefully you can open with MSVC 9.0 See the same there? Joaquín M López Muñoz Telefónica, Investigación y Desarrollo

Eli Zakashansky writes:

Joaquin hi,

I've opened your project in VS2005, added the required lib files (libboost_date_time-vc80-mt-s-1_41.lib and libboost_date_time-vc80-mt-sgd-1_41.lib) [...]

These libs are not really needed, you can get rid of them by defining BOOST_ALL_NO_LIB. Does this make any difference wrt to the 1, below?

I was surprised twice:

1. I've executed the program with no difficulties (didn't reproduced what you've mentioned about endless loops).

See above.

2. I didn't reproduced the initial problem which I've complained about (the assertions). Which makes me think that this a result of the VC9 compilation. [...] Any new ideas ?

Not really, your results are not consistent with mine and seemingly we're compiling the same project. I don't get why your program does not get stuck om startup. You might also want to try Andrew's sensible suggestion. I'm sorry I can't be more helpful, please keep me updated if some idea or new fact arises. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo