Thanks for your code, 

I am working on a multi-computer Open MPI cluster system. 

If I put these data files in /home/mypath/folder, is it possible that all non-head nodes can access the files in the folder ? 

I need to load some data to some nodes, if all nodes can access the data, I do not need to load them to each node one by one. 

If multiple nodes access the same file to get data, is there conflict ? 

For example, 

fopen(myFile) by node 1, at the same time fopen(myFile) by node 2. 

Is it allowed to do that on MPI cluster without conflict ? 

Are there some boost MPI routines to support this ? 

I cannot find them in boost.org.

Any help is appreciated. 

Jack

July 12  2010

From: binglongx@gmail.com
To: boost-users@lists.boost.org
Date: Tue, 13 Jul 2010 00:01:52 -0400
Subject: Re: [Boost-users] boost C++ load data from file

It's a wide shot without an example of your specific data organization, but this is something that might give your desired indexing (being efficient or not is disregarded here):
 
using namespace std; // excuse: just for brevity
typedef ... POWC; // What's the underlying type of this thing?
typedef map<POWC, float> PowcFloat;
typedef map<int, PF> YearPowcFloat;
typedef map<string, YearPowcFloat> Jumbo;
 
void load(const vector<string>& states, const vector<int>& years, const vector<POWC>& powcs, Jumbo& big)
{
 big.clear();
 BOOST_FOREACH(const string& state, states)
 {
  YearPowcFloat ypf;
  BOOST_FOREACH(int year, years)
  {
   PowcFloat pf;
   ifstream((state+boost::lexical_cast<string>(year)).c_str());
   for(size_t i=0; i<40; ++i)
   {
    pair<POWC,float> ppf = ...; // populate ppf with each float from file
    pf.insert(ppf);
   }
   ypf.insert(pair<int,PF>(year,pf));
  }
  big.insert(pair<string,YearPowcFloat>(state,ypf));
 }
}
 
void test_load()
{
 vector<string> states;
 states.push_back("New York");
 states.push_back("Pennsylvania");
 vector<int> years;
 years.push_back(2009);
 years.push_back(2010);
 vector<POWC> powcs(40); // ...
 
 Jumbo big;
 load(states, years, powcs, big);
 cout<<"New York 2010 powcs[0] = "<< big["NewYork"][2010][powcs[0]] << endl;
}
 
 
 
 

From: Jack Bryan
Sent: Monday, July 12, 2010 3:04 PM
To: boost-users@lists.boost.org
Subject: [Boost-users] boost C++ load data from file

Dear All,

I need to load data (24 X 28 X 40) from 24 X 28 files into a datastructure so that my program can use
them to do operations. 

Each file has 40 data items, which are floating numbers.

I hope that the data structure can be indexed by item name not only by integer.

For example, for 24 states in USA, each state's power consumption is POWC in year Y. 

I hope that the data structure indexing is like this  

datastructure[newYork][year][powerconsumption]

My question is :

(1) how to load the data from 24 x 28 files by boost C++? 

It is boring to type file's name one by one in program. 

Is there an alternative to load the data by indexing the file names ? 

for example, 

for (filename = name1; filename < filenum ; filename++)
{
load data from filename;
store loaded data in the datastructure.
}

(2) how to index the item of the datastructure by boost C++ ? 


Any help is appreciated. 

Jack 

July 11 2010

Hotmail is redefining busy with tools for the New Busy. Get more from your inbox. See how.

_______________________________________________
Boost-users mailing list
Boost-users@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users
The New Busy is not the old busy. Search, chat and e-mail from your inbox. Get started.