If you use the C standard library function fopen, your file IO has nothing to do with MPI. You'd better check the file sharing/mapping support of the underlying OS of your MPI nodes. Do you want to check with MPI-IO? From: Jack Bryan Sent: Monday, July 12, 2010 9:01 PM To: boost-users@lists.boost.org Subject: Re: [Boost-users] boost C++ load data from file thanks I am working on a multi-computer MPI cluster system. If I put some data files in /home/mypath/folder, is it possible that all non-head nodes can access the files in the folder ? I need to load some data to some nodes, if all nodes can access the data, I do not need to load them to each node one by one. If multiple nodes access the same file to get data, is there conflict ? For example, fopen(myFile) by node 1, at the same time fopen(myFile) by node 2. Is it allowed to do that on MPI cluster without conflict ? thansk Jinxu Ding July 12 2010

Date: Tue, 13 Jul 2010 02:35:53 +0200 From: theom3ga@gmail.com To: boost-users@lists.boost.org Subject: Re: [Boost-users] boost C++ load data from file

Hi Jack,

...

(1) how to load the data from 24 x 28 files by boost C++? It is boring to type file's name one by one in program. Is there an alternative to load the data by indexing the file names ?

You can use boost.filesystem to iterate over all the files contained in a folder.

...

(2) how to index the item of the datastructure by boost C++ ?

I don't really get how the file structure is supposed to be. Could you clarify it, maybe including a small example and/or file and folder hierachy?

...

Hotmail is redefining busy with tools for the New Busy.

I don't think hotmail is redefining anything. Well, yes, mediocrity. Get yourself a real mail account, like GMail.

-- José Tomás Tocino García _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

-------------------------------------------------------------------------------- Hotmail is redefining busy with tools for the New Busy. Get more from your inbox. See how. -------------------------------------------------------------------------------- _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Thanks for your code, I am working on a multi-computer Open MPI cluster system. If I put these data files in /home/mypath/folder, is it possible that all non-head nodes can access the files in the folder ? I need to load some data to some nodes, if all nodes can access the data, I do not need to load them to each node one by one. If multiple nodes access the same file to get data, is there conflict ? For example, fopen(myFile) by node 1, at the same time fopen(myFile) by node 2. Is it allowed to do that on MPI cluster without conflict ? Are there some boost MPI routines to support this ? I cannot find them in boost.org. Any help is appreciated. Jack July 12 2010 From: binglongx@gmail.com To: boost-users@lists.boost.org Date: Tue, 13 Jul 2010 00:01:52 -0400 Subject: Re: [Boost-users] boost C++ load data from file It's a wide shot without an example of your specific data organization, but this is something that might give your desired indexing (being efficient or not is disregarded here): using namespace std; // excuse: just for brevity typedef ... POWC; // What's the underlying type of this thing? typedef map PowcFloat; typedef map YearPowcFloat; typedef map Jumbo; void load(const vector<string>& states, const vector<int>& years, const vector<POWC>& powcs, Jumbo& big) { big.clear(); BOOST_FOREACH(const string& state, states) { YearPowcFloat ypf; BOOST_FOREACH(int year, years) { PowcFloat pf; ifstream((state+boost::lexical_cast<string>(year)).c_str()); for(size_t i=0; i<40; ++i) { pair ppf = ...; // populate ppf with each float from file pf.insert(ppf); } ypf.insert(pair(year,pf)); } big.insert(pair(state,ypf)); } } void test_load() { vector<string> states; states.push_back("New York"); states.push_back("Pennsylvania"); vector<int> years; years.push_back(2009); years.push_back(2010); vector<POWC> powcs(40); // ... Jumbo big; load(states, years, powcs, big); cout<<"New York 2010 powcs[0] = "<< big["NewYork"][2010][powcs[0]] << endl; } From: Jack Bryan Sent: Monday, July 12, 2010 3:04 PM To: boost-users@lists.boost.org Subject: [Boost-users] boost C++ load data from file Dear All, I need to load data (24 X 28 X 40) from 24 X 28 files into a datastructure so that my program can use them to do operations. Each file has 40 data items, which are floating numbers. I hope that the data structure can be indexed by item name not only by integer. For example, for 24 states in USA, each state's power consumption is POWC in year Y. I hope that the data structure indexing is like this datastructure[newYork][year][powerconsumption] My question is : (1) how to load the data from 24 x 28 files by boost C++? It is boring to type file's name one by one in program. Is there an alternative to load the data by indexing the file names ? for example, for (filename = name1; filename < filenum ; filename++) { load data from filename; store loaded data in the datastructure. } (2) how to index the item of the datastructure by boost C++ ? Any help is appreciated. Jack July 11 2010 Hotmail is redefining busy with tools for the New Busy. Get more from your inbox. See how. _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users _________________________________________________________________ The New Busy is not the old busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:W...