I'm wondering whether this can be generalized. Basically, I imagine a class "virtually" holding objects of some type T, with the restriction that it may not hold more than N actual objects of type T, and holds data<T> objects to create the other objects. You'd have a policy<T> which would create a T from a data<T> and later destroy it. The assumption is that a data<T> would be a lot cheaper than an actual T. Your case would be a specialization with T==FILE* and data<T> being just the file name. The policy<FILE*>::create(data<FILE*>) method would call fopen, the policy<FILE*>::destroy() would fclose it. The default policy<T>::create( data<T> data_T ) can probably call new T( data_T ) and return that. Regards, Michiel Salters
-----Original Message----- From: boost-bounces@lists.boost.org [mailto:boost-bounces@lists.boost.org]On Behalf Of Jody Hagins Sent: vrijdag 16 april 2004 1:57 To: boost@lists.boost.org Subject: [boost] library for managing a large number of FILE objects
I know that C++ IOStreams are supposed to take over the world, and cstdio with FILE is considered taboo by some C++ efficionados. However, due to many reasons, lots of code still uses cstdio FILE I/O. One issue that comes up from time to time is the need to manage more files than the OS will allow a single process to have open simultaneously. I process tons of data, and this is an issue for me. I have developed a small library that provides on-demand caching of FILE pointers, so that an application can "open" as many FILEs as necessary, and use them as normal. A simple LRU eviction algorithm is used to reclaim FILEs when "all" have been used.
I was discussing this library with another developer, and he said he has seen several questions recently about a similar issue, and he advised me to ask if there was interest here. This library, however, does not seem "on the edge" enough though...
A very simple example of how you can use the library (of course there are better ways to do the following, but it is meant to be a small easy to use example).
// Open 10,000 FILEs wjh::file_cache file_cache; std::vector<wjh::cached_fptr> fp; for (int i = 0; i < 10000; ++i) { std::stringstream strm; strm << "FILE_" << i; wjh::cached_fptr fptr = file_cache.open(strm.str().c_str(), "w"); if (!fptr) { std::cerr << strm.str() << ": " << strerror(errno) << std::endl; break; } fp.push_back(fptr); }
// Randomly write to a particular file. for (int i = 0; i < 200000; ++i) { int x = rand() % fp.size(); fprintf(fp[x], "file %d, iteration %d\n", x, i); }
Is it something useful for more than just me (and the pitiful souls who work for me and must use it) on this list?
Is it something worth posting here?
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.