RE: [boost] A library: out-of-core containers and algorithms
-----Original Message----- On Behalf Of Roman Dementiev Subject: [boost] A library: out-of-core containers and algorithms
Hello,
I am developing a library called Stxxl. It is an implementation of STL for external memory (out-of-core) computations, i.e. Stxxl implements containers and algorithms that can process huge volumes of data that only fit on disks.
Currently I have implemented vector, stack, and priority_queue. External memory map, list, and queue are coming soon. The containers take only specified (given) fixed amount of main memory, but can contain more elements than can fit into the main memory. The containers are compatible with STL algorithms, an example:
#include <stxxl>
// ten billion doubles stxxl::vector<double> HugeVector(10ULL * 1000000000ULL); std::fill(HugeVector.begin(), HugeVector.end(), 0.0);
[Brian Braatz] Roman- Thank you for submitting this. I am interested in this library and encourage you onward in bringing this to boost. Question: In a nutshell, if you could please describe * How it works * what the implications\dark corners are Are you using memory map files? What other libraries or os specific functions does your library need. Thank You Brian Braatz
Brian Braatz wrote:
-----Original Message----- On Behalf Of Roman Dementiev Subject: [boost] A library: out-of-core containers and algorithms
Hello,
I am developing a library called Stxxl. It is an implementation of STL for external memory (out-of-core) computations, i.e. Stxxl implements containers and algorithms that can process huge volumes of data that only fit on disks.
Currently I have implemented vector, stack, and priority_queue.
External
memory map, list, and queue are coming soon. The containers take only specified (given) fixed amount of main memory, but can
contain
more elements than can fit into the main memory. The containers are compatible with STL algorithms, an example:
#include <stxxl>
// ten billion doubles stxxl::vector<double> HugeVector(10ULL * 1000000000ULL); std::fill(HugeVector.begin(), HugeVector.end(), 0.0);
[Brian Braatz] Roman- Thank you for submitting this. I am interested in this library and encourage you onward in bringing this to boost.
thank you
Question: In a nutshell, if you could please describe * How it works * what the implications\dark corners are
what exactly do want to know? Try to look at the draft of the Stxxl tutorial: http://i10www.ira.uka.de/dementiev/files/stxxl_tutorial.pdf
Are you using memory map files?
What other libraries or os specific functions does your library need.
my library can use memory maped files, but it can also rely on a basic I/O calls: open/read/write ... the current implementation the library needs pthreads and support of POSIX file system calls (open/read/write/close). As I already mentioned in other postings, I would use boost threads instead of pthreads to make the library more portable and boost compliant. What concerns I/O, the library needs: 1) large file (>2GB) support (e.g. FAT32 does not support large files) 2) avoidance of superfluios buffering (like in std::fstream) I it also desirable to have (for performance reasons): 3*) unbuffered file access (ASAIK Windows does have it) Roman
On Tue, Feb 08, 2005 at 03:01:55PM +0100, Roman Dementiev wrote:
What other libraries or os specific functions does your library need.
Brian Braatz wrote: the current implementation the library needs pthreads and support of POSIX file system calls (open/read/write/close).
As I already mentioned in other postings, I would use boost threads instead of pthreads to make the library more portable and boost compliant.
What concerns I/O, the library needs: 1) large file (>2GB) support (e.g. FAT32 does not support large files) 2) avoidance of superfluios buffering (like in std::fstream)
std::fstream f; f.rdbuf()->setbuf(0,0); What's wrong with doing that? (apart from the fact it underflows for every character.) jon
participants (3)
-
Brian Braatz -
Jonathan Wakely -
Roman Dementiev