Jonathan Turkanis wrote:
Roman Dementiev wrote:
Thorsten Ottosen wrote:
"Roman Dementiev" <dementiev@ira.uka.de> wrote in message
yes. One must rewrite only the lower layer of Stxxl using native file access methods and native multithreading.
is that because you need mem.mapped files? Otherwise, why don't you use boost filsystem and boost threads?
I do not necessarily need memory mapped files.
The Boost filesystem library is more about manipulating files and directories. But what I need is the file access itself: create/open/read/write/close file. I would have been using std::fstream as a portable file access method, but it lacks support of files larger than 2 GB. It is really big disadvantage in my case.
The boost iostreams library, which is in CVS now and will be part of the next boost release, contains a class file_descriptor for accessing files using OS or runtime library file descriptors.
http://www.kangaroologic.com/iostreams/libs/iostreams/doc/?path=6.1.3
Currently, random access if performed using offsets of type std::streamoff, which is often implemented as a 32-bit unsigned long. However, I recently decided to relax this restriction and allow offsets to be specified using larger intergral type. On windows, seeking will be implemented using _lseek64 where available. On Posix, lseek will be used; it uses off_t, which is (I think) often 64 bits.
Under Linux if preprocessor macros _LARGEFILE_SOURCE, _LARGEFILE64_SOURCE, _FILE_OFFSET_BITS=64 are defined then off_t is 64 bit, otherwise it is 32 bit.
Access via iostreams will still have to use std::streamoff, since that is out of my control; but you should be able to use an instance of file_descriptor directly.
The new code should be available soon.
I think boost::iostreams::file_descriptor is what I need. Of course with 64 bit seek. Another reason why I do not like std::fstream is its bad performance. It introduces a superfluous copying into it's internal buffer when doing I/O on user buffer. Another issue is that the performance of my library will benefit from unbuffered file system access. Many operating system support it (O_DIRECT open option in Unix systems, FILE_FLAG_NO_BUFFERING|FILE_FLAG_WRITE_THROUGH option for "CreateFile" Windows API call). Roman