
After reviewing this, I think I spoke too soon. The above restrictions are consistent with Posix. I had thought that mmap would allow mapping beyond the file current size, but it seems not. One should use ftruncate to set the size before mmap instead.
Since ftruncate isn't portable, it seems something like this (warning: untested) would be appropriate for boost::filesystem (perhaps taking a path object as an argument instead of a string - and shouldn't the Boost IO library do the same?) #ifdef _WIN32 #include <io.h> #else #include <unistd.h> #endif bool create_file(const std::string& path,std::size_t size) { #ifdef _WIN32 int fh=::_open(path.c_str(),_O_CREAT|_O_SHORT_LIVED); if (fh == -1) return false; if (::_chsize(fh,size) == -1) return false; return ::_close(fh) != -1; #else return ::truncate(path.c_str(),size) != -1; #endif } I don't see how, in Windows, to atomically keep the file open before it's mmapped, because chsize takes an int fh and the (equivalent of) mmap takes a HANDLE (maybe there is some Win32 call to get the HANDLE from the int fh?). The abstraction/interface that seems to be missing here is a low level (operating system level) file descriptor that can be passed to platform specific functions like ftruncate or _chsize, which weren't anticipated and wrapped by the higher-level libraries. I'd also like to see a constructor for mapped_file that creates a new file of exactly the required size (this might be a reasonable request for mapped_file_sink as well). It might also be nice to be able to set MAP_SHARED vs MAP_PRIVATE, or to be allowed to atomically deactivate an existing mapped_file (substituting a new backing file), guaranteeing the same address space (this is, I think, feasible if you do no allocation between munmap and mmap, and you can always detect failure by setting MAP_FIXED). I guess there's enough work left to be done for a "proper" Windows/POSIX mmap wrapper that the IO library shouldn't attempt to promote it yet.

"Jonathan Graehl" <jonathan@graehl.org> wrote in message news:41367DC5.5050202@graehl.org...
Since ftruncate isn't portable, it seems something like this (warning: untested) would be appropriate for boost::filesystem (perhaps taking a path object as an argument instead of a string - and shouldn't the Boost IO library do the same?)
I considered using fs::path arguments for the file-based components in the iostreams library, but I decided against it (tentatively) because I didn't want to create a dependency between the libraries and because it made the components slightly harder to use (esp. in the regression tests.)
#ifdef _WIN32 #include <io.h> #else #include <unistd.h> #endif
bool create_file(const std::string& path,std::size_t size) { #ifdef _WIN32 int fh=::_open(path.c_str(),_O_CREAT|_O_SHORT_LIVED); if (fh == -1) return false; if (::_chsize(fh,size) == -1) return false; return ::_close(fh) != -1; #else return ::truncate(path.c_str(),size) != -1; #endif }
For portability it's best to stick to the win32 API. I managed to implement relatively portable file descriptor resources, but discovered that there is enormous variation in the low-level i/o functions provided by the runtime-libraries of different vendors.
I guess there's enough work left to be done for a "proper" Windows/POSIX mmap wrapper that the IO library shouldn't attempt to promote it yet.
Yeah, I'd like the discussion to focus on the architecture of the iostreams extension framework and is implementation. Jonathan

You mean file descriptor, not "int fh". "File handle" usually means a FILE or FILE pointer. Anyway, file descriptors are a Unix feature emulated by many C libraries and not by Win32. The VC++ library provides the function _get_osfhandle which returns the underlying Win32 file handle for a file descriptor. Other C/C++ libraries for Windows may provide similar functions.
I almost never do any Windows specific code, so I didn't realize those _functions were VC++ only; thanks for pointing that out. And yes, I meant descriptor, not handle; that's a useful distinction I'll keep in mind.

For portability it's best to stick to the win32 API.
Sorry ... I didn't realize that was VC++ only. It looks like the Win32 API call to set a file's length directly (without writing out bytes to a newly created file) is: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base... *BOOL* *SetFileValidData(* *HANDLE* /hFile/*, * *LONGLONG* /ValidDataLength/* **); * In any case it looks like this SetFileValidData for win32, and ftruncate for POSIX, could be used by memmap.c to support automatic creation of an appropriately sized backing file in the constructor for mapped_file (and the constructor for mapped output files in the IO lib could expose this). I still think a truncate/SetFileValidData wrapper would be nice for the filesystem lib. I'm willing to code this if people think it's a good idea. It frees users from having to first create an empty file of the right length first.

"Jonathan Graehl" <jonathan@graehl.org> wrote in message news:413795B2.4080301@graehl.org...
In any case it looks like this SetFileValidData for win32,
The docs say this works only for NTFS volumes. The description of the parameter ValidDataLength says: "This parameter must be a positive value that is greater than the current valid data length, but less than the current file size." It looks like SetEndOffFile is the correct function. The remarks say: "This function can be used to truncate or extend a file. If the file is extended, the contents of the file between the old EOF position and the new position are not defined."
and ftruncate for POSIX, could be used by memmap.c to support automatic creation of an appropriately sized backing file in the constructor for mapped_file (and the constructor for mapped output files in the IO lib could expose this).
This could be useful.
I still think a truncate/SetFileValidData wrapper would be nice for the filesystem lib.
I'm willing to code this if people think it's a good idea. It frees users from having to first create an empty file of the right length first.
I'm interested. It sounds like something people want for memory-mapped files. But be careful reading those windows docs! ;-) Jonathan
participants (2)
-
Jonathan Graehl
-
Jonathan Turkanis