memory mapped io and boost

Dear All, from what it looks like the memory mapped io classes in boost::interprocess cannot deal with readonly files. This is a big problem. I think opening readonly files should be possible but all write operations to the file should fail. Of course mapping a readonly file should be done with the matching flags so that the mapping can be shared and that the matching address space is readonly. Also another drawback is that at least on Windows disk-space is being allocated when growing a file. This is slow and it can be avoided by making the file sparse. On UNIXs this is the default. Thanks Peter

peter_foelsche@agilent.com wrote:
Dear All,
from what it looks like the memory mapped io classes in boost::interprocess cannot deal with readonly files.
Which memory mapped classes? Managed or not? For basic operations like obtain a file mapping: file_mapping(const char *filename, mode_t mode); And map it: template<class MemoryMappable> mapped_region(const MemoryMappable& mapping ,mode_t mode ,offset_t offset = 0 ,std::size_t size = 0 ,const void *address = 0); you can specify mode_t. For managed_mapped_file you can't use read-only, because there are mutexes built in the mapped file that have to be locked in case one process opens creates new elements. So I guess you are referring to managed classes, right? a)The problem with allowing read-only is that another process can open it for writting and you would have problems. b)The internal structure should be changed to avoid using mutexes and that can be difficult because internal structures and user-defined algorithms and indexes should avoid that (and also user-defined objects that write in some read-operations). You also say that disk-space is being allocated when growing a file. That's the idea. Otherwise, it would be the internal "offset_ptr" pointers would become useless unless we map all file portions contiguously and try to manage several mapped regions (which might be a bit tricky). Anyway, I don't see why growing a file should be slower than creating a new one. The OS looks for free disk sectors so file contents are not physically contiguous although you get a single file. Why is making the file sparse an advantage? Regards, Ion
This is a big problem.
I think opening readonly files should be possible but all write operations to the file should fail.
Of course mapping a readonly file should be done with the matching flags so that the mapping can be shared and that the matching address space is readonly.
Also another drawback is that at least on Windows disk-space is being allocated when growing a file.
This is slow and it can be avoided by making the file sparse.
On UNIXs this is the default.
Thanks
Peter
------------------------------------------------------------------------
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Dear All,
from what it looks like the memory mapped io classes in boost::interprocess cannot deal with readonly files.
Which memory mapped classes? Managed or not?
managed
For managed_mapped_file you can't use read-only, because there are mutexes built in the mapped file that have to be locked in case one process opens creates new elements.
I think this (synchronization) is overkill. Most people want memory mapped io to write/read from files. Synchronization is not necessary. Of course creating new elements in a readonly file would fail.
So I guess you are referring to managed classes, right? a)The problem with allowing read-only is that another process can open
it for writting and you would have problems.
this may cause a crash. The file needs to be opened in a way which prevents other openings for writing. Problem is that UNIXs don't allow for locking of files. But at least on windows this can be done.
b)The internal structure should be changed to avoid using mutexes and that can be difficult because internal structures and user-defined algorithms and indexes should avoid that (and also user-defined object s that write in some read-operations).
You also say that disk-space is being allocated when growing a file. That's the idea. Otherwise, it would be the internal "offset_ptr" pointers would become useless unless we map all file portions contiguously and try to manage several mapped regions (which might be a bit tricky). Anyway, I don't see why growing a file should be slower than creating a new one. The OS looks for free disk sectors so file contents are not physically contiguous although you get a single file.
Why is making the file sparse an advantage?
in a sparse file (the default on UNIXs) diskspace is only allocated when needed. So if you create a new file and do fd = open("filename", O_CREAT | O_TRUNC, 0600); seek(fd, 4*1024*1024*1024, SEEK_SET); write(fd, "", 1); only a single sector is being allocated. You can have a huge file on disk (according to the filesize) which in fact takes nearly no disk space. Only when writing into it (e.g. via memory mapped io) disk space is being allocated. If you do this (create an empty big file) on Windows the default is to allocate disk space -- thus the matching system call may be very slow. Only if you set some file attribute this is avoided. Problem is when the system runs out of disk space while writing into memory_mapped memory. On UNIXs you will get a useless signal which gives you the choice of terminating the process. I don't know about Windows but I guess you get a structured exception which can be caught in C++. Peter
Regards,
Ion
This is a big problem.
I think opening readonly files should be possible but all write operations to the file should fail.
Of course mapping a readonly file should be done with the matching f
lags
so that the mapping can be shared and that the matching address spac e is readonly.
Also another drawback is that at least on Windows disk-space is bein g allocated when growing a file.
This is slow and it can be avoided by making the file sparse.
On UNIXs this is the default.
Thanks
Peter
--------------------------------------------------------------------
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users

Sorry for the late reply, peter_foelsche@agilent.com wrote:
I think this (synchronization) is overkill. Most people want memory mapped io to write/read from files. Synchronization is not necessary. Of course creating new elements in a readonly file would fail.
I've added read-only managed classes to my to-do list. The idea would be to create both shared_memory_object/file_mapping/mapped_regions with read-only attributes and avoid internal locking. I'm afraid this will require some important refactoring because I'll need to avoid internal locking calls in the segment manager placed in shared memory, without storing a read-only bit in the segment manager (because it can be later opened for read-write). I can't say when it would be ready, though.
in a sparse file (the default on UNIXs) diskspace is only allocated when needed.
I'll also add this to my to-do list. Thanks for your suggestions. If you have more, please let me know. Regards, Ion

Hi Peter,
Have you looked at the iostreams mapped_file/ read_mapped_file ?
regards
jose
On Fri, May 2, 2008 at 12:38 AM,
Dear All,
from what it looks like the memory mapped io classes in boost::interprocess cannot deal with readonly files.
This is a big problem.
I think opening readonly files should be possible but all write operations to the file should fail.
Of course mapping a readonly file should be done with the matching flags so that the mapping can be shared and that the matching address space is readonly.
Also another drawback is that at least on Windows disk-space is being allocated when growing a file.
This is slow and it can be avoided by making the file sparse.
On UNIXs this is the default.
Thanks
Peter
_______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
participants (3)
-
Ion Gaztañaga
-
Jose
-
peter_foelsche@agilent.com