Large files ( > 2GB) and boost::filesystem library
In a number of places in the filesystem library on Linux, the ::stat file system call is used. When this call is executed on a file whose size is larger than 2GB, it fails because the file size cannot fit into the default "stat" structure. This problem happens even when the size information in the stat structure is not used. For example the stat information is used in the directory iterator to look for sub directories. This can be corrected by setting the macro _FILE_OFFSET_BITS to 64. What is the best way to set this macro on Linux? I have looked at the jam rules and cannot locate the appropriate place to include the definition. Also, what process should be followed if I want to have this macro defined either by default or at least as an option in the boost source tree so that I don't have to modify the build rules each time I install boost?
David Judson <djudson@houston.westerngeco.slb.com> writes:
In a number of places in the filesystem library on Linux, the ::stat file system call is used. When this call is executed on a file whose size is larger than 2GB, it fails because the file size cannot fit into the default "stat" structure. This problem happens even when the size information in the stat structure is not used. For example the stat information is used in the directory iterator to look for sub directories.
This can be corrected by setting the macro _FILE_OFFSET_BITS to 64. What is the best way to set this macro on Linux?
You can add it to your BUILD variable with bjam ... "-sBUILD=<define>_FILE_OFFSET_BITS ..." ... Or you can put <define>_FILE_OFFSET_BITS in your project requirements or default-BUILD.
I have looked at the jam rules and cannot locate the appropriate place to include the definition. Also, what process should be followed if I want to have this macro defined either by default or at least as an option in the boost source tree so that I don't have to modify the build rules each time I install boost?
I don't understand what you have in mind. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
At 08:01 AM 5/3/2004, you wrote:
David Judson <djudson@houston.westerngeco.slb.com> writes:
In a number of places in the filesystem library on Linux, the ::stat file system call is used. When this call is executed on a file whose size is larger than 2GB, it fails because the file size cannot fit into the default "stat" structure. This problem happens even when the size information in the stat structure is not used. For example the stat information is used in the directory iterator to look for sub directories.
This can be corrected by setting the macro _FILE_OFFSET_BITS to 64. What is the best way to set this macro on Linux?
You can add it to your BUILD variable with
bjam ... "-sBUILD=<define>_FILE_OFFSET_BITS ..." ...
Or you can put <define>_FILE_OFFSET_BITS in your project requirements or default-BUILD.
Thanks - that works for me
I have looked at the jam rules and cannot locate the appropriate place to include the definition. Also, what process should be followed if I want to have this macro defined either by default or at least as an option in the boost source tree so that I don't have to modify the build rules each time I install boost?
I don't understand what you have in mind.
I'm thought that it was likely that other users would run up against this same problem. I am suggesting that large file support should be enabled by default when building boost on Linux.
-- Dave Abrahams Boost Consulting http://www.boost-consulting.com
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
David Judson <djudson@houston.westerngeco.slb.com> writes:
I'm thought that it was likely that other users would run up against this same problem. I am suggesting that large file support should be enabled by default when building boost on Linux.
I'm not aware of the implications of that choice, so I have no opinion. -- Dave Abrahams Boost Consulting http://www.boost-consulting.com
David Judson wrote:
At 08:01 AM 5/3/2004, you wrote:
David Judson <djudson@houston.westerngeco.slb.com> writes:
In a number of places in the filesystem library on Linux, the ::stat file system call is used. When this call is executed on a file whose size is larger than 2GB, it fails because the file size cannot fit into the default "stat" structure. This problem happens even when the size information in the stat structure is not used. For example the stat information is used in the directory iterator to look for sub directories.
This can be corrected by setting the macro _FILE_OFFSET_BITS to 64. What is the best way to set this macro on Linux?
You can add it to your BUILD variable with
bjam ... "-sBUILD=<define>_FILE_OFFSET_BITS ..." ...
Or you can put <define>_FILE_OFFSET_BITS in your project requirements or default-BUILD.
Thanks - that works for me
I have looked at the jam rules and cannot locate the appropriate place to include the definition. Also, what process should be followed if I want to have this macro defined either by default or at least as an option in the boost source tree so that I don't have to modify the build rules each time I install boost?
I don't understand what you have in mind.
I'm thought that it was likely that other users would run up against this same problem. I am suggesting that large file support should be enabled by default when building boost on Linux.
-- Dave Abrahams Boost Consulting http://www.boost-consulting.com
This caught my eye since I have code that depends on Large File Support (LFS). Though, for reasons alluded to below, I get LFS via C stdio instead of C++ streams. For the record, the LFS standard is here: http://ftp.sas.com/standards/large.file/ I wonder if there would be complications if LFS were enabled in Boost by default? For instance, how would it work with the GNU standard C++ library? My dated understanding is that some (all?) versions of GNU C++ streams do not support LFS. -- Vince Virgilio
This caught my eye since I have code that depends on Large File Support (LFS). Though, for reasons alluded to below, I get LFS via C stdio instead of C++ streams.
For the record, the LFS standard is here:
http://ftp.sas.com/standards/large.file/
I wonder if there would be complications if LFS were enabled in Boost by default?
I think it is for Boost.Filesystem now - see http://lists.boost.org/MailArchives/boost/msg60670.php John.
At 06:31 AM 5/8/2004, you wrote:
This caught my eye since I have code that depends on Large File Support (LFS). Though, for reasons alluded to below, I get LFS via C stdio instead of C++ streams.
For the record, the LFS standard is here:
http://ftp.sas.com/standards/large.file/
I wonder if there would be complications if LFS were enabled in Boost by default?
I think it is for Boost.Filesystem now - see http://lists.boost.org/MailArchives/boost/msg60670.php
Yes, that's correct. The macro __USE_FILE_OFFSET64 is always defined when BOOST_POSIX is defined. So users of current CVS code don't have to do anything special to get 64-bit support. Note that the LFS standard document above seems to be out-of-date as far as the macro name is concerned. The name was determined on Linux by inspecting the actual header file, and verified by testing. If anyone runs into a system which uses a different macro name, please let us know. Thanks, --Beman
At 04:40 PM 5/8/2004, you wrote:
At 06:31 AM 5/8/2004, you wrote:
This caught my eye since I have code that depends on Large File Support (LFS). Though, for reasons alluded to below, I get LFS via C stdio instead of C++ streams.
For the record, the LFS standard is here:
http://ftp.sas.com/standards/large.file/
I wonder if there would be complications if LFS were enabled in Boost by default?
I think it is for Boost.Filesystem now - see http://lists.boost.org/MailArchives/boost/msg60670.php
Yes, that's correct. The macro __USE_FILE_OFFSET64 is always defined when BOOST_POSIX is defined. So users of current CVS code don't have to do anything special to get 64-bit support.
Note that the LFS standard document above seems to be out-of-date as far as the macro name is concerned. The name was determined on Linux by inspecting the actual header file, and verified by testing. If anyone runs into a system which uses a different macro name, please let us know.
Thanks,
--Beman
I believe that the macro __USE_FILE_OFFSET64 is an internal GNU macro and should not be defined in user code. If you examine the header file "features.h" you will see that this macro is undefined and then possibly redefined based on the value of various standard macros. The LFS standard says that to enable "large" off_t, use the compiler option returned by "getconf LFS_CFLAGS" which on Linux returns -D_FILE_OFFSET_BITS=64. In "features.h", this macro will eventually cause __USE_FILE_OFFSET64 to be defined. The only way that __USE_FILE_OFFSET64 would work is if you set it after including the system header files and it won't work on non-GNU systems. The standard way to do enable 64 bit offsets is set the value returned by "getconf LFS_CFLAGS" . This should be done in the jam build script. This works on Linux, Solaris and Irix. I don't have access to a recent AIX system (5.x) to test.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Fri, May 07, 2004 at 11:09:24PM -0500, Vincent N. Virgilio wrote:
For instance, how would it work with the GNU standard C++ library? My dated understanding is that some (all?) versions of GNU C++ streams do not support LFS.
In G++ 3.4 onwards the std::streamoff type is a 64bit value, so LFS should be enabled on all systems that support it. See the "Runtime Library" section here: http://gcc.gnu.org/gcc-3.4/changes.html jon
participants (6)
-
Beman Dawes -
David Abrahams -
David Judson -
John Maddock -
Jonathan Wakely -
Vincent N. Virgilio