On 2012-08-07 06:38, Vitaly Budovski wrote:

On 7 August 2012 02:43, Heiko Klein wrote:

...

Hi,

I wanted to test the mapped_file_source iostream device. My code is working well with a normal file_source device and files >= 6GB. When I change the constructor to a mapped_file_source, it works still well for smaller files (e.g. 150MB), but for files > 2GB I get (tested with 2.5GB):

exception occured: failed getting file size: Value too large for defined data type

And for files > 4GB (tested with 6GB) I get:

exception occured: failed opening file: Bad file descriptor

My constructor is as follows:

using namespace boost::iostreams; typedef stream mmStream; mmStream* mmistream = new mmStream(); mmistream->open(mapped_file_source(fileName));

I've tested against boost 1.44 and a 32bit linux (Ubuntu 10.04).

Are these general mmap problems, or am I missing some flags for mapped_file_source?

Best regards,

Heiko

Hello,

32bit systems have only approximately 3GB of user addressable memory. You should upgrade to a 64bit system.

Regards,

Vitaly

Vitaly, thanks for your explanation. This makes sense for the problems with the 6GB file. But it does not explain the 'failed getting file size: Value too large for defined data type' with the 2.5GB file. Do you have an idea why this operation fails? Since I cannot upgrade to 64bit everywhere, I catch now the exception and open the stream again as file_source. Heiko

I checked now the boost-sources. My version was 1.40 not 1.44 as I thought previously. 1.40 does not have build-in large-file support, restricting it to 2GB files, this has been corrected in later version, e.g. from a diff to 1.41: #ifdef _LARGEFILE64_SOURCE flags |= O_LARGEFILE; #endif Heiko On 2012-08-07 08:58, Heiko Klein wrote:

On 2012-08-07 06:38, Vitaly Budovski wrote:

...

On 7 August 2012 02:43, Heiko Klein wrote:

...

Hi,

I wanted to test the mapped_file_source iostream device. My code is working well with a normal file_source device and files >= 6GB. When I change the constructor to a mapped_file_source, it works still well for smaller files (e.g. 150MB), but for files > 2GB I get (tested with 2.5GB):

exception occured: failed getting file size: Value too large for defined data type

And for files > 4GB (tested with 6GB) I get:

exception occured: failed opening file: Bad file descriptor

My constructor is as follows:

using namespace boost::iostreams; typedef stream mmStream; mmStream* mmistream = new mmStream(); mmistream->open(mapped_file_source(fileName));

I've tested against boost 1.44 and a 32bit linux (Ubuntu 10.04).

Are these general mmap problems, or am I missing some flags for mapped_file_source?

Best regards,

Heiko

Hello,

32bit systems have only approximately 3GB of user addressable memory. You should upgrade to a 64bit system.

Regards,

Vitaly

Vitaly,

thanks for your explanation. This makes sense for the problems with the 6GB file. But it does not explain the 'failed getting file size: Value too large for defined data type' with the 2.5GB file. Do you have an idea why this operation fails?

Since I cannot upgrade to 64bit everywhere, I catch now the exception and open the stream again as file_source.

Heiko

-- Dr. Heiko Klein Tel. + 47 22 96 32 58 Development Section / IT Department Fax. + 47 22 69 63 55 Norwegian Meteorological Institute http://www.met.no P.O. Box 43 Blindern 0313 Oslo NORWAY