Thread Subject: strangle limitation of memory mapped files

Under the title "Accessing Files with Memory-Mapping", we read in the
"Limitations..."

"The 2 GB limit also applies to 64-bit platforms"

I can't think of any good reason for this. On of the good point of 64-
bit OS is to be able to map an entire (very large) file to the address
space and let the OS do the paging operations for us. It takes 30
minutes to code it in C++ (using boost library), but it'd be nice to
have it "native" in Matlab. But why again do I see the nagging 2GB
limit in the Matlab implementation? Is it something that can/will be
fixed in the future releases? Thanks.

In article <eb9b26b2-bc49-487e-a538-bca7e632b873@k37g2000hsf.googlegroups.com>,
rych <rychphd@gmail.com> wrote:
>Under the title "Accessing Files with Memory-Mapping", we read in the
>"Limitations..."

>"The 2 GB limit also applies to 64-bit platforms"

>I can't think of any good reason for this. On of the good point of 64-
>bit OS is to be able to map an entire (very large) file to the address
>space and let the OS do the paging operations for us.


Eeee, gross. :( MS Windows does not have a POSIX mmap function;
the function that it does have is CreateFileMapping(), which takes
a pair of DWORD (32 bit values) as the high and low words of
the size of the object. So MS Windows could do it, even if not
in the same way as POSIX.

In POSIX, the final parameter, the offset to be mapped, is of
off_t, which is an implementation defined type. Older Unix versions
use a 32 bit off_t; newer versions use a 64 bit off_t. Some
unixes have mmap64() calls that use a 64 bit offset with
mmap() being stuck at 32 bits for backwards compatability.

Looks like it could possibly be done, with some special casing.
I am, though, not taking into account issues about the placement
of DLLs or shared objects in the virtual address space that might
interfer with the objective.
--
  "Allegories are in the realm of thoughts, what ruins are in
  the realm of things." -- Walter Benjamin

Walter Roberson wrote:
> In article <eb9b26b2-bc49-487e-a538-bca7e632b873@k37g2000hsf.googlegroups.com>,
> rych <rychphd@gmail.com> wrote:
>> Under the title "Accessing Files with Memory-Mapping", we read in the
>> "Limitations..."
>
>> "The 2 GB limit also applies to 64-bit platforms"
>
>> I can't think of any good reason for this. On of the good point of 64-
>> bit OS is to be able to map an entire (very large) file to the address
>> space and let the OS do the paging operations for us.
>
>
> Eeee, gross. :( MS Windows does not have a POSIX mmap function;
> the function that it does have is CreateFileMapping(), which takes
> a pair of DWORD (32 bit values) as the high and low words of
> the size of the object. So MS Windows could do it, even if not
> in the same way as POSIX.
>
> In POSIX, the final parameter, the offset to be mapped, is of
> off_t, which is an implementation defined type. Older Unix versions
> use a 32 bit off_t; newer versions use a 64 bit off_t. Some
> unixes have mmap64() calls that use a 64 bit offset with
> mmap() being stuck at 32 bits for backwards compatability.
>
> Looks like it could possibly be done, with some special casing.
> I am, though, not taking into account issues about the placement
> of DLLs or shared objects in the virtual address space that might
> interfer with the objective.


First off, sorry for the late response to this. This appears to be an
oversight in our documentation; The 32-bit limitation on memmapfile for
64-bit operating systems no longer exists as of R2007a (MATLAB version 7.4).

Sincerely,
Scott French
The MathWorks, Inc.

On Jul 16, 3:28=A0pm, Scott French <sc...@mathworks.com> wrote:
> Walter Roberson wrote:
> > In article <eb9b26b2-bc49-487e-a538-bca7e632b...@k37g2000hsf.googlegrou=
ps.com>,
> > rych =A0<rych...@gmail.com> wrote:
> >> Under the title "Accessing Files with Memory-Mapping", we read in the
> >> "Limitations..."
>
> >> "The 2 GB limit also applies to 64-bit platforms"
>
> >> I can't think of any good reason for this. On of the good point of 64-
> >> bit OS is to be able to map an entire (very large) file to the address
> >> space and let the OS do the paging operations for us.
>
> > Eeee, gross. :( MS Windows does not have a POSIX mmap function;
> > the function that it does have is CreateFileMapping(), which takes
> > a pair of DWORD (32 bit values) as the high and low words of
> > the size of the object. So MS Windows could do it, even if not
> > in the same way as POSIX.
>
> > In POSIX, the final parameter, the offset to be mapped, is of
> > off_t, which is an implementation defined type. Older Unix versions
> > use a 32 bit off_t; newer versions use a 64 bit off_t. Some
> > unixes have mmap64() calls that use a 64 bit offset with
> > mmap() being stuck at 32 bits for backwards compatability.
>
> > Looks like it could possibly be done, with some special casing.
> > I am, though, not taking into account issues about the placement
> > of DLLs or shared objects in the virtual address space that might
> > interfer with the objective.
>
> First off, sorry for the late response to this. This appears to be an
> oversight in our documentation; The 32-bit limitation on memmapfile for
> 64-bit operating systems no longer exists as of R2007a (MATLAB version 7.=
4).
>
> Sincerely,
> Scott French
> The MathWorks, Inc.

Oh yes indeed, it seems I can actually map a multi-GB file without a
problem. Great! Thank you, Scott.

In article <eb9b26b2-bc49-487e-a538-bca7e632b873@k37g2000hsf.googlegroups.com>,
rych <rychphd@gmail.com> wrote:
>Under the title "Accessing Files with Memory-Mapping", we read in the
>"Limitations..."

>"The 2 GB limit also applies to 64-bit platforms"

>I can't think of any good reason for this. On of the good point of 64-
>bit OS is to be able to map an entire (very large) file to the address
>space and let the OS do the paging operations for us.


Eeee, gross. :( MS Windows does not have a POSIX mmap function;
the function that it does have is CreateFileMapping(), which takes
a pair of DWORD (32 bit values) as the high and low words of
the size of the object. So MS Windows could do it, even if not
in the same way as POSIX.

In POSIX, the final parameter, the offset to be mapped, is of
off_t, which is an implementation defined type. Older Unix versions
use a 32 bit off_t; newer versions use a 64 bit off_t. Some
unixes have mmap64() calls that use a 64 bit offset with
mmap() being stuck at 32 bits for backwards compatability.

Looks like it could possibly be done, with some special casing.
I am, though, not taking into account issues about the placement
of DLLs or shared objects in the virtual address space that might
interfer with the objective.
--
  "Allegories are in the realm of thoughts, what ruins are in
  the realm of things." -- Walter Benjamin

Walter Roberson wrote:
> In article <eb9b26b2-bc49-487e-a538-bca7e632b873@k37g2000hsf.googlegroups.com>,
> rych <rychphd@gmail.com> wrote:
>> Under the title "Accessing Files with Memory-Mapping", we read in the
>> "Limitations..."
>
>> "The 2 GB limit also applies to 64-bit platforms"
>
>> I can't think of any good reason for this. On of the good point of 64-
>> bit OS is to be able to map an entire (very large) file to the address
>> space and let the OS do the paging operations for us.
>
>
> Eeee, gross. :( MS Windows does not have a POSIX mmap function;
> the function that it does have is CreateFileMapping(), which takes
> a pair of DWORD (32 bit values) as the high and low words of
> the size of the object. So MS Windows could do it, even if not
> in the same way as POSIX.
>
> In POSIX, the final parameter, the offset to be mapped, is of
> off_t, which is an implementation defined type. Older Unix versions
> use a 32 bit off_t; newer versions use a 64 bit off_t. Some
> unixes have mmap64() calls that use a 64 bit offset with
> mmap() being stuck at 32 bits for backwards compatability.
>
> Looks like it could possibly be done, with some special casing.
> I am, though, not taking into account issues about the placement
> of DLLs or shared objects in the virtual address space that might
> interfer with the objective.


First off, sorry for the late response to this. This appears to be an
oversight in our documentation; The 32-bit limitation on memmapfile for
64-bit operating systems no longer exists as of R2007a (MATLAB version 7.4).

Sincerely,
Scott French
The MathWorks, Inc.

On Jul 16, 3:28=A0pm, Scott French <sc...@mathworks.com> wrote:
> Walter Roberson wrote:
> > In article <eb9b26b2-bc49-487e-a538-bca7e632b...@k37g2000hsf.googlegrou=
ps.com>,
> > rych =A0<rych...@gmail.com> wrote:
> >> Under the title "Accessing Files with Memory-Mapping", we read in the
> >> "Limitations..."
>
> >> "The 2 GB limit also applies to 64-bit platforms"
>
> >> I can't think of any good reason for this. On of the good point of 64-
> >> bit OS is to be able to map an entire (very large) file to the address
> >> space and let the OS do the paging operations for us.
>
> > Eeee, gross. :( MS Windows does not have a POSIX mmap function;
> > the function that it does have is CreateFileMapping(), which takes
> > a pair of DWORD (32 bit values) as the high and low words of
> > the size of the object. So MS Windows could do it, even if not
> > in the same way as POSIX.
>
> > In POSIX, the final parameter, the offset to be mapped, is of
> > off_t, which is an implementation defined type. Older Unix versions
> > use a 32 bit off_t; newer versions use a 64 bit off_t. Some
> > unixes have mmap64() calls that use a 64 bit offset with
> > mmap() being stuck at 32 bits for backwards compatability.
>
> > Looks like it could possibly be done, with some special casing.
> > I am, though, not taking into account issues about the placement
> > of DLLs or shared objects in the virtual address space that might
> > interfer with the objective.
>
> First off, sorry for the late response to this. This appears to be an
> oversight in our documentation; The 32-bit limitation on memmapfile for
> 64-bit operating systems no longer exists as of R2007a (MATLAB version 7.=
4).
>
> Sincerely,
> Scott French
> The MathWorks, Inc.

Oh yes indeed, it seems I can actually map a multi-GB file without a
problem. Great! Thank you, Scott.