| MATLAB® | |
| On this page… |
|---|
Constructing a memmapfile Object |
Memory-mapping is a mechanism that maps a portion of a file, or an entire file, on disk to a range of addresses within an application's address space. The application can then access files on disk in the same way it accesses dynamic memory. This makes file reads and writes faster in comparison with using functions such as fread and fwrite.
Another advantage of using memory-mapping in your MATLAB application is that it enables you to access file data using standard MATLAB indexing operations. Once you have mapped a file to memory, you can read the contents of that file using the same type of MATLAB statements used to read variables from the MATLAB workspace. The contents of the mapped file appear as if they were an array in the currently active workspace. You simply index into this array to read or write the desired data from the file.
This section describes the benefits and limitations of memory-mapping in MATLAB. The last part of this section gives details on which types of applications derive the greatest advantage from using memory-mapping:
The principal benefits of memory-mapping are efficiency, faster file access, the ability to share memory between applications, and more efficient coding.
Faster File Access. Accessing files via memory map is faster than using I/O functions such as fread and fwrite. Data are read and written using the virtual memory capabilities that are built in to the operating system rather than having to allocate, copy into, and then deallocate data buffers owned by the process.
MATLAB does not access data from the disk when the map is first constructed. It only reads or writes the file on disk when a specified part of the memory map is accessed, and then it only reads that specific part. This provides faster random access to the mapped data.
Efficiency. Mapping a file into memory allows access to data in the file as if that data had been read into an array in the application's address space. Initially, MATLAB only allocates address space for the array; it does not actually read data from the file until you access the mapped region. As a result, memory-mapped files provide a mechanism by which applications can access data segments in an extremely large file without having to read the entire file into memory first.
Efficient Coding Style. Memory-mapping eliminates the need for explicit calls to the fread and fwrite functions. In MATLAB, if x is a memory-mapped variable, and y is the data to be written to a file, then writing to the file is as simple as
x.Data = y;
Sharing Memory Between Applications. Memory-mapped files also provide a mechanism for sharing data between applications, as shown in the figure below. This is achieved by having each application map sections of the same file. You can use this feature to transfer large data sets between MATLAB and other applications.
Also, within a single application, you can map the same segment of a file more than once.
MATLAB restricts the size of a memory map to 2 gigabytes, and on some platforms, requires that you set up your memory-mapping so that all data access is aligned properly. See the following section, "Maximum Size of a Memory Map", for more information.
Maximum Size of a Memory Map. Due to limits set by the operating system and MATLAB, the maximum amount of data you can map with a single instance of a memory map is 2 gigabytes on 32-bit systems, and 256 terabytes on 64-bit systems. If you need to map more than this limit, you can either create separate maps for different regions of the file, or you can move the window of one map to different locations in the file.
Aligned Access on Sol64. The Sol64 platform only supports aligned data access. This means that numeric values of type double that are to be read from a memory-mapped file must start at some multiple of 8 bytes from the start of the file. (Note that this is from the start of the file, and not the start of the mapped region.) Furthermore, numeric values of type single and also 32-bit integers must start at multiples of 4 bytes, and 16-bit integers at 2-byte multiples.
If you attempt to map a file on Sol64, which does not take into account these alignment considerations, MATLAB generates an error.
Memory-mapping works only with data that have the same byte ordering scheme as the native byte ordering of your operating system. For example, because both Linus Tolvald's Linux® and Microsoft Windows systems use little-endian byte ordering, data created on a Linux system can be read on Windows systems. You can use the computer function to determine the native byte ordering of your current system.
Just how much advantage you get from mapping a file to memory depends mostly on the size and format of the file, the way in which data in the file is used, and the computer platform you are using.
When Memory-Mapping Is Most Useful. Memory-mapping works best with binary files, and in the following scenarios:
For large files that you want to access randomly one or more times
For small files that you want to read into memory once and access frequently
For data that you want to share between applications
When you want to work with data in a file as if it were a MATLAB array
When the Advantage Is Less Significant. The following types of files do not fully use the benefits of memory-mapping:
Formatted binary files like HDF or TIFF that require customized readers are not good for memory-mapping. Describing the data contained in these files can be a very complex task. Also, you cannot access data directly from the mapped segment, but must instead create arrays to hold the data.
Text or ASCII files require that you convert the text in the mapped region to an appropriate type for the data to be meaningful. This takes up additional address space.
Files that are larger than several hundred megabytes in size consume a significant amount of the virtual address space needed by MATLAB to process your program. Mapping files of this size may result in MATLAB reporting out-of-memory errors more often. This is more likely if MATLAB has been running for some time, or if the memory used by MATLAB becomes fragmented.
MATLAB implements memory-mapping using an object-oriented class called memmapfile. The memmapfile class has the properties and methods you need to map to a file, read and write the file via the map, and remove the map from memory when you are done.
There are six properties defined for the memmapfile class. These are shown in the table below. These properties control which file is being mapped, where in the file the mapping is to begin and end, how the contents of the file are to be formatted, and whether or not the file is writable. One property of the file contains the file data itself.
Property | Description | Data Type | Default |
|---|---|---|---|
Data | Contains the data read from the file or to be written to the file. (See Reading a Mapped File and Writing to a Mapped File) | Any of the numeric types | None |
Filename | Path and name of the file to map into memory. (See Selecting the File to Map) | char array | None |
Format | Format of the contents of the mapped region, including class, array shape, and variable or field name by which to access the data. (See Identifying the Contents of the Mapped Region) | char array or N-by-3 | uint8 |
Offset | Number of bytes from the start of the file to the start of the mapped region. This number is zero-based. That is, offset 0 represents the start of the file. Must be a nonnegative integer value. (See Setting the Start of the Mapped Region) | double | 0 |
Repeat | Number of times to apply the specified format to the mapped region of the file. Must be a positive integer value or Inf. (See Repeating a Format Scheme) | double | Inf |
Writable | Type of access allowed to the mapped region. Must be logical 1 or logical 0. (See Setting the Type of Access) | logical | false |
You can set the values for any property except for Data at the time you call the memmapfile constructor, or at any time after that while the map is still valid. Any properties that are not explicitly set when you construct the object are given their default values as shown in the table above. For information on calling the constructor, see Constructing a memmapfile Object.
Once a memmapfile object has been constructed, you can change the value of any of its properties. Use the objname.property syntax in assigning the new value. For example, to set a new Offset value for memory map object m, type
m.Offset = 2048;
Note Property names are not case sensitive. For example, MATLAB considers m.offset to be the same as m.Offset. |
To display the value of all properties of a memmapfile object, simply type the object name. For a memmapfile object m, typing the variable name m displays the following. Note that this example requires the file records.dat which you will create at the beginning of the next section.
m =
Filename: 'records.dat'
Writable: true
Offset: 1024
Format: 'uint32'
Repeat: Inf
Data: 4778x1 uint32 arrayTo display the value of any individual property, for example the Writable property of object m, type
m.Writable ans = true
The first step in mapping to any file is to construct an instance of the memmapfile class using the class constructor function. You can have MATLAB assign default values to each of the new object's properties, or you can specify property values yourself in the call to the memmapfile constructor.
For information on how to set these values, see these sections:
All the examples in this section use a file named records.dat that contains a 5000-by-1 matrix of double-precision floating point numbers. Use the following code to generate this file before going on to the next sections of this documentation.
First, save this function in your current working directory:
function gendatafile(filename, count)
dmax32 = double(intmax('uint32'));
rand('state', 0)
fid = fopen(filename,'w');
fwrite(fid, rand(count,1)*dmax32, 'double');
fclose(fid);
Now execute the gendatafile function to generate the records.dat file that is referenced in this section. You can use this function at any time to regenerate the file:
gendatafile('records.dat', 5000);The simplest and most general way to call the constructor is with one input argument that specifies the name of the file you want to map. All other properties are optional and are given their default values. Use the syntax shown here:
objname = memmapfile(filename)
To construct a map for the file records.dat that resides in your current working directory, type the following:
m = memmapfile('records.dat')
m =
Filename: 'd:\matlab\mfiles\records.dat'
Writable: false
Offset: 0
Format: 'uint8'
Repeat: Inf
Data: 40000x1 uint8 arrayMATLAB constructs an instance of the memmapfile class, assigns it to the variable m, and maps the entire records.dat file to memory, setting all object properties to their default values. In this example, the command maps the entire file as a sequence of unsigned 8-bit integers and gives the caller read-only access to its contents.
You can make the memory map more specific to your needs by including more information when calling the constructor. In addition to the filename argument, there are four other parameters that you can pass to the constructor. Each of these parameters represents a property of the object, and each requires an accompanying value to be passed, as well:
objname = memmapfile(filename, prop1, value1, prop2, value2, ...)
For example, to construct a map using nondefault values for the Offset, Format, and Writable properties, type the following, enclosing all property names and string parameter values in quotes:
m = memmapfile('records.dat', ...
'Offset', 1024, ...
'Format', 'double', ...
'Writable', true);Type the object name to see the current settings for all properties:
m
m =
Filename: 'd:\matlab\mfiles\records.dat'
Writable: true
Offset: 1024
Format: 'double'
Repeat: Inf
Data: 4872x1 double arrayYou can also change the value of any property after the object has been constructed. Use the syntax:
objname.property = newvalue;
For example, to set the format to uint16, type the following. (Property names, like Format, are not case sensitive.)
m.format = 'uint16'
m =
Filename: 'd:\matlab\mfiles\records.dat'
Writable: true
Offset: 1024
Format: 'uint16'
Repeat: Inf
Data: 19488x1 uint16 arrayFurther read and write operations to the region mapped by m now treat the data in the file as a sequence of unsigned 16-bit integers. Whenever you change the value of a memmapfile property, MATLAB remaps the file to memory.
filename is the only required argument when you call the memmapfile constructor. When you call the memmapfile constructor, MATLAB assigns the file name that you specify to the Filename property of the new object instance.
Specify the file name as a quoted string, (e.g., 'records.dat'). It must be first in the argument list and not specified as a parameter-value pair. filename must include a file name extension if the name of the file being mapped has an extension. The filename argument cannot include any wildcard characters (e.g., * or ?), and is not case sensitive.
Note Unlike the other memmapfile constructor arguments, you must specify filename as a single string, and not as a parameter-value pair. |
If the file to be mapped resides somewhere on the MATLAB path, you can use a partial pathname. If the path to the file is not fully specified, MATLAB searches for the file in your current working directory first, and then on the MATLAB path.
Once memmapfile locates the file, MATLAB stores the absolute path name for the file internally, and then uses this stored path to locate the file from that point on. This enables you to work in other directories outside your current work directory and retain access to the mapped file.
You can change the value of the Filename property at any time after constructing the memmapfile object. You might want to do this if:
You want to use the same memmapfile object on more than one file.
You save your memmapfile object to a MAT-file, and then later load it back into MATLAB in an environment where the mapped file has been moved to a different location. This requires that you modify the path segment of the Filename string to represent the new location.
For example, save memmapfile object m to file mymap.mat:
disp(m.Filename)
d:\matlab\mfiles\records.dat
save mymat mNow move the file to another location, load the object back into MATLAB, and update the path in the Filename property:
load mymat m m.Filename = 'f:\testfiles\oct1\records.dat'
Note You can only map an existing file. You cannot create a new file and map that file to memory in one operation. Use the MATLAB file I/O functions to create the file before attempting to map it to memory. |
By default, MATLAB begins a memory map at the start of the file. To begin the mapped region at some point beyond the start of the file, specify an Offset parameter in the call to the memmapfile constructor:
objname = memmapfile(filename, 'Offset', bytecount)
The bytecount value is the number of bytes from the beginning of the file to the point in the file where you want the memory map to start (a zero-based offset). To map the file records.dat from a point 1024 bytes from the start and extending to the end of the file, type
m = memmapfile('records.dat', 'Offset', 1024);You can change the starting position of an existing memory map by setting the Offset property for the associated object to a new value. The following command sets the offset of memmapfile object m to be 2,048 bytes from the start of the mapped file:
m.Offset = 2048;
Note The Sol64 platform supports aligned data access only. If you attempt to use a memmapfile offset on Sol64 that does not take the necessary alignment considerations into account, MATLAB generates an error. (See Aligned Access on Sol64). |
By default, MATLAB considers all the data in a mapped file to be a sequence of unsigned 8-bit integers. To have the data interpreted otherwise as it is read or written to in the mapped file, specify a Format parameter and value in your call to the constructor:
objname = memmapfile(filename, 'Format', formatspec)
The formatspec argument can either be a character string that identifies a single class used throughout the mapped region, or a cell array that specifies more than one class.
For example, say that you map a file that is 12 kilobytes in length. Data read from this file could be treated as a sequence of 6,000 16-bit (2-byte) integers, or as 1,500 8-byte double-precision floating-point numbers, to name just a couple of possibilities. Or you could read this data in as a combination of different types: for example, as 4,000 8-bit (1-byte) integers followed by 1,000 64-bit (8-byte) integers. You determine how MATLAB will interpret the mapped data by setting the Format property of the memmapfile object when you call its constructor.
MATLAB arrays are stored on disk in column-major order. (The sequence of array elements is column 1, row 1; column 1, row 2; column 1, last row; column 2, row 1, and so on.) You might need to transpose or rearrange the order of array elements when reading or writing via a memory map.
Note The Sol64 platform supports aligned data access only. If you attempt to use a memmapfile format on Sol64 that does not take the necessary alignment considerations into account, MATLAB generates an error. (See Aligned Access on Sol64). |
For a list of data types supported for the Format property, see Supported Data Types for the Format Property.
For more information on format options see these sections:
Mapping a Single Data Type. If the file region being mapped contains data of only one type, specify the Format value as a character string identifying that type:
objname = memmapfile(filename, 'Format', datatype)
The following command constructs a memmapfile object for the entire file records.dat, and sets the Format property for that object to uint64. Any read or write operations made via the memory map will read and write the file contents as a sequence of unsigned 64-bit integers:
m = memmapfile('records.dat', 'Format', 'uint64')
Filename: 'd:\matlab\mfiles\records.dat'
Writable: false
Offset: 0
Format: 'uint64'
Repeat: Inf
Data: 5000x1 uint64 arrayYou can change the value of the Format property at any time after the memmapfile object is constructed. Use the object.property syntax shown here in assigning the new value:
m.Format = 'int32';
Further read and write operations to the region mapped by m now treat the data in the file as a sequence of signed 32-bit integers.
Property names, like Format, are not case sensitive.
Formatting the Mapped Data to an Array. You can also specify an array shape to be applied to the data read or written to the mapped file, and a field name to be used in referencing this array. Use a cell array to hold these values either when calling the memmapfile constructor or when modifying m.Format after the object has been constructed. The cell array contains three elements: the class to be applied to the mapped region, the dimensions of the array shape that is applied to the region, and a field name to use in referencing the data:
objname = memmapfile(filename, ...
'Format', {datatype, dimensions, varname})The following command constructs a memmapfile object for a region of records.dat such that MATLAB handles the contents of the region as a 4-by-10-by-18 array of unsigned 32-bit integers, which you can reference in the structure of the returned object using the field name x:
m = memmapfile('records.dat', ...
'Offset', 1024, ...
'Format', {'uint32' [4 10 18] 'x'})
m =
Filename: 'd:\matlab\mfiles\records.dat'
Writable: false
Offset: 1024
Format: {'uint32' [4 10 18] 'x'}
Repeat: Inf
Data: 13x1 struct array with fields:
x
A = m.Data(1).x;
whos A
Name Size Bytes Class
A 4x10x18 2880 uint32 array
Grand total is 720 elements using 2880 bytesYou can change the class, array shape, or field name that MATLAB applies to the mapped region at any time by setting a new value for the Format property of the object:
m.Format = {'uint64' [30 4 10] 'x'};
A = m.Data(1).x;
whos A
Name Size Bytes Class
A 30x4x10 9600 uint64 array
Grand total is 1200 elements using 9600 bytesMapping Multiple Data Types and Arrays. If the region being mapped is composed of segments of varying classes or array shapes, you can specify an individual format for each segment using an N-by-3 cell array, where N is the number of segments. The cells of each cell array row identify the class for that segment, the array dimensions to map the data to, and a field name by which to reference that segment:
objname = memmapfile(filename, ...
'Format', { ...
datatype1, dimensions1, fieldname1; ...
datatype2, dimensions2, fieldname2; ...
: : : ...
datatypeN, dimensionsN, fieldnameN})The following command maps a 24-kilobyte file containing data of three different classes: int16, uint32, and single. The int16 data is mapped as a 2-by-2 matrix that can be accessed using the field name model. The uint32 data is a scalar value accessed as field serialno. The single data is a 1-by-3 matrix named expenses.
Each of these fields belongs to the 800-by-1 structure array m.Data:
m = memmapfile('records.dat', ...
'Offset', 2048, ...
'Format', { ...
'int16' [2 2] 'model'; ...
'uint32' [1 1] 'serialno'; ...
'single' [1 3] 'expenses'});
The figure below shows the ordering of the array elements more closely. In particular, it illustrates that MATLAB arrays are stored on the disk in column-major order. The sequence of array elements in the mapped file is row 1, column 1; row 2, column 1; row 1, column 2; and row 2, column 2.
If the data in your file is not stored in this order, you might need to transpose or rearrange the order of array elements when reading or writing via a memory map.
Supported Data Types for the Format Property. You can use any of the following classes when you specify a Format value. The default type is uint8.
Format String | Data Type Description |
|---|---|
'int8' | Signed 8-bit integers |
'int16' | Signed 16-bit integers |
'int32' | Signed 32-bit integers |
'int64' | Signed 64-bit integers |
'uint8' | Unsigned 8-bit integers |
'uint16' | Unsigned 16-bit integers |
'uint32' | Unsigned 32-bit integers |
'uint64' | Unsigned 64-bit integers |
'single' | 32-bit floating-point |
'double' | 64-bit floating-point |
After you set a Format value for the memmapfile object, you can have MATLAB apply that format to the file data multiple times by specifying a Repeat value when you call the memmapfile constructor:
objname = memmapfile(filename, ...
'Format', formatspec, ...
'Repeat', count)The Repeat value applies to the whole format specifier, whether that specifier describes just a single class that repeats, or a more complex format that includes various classes and array shapes. The default Repeat value is infinity (inf), which means that the full extent of the Format specifier repeats as many times as possible within the mapped region.
The next example maps a file region identical to that of the previous example, except the pattern of int16, uint32, and single classes is repeated only three times within the mapped region of the file:
m = memmapfile('records.dat', ...
'Offset', 2048, ...
'Format', { ...
'int16' [2 2] 'model'; ...
'uint32' [1 1] 'serialno'; ...
'single' [1 3] 'expenses'}, ...
'Repeat', 3);You can change the value of the Repeat property at any time. To change the repeat value to 5, type
m.Repeat = 5;
Property names, like Repeat, are not case sensitive.
Keeping the Repeated Format Within the Mapped Region. MATLAB maps only the full pattern specified by the Format property. If you repeat a format such that it would cause the map to extend beyond the end of the file, then either of two things can happen:
If you specify a repeat value of Inf, MATLAB applies to the map only those repeated segments that fit within the file in their entirety.
If you specify a repeat value other than Inf, and that value would cause the map to extend beyond the end of the file, MATLAB generates an error.
Considering the last example, if the part of the file from m.Offset to the end were 70 bytes (instead of the 72 bytes required to repeat m.Format three times) and you used a Repeat value of Inf, then only two full repetitions of the specified format would have been mapped. The end result is as if you had constructed the map with this command:
m = memmapfile('records.dat', ...
'Offset', 2048, ...
'Format', { ...
'int16' [2 2] 'model'; ...
'uint32' [1 1] 'serialno'; ...
'single' [1 3] 'expenses'}, ...
'Repeat', 2);If Repeat were set to 3 and you had only 70 bytes to the end of the file, you would get an error.
Note memmapfile does not expand or append to a mapped file. Use standard file I/O functions like fopen and fwrite to do this. |
You can map a file region to allow either read-only or read and write access to its contents. Pass a Writable parameter and value in the memmapfile constructor, or set m.Writable on an existing object to set the type of access allowed:
objname = memmapfile(filename, 'Writable', trueorfalse)
The value passed can be either true (equal to logical(1)) or false (equal to logical(0)). By default, it is false, meaning that the mapped region is read only.
To map a read and write region of the file records.dat in memory, type
m = memmapfile('records.dat', 'Writable', true);Note To successfully modify the file you are mapping to, you must have write permission for that file. If you do not have write permission, you can still set the Writable property to true, but attempting to write to the file generates an error. |
You can change the value of the Writable property at any time. To make the memory map to records.dat read only, type:
m.Writable = false;
Property names, like Writable, are not case sensitive.
The most commonly used property of the memmapfile class is the Data property. It is through this property of the memory-map object that MATLAB provides all read and write access to the contents of the mapped file.
The actual mapping of a file to the MATLAB address space does not take place when you construct a memmapfile object. A memory map, based on the information currently stored in the mapped object, is generated the first time you reference or modify the Data property for that object.
After you map a file to memory, you can read the contents of that file using the same MATLAB statements used to read variables from the MATLAB workspace. By accessing the Data property of the memory map object, the contents of the mapped file appear as if they were an array in the currently active workspace. You simply index into this array to read the desired data from the file.
This section covers the following topics:
MATLAB accesses data in structures more efficiently than it does data contained in objects. The main reason is that structures do not require the extra overhead of a subsref routine. Instead of reading directly from the memmapfile object, as shown here:
for k = 1 : N y(k) = m.Data(k); end
you will get better performance when you assign the Data field to a variable, and then read or write the mapped file through this variable, as shown in this second example:
dataRef = m.Data; for k = 1 : N y(k) = dataRef(k); end
This example maps a file of 100 double-precision floating-point numbers to memory. The map begins 1024 bytes from the start of the file, and ends 800 bytes (8 bytes per double times a Repeat value of 100) from that point.
If you haven't done so already, generate a test data file for use in the following examples by executing the gendatafile function defined under Constructing a memmapfile Object:
gendatafile('records.dat', 5000);Now, construct the memmapfile object m, and show the format of its Data property:
m = memmapfile('records.dat', 'Format', 'double', ...
'Offset', 1024, 'Repeat', 100);
d = m.Data;
whos d
Name Size Bytes Class
d 100x1 800 double array
Grand total is 100 elements using 800 bytesRead a selected set of numbers from the file by indexing into the single-precision array m.Data:
d(15:20)
ans =
1.0e+009 *
3.6045
2.7006
0.5745
0.8896
2.6079
2.7053This example is similar to the last, except that the constructor of the memmapfile object now specifies an array shape of 4-by-6 to be applied to the data as it is read from the mapped file. MATLAB maps the file contents into a structure array rather than a numeric array, as in the previous example:
m = memmapfile('records.dat', ...
'Format', {'double', [4 6], 'x'}, ...
'Offset', 1024, 'Repeat', 100);
d = m.Data;
whos d
Name Size Bytes Class
d 100x1 25264 struct array
Grand total is 2500 elements using 25264 bytesWhen you read an element of the structure array, MATLAB presents the data in the form of a 4-by-6 array:
d(5).x
ans =
1.0e+009 *
3.1564 0.6684 2.1056 1.9357 1.2773 4.2219
2.9520 0.8208 3.5044 1.7705 0.2112 2.3737
1.4865 1.8144 1.9790 3.8724 2.9772 1.7183
0.7131 3.6764 1.9643 0.0240 2.7922 0.8538To index into the structure array field, use:
d(5).x(3,2:6)
ans =
1.0e+009 *
1.8144 1.9790 3.8724 2.9772 1.7183This example maps a file containing more than one class. The different classes contained in the file are mapped as fields of the returned structure array m.Data.
The Format parameter passed in the constructor specifies that the first 80 bytes of the file are to be treated as a 5-by-8 matrix of uint16, and the 160 bytes after that as a 4-by-5 matrix of double. This pattern repeats until the end of the file is reached. The example shows different ways of reading the Data property of the object.
Start by calling the memmapfile constructor to create a memory map object, m:
m = memmapfile('records.dat', ...
'Format', { ...
'uint16' [5 8] 'x'; ...
'double' [4 5] 'y' });If you examine the Data property, MATLAB shows a 166-element structure array with two fields, one for each format specifier in the constructor:
d = m.Data
ans =
166x1 struct array with fields:
x
yExamine one structure in the array to show the format of each field:
d(3)
ans =
x: [5x8 uint16]
y: [4x5 double]Now read the x and y fields of that structure from the file. MATLAB formats the first block of data as a 5-by-8 matrix of uint16, as specified in the Format property, and the second block as a 4-by-5 matrix of double:
d(3).x
ans =
34432 47500 19145 16868 38165 47956 35550 16853
60654 51944 16874 47166 35397 58072 16850 56576
51075 16876 12471 34369 8341 16853 44509 57652
16863 16453 6666 11480 16869 58695 36217 5932
57883 15551 41755 16874 37774 31693 54813 16865
d(3).y
ans =
1.0e+009 *
3.1229 1.5909 2.9831 2.2445 1.1659
1.3284 3.0182 2.6685 3.7802 1.0837
3.6013 2.3475 3.4137 0.7428 3.7613
2.4399 1.9107 4.1096 4.2080 3.1667This example plots the Fourier transform output of data read from a file via a memory map. It then modifies several parameters of the existing map, reads from a different part of the data file, and plots a histogram from that data.
Create a memory-mapped object, mapping 1,000 elements of type double starting at the 1025th byte:
m = memmapfile('mybinary.bin', 'Offset', 1024, ...
'Format', 'double', 'Repeat', 1000);
Get data associated with the map and plot the FFT of the first 1000 values of the map. This is when the map is actually created, because no data has been referenced until this point:
plot(abs(fft(m.Data(1:1000))));
Get information about the memory map:
mapStruct = get(m)
mapStruct =
Filename: 'd:\matlab\mfiles\mybinary.bin'
Writable: 0
Offset: 1024
Format: 'double'
Repeat: 1000
Data: [1000x1 double]Change the map, but continue using the same file:
m.Offset = 4096; m.Format = 'single'; m.Repeat = 800;
Read from a different area of the file, and plot a histogram of the data. This maps a new region and unmaps the previous region:
hist(m.Data)
Writing to a mapped file is done with standard MATLAB subscripted assignment commands. To write to a particular location in the file mapped to memmapfile object m, assign the value to the m.Data structure array index and field that map to that location.
If you haven't done so already, generate a test data file for use in the following examples by executing the gendatafile function defined under Constructing a memmapfile Object:
gendatafile('records.dat', 5000);Now call the memmapfile constructor to create the object:
m = memmapfile('records.dat', ...
'Format', { ...
'uint16' [5 8] 'x'; ...
'double' [4 5] 'y' });If you are going to modify the mapped file, be sure that you have write permission, and that you set the Writable property of the memmapfile object to true (logical 1):
m.Writable = true;
Note You do not have to set Writable as a separate command, as done here. You can include a Writable parameter-value argument in the call to the memmapfile constructor. |
Read from the 5-by-8 matrix x at m.Data(2):
d = m.Data; d(2).x ans = 35330 4902 31861 16877 23791 61500 52748 16841 51314 58795 16860 43523 8957 5182 16864 60110 18415 16871 59373 61001 52007 16875 26374 28570 16783 4356 52847 53977 16858 38427 16067 33318 65372 48883 53612 16861 18882 39824 61529 16869
Update all values in that matrix using a standard MATLAB assignment statement:
d(2).x = d(2).x * 1.5;
Verify the results:
d(2).x ans = 52995 7353 47792 25316 35687 65535 65535 25262 65535 65535 25290 65285 13436 7773 25296 65535 27623 25307 65535 65535 65535 25313 39561 42855 25175 6534 65535 65535 25287 57641 24101 49977 65535 65535 65535 25292 28323 59736 65535 25304
This section covers the following topics:
The dimensions of a memmapfile object's Data field are set at the time you construct the object and cannot be changed. This differs from other MATLAB arrays that have dimensions you can modify using subscripted assignment.
For example, you can add a new column to the field of a MATLAB structure:
A.s = ones(4,5);
A.s(:,6) = [1 2 3 4]; % Add new column to A.s
size(A.s)
ans =
4 6But you cannot add a new column to a similar field of a structure that represents data mapped from a file. The following assignment to m.Data(60).y does not expand the size of y, but instead generates an error:
m.Data(60)
ans =
x: [5x8 uint16]
y: [4x5 double]
m.Data(60).y(:,6) = [1 2 3 4]; % Generates an error.Thus, if you map an entire file and then append to that file after constructing the map, the appended data is not included in the mapped region. If you need to modify the dimensions of data that you have mapped to a memmapfile object, you must either modify the Format or Repeat properties for the object, or reconstruct the object.
Examples. Two examples of statements that attempt to modify the dimensions of a mapped Data field are shown here. These statements result in an error.
The first example attempts to diminish the size of the array by removing a row from the mapped array m.Data.
m.Data(5) = [];
The second example attempts to expand the size of a 50-row mapped array x by adding another row to it:
m.Data(2).x(1:51,31) = 1:51;
The syntax to use when writing to mapped memory can depend on what format was used when you mapped memory to the file.
When Memory Is Mapped in Nonstructure Format. When you map a file as a sequence of a single class (e.g., a sequence of uint16), you can use the following syntax to write matrix X to the file:
m.Data = X;
This statement is valid only if all of the following conditions are true:
The file is mapped as a sequence of elements of the same class, making m.Data an array of a nonstructure type.
The class of X is the same as the class of m.Data.
The number of elements in X equals the number of elements in m.Data.
This example maps a file as a sequence of 16-bit unsigned integers, and then uses the syntax shown above to write a matrix to the file. Map only a small part of the file, using a uint16 format for this segment:
m = memmapfile('records.dat', 'Writable', true', ...
'Offset', 2000, 'Format', 'uint16', 'Repeat', 15);Create a matrix X of the same size and write it to the mapped part of the file:
X = uint16(5:5:75); % Sequence of 5 to 75, counting by fives. m.data = X;
Verify that new values were written to the file:
m.offset = 1980; m.repeat = 35;
reshape(m.data,5,7)'
ans =
29158 16841 32915 37696 421 % <== At offset 1980
16868 51434 17455 30645 16871
5 10 15 20 25 % <== At offset 2000
30 35 40 45 50
55 60 65 70 75
16872 50155 51100 26469 16873
56776 6257 28746 16877 34374When Memory Is Mapped in Scalar Structure Format. When you map a file as a sequence of a single class (e.g., a sequence of uint16), you can use the following syntax to write matrix X to the file:
m.Data.f = X;
This statement is valid only if all of the following conditions are true:
The file is mapped as containing multiple classes that do not repeat, making m.Data a scalar structure.
The class of X is the same as the class of m.Data.f.
The number of elements in X equals that of m.Data.f.
This example maps a file as a 300-by-8 matrix of type uint16 followed by a 200-by-5 matrix of type double, and then uses the syntax shown above to write a matrix to the file.
m = memmapfile('records.dat', ...
'Format', { ...
'uint16' [300 8] 'x'; ...
'double' [200 5] 'y' }, ...
'Repeat', 1, 'Writable', true);
m.Data.x = ones(300, 8, 'uint16');When Memory Is Mapped in Nonscalar Structure Format. When you map a file as a repeating sequence of multiple classes, you can use the following syntax to write matrix X to the file, providing that k is a scalar index:
m.Data(k).field = X;
To do this, the following conditions must be true:
The file is mapped as containing multiple classes that can repeat, making m.Data a nonscalar structure.
k is a scalar index.
The class of X is the same as the class of m.Data(k).field.
The number of elements in X equals that of m.Data(k).field.
This example maps a file as a matrix of type uint16 followed by a matrix of type double that repeat 20 times, and then uses the syntax shown above to write a matrix to the file.
m = memmapfile('records.dat', ...
'Format', { ...
'uint16' [25 8] 'x'; ...
'double' [15 5] 'y' }, ...
'Repeat', 20, 'Writable', true);
d = m.Data;
d(12).x = ones(25,8,'uint16');You can write to specific elements of field x as shown here:
d(12).x(3:5,1:end) = uint16(500);
d(12).x(3:5,1:end)
ans =
500 500 500 500 500 500 500 500
500 500 500 500 500 500 500 500
500 500 500 500 500 500 500 500All of the usual MATLAB indexing and class rules apply when assigning values to data via a memory map. The class that you assign to must be big enough to hold the value being assigned. For example,
m = memmapfile('records.dat', 'Format', 'uint8', ...
'Writable', true);
d = m.Data;
d(5) = 300;saturates the x variable because x is defined as an 8-bit integer:
d(5) ans = 255
In the following code, the data in variable block2 is a copy of the file data mapped by m.Data(2). Because it is a copy, modifying array data in block2 does not modify the data contained in the file:
First, destroy the memmapfile object and restore the test file records.dat, since you modified it by running the previous examples:
clear m
gendatafile('records.dat',50000);Map the file as a series of uint16 and double matrices and make a copy of m.Data(2) in block2:
m = memmapfile('records.dat', ...
'Format', { ...
'uint16' [5 8] 'x'; ...
'double' [4 5] 'y' });
d = m.Data;
Write all zeros to the copy:
d(2).x(1:5,1:8) = 0;
d(2).x
ans =
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0Verify that the data in the mapped file is not changed even though the copy of m.Data(2).x is written with zeros:
m.Data(2).x ans = 35330 4902 31861 16877 23791 61500 52748 16841 51314 58795 16860 43523 8957 5182 16864 60110 18415 16871 59373 61001 52007 16875 26374 28570 16783 4356 52847 53977 16858 38427 16067 33318 65372 48883 53612 16861 18882 39824 61529 16869
Although you can expand the dimensions of a typical MATLAB array by assigning outside its current dimensions, this does not apply to the Data property of a memmapfile object. The following operation is invalid if m.Data has only 100 elements:
m.Data(120) = x;
If you need to expand the size of the mapped data region, first extend the map by updating the Format or Repeat property of the memmapfile object to reflect the new structure of the data in the mapped file.
You can use the following methods on objects constructed from the memmapfile class.
Syntax | Description |
|---|---|
| disp | Displays properties of the object. The display does not include the object's name. |
get(obj) | Returns the values of all properties of the memmapfile object in a structure array. |
get(obj, property) | Returns the value of the specified property. property can be a string or cell array of strings, each containing a property name. |
Use the disp method to display all properties of a memmapfile object. The text displayed includes only the property value, and not the object name or the MATLAB response string, ans =.
Construct object m:
m = memmapfile('records.dat', ...
'Offset', 2048, ...
'Format', { ...
'int16' [2 2] 'model'; ...
'uint32' [1 1] 'serialno'; ...
'single' [1 3] 'expenses'});
and display all of its properties:
disp(m)
Filename: 'd:\matlab\mfiles\records.dat'
Writable: false
Offset: 2048
Format: {'int16' [2 2] 'model'
'uint32' [1 1] 'serialno'
'single' [1 3] 'expenses'}
Repeat: Inf
Data: 16581x1 struct array with fields:
model
serialno
expensesYou can use the get method of the memmapfile class to return information on any or all of the object's properties. Specify one or more property names to get the values of specific properties.
This example returns the values of the Offset, Repeat, and Format properties for a memmapfile object. Use the get method to return the specified property values in a 1-by-3 cell array, m_props:
m_props = get(m, {'Offset', 'Repeat', 'Format'})
m_props =
[2048] [Inf] {3x3 cell}
m_props{3}
ans =
'int16' [1x2 double] 'model'
'uint32' [1x2 double] 'serialno'
'single' [1x2 double] 'expenses'You also can choose to use the objname.property syntax:
m_props = {m.Offset, m.Repeat, m.Format}
m_props =
[2048] [Inf] {3x3 cell}To return the values for all properties with get, pass just the object name:
get(m)
Filename: 'd:\matlab\mfiles\records.dat'
Writable: 0
Offset: 2048
Format: {3x3 cell}
Repeat: Inf
Data: [16581x1 struct]It is not necessary to explicitly call a destructor method to clear a memmapfile object from memory when you no longer need it. MATLAB calls the destructor for you whenever you do any of the following:
Reassign another value to the memmapfile object's variable
Clear the object's variable from memory
Exit the function scope in which the object was created
When you assign the Data field of the memmapfile object to a variable, MATLAB makes a shared data copy of the mapped data. This is very efficient as no memory actually gets copied. In the following statement, memdat is a shared data copy of the data mapped from the file:
memdat = m.Data;
When you finish using the mapped data, make sure to clear any variables that shared data with the mapped file before clearing the object itself. If you clear the object first, then the sharing of data between the file and dependent variables is broken, and the data assigned to such variables must be copied into memory before the object is destroyed. If access to the mapped file was over a network, then copying this data to local memory can take considerable time. So, if the statement shown above assigns data to the variable memdat, you should be sure to clear memdat before clearing m when you are finished with the object.
Note Keep in mind that the memmapfile object can be cleared in any of the three ways described under Deleting a Memory Map. |
In this demonstration, two separate MATLAB processes communicate with each other by writing and reading from a shared file. They share the file by mapping part of their memory space to a common location in the file. A write operation to the memory map belonging to the first process can be read from the map belonging to the second, and vice versa.
One MATLAB process (running send.m) writes a message to the file via its memory map. It also writes the length of the message to byte 1 in the file, which serves as a means of notifying the other process that a message is available. The second process (running answer.m) monitors byte 1 and, upon seeing it set, displays the received message, puts it into uppercase, and echoes the message back to the sender.
This function prompts you to enter a string and then, using memory-mapping, passes the string to another instance of MATLAB that is running the answer function.
Copy the send and answer functions to files send.m and answer.m in your current working directory. Begin the demonstration by calling send with no inputs. Next, start a second MATLAB session on the same machine, and call the answer function in this session. To exit, press Enter.
function send
% Interactively send a message to ANSWER using memmapfile class.
filename = fullfile(tempdir, 'talk_answer.dat');
% Create the communications file if it is not already there.
if ~exist(filename, 'file')
[f, msg] = fopen(filename, 'wb');
if f ~= -1
fwrite(f, zeros(1,256), 'uint8');
fclose(f);
else
error('MATLAB:demo:send:cannotOpenFile', ...
'Cannot open file "%s": %s.', filename, msg);
end
end
% Memory map the file.
m = memmapfile(filename, 'Writable', true, 'Format', 'uint8');
while true
% Set first byte to zero, indicating a message is not
% yet ready.
m.Data(1) = 0;
str = input('Enter send string (or RETURN to end): ', 's');
len = length(str);
if (len == 0)
disp('Terminating SEND function.')
break;
end
str = str(1:min(len, 255)); % Message limited to 255 chars.
% Update the file via the memory map.
m.Data(2:len+1) = str;
m.Data(1)=len;
% Wait until the first byte is set back to zero,
% indicating that a response is available.
while (m.Data(1) ~= 0)
pause(.25);
end
% Display the response.
disp('response from ANSWER is:')
disp(char(m.Data(2:len+1))')
endThe answer function starts a server that, using memory-mapping, watches for a message from send. When the message is received, answer replaces the message with an uppercase version of it, and sends this new message back to send.
To use answer, call it with no inputs:
function answer
% Respond to SEND using memmapfile class.
disp('ANSWER server is awaiting message');
filename = fullfile(tempdir, 'talk_answer.dat');
% Create the communications file if it is not already there.
if ~exist(filename, 'file')
[f, msg] = fopen(filename, 'wb');
if f ~= -1
fwrite(f, zeros(1,256), 'uint8');
fclose(f);
else
error('MATLAB:demo:answer:cannotOpenFile', ...
'Cannot open file "%s": %s.', filename, msg);
end
end
% Memory map the file.
m = memmapfile(filename, 'Writable', true, 'Format', 'uint8');
while true
% Wait till first byte is not zero.
while m.Data(1) == 0
pause(.25);
end
% The first byte now contains the length of the message.
% Get it from m.
msg = char(m.Data(2:1+m.Data(1)))';
% Display the message.
disp('Received message from SEND:')
disp(msg)
% Transform the message to all uppercase.
m.Data(2:1+m.Data(1)) = upper(msg);
% Signal to SEND that the response is ready.
m.Data(1) = 0;
endTo see what the demonstration looks like when it is run, first, start two separate MATLAB sessions on the same computer system. Call the send function in one and the answer function in the other to create a map in each of the processes' memory to the common file:
% Run SEND in the first MATLAB session. send Enter send string (or RETURN to end): % Run ANSWER in the second MATLAB session. answer ANSWER server is awaiting message
Next, enter a message at the prompt displayed by the send function. MATLAB writes the message to the shared file. The second MATLAB session, running the answer function, loops on byte 1 of the shared file and, when the byte is written by send, answer reads the message from the file via its memory map. The answer function then puts the message into uppercase and writes it back to the file, and send (waiting for a reply) reads the message and displays it:
% SEND writes a message and reads the uppercase reply. Hello. Is there anybody out there? response from ANSWER is: HELLO. IS THERE ANYBODY OUT THERE? Enter send string (or RETURN to end): % ANSWER reads the message from SEND. Received message from SEND: Hello. Is there anybody out there?
send writes a second message to the file. answer reads it, put it into uppercase, and then writes the message to the file:
% SEND writes a second message to the shared file. I received your reply. response from ANSWER is: I RECEIVED YOUR REPLY. Enter send string (or RETURN to end): <Enter> Terminating SEND function. % ANSWER reads the second message. Received message from SEND: I received your reply.
| Using the Import Wizard | Exporting Data to MAT-Files | |
| © 1984-2008- The MathWorks, Inc. - Site Help - Patents - Trademarks - Privacy Policy - Preventing Piracy - RSS |