Get from Ico-github-logo

Download apps, toolboxes, and other File Exchange content using Add-On Explorer in MATLAB.

» Watch video

Highlights from

  • MappedTensor
    MappedTensor - CLASS Create and map a new file to a large variable
  • View all files
5.0 | 3 ratings Rate this file 4 Downloads (last 30 days) File Size: 772 KB File ID: #29694 Version: 1.31
image thumbnail



Dylan Muir (view profile)


13 Dec 2010 (Updated )

A better, transparent memmapfile, with complex number support.

| Watch this File

File Information

See also
If this function is useful to your academic work, please cite the publication in lieu of thanks:
Muir and Kampa, 2015. "FocusStack and StimServer: A new open source MATLAB toolchain for visual stimulation and analysis of two-photon calcium neuronal imaging data". Frontiers in Neuroinformatics.

This class transparently maps large tensors of arbitrary dimensions to temporary files on disk. Referencing is identical to a standard matlab tensor, so a MappedTensor can be passed into functions without requiring that the function be written specifically to use MappedTensors. This is opposed to memmapfile objects, which cannot be used in such a way. Being able to used MappedTensors as arguments requires that the tensor is indexed inside the function (as opposed to using the object with no indices). This implies that a function using a MappedTensor must not be fully vectorised, but must operate on the mapped tensor in segments inside a for loop.

MappedTensor also offers support for basic operations such as permute and sum, without requiring space for the tensor to be allocated in memory. memmapfile sometimes runs out of virtual addressing space, even if the data is stored only on disk. MappedTensor does not suffer from this problem.

Functions that work on every element of a tensor, with an output the same size as the input tensor, can be applied to a MappedTensor without requiring the entire tensor to be allocated in memory. This is done with the convenience function "SliceFunction".
An existing binary file can also be mapped, similarly to memmapfile. However, memmapfile offers more flexibility in terms of file format. MappedTensors transparently support complex numbers, which is an advantage over memmapfile.

mtVar = MappedTensor(500, 500, 1000, 'Class', 'single');
% A new tensor is created, 500x500x1000 of class 'single'.
% A temporary file is generated on disk to contain the data for this tensor.

for (i = 1:1000)
   mtVar(:, :, i) = rand(500, 500);
   mtVar(:, :, i) = abs(fft(mtVar(:, :, i)));

mtVar = mtVar';


mtVar(:, 1, 1)

mfSum = sum(mtVar, 3);
% The sum is performed without allocating space for mtVar in
% memory.

mtVar2 = SliceFunction(mtVar, @(m)(fft2(m), 3);
% 'fft2' will be applied to each Z-slice of mtVar
% in turn, with the result returned in the newly-created
% MappedTensor mtVar2.

clear mtVar mtVar2
% The temporary files are removed

mtVar = MappedTensor('DataDump.bin', 500, 500, 1000);
% The file 'DataDump.bin' is mapped to mtVar.

SliceFunction(mtVar, @()(randn(500, 500)), 3);
% "Slice assignment" is supported, by using "generator" functions that accept no arguments. The assignment occurs while only allocating space for a single tensor slice in memory.

mtVar = -mtVar;
mtVar = 5 + mtVar;
mtVar = 5 - mtVar;
mtVar = 12 .* mtVar;
mtVar = mtVar / 5;
% Unary and binary mathematical operations are supported, as long as they are performed with a scalar. Multiplication, division and negation take O(1) time; addition and subtraction take O(N) time.

Required Products MATLAB
MATLAB release MATLAB 7.8 (R2009a)
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (7)
25 Apr 2016 fvff

fvff (view profile)

05 Apr 2016 Dylan Muir

Dylan Muir (view profile)

Hi Avinash,

MappedTensor can't handle matlab format (.mat) data files, which can be compressed. MappedTensor can only handle uncompressed, unpacked binary data formats. If you write your tensor out to disk in a binary format using fprintf then MappedTensor should work just fine.

Let me know if you have any other questions.

Comment only
04 Apr 2016 Avinash Pujala

I've run into a problem with the use of MappedTensor that I can't quite figure out. I have a large tensor matrix in matlab of size M x N x T. I saved this to the path fPath as
Then, I try to acess this using MappedTensor as follows
MT = MappedTensor(fPath,[M N T]);
However, when I look at any frame in MT as
imagesc(MT(:,:,10)), axis image
The image appears circularly shifted along the 1st dimension. I am not sure why this is happening. Anyone have any ideas?

Comment only
10 Feb 2015 Wouter

Wouter (view profile)

Just what I needed in order to process huge (18GB) files in a memory map fashion without the excessive virtual memory usage from matlab's builtin memmapfile functionality.

07 Jul 2014 Dylan Muir

Dylan Muir (view profile)

Hi Holger,
Thanks a lot for your bug report.
I converted the single-line // comments to /*-style comments. I also fixed the htobe16 issue. However, I can't replicate the "fopen" error, and can't see what the problem would be with the line. Could you please try the updated code and let me know?

Comment only
24 Jun 2014 nightrome

I found various bugs in the code:
- Line 1590 in mapped_tensor_shim_nomex: One or more output arguments not assigned during call to "fopen".

- In the mex files: The //-style comments are not allowed in ANSI C-code, which is why I had to specify that as a compiler flag (would have been good to know).

- mapped_tensor_shim doesn't compile at all, since htobe16 passed 2 arguments, but takes just 1.

Is it possible that you rework the current for Matlab 2014?

Comment only
06 Nov 2012 Stanislas Rapacchi

14 Dec 2010 1.2

Added a brief example, more details of restrictions.

15 Dec 2010 1.3

Added support for "sum"; added SliceFunction.

21 Dec 2010 1.4

Fixed a bug in linear indexing of a permuted tensor; added support for slice assignment; added support for complex values.

23 Dec 2010 1.5

Added support for unary uplus, uminus; binary plus, minus, times, mtimes, m/l/r/divide (all with a scalar).

16 Aug 2011 1.7

Updated image

07 Nov 2011 1.8

Updated description

29 May 2012 1.9

MappedTensor now does not rely internally on memmapfile, but performs optimised direct binary file reads. It is now much faster than memmapfile, for some tasks. You can now specify a header offset to skip, when mapping an existing file.

31 May 2012 1.10

Accelerated SliceFunction; SliceFunction now provides a slice index argument; better error reporting when too many dimensions were used for indexing; SliceFunction now provides feedback during operation

09 Jul 2012 1.11

Fixed a referencing bug, where repeated indices and multi-dimensional indices were not referenced correctly on reads.

07 Aug 2012 1.13

MappedTensor now uses mex-accellerated internal functions, if possible. MappedTensor is now much faster.

25 Sep 2012 1.15

Minor bug fixes

09 Nov 2012 1.16

Fixed a regression, such that SliceFunction no longer worked. Thanks to Stanislas Rapacchi for the bug report.

02 Dec 2013 1.17

Updated description

26 Mar 2014 1.18

Updated description

18 Jun 2014 1.19

Updated summary

07 Jul 2014 1.20

Fixed several mex compilation bugs.

07 Jul 2014 1.21

Removed compiled MEX files from archive

07 Jul 2014 1.22

Fixed bug where call to fopen failed

27 Oct 2014 1.23

Updated description

05 Nov 2014 1.24

Accelerated reading of data, especially when accessing chunks of data in sequential order.

05 Dec 2014 1.25

Indexing improvement

08 Dec 2014 1.26

Added paper reference.

21 Jan 2015 1.27

Updated usage notes.

23 Apr 2015 1.28

Improved referencing to make it more similar to matlab tensors. FIxed a referencing bug involving a confusion between "58" and ":".

23 Apr 2015 1.29

Improved referencing for edge cases.

11 Nov 2015 1.30

Moved to github hosting

18 Jun 2016 1.30

Added "neuroscience" tag

22 Sep 2016 1.31

Updated formatting

04 Oct 2016 1.31

Updated description

16 Feb 2017 1.31

Updated description

Contact us