No BSD License  

Highlights from
Handling Large Data Sets Efficiently in MATLAB

3.7619

3.8 | 22 ratings Rate this file 55 Downloads (last 30 days) File Size: 1.23 MB File ID: #9060
image thumbnail

Handling Large Data Sets Efficiently in MATLAB

by

 

16 Nov 2005 (Updated )

Handling Large Data Sets Efficiently in MATLAB®

Editor's Notes:

This file was a File Exchange Pick of the Week

| Watch this File

File Information
Description

Example code from "Handling Large Data Sets Efficiently in MATLAB " webinar (http://www.mathworks.com/company/events/webinars/wbnr30435.html) describing strategies for handling large amounts of data in MATLAB and avoiding "out-of-memory" errors. It will provide you with an understanding of the causes of memory limitations in MATLAB and a set of techniques to increase the available memory in MATLAB. It will also show techniques for minimizing memory usage in MATLAB while accessing, storing, processing, and plotting data.

This information is also available (and updated) in technical note 1107:
http://www.mathworks.com/support/tech-notes/1100/1107.html

Specific topics include:
* Understanding the maximum size of an array and the workspace in MATLAB
* Using undocumented features to show you the available memory in MATLAB
* Setting the 3GB switch under Windows XP to get 1GB more memory for MATLAB
* Using textscan to read large text files and memory mapping feature to read large binary files

Acknowledgements

This file inspired A Powerful Sidekick: Using My Sql For High Volume Data Manipulation In Matlab.

MATLAB release MATLAB 7.0.4 (R14SP2)
Other requirements Most techniques for available memory maxmimization (first part of presentation) are for Windows XP. Many examples need at least 1G RAM.
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (26)
29 Jul 2013 Sarah Wait Zaranek

This webinar has been recently updated. Here is a link to the latest version:

https://www.mathworks.com/company/events/webinars/wbnr33692.html

30 Jul 2010 Iqbal Nouyed

Here's the updated link to the webinar mentioned above:

http://www.mathworks.com/company/events/webinars/wbnr30435.html?id=30435&p1=32517&p2=32518

09 May 2010 Anthony

Good, but the presentation stops after processing and doesn't get to plotting.

12 Jan 2010 Stuart McGarrity

Please watch the webinar for which this is the code. The first 40 min, as you correctly say, focuses on maximizing the available memory for MATLAB. The latter 25 min focuses on minimizing MATLAB's memory usage, processing a large data set in a few ways. BTW MATLAB's out of memory errors are not directly affected by available RAM.

12 Jan 2010 Aldemar

Okay... the title says: "Handling Large Data Sets Efficiently in MATLAB".
Actually, it just shows information on how to "Handle" the system memory in order to provide more RAM for MATLAB to work with.
Not even one word is spent on an arbitrary "Data Set".
Nevertheless, this guide is useful for the above.

Conclusion:
Good work, wrong topic.

Suggestion:
Replace the title by something like "Providing MATLAB with more RAM in order not to run out of memory" or something similar.

25 Nov 2009 Danila  
29 Sep 2009 Chamane

Helpfull and really interesting

07 Aug 2009 Danila  
22 Jul 2009 Steen Johansen

Very interesting. This should be a part of the standard documentation.
I only wish I had found this when it was posted.

02 Jun 2008 Lin Dong

great

24 Mar 2008 Lizhu Qi

Very helpful. I really appreciate your work.

03 Mar 2008 Gideon Alon

Very good and useful information.

27 Feb 2008 Chinara Griffith

The information here is timely and accurate. I learnt a lot from this presentation.

26 Sep 2007 Kurt Sanger

Video for setting 3GB Switch does not play. What do I do now?

23 Aug 2007 mohamed el_amawy  
12 Jul 2007 anu puri  
13 Jun 2007 Roland May  
09 Mar 2007 syaidin dien  
05 Nov 2006 me again

setting the 3GB switch too stupid slow/big download/view

10 Oct 2006 Lorena patiño  
05 Jun 2006 Parul Gupta

I found this webinar tremendously useful - there were lots of tips and tricks I didn't know that can help better memory management

22 Apr 2006 hamed mehdizadeh  
28 Feb 2006 Gurk Peter

Gives some information on how to resolve out of memory errors, but all the tips are more or less straightforward for anyone knowing basic things about computers and operating systems.

02 Feb 2006 Jérôme Briot

I agree with John comments.

Note that the "/3gb" method only works with the Service Pack 2 of Window$ XP (SP2).

A patch is available for the SP1 and the problem seems to be fixed with the SP2 .

See there :

http://support.microsoft.com/default.aspx?scid=kb;en-us;328269

Jérôme

23 Dec 2005 FIKAYO ADERINLEWO  
18 Nov 2005 John D'Errico

A superb review of memory management for those really
big problems. He explains what the limits are, why those
limits exist, and what to do if its possible to do anything.
Its well worth your time to read.

Updates
17 Jul 2009

Removed Slides

21 Jul 2009

Removed PPT slides.

22 Jul 2009

Error in description.

04 Aug 2009

Copyright work.

04 Aug 2009

Contact us