MATLAB Answers

Matlab slow when too many files in directory

50 views (last 30 days)
Amritsari sandhu
Amritsari sandhu on 11 Jan 2014
Commented: Stephen Cobeldick on 13 Jan 2019
I have a large number of data files in the my working directory. About 10000 - 30000. Whenever the number of files gets to be too many (looks like a few thousand will do it), matlab becomes very slow to do anything. Is there a way to prevent this ? Making sub-directories is not an option. The files are on a local disk on the computer, not on a network disk. I only load or save to one file at a time.

  9 Comments

Show 6 older comments
renaud
renaud on 22 May 2014
Same problem with Matlab R2013a and Ubuntu 14.04.I tried to run with the -nodesktop option, it does help but not for long. If i downgrade (R2012b, R2011b, R2010a or R2009a) will i still get the same problem?
Aaron Sigut
Aaron Sigut on 18 Jun 2014
I have encountered the same problem with R2013a under Ubuntu 12.04. But it is easy to resolve. Don't run Matlab in a directory containing thousands of files. Just place all of the files in a Data subdirectory and give the path. Say I want all of files of the form SPEC.* from the subdirectory MyData. I would set STEM='MyData' and then
d=dir(STEM,'/','SPEC.*');
fname=[STEM,'/',d(1).name];
a=load(filename);
etc...
Make STEM a varargin, set to a default of '.' for flexibility.
Stephen Cobeldick
Stephen Cobeldick on 13 Jan 2019
"As soon as I cd to the directory..."
And that is the problem right there. Using cd in order to access data files slows MATLAB down, makes debugging harder, and is totally unnecessary. All MATLAB functions that read/write data files accept absolute/relative filenames, so you can simply avoid this whole problem by writing better code:
  1. keep data and code in separate folders,
  2. use absolute/relative filenames instead of cd.
Note that it is recommended to use fullfile to create filenames, rather than string concatenation.

Sign in to comment.

Answers (2)

Matt J
Matt J on 12 Jan 2014
Edited: Matt J on 12 Jan 2014
Do you have the Current Folder window open? Do things improve if you close it (before CD-ing to the directory)?

  0 Comments

Sign in to comment.


James Tursa
James Tursa on 18 Jun 2014
Every time you change the current directory, MATLAB has to examine each and every file in the new directory to see if there are any m-files, mex-files, etc. If so, these will shadow any other m-files or mex-files etc that are on the MATLAB path. This can take some time for MATLAB to determine. Bottom line is don't have a huge number of files in these directories (e.g., see the last comment above).

  1 Comment

Aaron Sigut
Aaron Sigut on 19 Jun 2014
Yes, I thought this is what was happening. One additional way to handle this would be to remove the current directory from the search path, but this does not seem to be possible. The current directory seems to be hard-wired, perhaps because of naming issues (i.e. m-files in the current directory take precedent over other m-files with the same name). The bottom line is that this behaviour by Matlab actually serves to enforce good programming practice: don't mix your source codes in with your data.

Sign in to comment.

Sign in to answer this question.

Tags