Matlab slow when too many files in directory

35 views (last 30 days)
Amritsari sandhu
Amritsari sandhu on 11 Jan 2014
Commented: Stephen on 13 Jan 2019
I have a large number of data files in the my working directory. About 10000 - 30000. Whenever the number of files gets to be too many (looks like a few thousand will do it), matlab becomes very slow to do anything. Is there a way to prevent this ? Making sub-directories is not an option. The files are on a local disk on the computer, not on a network disk. I only load or save to one file at a time.
  9 Comments
Stephen
Stephen on 13 Jan 2019
"As soon as I cd to the directory..."
And that is the problem right there. Using cd in order to access data files slows MATLAB down, makes debugging harder, and is totally unnecessary. All MATLAB functions that read/write data files accept absolute/relative filenames, so you can simply avoid this whole problem by writing better code:
  1. keep data and code in separate folders,
  2. use absolute/relative filenames instead of cd.
Note that it is recommended to use fullfile to create filenames, rather than string concatenation.

Sign in to comment.

Answers (2)

Matt J
Matt J on 12 Jan 2014
Edited: Matt J on 12 Jan 2014
Do you have the Current Folder window open? Do things improve if you close it (before CD-ing to the directory)?

James Tursa
James Tursa on 18 Jun 2014
Every time you change the current directory, MATLAB has to examine each and every file in the new directory to see if there are any m-files, mex-files, etc. If so, these will shadow any other m-files or mex-files etc that are on the MATLAB path. This can take some time for MATLAB to determine. Bottom line is don't have a huge number of files in these directories (e.g., see the last comment above).
  1 Comment
Aaron Sigut
Aaron Sigut on 19 Jun 2014
Yes, I thought this is what was happening. One additional way to handle this would be to remove the current directory from the search path, but this does not seem to be possible. The current directory seems to be hard-wired, perhaps because of naming issues (i.e. m-files in the current directory take precedent over other m-files with the same name). The bottom line is that this behaviour by Matlab actually serves to enforce good programming practice: don't mix your source codes in with your data.

Sign in to comment.

Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!