Unexplained error on dir

I'm trying to import data files while skipping headers. This is the entirety of my code:

function[dataCurves,curveFiles,curveFilenames]=loadData(headersize,Average)
  curveFiles = dir;
  curveFilenames = {curveFiles.name};
  numberOfDataFiles=size(curveFilenames,1);
  try
    for counterCurves= 4:numberOfDataFiles
      filecell=curveFilenames(counterCurves,1);
      file=filecell{1};
      dataCurves(counterCurves-3,:,:)=transpose(dlmread(file,'\t',headersize,0));
    end
    dataCurves(:,3,:)=[]
  end

It quits, saying this:

>> [dataCurves,curveFiles,curveFilenames]=loadData(396,0)
Error in loadData (line 3)
curveFiles = dir;

Output argument "dataCurves" (and maybe others) not assigned during call to
"loadData".

>>

It worked earlier today (not very well, but at least it ran). Now, I can't get it to work, even if I restart MATLAB. If I enter the [dir] command into the MATLAB terminal it executes fine, but it will not execute from inside this function.

Any help gratefully received.

Edit:

After reading some comments and Walter's suggested answer, I removed the 'try' error handler. This caused it to throw an error at the second last line, so I removed that too, which brought it back to the same (empty) error. If I remove the offset to the counter, it throws a different error (about '.' being a directory, so can't be read - which is why I added the offset in the first place.) Any offset causes the original error to be thrown again.

I'm using 2015b on Mac OS X. The number of files in the folder is 240, and they all have the same filename stem, starting with 'i(v)'. I can run dir from the command line, no problem:

>> [dataCurves,curveFiles,curveFilenames]=loadData(396,0)
Error in loadData (line 12)
curveFiles = dir;

Output argument "dataCurves" (and maybe others) not assigned during call to
"loadData".

>> curveFiles = dir;
>>

The cell matrix resulting from this has '.', '..', and '.DS_Store' as the first three values in the first column. Thanks for your suggestions, but if I understand them correctly, they don't fit the symptoms (of my enhanced disclosure in this edited request for help). If I figure it out, I'll leave my own answer.

Edit 2: Reading Edit 1 over, I realised that it wouldn't throw an error at the second last line if it hadn't already managed dlmread, because it wouldn't have gotten there. I tackled this error ('Deletion requires an existing variable.') by dimensioning dataCurves at the beginning. Now the whole thing works. I don't know why this fixed the error with 'dir'.

Edit 3: The above didn't throw an error, but it didn't actually work. The below is what eventually worked, so I'm happy. However, I'm still mystified as to why I got the error in the first place. But thanks all for your help. I didn't actually know that dir took arguments ...

function[dataCurves,curveFiles,curveFilenames,Average]=loadData(headersize);

curveFiles = dir('*(*)*');
curveFilenames = {curveFiles.name};

numberOfDataFiles=size(curveFilenames,2)

dataCurves=zeros(numberOfDataFiles,3,200);

for counterCurves= 1:numberOfDataFiles
    filecell=curveFilenames(1,counterCurves);
    file=filecell{1};
    dataCurves(counterCurves,:,:)=transpose(dlmread(file,'\t',headersize,0));
end

dataCurves(:,3,:)=[];
dataCurves=permute(dataCurves,[3 2 1]);
Average=mean(dataCurves,3);

[Star Strider]

I couldn’t reproduce your problem with dir with my own test function. It worked correctly every time.

Does the textscan (link) function not work with your files? It would allow you to skip a constant number of headers in each file, and it can read tab-delimited files.

[dpb]

I don't think the error has anything to do with dir itself; it (the error) is saying the function didn't assign any output to the return variable dataCurves not that something else occurred. Since you don't have a catch clause for try, if it errors that's precisely what one would expect the error to be because you won't have executed any code to assign anything (even []) to the output variables.

One must presume that numberOfDataFiles <4 for the case you're trying.

ADDENDUM Of course, as Walter points out it could also error on the dlmwrite internal to the loop besides the count...

Set a breakpoint at the entry line and step through and I suspect you'll find the problem.

[dpb]

Also, I'd recast a little--

d=dir; d([d.isdir])=[];  % remove directory entries
for i=1:length(d)        % iterate over remaining files

If you need any other qualifications, either

1. use an appropriate wildcard in the filespec argument to dir or
2. use pattern matching on d.name to filter out what don't want; regexp can helpfor more selective searching than what can do with wildcards.

[Stephen23]

I suspect that using try is not helping this code at all. It would likely be more robust to use dir correctly with a match string and filter the results accordingly (e.g. using the isdir field, and/or ismember to remove names). Using dir without a match string solves this delicate task with a sledgehammer.

[dpb]

The problem is NOT in dir itself but in the handling of the returned values since you seem fixated on not using wildcard patterns or any other parsing to eliminate the unwanted and directory names from the list.

The symptoms are precisely consistent with the error analyses Walter and others have made with or without your edit; you're misinterpreting the root cause as something related to dir when it has nothing to do with it, specifically. It's done precisely what was asked of it; you're just making it much more difficult than need be.

curveFiles = dir('*(*)*');
curveFilenames = {curveFiles.name};

"... all have the same filename stem, starting with 'i(v)'"

So make use of that piece of information and make life much simpler for yourself--

curveFiles = dir('i(v)*.ext');
for counterCurves= 1:length(curveFiles)
  data=dlmread(d(i).name, ...
  ...

presuming the '(v)' is a literal string and not some numeric or other variable. If the latter, then just use 'i*.ext' where 'ext' is a placeholder for whatever the file extension actually is; put that literal string in there.

Then, of course, if you're going to be saving these in a larger array, preallocate and fill rather than dynamically reallocating, but that's a detail.

[Joseph Smerdon]

I shifted to using wildcards, as you mention, as soon as I was made aware this was an option. The case of the 'i(v)' is not fixed, hence the '*(*)*'. Without the wildcards, 'dir' was generating a file list with, on my system, the navigation dots at the top, every time, regardless of what Walter suggested, which is why I didn't accept it as an answer. It just wouldn't generate it from within the function, which is still a mystery to me. So, of the suggestions, (a) the dots were at the top of the file list, (b) the folder had more than 4 files in it and (c) the use of wildcards fixed the problem. So thanks again for the help.

[dpb]

I'm not surprised the "dot" entries were first; Walter is merely noting they're not required to be and to write code that makes that assumption is, while it may work for a given case, fragile at best.

Ah, I misread the '*()' string and was thinking it wouldn't help initially--that it would be essentially the same as '*'. My bad. It will find anything with a pair of parentheses.

That'll certainly help; at least it will cull the two relative directory "dot" entries and anything else without the parens. As long as no directory has such a name and there aren't two or more classes of these files in the subdirectory, it'll likely work without further vetting of the names before calling the reading routine.

[Joseph Smerdon]

Yes, I hadn't explained that this was for one known system where I had seen that the 'dot' entries were always first. As often happens, I embarked on this yesterday morning expecting to have a functional kludge done in ten minutes. Twenty six hours later :)....

[Walter Roberson]

!touch ! \# \$ % \& \( \) \+ \,
d = dir;
{d.name}

 ans =
  1×11 cell array
    {'!'}    {'#'}    {'$'}    {'%'}    {'&'}    {'('}    {')'}    {'+'}    {','}    {'.'}    {'..'}

OS-X El Capitan.

Filenames that begin with '-' or '*' or '"' or "'" also sort before . and ..

Answer 1

[Walter Roberson]

Either the dlmread of the very first entry attempted failed or the number of entries was less than 4.

Please keep in mind that the order of the entries is not defined by dir(), and reflects whatever the operating system tells MATLAB. In turn, the operating system relies upon whatever the file system subsystem tells it, which can depend upon what kind of file system it is and upon administrator-configurable settings such as whether the file system is case sensitive.

In practice, NTFS appears to return file names sorted by byte value. That is not the same as "." and ".." always being the first two entries: there are a series of other characters that can appear in filenames on NTFS file systems that can sort before "." (I posted a specific list a few weeks ago.)

You should not be counting on dir() returning the files in any particular order. If you want to skip some entries such as "." and ".." then you can remove directories from the list (if that is the criteria) and you can sort names yourself, and it sometimes makes sense to use the natsortorder() File Exchange Contribution.

With regards to individual files: they can be unreadable due to permission problems, or due to being in an unexpected format for dlmread. With older versions of MATLAB (ending roughly R2014a) dlmread cannot handle numeric files that have text in them, even if the text is within the header lines.

Also I recently discovered that at least on OS-X, some of the older routines such as textread() cannot handle names with non-ASCII characters such as µ . That should, however, not affect dlmread() which calls into textscan() rather than textread().

[dpb]

"...dlmread of the very first entry attempted failed.." good catch, Walter, neglected that can error in the first loop. Probably the most likely case, indeed.