How to read all CSV files from specific folders?

Hi, I want to open all CSV files and do a calculation on them. I used the code below and it worked

files = subdir('C:\Users\roozm\Desktop\New folder\*.csv');

Subdir function can do it easily. Now I want to be more specific and only open folder with the name of BIN and then read CSV files in only BIN folders.

How can I do that?

[Paolo]

Do you have one folder with the term 'BIN' or multiple folders, let's say 'BIN1','BIN2' etc?

[Roozbeh Yousefnejad]

all the sub-folders are named BIN

I have one Newfolder. in it I have different folders with different test name. for example: F21802010055 , F21802010058 and in each of these folders I have file name BIN and I only want to read CSV files in the BIN folder.

[Walter Roberson]

subdir() probably refers to https://www.mathworks.com/matlabcentral/fileexchange/15859-subdir--a-recursive-file-search

[Walter Roberson]

Roozbeh, which MATLAB release are you using? Some of the newer releases have better directory facilities.

[Roozbeh Yousefnejad]

I don't know how use subdir for this purpose. I am using 2017b. would be great if you can tell me a better function for this purpose

[Paolo]

Roozbeh,

I suspect you can actually use Matlab's dir for this purpose. I created three different folders on my Desktop:

1. F21802010055
2. F21802010058
3. F2180201010

Within each folder, I created a folder named BIN. Within each BIN folder, I created a .csv file.

If you run the following code:

 dir('*/BIN*/*.csv')

The output is as follows:

 Files Found in: F21802010055\BIN

 test3.csv  

 Files Found in: F21802010058\BIN

 test.csv  

 Files Found in: F2180201010\BIN

 test2.csv

Does this solve the problem?

EDIT Added some folders with random names in the same directory as BIN folders and they are successfully ignored by the search.

[Roozbeh Yousefnejad]

Unfortunately not. when i use it I got this message

No matches for pattern '*/BIN*/*.csv'.

I am thinking of deleting all other folders except BIN but that could cause other problems

[Paolo]

Where are you running the command from? Are the F21802010055 folders present in your current working directory?

I am also using 2017b and the code runs fine for me.

[Paolo]

I moved up in my working directory to my User folder, ran the command:

 dir('Desktop/*/BIN*/*.csv')

And got the correct output:

 Files Found in: Desktop\F21802010055\BIN

 test3.csv  

 Files Found in: Desktop\F21802010058\BIN

 test.csv  

 Files Found in: Desktop\F2180201010\BIN

 test2.csv

[Roozbeh Yousefnejad]

I am just entering it in the command window. Honestly, I do not know how to check if the file is in my current directory.

[Paolo]

I'm not sure I understand what you mean by 'I do not know how to check if the file is in my current directory.'

Lets say your current working directory is :

 C:\Users\roozm\

as from your OP. Could you list the full path of the .csv files you want, this way we can build the dir command?

Answer 1

[Walter Roberson]

files = dir('C:\Users\roozm\Desktop\New folder\**\BIN*\*.csv');
fullpaths = fullfile({files.folder}, {files.name});

Now fullpaths is a cell array of fully qualified .csv files that are directly under a BIN* folder anywhere under "New folder"

[Roozbeh Yousefnejad]

Unfortunately, fullpath is 0*0 cell for me

[Walter Roberson]

filelocation = 'C:\Users\roozm\Desktop\New folder\F21802010055\BIN*\CSV\*.csv';
files = dir(filelocation);
fullpaths = fullfile({files.folder}, {files.name});

[Roozbeh Yousefnejad]

Thanks. It works perfectly.

[Roozbeh Yousefnejad]

The only problem I have is that it seems it can't search between all the forlders. Now I have this code

clc
clear
filelocation = 'C:\Users\roozm\Desktop\New folder\*\BIN*\CSV\*.csv';
files = dir(filelocation);
fullpaths = fullfile({files.folder}, {files.name});
M=zeros(numel(files),10); % number must change based on the column total number
for i=1:numel(files)  
  filename = files(i).name;
  sheet = 1;
[filepath,name,ext] = fileparts(filename);
N{i,1}= name;
subset = xlsread(filename,sheet,'B:F'); %AE
subset2 = xlsread(filename,sheet,'AL:AN');
subset3 = xlsread(filename,sheet,'BM:BN');
subset_merged=[subset,subset2,subset3];
subset_tot=subset_merged([end-6:end-2],:);
M(i,:)=mean(subset_tot);
 i
end
xlswrite('resultset',N,sheet,'A1')
xlswrite('resultset',M,sheet,'B1')

when I run it, I get this error

Error using xlsread (line 139)
XLSREAD unable to open file 'F21802010055_201801-21_00-17.csv'.
File 'C:\Users\roozm\Desktop\New folder\F21802010055_201801-21_00-17.csv' not found.

Error in new (line 12)
subset = xlsread(filename,sheet,'B:F'); %AE

in your code, I changed the folder name (F21802010055) that you've mentioned with * in order to cover all the subfolders in Newfolder, but it seems it doesn't work. Can you please tell how to fix it?

[Walter Roberson]

filename = fullpaths{i};

[Roozbeh Yousefnejad]

Sorry but again I got an error

Error using xlsread (line 260)
Excel Worksheet could not be activated.

Error in new (line 12)
subset = xlsread(filename,sheet,'B:F'); %AE

[Roozbeh Yousefnejad]

I do not know what is going on. with my old code I get the same name as the name I get from your code it is working, but for yours, it doesn't work

[Walter Roberson]

When you get told that a worksheet could not be activated, it had to have found the file, because it would have errored out if it had not found the file.

Not being able to activate a worksheet can occur for different reasons, including:

• the worksheet is in use by something (possibly a different process)
• sometimes when you are looping using xlsread(), you need to put in a brief pause() to give xlsread() time to completely clean up old COM objects
• activation failure can occur if you have Excel Add-ins enabled, especially FoxIt.

There might be other reasons; I do not use the ActiveX interface myself, so I have not kept close track of the possibilities.

[Roozbeh Yousefnejad]

I am afraid that there is another reason behind it as I am running the code in the debugging mode and step by step (I've attached a pic) So I believe xlsread() doesn't have a problem with the loop. Also, the files are closed.

As you can see in the picture, the curser is on the subset line and it is only the first xlsread command that I get error.

[Behtom]

Thank You !

Answer 2

[Roozbeh Yousefnejad]

I mean, I don't know how to find my directory path. The file that I want to open is in this path

C:\Users\roozm\Desktop\New folder\F21802010055\BIN\CSV

which is on my desktop.

[Paolo]

You can find the current working directory with pwd.

Change directory to new folder

 cd('C:\Users\roozm\Desktop\New folder')

and run the command I posted from the comment.

[Roozbeh Yousefnejad]

I've done this and got this message

ans =

    'C:\Users\roozm\Desktop\New folder'

so this is my directory. Then my script is also in this directory but when I run it I get this message

No matches for pattern 'Desktop/*/BIN*/*.csv'.

[Paolo]

I was referring to this command:

 dir('*/BIN*/*.csv')

Use this one if you are running from 'New folder'

[Roozbeh Yousefnejad]

same error ...

Answer 3

[Roozbeh Yousefnejad]

I replaced filename = filepaths{i}; instead of filename=files(i).name but got this error

Undefined variable "filepaths" or class "filepaths".

Error in new (line 8)
filename = filepaths{i};

[Roozbeh Yousefnejad]

is filepaths a function? should I download it and add it to my code?