MATLAB Answers

1

How to move (and copy) specific csv files from multiple sub folders into a single folder

Asked by Richard Reees on 25 Mar 2019
Latest activity Answered by Rylan Dmello on 26 Mar 2019
Good evening, I have a series of subfolders representing time steps for a set of simulations I am running e.g. 000 for start and 2800 for the end, with a single CSV file outside of the subfolder domain representing iteration counts. Within each subfolder I have a "node" csv file, that is entilled the same for each time step i.e. subfolder. What I would like to do is the following:
  1. Automatically copy each node.CSV file from each subfolder and place into a seperate one (ensuring I do not overwrite the primary files)
  2. Extract the data from the data form the 1st and 2nd column for the first time step and the 2nd from thereafter, compiling them within a single CSV file, with each column presentating a timestep
My friend kindly helped me today. The position of the "node.csv" file is not static for each folder, so the code compensates for this. I was wondering if someone could help with the efficency of the code.
% Start with a folder and get a list of all subfolders.
% Finds and prints names of all PNG, JPG, and TIF images in
% that folder and all of its subfolders.
close;clc;clear % Clear the command window.
workspace; % Make sure the workspace panel is showing.
format longg;
format compact;
% Define a starting folder.
start_path = fullfile(matlabroot);
% Ask user to confirm or change.
topLevelFolder = uigetdir(start_path);
if topLevelFolder == 0
return;
end
% Get list of all subfolders.
allSubFolders = genpath(topLevelFolder);
% Parse into a cell array.
remain = allSubFolders;
listOfFolderNames = {};
while true
[singleSubFolder, remain] = strtok(remain, ';');
if isempty(singleSubFolder)
break;
end
listOfFolderNames = [listOfFolderNames singleSubFolder];
end
numberOfFolders = length(listOfFolderNames)
%%
for k = 1 : numberOfFolders
% Get this folder and print it out.
thisFolder = listOfFolderNames{k};
fprintf('Processing folder %s\n', thisFolder);
% Get csv files.
filePattern = sprintf('%s/*.csv', thisFolder);
baseFileNames = dir(filePattern);
numberCSV = length(baseFileNames)
% Now we have a list of all files in this folder.
if numberCSV >= 1
% Go through all those image files.
for f = 1
fullFileName = fullfile(thisFolder, baseFileNames(f).name);
fprintf(' Processing CSV files %s\n', fullFileName);
end
else
fprintf(' Folder %s has no CSV files in it.\n', thisFolder);
end
end
%%
for k = 1 : numberOfFolders
% Get this folder and print it out.
thisFolder = listOfFolderNames{k};
fprintf('Processing folder %s\n', thisFolder);
% Get csv files.
filePattern = sprintf('%s/*.csv', thisFolder);
baseFileNames = dir(filePattern);
numberCSV = length(baseFileNames)
% Now we have a list of all files in this folder.
if numberCSV >= 5
% Go through all those image files.
for f = 5
fullFileName{k} = fullfile(thisFolder, baseFileNames(f).name);
%fprintf(' Processing CSV files %s\n', fullFileName);
end
else
%fprintf(' Folder %s has no CSV files in it.\n', thisFolder);
end
end
%%
nameee = 'node.csv'
for k = 1 : numberOfFolders
% Get this folder and print it out.
thisFolder = listOfFolderNames{k};
fprintf('Processing folder %s\n', thisFolder);
% Get csv files.
filePattern = sprintf('%s/*.csv', thisFolder);
baseFileNames = dir(filePattern);
numberCSV = length(baseFileNames)
% Now we have a list of all files in this folder.
for f = 1:length(baseFileNames)
if strcmp(nameee,baseFileNames(f).name)==1
fullFileName{k} = fullfile(thisFolder, baseFileNames(f).name);
else
continue
end
end
end
%%
node = nan(515,length(fullFileName))
node(:,1) = (1:515);
for k = 2:length(fullFileName)
data = xlsread(fullFileName{k});
node(:,k) = data(:,2);
end
disp('finito')
%%
xlsread(fullFileName{1})

  0 Comments

Sign in to comment.

Products


Release

R2018b

1 Answer

Answer by Rylan Dmello on 26 Mar 2019

If you have a recent version of MATLAB, you could try using TabularTextDatastore which can make your code simpler (and possibly faster, although you'll need to measure and check if this is the case with your setup). Something like the following might work for you:
% Create a datastore to refer to the CSV files.
ttds = tabularTextDatastore(topLevelFolder, "IncludeSubFolders", true);
% Make sure that only files that end with "node.csv" are in the datastore.
file_indices = endsWith(ttds.Files, "node.csv");
ttds.Files = ttds.Files(file_indices);
t = table([1:515]');
% Iterate through the files in the dataset and append to a table.
reset(ttds);
while hasdata(ttds)
% Read from each node.csv file.
[data, info] = read(ttds);
% Copy the second column of each input file into the output table.
t(:, end+1) = data(:, 2);
end
% Rename the variables in the table so that it is nicer to access each iteration's data.
t.Properties.VariableNames = ["Node", [ "Iteration" + [1:length(ttds.Files)] ]];
This creates a MATLAB table called t that contains variables for each iteration column read from each file. You can index into this using t.Node, t.Iteration1, t.Iteration2, etc.
You can also turn the table into a normal MATLAB matrix using:
node = t{:, :};
I hope this helps!

  0 Comments

Sign in to comment.