MATLAB Answers

Mat1
0

How can I load multiple dat files of different name pattern and from different directory consecutively to do something

Asked by Mat1
on 16 May 2018
Latest activity Commented on by Mat1
on 23 May 2018

Hi, I have some dat files of row*Column( 9999*10 )of names:

dir1/a_0.1_1
dir1/b_0.1_2
dir1/c_0.3_1
dir1/c_1.5_1
dir2/a_0.1_1
dir2/a_0.5_1

Now How can I load and recall then sequentially to do something? is there any short cut for loop for this specially when they are in different folder? Thank you

  0 Comments

Sign in to comment.

6 Answers

Answer by Ameer Hamza
on 16 May 2018
 Accepted Answer

You can use this FEX submission. Download it and place it in MATLAB path. To get the list of all .dat files, use it as follow

files = subdir('*.mat');

this will work, if you are present in the top folder of dir1, dir2 etc. This will give you name and path of all the '.dat' files in the subfolder. Then read and process like this

data = cell(1, length(files))
for i=1:length(files)
  filename = files(i).name;
  % load your file here, use load(), tableread() or any appropriate function depending on the type of data in .dat files
  data{i} = readData; % save your read data or do any other processing
end

  7 Comments

Hello Mr. Hamza, it Works !!!!!

files = dir('B:/et/*.dat');

thank you so much. Now it takes just several minutes by your code, what takes several hour for me. God bless you for help me learn.

You are welcome. Yes, I noticed that your code requires huge memory several 10s of GBs. I would have taken a long time. You can further decrease time by making interpolation resolution of 10^8 to smaller values, but this depends on your requirement

In case you haven't noticed, the 2 lines at end should be

ave_time_extr = ave_time/length(files);
ave_radius_extr = ave_radius/length(files);

I forget to change those lines. Therefore right now you are just adding all the columns of X and Y. To get mean value, you need to divide them with length(files).

Sign in to comment.


Answer by Mat1
on 22 May 2018

Yes Mr. Hamza you are exactly right. Now on the first day you advised to download this FEX submission if in case .dat files are in different folder. I have downloaded FEX but do not know how to put that code on the MATLAB path. I have to spent lot of time just to bring the files in the same folder then execute your code. Your code is excellent for this job, but the .dat files that I have being generated in a computer cluster where they all came with the same name 'event_time'. So when I try to put them in one folder to plot in MATLAB I have to change their name every time or they overwrite. So It would be quite robust if I could load the .dat files from different sub-folders keeping the code in the main folder. Thank you very much.

  3 Comments

You need to download the files from FEX and place it in the current folder of MATLAB. The current folder is the folder which is displayed in Current Folder window in MATLAB.

Now, suppose that you have a folder structure like this

B:  <- B drive for you computer
  other folders in the path
    folderTop <--- this folder contains all the .dat files
      folder1
        file1.dat
        file2.dat
        ...
      folder2
        file1.dat
        file2.dat
        ...
      folder3
        file1.dat
        file2.dat
        ...

You can use subdir() as follow

files = subdir('B:/other folders/folderTop/*.dat');

It will give you the similar struct given by dir but containing files in all the subfolders. You can use the remaining code same as before.

Wow!! It works like a magic. Thank you so much Mr. Hamza.

Had I knew this or when first time you mention about this if then I would understood this then It save me a lot of time. Now it seems I am learning something about matlab.

Now one last question, in line 89 where you put this command

index = find(isinf(x)); x(index) = x(index-1);

it works much better, but the reason I used this to avoid the inf from data as interpolation does not work if there are NAN, Inf, and same repetitive value in consecutive row. is there any general command to avoid that type of data value from the table entirely so that the row number remain the same for all the column.

For NaN you can use fillmissing() to replace values. Since you also want to deal with inf in the same way, so first replace all inf with NaN and then use fillmissing(). For example

x = [1 2 inf 8 9 nan 15];
x(isinf(x)) = nan;
fillmissing(x, 'linear') % using linear will avoid repeated values.
ans =
   1     2     5     8     9    12    15

Sign in to comment.


Answer by Mat1
on 23 May 2018

Thank you Mr. Hamza it is working.

  0 Comments

Sign in to comment.


Answer by Mat1
on 23 May 2018

Hello Mr. Hamza, one thing, In your corrected code if in line 98 instead of "interploation_points' if I decide to use m1 values of line 92, then it gives me unequal row length among 20 interpolation X Y pairs after interpolation has been done. the problem is then matlab can not take average of columns with unequal row length.

Is there any how it can be done in matlab with unequal matrix dimension? If not then by putting zeros on that rows which comes shorter_length than others. So basically then making the matrix of dimension of the max length row with a code and filled other shorter row length with zeros.

  1 Comment

Can you explain again, which variable have different lengths and which function is causing the error?

Sign in to comment.


Answer by Mat1
on 23 May 2018

In this attachment the value of 'm' that I am using to interpolate in line 34. Now everything remain as of your corrected code just instead of interploation_points I am using m values. now m is changing as every data has been loaded. So at the end ave_time and ave_radius is not calculated as the because of variable m the interplotion points are different and so every X Y pair has different row length.

  0 Comments

Sign in to comment.


Answer by Mat1
on 23 May 2018

I am sending the whole folder with the code and files. Thank you

  4 Comments

Show 1 older comment

Yes you are right. I think I have made the code complicated. So if instead of using any fixed interploation_poind if Ii want the m, a variable that will change the value in side the loop for every data file, then the ave_time matrix will be consist of colum vectors of different length. Will then the it possible to take the mean?

No, You must have all columns of equal length to take mean. Mean will only make sense if all columns have equal lengths. That is why you need to use fixed interpolation.

Sign in to comment.