http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958
MATLAB Central Newsreader  Loading multiple files into a dataset
Feed for thread: Loading multiple files into a dataset
enus
©19942015 by MathWorks, Inc.
webmaster@mathworks.com
MATLAB Central Newsreader
http://blogs.law.harvard.edu/tech/rss
60
MathWorks
http://www.mathworks.com/images/membrane_icon.gif

Tue, 12 Jun 2012 13:08:08 +0000
Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#879818
Daniel Robbins
Hi,<br>
<br>
I have a series of .txt files which contain data I would like to load into MATLAB and combine to form a dataset. Each file contains 2 columns: time and velocity.<br>
<br>
Ideally I would like to have a dataset which has one column for time then a series of columns for the velocities of the various texts files e.g. attempt one velocity, attempt 2 velocity, attempt 3 velocity etc.<br>
<br>
I have spent some time Googling and seen a few examples of loading multiple files but not into a dataset.<br>
<br>
Please can anyone offer help?<br>
Dan

Tue, 12 Jun 2012 14:38:54 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#879831
Tom Lane
> I have a series of .txt files which contain data I would like to load into <br>
> MATLAB and combine to form a dataset. Each file contains 2 columns: time <br>
> and velocity.<br>
<br>
Dan, take a look at "help dataset/join". You might want something like<br>
<br>
join(d1,d2,'key','time')<br>
<br>
 Tom

Tue, 12 Jun 2012 14:59:08 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#879836
Daniel Robbins
Hi Tom,<br>
<br>
The join function looks very good, I think that will be great for bringing the data together. <br>
<br>
What would your recommend for loading multiple text files in to the workspace?<br>
<br>
I would like to write a script that could complete the whole process once I have saved the .txt files into a directory of their own.<br>
<br>
Thanks for your help<br>
Dan

Tue, 12 Jun 2012 16:55:42 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#879848
Steven_Lord
<br>
<br>
"Daniel Robbins" <d.w.e.robbins@gmail.com> wrote in message <br>
news:jr7lfr$au4$1@newscl01ah.mathworks.com...<br>
> Hi Tom,<br>
><br>
> The join function looks very good, I think that will be great for bringing <br>
> the data together.<br>
> What would your recommend for loading multiple text files in to the <br>
> workspace?<br>
<br>
Question 12 in the Programming section of the FAQ describes several <br>
techniques with which you can process a list of files.<br>
<br>
<a href="http://matlab.wikia.com/wiki/FAQ">http://matlab.wikia.com/wiki/FAQ</a><br>
<br>
 <br>
Steve Lord<br>
slord@mathworks.com<br>
To contact Technical Support use the Contact Us link on <br>
<a href="http://www.mathworks.com">http://www.mathworks.com</a>

Wed, 13 Jun 2012 16:37:07 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#879955
Daniel Robbins
Hi Steve,<br>
<br>
I have tried both the methods on the link you provided, but I find that all of the data is not stored, only the first number in the text file is stored. If I use the importdata function then I can load the entire text file e.g.<br>
<br>
textfiles=dir('*.txt');<br>
<br>
for k = 1: length(textfiles)<br>
textfilename = [num2str (k), '.txt];<br>
a = importdata (textfilename);<br>
end<br>
<br>
However this over writes the data with each loop. I have googled around to look if there is anyway to create sequential variables e.g. a1,a2,a3...etc. However there are lots of pages indicating this is not the way to go.<br>
<br>
The ultimate goal is to get all the text files loaded and combined in to one dataset, however I would like to be able to get MATLAB to read direct from the text files rather then load them each into excel organise them and then load using the dataset function. Obviously this would be very time consuming... <br>
<br>
Any help will be much appreciated.<br>
Thanks<br>
Dan

Wed, 13 Jun 2012 17:10:05 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#879965
Peter Perkins
Daniel, I assume you're talking about the dataset array class in the <br>
Statistics Toolbox. If you're looking for one line of code to do what <br>
you ask, I think you're out of luck. But here's a suggestion.<br>
<br>
If you read each file into a dataset, and they all have the same time <br>
vector (named the same, same exact values), you can horzcat them all <br>
together and get a single array with a single time variable:<br>
<br>
>> d1 = dataset((1:5)',rand(5,1),'VarNames',{'time' 'velocity1'});<br>
>> d2 = dataset((1:5)',rand(5,1),'VarNames',{'time' 'velocity2'});<br>
>> d3 = dataset((1:5)',rand(5,1),'VarNames',{'time' 'velocity3'});<br>
<br>
>> [d1 d2 d3]<br>
ans =<br>
time velocity1 velocity2 velocity3<br>
1 0.81472 0.09754 0.15761<br>
2 0.90579 0.2785 0.97059<br>
3 0.12699 0.54688 0.95717<br>
4 0.91338 0.95751 0.48538<br>
5 0.63236 0.96489 0.80028<br>
<br>
You'll have to give each velocity variable a unique name, but if you're <br>
reading files in a loop, that's a oneliner:<br>
<br>
>> d.Properties.VarNames{2} = num2str(i,'velocity%d')<br>
<br>
Of course, if these are really long files, you'll have spent time <br>
checking to make sure that all the time variables are the same thing, <br>
and you already know that. It may be simpler to just create one dataset <br>
from the first file, then read into a temp dataset,and do something like <br>
this:<br>
<br>
>> d.(num2str(i,'velocity%d')) = dTmp.velocity;<br>
<br>
That ".()" is the same "dynamic dot indexing" that you may have seen <br>
with struture arrays. Hope this helps.<br>
<br>
On 6/12/2012 9:08 AM, Daniel Robbins wrote:<br>
> Hi,<br>
><br>
> I have a series of .txt files which contain data I would like to load<br>
> into MATLAB and combine to form a dataset. Each file contains 2 columns:<br>
> time and velocity.<br>
><br>
> Ideally I would like to have a dataset which has one column for time<br>
> then a series of columns for the velocities of the various texts files<br>
> e.g. attempt one velocity, attempt 2 velocity, attempt 3 velocity etc.<br>
><br>
> I have spent some time Googling and seen a few examples of loading<br>
> multiple files but not into a dataset.<br>
><br>
> Please can anyone offer help?<br>
> Dan

Wed, 13 Jun 2012 17:28:06 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#879969
Daniel Robbins
Hi Peter,<br>
<br>
Thanks for your help, I think what your suggesting is the answer to the second part of of my problem  which is great btw, thank you.<br>
<br>
The main issue I have at the moment is actually loading the data into MATLAB from various textfiles without them over writing in the 'for' loop. I have managed to acheive this with the following code:<br>
<br>
textfiles=dir('*.txt');<br>
<br>
for k = 1:length(textfiles);<br>
textfilename = [num2str(k),'.txt'];<br>
a= importdata(textfilename);<br>
eval(['a', int2str(k) '= importdata(textfilename)']);<br>
end<br>
<br>
However, I have seen many warnings for this use of the eval function so I am assume its not a good way to go. Once I am confident that I have a good way of getting the data into the workspace I will follow your advice, perhaps combined with the earlier advice of Tom Lane. <br>
<br>
I have saved some .txt files with no headers and two columns of data with ';' deliminators and I am currently working with these so that I can run quick tests. So far the above code is accurate  but the various warnings around make me nervous of this code.<br>
<br>
I dont mind if the code gets long and is not the fastest (provided it doesn't take minutes to process). I just want to be sure that the data is accurate. <br>
<br>
Any and all advice is greatly appreciated.<br>
Many thanks<br>
Dan

Wed, 13 Jun 2012 17:37:41 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#879973
Steven_Lord
<br>
<br>
"Daniel Robbins" <d.w.e.robbins@gmail.com> wrote in message <br>
news:jraij6$eal$1@newscl01ah.mathworks.com...<br>
> Hi Peter,<br>
><br>
> Thanks for your help, I think what your suggesting is the answer to the <br>
> second part of of my problem  which is great btw, thank you.<br>
><br>
> The main issue I have at the moment is actually loading the data into <br>
> MATLAB from various textfiles without them over writing in the 'for' loop. <br>
> I have managed to acheive this with the following code:<br>
><br>
> textfiles=dir('*.txt');<br>
><br>
> for k = 1:length(textfiles);<br>
> textfilename = [num2str(k),'.txt'];<br>
> a= importdata(textfilename);<br>
> eval(['a', int2str(k) '= importdata(textfilename)']);<br>
> end<br>
><br>
> However, I have seen many warnings for this use of the eval function so I <br>
> am assume its not a good way to go. Once I am confident that I have a good <br>
> way of getting the data into the workspace I will follow your advice, <br>
> perhaps combined with the earlier advice of Tom Lane.<br>
<br>
Use a cell array (or if all your data is the same size, a 3dimensional <br>
array) as described in question 6 in the Programming section of the <br>
newsgroup FAQ.<br>
<br>
<a href="http://matlab.wikia.com/wiki/FAQ">http://matlab.wikia.com/wiki/FAQ</a><br>
<br>
Or read in the data file to a temporary variable, append it to the dataset <br>
in question, then overwrite that temporary variable at the next iteration. <br>
This technique is used in question 12 in the Programming section of the FAQ.<br>
<br>
 <br>
Steve Lord<br>
slord@mathworks.com<br>
To contact Technical Support use the Contact Us link on <br>
<a href="http://www.mathworks.com">http://www.mathworks.com</a>

Wed, 13 Jun 2012 17:57:07 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#879981
Daniel Robbins
Hi Steven,<br>
<br>
Thanks for the advice. I have actually looked at these sections before. I tried writing the code in as advised iin section 6 (see below)<br>
<br>
textfiles=dir('*.txt');<br>
a= zeros (1,10);<br>
<br>
for i = 1:length(textfiles);<br>
textfilename = [num2str(i),'.txt'];<br>
a{i}= importdata(textfilename);<br>
end<br>
<br>
however I get the following error message:<br>
<br>
??? Cell contents assignment to a noncell array object.<br>
<br>
Error in ==> urlgh at 8<br>
a{i}= importdata(textfilename);<br>
<br>
I have also looked at section 12, however while the code runs it does not store all the data from the text files, only the final interger stored in the text file.<br>
<br>
To date the only code I have managed to get to run and work is the one with the eval function.<br>
<br>
Any ideas?<br>
<br>
Thanks<br>
Dan

Wed, 13 Jun 2012 18:02:57 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#879984
Steven_Lord
<br>
<br>
"Daniel Robbins" <d.w.e.robbins@gmail.com> wrote in message <br>
news:jrak9j$lnn$1@newscl01ah.mathworks.com...<br>
> Hi Steven,<br>
><br>
> Thanks for the advice. I have actually looked at these sections before. I <br>
> tried writing the code in as advised iin section 6 (see below)<br>
><br>
> textfiles=dir('*.txt');<br>
> a= zeros (1,10);<br>
<br>
Preallocate a as a cell array, not a numeric array.<br>
<br>
a = cell(1, 10); % or even better<br>
a = cell(1, length(textfiles));<br>
<br>
> for i = 1:length(textfiles);<br>
> textfilename = [num2str(i),'.txt'];<br>
> a{i}= importdata(textfilename);<br>
> end<br>
><br>
> however I get the following error message:<br>
><br>
> ??? Cell contents assignment to a noncell array object.<br>
<br>
You can't assign into a numeric array using curly braces; those are only for <br>
cell arrays and objects that define what curly brace indexing means for <br>
them.<br>
<br>
 <br>
Steve Lord<br>
slord@mathworks.com<br>
To contact Technical Support use the Contact Us link on <br>
<a href="http://www.mathworks.com">http://www.mathworks.com</a>

Wed, 13 Jun 2012 18:11:07 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#879990
Daniel Robbins
Ok,<br>
<br>
I changed the brackets as suggested:<br>
<br>
textfiles=dir('*.txt');<br>
a= zeros (1,10);<br>
%for i = 1:10<br>
% a(i)=i*2;<br>
%end<br>
for i = 1:length(textfiles);<br>
textfilename = [num2str(i),'.txt'];<br>
a(i)= importdata(textfilename);<br>
end<br>
<br>
I now get the follwing error message:<br>
<br>
??? In an assignment A(I) = B, the number of elements in B and<br>
I must be the same.<br>
<br>
Error in ==> urlgh at 10<br>
a(i)= importdata(textfilename);<br>
<br>
I also noted that in section 12 of the FAQ page there is a suggestion to save within the loop but no example code explaining how to do this. For example if I were to save each section to the dataset as previously suggested then how would I tell matlab how name each section e.g. velocity1, velocity2, velocity3...etc<br>
<br>
Thanks again!

Wed, 13 Jun 2012 20:12:05 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#880004
Daniel Robbins
Apologies Steven,<br>
<br>
I missed the code in the middle of your last message (think I have been staring at the screen for too long...)<br>
<br>
I have now managed to get the following code to run:<br>
<br>
textfiles=dir('*.txt');<br>
<br>
a = cell(1, length(textfiles));<br>
<br>
for i = 1:length(textfiles);<br>
textfilename = [num2str(i),'.txt'];<br>
a{i}= importdata(textfilename);<br>
end<br>
<br>
However, while calling a provides the following display:<br>
<br>
a = <br>
[6x2 double] [6x2 double] [6x2 double] [6x2 double] [6x2 double] [6x2 double]<br>
<br>
Which presumably is the data from the text files, I have no idea how to access the data from each file or how to get them into a dataset.<br>
<br>
Please can you help me with this last issue?<br>
<br>
Many thanks for all your help so far!!<br>
Dan

Wed, 13 Jun 2012 22:29:35 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#880013
dpb
On 6/13/2012 3:12 PM, Daniel Robbins wrote:<br>
....<br>
<br>
> a{i}= importdata(textfilename);<br>
> end<br>
><br>
> However, while calling a provides the following display:<br>
><br>
> a = [6x2 double] [6x2 double] [6x2 double] [6x2 double] [6x2 double]<br>
> [6x2 double]<br>
><br>
> Which presumably is the data from the text files, I have no idea how to<br>
> access the data from each file or how to get them into a dataset.<br>
<br>
> Please can you help me with this last issue?<br>
<br>
Each subset...<br>
<br>
x=a{1}; % use the {} to dereference a cell array<br>
<br>
alldat=cat(1,a{:}); % put them all in a 2D array<br>
<br>
Read the Getting Started section on cell arrays and the section on cells <br>
in the data types section to get a better grasp on using cell arrays. <br>
In short, the cell is essentially a pointer to the data within it; <br>
referring to the cell w/ the () gets that cell but to get to the <br>
contents you must use the {}.<br>
<br>


Thu, 14 Jun 2012 08:24:07 +0000
Re: Loading multiple files into a dataset
http://www.mathworks.com/matlabcentral/newsreader/view_thread/320958#880036
Daniel Robbins
Thanks dpb, appreciate the help!