1d interpolation from multiple text files
Show older comments
I want to apply these two interpolations for missing temperature data in text files
- nearest neighbor interpolation
- bilinear interpolation
My text files are named with Output_00 to Output_23 in a folder. Each text file contains three columns
- First column is latitude
- Second column is longitude
- Third column consist on air temperature corresponding to that latitude and longitude.
In temperature column there are -9999.000 values which representing missing or NaN values.
I want to interpolate these values with above mentioned techniques in such a way that
- Each interpolation technique interpolate temperature(3rd column) in text file and save this interpolated text file with inter_NNH(means this text file has been interpolated with nearest neighbor) and inter_BILIN(means this text file has been interpolated with bi-linear interpolations).
- After that, there will be 72 text files remains in my folder. (24 for uninterpreted and 48 of interpolated.)
I able to attached ten text files(due to size limit) with this question. If any more requirement need tell me first.
Thanks in advance for this kind effort.
6 Comments
Muhammad Usman Saleem
on 31 Jan 2016
Muhammad Usman Saleem
on 31 Jan 2016
Stephen23
on 1 Feb 2016
You indicate that you want to interpolate using 1D interpolation, yet bilinear interpolation is 2D interpolation method.
Please clarify if you want 1D or 2D interpolation.
Muhammad Usman Saleem
on 2 Feb 2016
Muhammad Usman Saleem
on 3 Feb 2016
Stephen23
on 9 Mar 2016
Question is continued here:
Accepted Answer
Although beginners love using numbered variables and think that using numbered variables will solve all of their problems they actually just make writing code more difficult. To process these you will need to either write lots of similar lines by hand, or use dynamic variable names. These are both BAD ways to program! No matter how obvious it seems to beginners, they need to learn to avoid using numbered variables. Here is why:
Beginners need to learn that if they are numbering their variables then this is a de facto index, and that they should actually use a real index into just one variable. Then your task becomes trivial to achieve. The reason that you are stuck now is because you have chosen the slow and amateur way of coding, by using numbered variables, which makes the task very difficult.
Here is some simple code which reads the files into one variable, and then rearranges it for you to run your interpolation on:
% Read the file data:
S = dir('*Output_*.txt');
N = sort({S.name});
P = numel(N);
R = size(load(N{1},'-ascii'),1);
M = zeros(R,3,P);
for k = 1:P
M(:,:,k) = load(N{k},'-ascii');
end
tmp = 0==diff(M(:,1:2,:),1,3);
assert(all(tmp(:)),'First columns do not match')
% Rearrange:
[~,~,X] = unique(M(:,1,1));
[~,~,Y] = unique(M(:,2,1));
Z = reshape(M(:,3,:),max(Y),max(X),P);
% Detect -9999
idx = Z<-9998;
% Try your interpolation here!
29 Comments
Muhammad Usman Saleem
on 1 Feb 2016
What version of MATLAB are you using?
[NA,NA,X] = unique(M(:,1,1));
Muhammad Usman Saleem
on 1 Feb 2016
Muhammad Usman Saleem
on 2 Feb 2016
What is your question? You have supplied some input data (with NaN's) to interp1. How do you expect it to interpolate NaN values? You need to actually think about the problem you are trying to solve here: those NaN values are at the locations that you are trying to interpolate values at, and yet you are trying to supply them as input values. Interpolation algorithms do not interpolate NaN's , only finite numeric values, therefore you need to supply the interpolation tool with finite numeric values.
Hint: you don't want to pass all of the values to interp1, because some of them are NaN. Remember you can use indexing to select the non-NaN elements, and interpolate where the NaN elements are. You could use a loop, and logical indexing.
Muhammad Usman Saleem
on 2 Feb 2016
Edited: Muhammad Usman Saleem
on 2 Feb 2016
Muhammad Usman Saleem
on 3 Feb 2016
Edited: Muhammad Usman Saleem
on 3 Feb 2016
Muhammad Usman Saleem
on 3 Feb 2016
You need to learn to check your code as you write it. In particular this means checking that the results are what you expect to get. There is no point in writing these data to files until you have checked that the result is sensible. Just because interp1 is giving you an output does not mean that that output is meaningful or correct. Did you check its output? If so, how?
For example when I checked the array (by comparing it to the data files), I found a bug, and had to change this line:
Z = reshape(M(:,3,:),max(Y),max(X),P);
Your interpolation does not take into account the dimensions of the data array. You have a 3D array, and yet you are interpolating it as 1D data, thus placing some data adjacently that are not adjacent in the data array, like this:
>> X = [1,2,3;4,NaN,6;7,8,NaN].'
X =
1 4 7
2 NaN 8
3 6 NaN
>> X(~isnan(X))also
ans =
1
2
3
4
6
7
8
Do you see that the elements 3 and 4 are not close to each other in the data array, and yet get rearranged next to each other. This makes no sense.
This is why I suggested using a loop (because there can be a different number of input data values along each vector in the array).
Before you interpolate you need to answer a very important question: What dimensions do you want to interpolate over? Over the Latitude (1st dimension), Longitude (2nd dimension), the files (3rd dimension), or some combination of these? If you make it clear what dimensions you want to interpolate over than I can show you how.
Muhammad Usman Saleem
on 3 Feb 2016
"I have checked this code it is running fine" just because code is running does not mean that its output is correct. You are the code's author, you need to understand what it is doing, and you need to check that it is doing it correctly.
Your code will only interpolate the NaN values, this is true. But for NaN's at the end of a sequence of files it will interpolate with beginning of the next sequence. That is why you cannot simply interpolate all values at once, in one long vector, because this mixes the file sequence into one vector and can give meaningless values. Consider this simple example of two independent columns and we want to interpolate/extrapolate along the columns to get rid of the NaN's, using your method:
>> M = [1,2,3,NaN;NaN,NaN,0,0]'
M =
1 NaN
2 NaN
3 0
NaN 0
>> X = isnan(M);
>> Z = 1:numel(M);
>> M(X) = interp1(Z(~X),M(~X),Z(X))
M =
1.00000 1.50000
2.00000 0.75000
3.00000 0.00000
2.25000 0.00000
Ouch! Where did 2.25 come from? Or 1.5, or 0.75? Do these look correct to you? This is totally wrong! In fact our desired result is actually this matrix:
1 0
2 0
3 0
4 0
The reason why it has not interpolated along the columns is because your methods puts all of the (input) Y values into one vector, like this:
>> M(~X)
ans =
1
2
3
0
0
There are the input Y values, and interp1 provides you with some new Y values between the 3 and 0. This does not conceptually make any sense though, because the 3 and 0 are in different columns and have nothing to do with each other (remember, the columns are supposed to be independent).
So your concept is fundamentally flawed, unless you don't mind that the ends can end up with garbage values. Like I wrote in my earlier comment, you will need to use a loop and interpolate each page (3rd dimension) of the data array separately. Like this:
% Read the file data:
S = dir('Output_*.txt');
N = sort({S.name});
nmf = numel(N);
nmr = size(load(N{1},'-ascii'),1);
mat = zeros(nmr,3,nmf);
for k = 1:nmf
mat(:,:,k) = load(N{k},'-ascii');
end
tmp = 0==diff(mat(:,1:2,:),1,3);
assert(all(tmp(:)),'First columns do not match')
% Rearrange:
[VX,NA,IX] = unique(mat(:,1,1));
[VY,NA,IY] = unique(mat(:,2,1));
out = reshape(mat(:,3,:),max(IY),max(IX),nmf);
% Detect -9999
idx = out<(-9998);
% Interpolation:
vec = 1:nmf;
for r = 1:size(out,1)
for c = 1:size(out,2)
idy = squeeze(idx(r,c,:)).';
if any(idy)
Xold = vec(~idy);
Yold = squeeze(out(r,c,~idy)).';
Xnew = vec(idy);
out(r,c,idy) = interp1(Xold,Yold,Xnew,'linear','extrap');
end
end
end
Note that if the number of NaN's was the same along each page (3rd dimension), then we would not need the loops, and could use interp1's ability to works with arrays. I fixed some other bugs too, so use this code, not the one in my answer.
Muhammad Usman Saleem
on 3 Feb 2016
Edited: Muhammad Usman Saleem
on 4 Feb 2016
Muhammad Usman Saleem
on 3 Feb 2016
Many of your other questions have shown you how to save data to files, including these ones:
I am confident that you can figure out how to do that, based on the information from all of your other questions.
"I try to understand it by running each commands" that is great way to learn how code works. You should also read the documentation.
The second-to-last sentence just points out that iterp1 can also accept array inputs, and that a loop is not required when the Xin and Xout points are the same. This does not apply in your case. This is why we had to use two loops.
Muhammad Usman Saleem
on 4 Feb 2016
Edited: Muhammad Usman Saleem
on 4 Feb 2016
"Loop implementation is creating trouble" What trouble is it creating? Does it run? Do you get a result? Do you get an error message?
"How many times inner loop run" presumably someone with a "BS(Hon's) in space science" and a "M.Phil in Geomatics" can also figure this one out. Why not look at the code and figure it out:
- what the loops are iterating over
- then you will know how many times the inner loop runs.
If you are still confused about loops, then try doing the introductory tutorials:
"what check is doing here" perhaps you are referring to that if statement: it checks if any of the selected values (i.e. along the 3rd dimension, the page) need to be interpolated. If there are none (i.e. no -9999 values), then there is no point in running interp1.
"searching on google for new function" this is very good idea. Then you can learn to read the official MATLAB documentation.
Muhammad Usman Saleem
on 4 Feb 2016
Edited: Muhammad Usman Saleem
on 4 Feb 2016
A cup of tea would be nice.... could you send me some?
Here are ways to figure out how many times the inner loop runs:
- add a counter variable, and check its value after the code finishes (a universal solution).
- multiply the number of times the outer and inner loops run. This is easy because you can read the code (hint: look for size(out,...), and note that this is the number of rows and columns of the 3D data array).
- Understand the algorithm: each latitude and longitude is independently interpolated (as you requested), therefore it must iterate for each latitude and longitude...
- which is incidentally the number of lines of data in the data files.
Just out of curiosity, what do you need this value for?
Muhammad Usman Saleem
on 5 Feb 2016
Edited: Muhammad Usman Saleem
on 5 Feb 2016
Try this:
% Read the file data:
S = dir('Output_*.txt');
N = sort({S.name});
nmf = numel(N);
nmr = size(load(N{1},'-ascii'),1);
mat = zeros(nmr,3,nmf);
for k = 1:nmf
mat(:,:,k) = load(N{k},'-ascii');
end
tmp = 0==diff(mat(:,1:2,:),1,3);
assert(all(tmp(:)),'First columns do not match')
% Rearrange:
[VC,NA,IC] = unique(mat(:,1,1));
[VR,NA,IR] = unique(mat(:,2,1));
out = reshape(mat(:,3,:),numel(VR),numel(VC),nmf);
% Detect -9999:
idx = out<(-9998);
% Interpolate:
vec = 1:nmf;
for r = 1:numel(VR)
for c = 1:numel(VC)
idy = squeeze(idx(r,c,:)).';
if any(idy)
Xold = vec(~idy);
Yold = squeeze(out(r,c,~idy)).';
Xnew = vec(idy);
out(r,c,idy) = interp1(Xold,Yold,Xnew,'linear','extrap');
end
end
end
% Write to file:
pressures = [1000, 925, 850, 700, 600, 500, 400, 300, 250, 200, 150, 100, 70, 50, 30, 20, 15, 10, 7, 5, 3, 2, 1.5, 1] ;
for k = 1:nmf
fprintf('Export Interpolation. %d -> p=%.1f hpa.\n', k-1, pressures(k) );
str = sprintf( 'Interpolated_%d.txt', k-1);
fid = fopen(str,'wt');
fprintf(fid,' \t \t \n');
tmp = out(:,:,k);
fprintf(fid,'%.3f\t%.3f\t%.3f\n',[mat(:,1:2),tmp(:)].');
fclose(fid);
end
Note that I used fopen with the 't' option selected, so that you do not need to specify the carriage-return character with every newline.
Muhammad Usman Saleem
on 8 Feb 2016
@Muhammad Usman Saleem: don't repeat code like that. The one thing computers are good at is repeatedly doing tasks, so why waste your time doing the computers work. Use a for-loop and loop over a cell array of the interpolation methods (as strings). This will be easier to understand and easier to maintain. Whenever you find yourself copying code like that either:
- Use a loop.
- Write a function.
Basically do this:
metC = {'linear','pchip','...'};
for m = 1:numel(metC)
metS = metC{m} % the method
... my code
out(r,c,idy) = interp1(Xold,Yold,Xnew,metS,'extrap');
...
end
I am sure you can figure out how to put the method in the filenames too (hint: the best method is to use sprintf).
Muhammad Usman Saleem
on 8 Feb 2016
Muhammad Usman Saleem
on 8 Feb 2016
Edited: Muhammad Usman Saleem
on 8 Feb 2016
The string (interpolation method) needs to be its own input into sprintf:
str = sprintf('Interpolated_%s_%d.txt', metS, k-1);
Read the sprintf documentation to know how this works: search for %s.
Optional Change: note that the code currently reads the original data files for every interpolation method. To speed up the code we could read the original files just once and then interpolate on the same data multiple times. This works because we do not change the original data points.
It just requires moving that for-loop from the start to here:
... all of the file reading
% Interpolate:
vec = 1:nmf;
for m = 1:numel(metC) % <- insert |for| here
metS = metC{m} % <- and the method here
for r = 1:numel(VR)
... all of the interpolation
Now the files only get read once, and the code will be quicker.
Muhammad Usman Saleem
on 8 Feb 2016
Your code looks okay. You should test it thoroughly yourself by comparing with a few hand-calculated interpolations.
PS: "Read series of characters, until find white space" is from sscanf, not sprintf. You used the wrong function documentation.
Muhammad Usman Saleem
on 15 Feb 2016
Edited: Muhammad Usman Saleem
on 16 Feb 2016
This is a slightly more complicated task because the interp2 function only accepts "2-D gridded data in meshgrid format". Because of your missing data you do not have gridded data, but scattered data points.
You could resolve this yourself by browsing the MATLAB documentation. Start at the link for interp2 above. You read the very first line and realize that this function will not work (because you do not have gridded data). So you look at the contents on the left-hand side of the page. You click on "Interpolation", because that seems to be the best place to look. Then you see a page with a list of three subcategories: 1-D Interpolation, Gridded Data Interpolation, and Scattered Data Interpolation. Which of these do you think would be best for interpolating your scattered data? Click on the link... there are several functions, and pages of examples.
Try the examples. Think about your task and how to solve it using those examples.
If you get stuck, then ask a new question (with a link to this one) showing how you modified those examples to fit your task.
More Answers (0)
Categories
Find more on Interpolation of 2-D Selections in 3-D Grids in Help Center and File Exchange
Products
on 30 Jan 2016
on 9 Mar 2016
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
