using findstr to find what you are looking for

Here is a function that is suppose to take a datetime, reformat it in a way where it can search for a date on the following webiste, https://www.celestrak.com/SpaceData/EOP-Last5Years.txt. I am getting an error 'Error using findstr

Conversion to double from datetime is not possible.' and do not know how to fix it.

Here is the script:

datestr = datetime('now');
%startdatestr = datestr(get(handles.startedit,'string'),'yyyy mm dd');
startdatestr = datetime (('now'), 'Format','yyyy MM dd')
internet = inputdlg('Are you connected to the internet? (y/n)','Connectivity');
if(strcmp(internet,'y'))
    eopdata = urlread('https://www.celestrak.com/SpaceData/EOP-Last5Years.txt');
    ind = findstr(eopdata,startdatestr);
    xp = str2num(eopdata(ind+18:ind+26))*convrt;
    yp = str2num(eopdata(ind+28:ind+36))*convrt;
    dut1 = str2num(eopdata(ind+38:ind+47));
    lod = str2num(eopdata(ind+49:ind+58));
    % dPsi = str2num(eopdata(ind+60:ind+68));
    % dEpsilon = str2num(eopdata(ind+70:ind+78));
    % dx = str2num(eopdata(ind+80:ind+88));
    % dy = str2num(eopdata(ind+90:ind+98));
    dat = str2num(eopdata(ind+100:ind+102));
else
    localeopfile = inputdlg('Do you have the latest EOP file from https://www.celestrak.com/SpaceData/EOP-Last5Years.txt (y/n)','EOP File');
    if(strcmp(localeopfile,'y'))
        [filename,pathname] = uigetfile('*.txt', 'Select an EOP file');
        if(isempty(filename))
            return;
        end
        fid = fopen(fullfile(pathname,filename));
        datastartstr = 'BEGIN OBSERVED';
        while 1
            tline = fgetl(fid);
            if(strcmp(datastartstr,tline))
                break;
            end
        end
        while 1
            tline = fgetl(fid);
            if(~isempty(tline))
                if(strcmp(startdatestr,tline(1:10)))
                    break;
                end
            end
        end
        eopdata = tline;
        xp = str2num(eopdata(18:26))*convrt;
        yp = str2num(eopdata(28:36))*convrt;
        dut1 = str2num(eopdata(38:47));
        lod = str2num(eopdata(49:58));
        % dPsi = str2num(eopdata(60:68));
        % dEpsilon = str2num(eopdata(70:78));
        % dx = str2num(eopdata(80:88));
        % dy = str2num(eopdata(90:98));
        dat = str2num(eopdata(100:102));
    else
        xp=0;
        yp=0;
        lod=0; 
        dut1=0;
        dat=0;
        msgbox('EOP data set to zeros. EOP file can be downloaded from http://celestrak.com/SpaceData');
    end
end

Answer 1

[dpb]

Looks like hard way to go at it to me...I'd just download the data to local file and snarf it up ...

eopdata = urlread('https://www.celestrak.com/SpaceData/EOP-Last5Years.txt');
fid=fopen('EOP-Last5Years.txt','w');
fwrite(fid,eopdata,"uchar");
fid=fclose(fid);
eopdata=split(eopdata,newline);             % convert to cell array from char string
ixVNames=find(contains(eopdata,'MJD'));     % find the record with the variable names as marker
tEOP=readtable('EOP-Last5Years.txt','NumHeaderLines',ixVNames+7); % I just counted here...
tEOP.Date=datetime(tEOP.Var1,tEOP.Var2,tEOP.Var3);
vNames=split(strtrim(extractBetween(eopdata(ixVNames),'#',char(13))));
tEOP=tEOP(:,[end 5:end-1]);
tEOP.Properties.VariableNames=vNames([1 3:end]);

Above left with

>> head(tEOP)
ans =
  8×10 table
       Date         x       y      UT1-UTC    LOD     dPsi     dEpsilon     dX       dY       DAT 
    ___________    ____    ____    _______    ____    _____    ________    _____    _____    _____
    01-Jan-2016    0.05    0.26     0.08      0.00    -0.09     -0.01      -0.00     0.00    36.00
    02-Jan-2016    0.05    0.26     0.08      0.00    -0.09     -0.01      -0.00     0.00    36.00
    03-Jan-2016    0.05    0.26     0.08      0.00    -0.09     -0.01      -0.00     0.00    36.00
    04-Jan-2016    0.05    0.26     0.08      0.00    -0.09     -0.01      -0.00    -0.00    36.00
    05-Jan-2016    0.04    0.26     0.07      0.00    -0.09     -0.01      -0.00    -0.00    36.00
    06-Jan-2016    0.04    0.26     0.07      0.00    -0.09     -0.01      -0.00    -0.00    36.00
    07-Jan-2016    0.04    0.26     0.07      0.00    -0.09     -0.01      -0.00    -0.00    36.00
    08-Jan-2016    0.04    0.27     0.07      0.00    -0.09     -0.01      -0.00    -0.00    36.00
>> 

One can read the header info and do whatever it is that is needed with the NGA_COEFFICIENTS to use the data in similar manner or use textscan to parse the in-memory copy.

At this point you can do all your searching for times directly on the datetime Date value; much simpler than finding strings.

[dpb]

ADDENDUM:

You can, of course, create the table from the in-memory copy by reading it to variables via textscan, but readtable and friends haven't been made to accept in-memory text as their input, unfortunately, only physical files.

[Christina Reid]

mhhm, you were able to run this? My first error:

Unrecognized function or variable 'eop'.
Error in find_eop_v2 (line 5)
ixVNames=find(contains(split(eop,newline),'MJD'));

So I change eop to eopdata (from line 1)

and then the next error is:

vNames =

  0×1 empty cell array

Index exceeds the number of array elements (0).

Error in find_eop_v2 (line 9)
tEOP.Properties.VariableNames=vNames([1 3:end]);

[dpb]

Sorry, I tried to clean up the test code on the fly in Answer -- missed a couple of things by smooshing the conversion from the char() string returned by urlread to a cellstr() array so can find the record for variable names...that I had called the eop array, but that way I had two copies of the raw data which was wasteful.

I have now just rerun the above corrected code that does the conversion in place on eopdata array. It did return the table as shown...

[Christina Reid]

Thank you! This is perfect!

[Christina Reid]

So this is the part I struggle with . How do I set it up where it searches for my date in the format

datestr = datetime('now');
%startdatestr = datestr(get(handles.startedit,'string'),'yyyy mm dd');
startdatestr = datetime (('now'), 'Format','yyyy MM dd')

It also looks like the date format from the website and Matlab do not exactly match. Which is okay, because I can just change the format in my startdatestr variable

[dpb]

Why revert to strings to search? Use the tEOP.Date field and select whatever date(s) you wish.

>> tstart=datetime(2021,1,1)                % set a starting date
tstart = 
  datetime
   01-Jan-2021
>> tEOP_2021=tEOP(tEOP.Date>=tstart,:);     % retrieve the data starting from that point on
>> [head(tEOP_2021);tail(tEOP_2021)]
ans =
  16×10 table
       Date           x           y        UT1-UTC        LOD          dPsi      dEpsilon        dX          dY        DAT
    ___________    ________    _______    _________    __________    ________    _________    ________    _________    ___
    01-Jan-2021    0.068685    0.30411     -0.17537    -0.0005759    -0.10966    -0.005849     3.5e-05     0.000126    37 
    02-Jan-2021    0.067783    0.30545     -0.17484    -0.0004419    -0.10921    -0.005801     6.6e-05     0.000106    37 
    03-Jan-2021    0.066068    0.30671     -0.17448    -0.0002389    -0.10891    -0.005785     9.8e-05      9.6e-05    37 
    04-Jan-2021    0.063906    0.30781     -0.17432     -4.72e-05     -0.1087    -0.005813    0.000129      8.1e-05    37 
    05-Jan-2021    0.061685    0.30862     -0.17438     0.0001362    -0.10854    -0.005819    0.000161      6.6e-05    37 
    06-Jan-2021    0.060391     0.3095      -0.1746     0.0002499    -0.10846    -0.005682    0.000217     0.000116    37 
    07-Jan-2021    0.059466     0.3103     -0.17488     0.0002552    -0.10858    -0.005465     0.00028     0.000183    37 
    08-Jan-2021    0.059005    0.31105     -0.17508     0.0001439    -0.10888    -0.005272    0.000343      0.00025    37 
    23-Sep-2021     0.22239    0.31442    -0.062067    -0.0005718    -0.11791    -0.008529    0.000192    -0.000144    37 
    24-Sep-2021     0.22127    0.31312    -0.061402    -0.0006906    -0.11802    -0.008595    0.000199    -0.000143    37 
    25-Sep-2021     0.22003    0.31182    -0.060594    -0.0008548    -0.11811    -0.008748    0.000204    -0.000146    37 
    26-Sep-2021     0.21877    0.31055    -0.059618    -0.0010133    -0.11804    -0.008855    0.000207     -0.00015    37 
    27-Sep-2021     0.21755    0.30928    -0.058499     -0.001151    -0.11787    -0.008871     0.00021    -0.000151    37 
    28-Sep-2021     0.21638    0.30809    -0.057254    -0.0012513    -0.11772     -0.00887    0.000213    -0.000146    37 
    29-Sep-2021     0.21526    0.30694    -0.055945    -0.0012821    -0.11764    -0.008932    0.000215    -0.000134    37 
    30-Sep-2021     0.21422    0.30573    -0.054636    -0.0012521    -0.11746    -0.009039    0.000213     -0.00012    37 
>> 

You can also convert to a timetable in which case there's retime that lets you do all kinds of neat things or you can use grouping variables and rowfun to process the table by day, year, quarters, ...

Read up on using tables and splitapply and friends to get some ideas out of the just plain thinking...MATLAB has all kinds of tools built in, but you have to be willing to look under the hood a little...

[dpb]

ADDENDUM

" the date format from the website and Matlab do not exactly match. "

While searching for dates by matching strings is a fools errand, you can set the display format for the .Date field however you want it to look.

But, there's no reason to be going back to the web site; you've got all the data in hand and know how to get more any time you want it.

It's much more efficient to just load it all and use it than it would be to try to add on the next day's readings or such by trying to only download what you don't already have--just get it all.

If the archive "rolls off the top" and loses past history, then save your data to a .mat file and augment it if needed.

But, even if you want to add to the existing, do it by downloading everything the way we just demonstrated, then select dates after the last in the original that are in the new and add them is the way to do it. join and friends are useful there.

[dpb]

ADDENDUM 2:

OBTW, there are some missing records it seems; you'll probably want to clean up the data as follows:

>> tail(tEOP)  % original -- note last record
ans =
  8×10 table
       Date           x          y        UT1-UTC        LOD          dPsi      dEpsilon        dX          dY        DAT
    ___________    _______    _______    _________    __________    ________    _________    ________    _________    ___
...
    29-Sep-2021    0.21526    0.30694    -0.055945    -0.0012821    -0.11764    -0.008932    0.000215    -0.000134     37
    30-Sep-2021    0.21422    0.30573    -0.054636    -0.0012521    -0.11746    -0.009039    0.000213     -0.00012     37
            NaT        NaN        NaN          NaN           NaN         NaN          NaN         NaN          NaN    NaN
>> sum(isnat(tEOP.Date))    % how many are there altogether?
ans =
     4
>> tEOP(isnat(tEOP.Date),:) % look to see if there's data or what's going on...
ans =
  4×10 table
    Date     x      y     UT1-UTC    LOD    dPsi    dEpsilon    dX     dY     DAT
    ____    ___    ___    _______    ___    ____    ________    ___    ___    ___
    NaT     NaN    NaN      NaN      NaN    NaN       NaN       NaN    NaN    NaN
    NaT     NaN    NaN      NaN      NaN    NaN       NaN       NaN    NaN    NaN
    NaT     NaN    NaN      NaN      NaN    NaN       NaN       NaN    NaN    NaN
    NaT     NaN    NaN      NaN      NaN    NaN       NaN       NaN    NaN    NaN
>> tEOP=tEOP(~isnat(tEOP.Date),:);      % all are null records so delete 'em all
>> whos tEOP                            % who's left info...
  Name         Size             Bytes  Class    Attributes

  tEOP      2100x10            170918  table              

>>