Automatic Download from a Url when the file of the date is inputted by the user

19 views (last 30 days)
I have written a code to download a specific data from cosmic satellite website, the code is as follows
url_https='https://data.cosmic.ucar.edu';
dataUrl=strcat(url_https,"/gnss-ro/cosmic2/nrt/level1b/2022/004/podTc2_nrt_2022_004.tar.gz");
dataFile="podTc2_nrt_2022_005.tar.gz";
FileFullPath=websave(dataFile,dataUrl);
Files=untar("F:\podtc2_data\podTc2_nrt_2022_005.tar.gz",'podTc2_nrt_2022_005');
Now this code only works for one day of the data that is provided by the data portal. is there a way such that I could create a loop that modifies a string of characters corresponent to the desired date that I am looking to work with, then the web function can read the string automatically created by MATLAB.
I was wondering if a web crawler or something similar has been done on matlab?
Overall, I have currently created a code that download a specific date of data, I am looking to code a set of loops where if I enter a specific date the code would automatically crawl the web and download the specific file that I am looking for.
I have been stuck on this section for a quite a long time, hopefully you guys could give me some advice and ideas on this matter.
  2 Comments
Tianchu Lu
Tianchu Lu on 10 Aug 2022
Thank you very much for your answer benearth, so the part that should be modified would be the part of _005, as 005 means 5th of Jan 2022 and hence 001 is 1st of Jan and so on and so forth. So the pusedocode that I had in mind was that the user( myself) would enter a specific date that the I would like to investigate into, and then a loop where matlab would urlread (https://data.cosmic.ucar.edu/gnss-ro/cosmic2/nrt/level1b/2022/) and identify the date that the user want and would then download the specific zipped file of that date. so a loop where it would urlread and identify the date inputted and download the file correspond to the inputted information. But I will defo have a play with the code that you submitted below and let you know how I got on.

Sign in to comment.

Accepted Answer

Jan
Jan on 10 Aug 2022
This can be implemented with sprintf() or compose() easily.
url_https='https://data.cosmic.ucar.edu';
for year = 2020:2022 % ??? Bold guess
for day = 1:365 % ??? Bold guess
dataUrl = sprintf('%s/gnss-ro/cosmic2/nrt/level1b/%d/%03d/podTc2_nrt_%d_%03d.tar.gz', ...
url_https, year, day, year, day);
dataFile = sprintf('podTc2_nrt_%d_%03d.tar.gz', year, day);
FileFullPath = websave(dataFile, dataUrl);
end
end
  3 Comments
Débora Rodrigues
Débora Rodrigues on 20 Sep 2022 at 8:20
Edited: Débora Rodrigues on 20 Sep 2022 at 8:45
Hello! So i solved my problem!
I changed the function websave for urlwrite, and it worked! I am living it in here in case someone with similar problens appear haha.
url = 'http://ftp1.cptec.inpe.br/modelos/tempo/MERGE/GPM/DAILY';
for ano= 2010:2021
for mes= 1:12
for dia=1:30
data_url = sprintf('%s/%3d/%02d/MERGE_CPTEC_%d%02d%02d.grib2', ...
url, ano, mes, ano, mes, dia);
data_file= sprintf('MERGE_CPTEC_%d%02d%02d.grib2', ano, mes, dia);
%data = websave(data_url,data_file);
a= urlwrite(data_url,data_file);
end
end
end

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!