4 views (last 30 days)
Timothy Keane on 5 Dec 2019
Commented: Timothy Keane on 6 Dec 2019
Working with large sets of data
Goals
Extract weather station data.
Determine the top 10 weather station names that have highest single day average temperature.
Steps
https://www.ncdc.noaa.gov/orders/qclcd/ using the websave()
function in MATLAB. For example, you will find the January 2017 data in a file called QCLCD201701.zip. (QCLCD
stands for Quality Controlled Local Climate Data)
Unzip each monthly file using the unzip() function in MATLAB.
o Each zip file contains several text files. We will need two files, the daily file and the station file. For the January 2017 data, those files would be “201701daily.txt” and “201701station.txt”.
o The station file identifies each reporting weather station and assigns each a unique WBAN (Weather
Bureau Army Navy) id. When identifying a station, use the Name, State and Location fields.
o The daily file contains weather data for each day of the month from all of the weather stations. We’ll be collecting the average temperature data from the “Tavg” column.
For this project you’ll be collecting temperature data for all the weather stations and determining which
weather station has the hottest temperature for any given day in 2017. Be sure that you don’t include the same
weather station multiple times. You should report 10 unique weather stations.
There are multiple ways to process the daily weather files in MATLAB. Here is my suggestion:
o Use fopen() and fclose() to open and close the various daily files.
o Use fgets() to read one line of data from the daily weather data files that was already opened.
o Use the textscan() function and comma as the delimiter to split up the data you just read.
o textscan() returns data as a cell. A cell can contain different types of data. cell2mat() can convert cell
data to a matrix.
o If you need to convert string data to a number, use str2num().
strData = fgets (fileID);
cellData = textscan(strData, '%s', 'Delimiter', ',');
wbanID = cell2mat(cellData{1}(1)); % first field in the data
day = cell2mat(cellData{1}(2)); % second field in the data
The data is not perfect. If any data is missing (or non-numeric) do not consider that value.
Determine which weather station (Name, State & Location) has the hottest daily temperature. Display the
Name, State and Location and the highest daily temperature recorded for the top 10 stations.
If you want a sanity check of your answer, the answer will most likely be somewhere near the equator
Do not try to do everything at once. I suggest you get January to work before trying to loop over all the months.
If you don’t have a lot of disk space on your computer, use a MATLAB command to delete all the data files after
unzipping the data except for the daily and station files.
Since the processing may take some time, it may be helpful to display to the screen what is happening. An
example of how to do this is shown below.
Pay close attention to the arguments/parameters to the websave() function. You want to save a zip file, not an
html file.
When specifying full path names in Windows, remember that the \ character starts an escape character such as
‘\n’ (newline) and ‘\t’ (tab). To specify the actual \ character, it needs to be \\.
Annotations