Downloading Data and Extracting Specific Data



Working with large sets of data

Goals

Download weather data

Extract weather station data.

Determine the top 10 weather station names that have highest single day average temperature.

Steps

Download all the 2017 weather data from

https://www.ncdc.noaa.gov/orders/qclcd/ using the websave()

function in MATLAB. For example, you will find the January 2017 data in a file called QCLCD201701.zip. (QCLCD

stands for Quality Controlled Local Climate Data)

Unzip each monthly file using the unzip() function in MATLAB.

o Each zip file contains several text files. We will need two files, the daily file and the station file. For the January 2017 data, those files would be “201701daily.txt” and “201701station.txt”.

o The station file identifies each reporting weather station and assigns each a unique WBAN (Weather

Bureau Army Navy) id. When identifying a station, use the Name, State and Location fields.

o The daily file contains weather data for each day of the month from all of the weather stations. We’ll be collecting the average temperature data from the “Tavg” column.

For this project you’ll be collecting temperature data for all the weather stations and determining which

weather station has the hottest temperature for any given day in 2017. Be sure that you don’t include the same

weather station multiple times. You should report 10 unique weather stations.

There are multiple ways to process the daily weather files in MATLAB. Here is my suggestion:

o Use fopen() and fclose() to open and close the various daily files.

o Use fgets() to read one line of data from the daily weather data files that was already opened.

o Use the textscan() function and comma as the delimiter to split up the data you just read.

o textscan() returns data as a cell. A cell can contain different types of data. cell2mat() can convert cell

data to a matrix.

o If you need to convert string data to a number, use str2num().



strData = fgets (fileID);



cellData = textscan(strData, '%s', 'Delimiter', ',');



wbanID = cell2mat(cellData{1}(1)); % first field in the data



day = cell2mat(cellData{1}(2)); % second field in the data



The data is not perfect. If any data is missing (or non-numeric) do not consider that value.



Determine which weather station (Name, State & Location) has the hottest daily temperature. Display the

Name, State and Location and the highest daily temperature recorded for the top 10 stations.



If you want a sanity check of your answer, the answer will most likely be somewhere near the equator

Do not try to do everything at once. I suggest you get January to work before trying to loop over all the months.

If you don’t have a lot of disk space on your computer, use a MATLAB command to delete all the data files after

unzipping the data except for the daily and station files.

Since the processing may take some time, it may be helpful to display to the screen what is happening. An

example of how to do this is shown below.

o fprintf('%s Downloading file: %s\n', datetime('now'), fileName)



Pay close attention to the arguments/parameters to the websave() function. You want to save a zip file, not an

html file.



When specifying full path names in Windows, remember that the \ character starts an escape character such as

‘\n’ (newline) and ‘\t’ (tab). To specify the actual \ character, it needs to be \\.

Annotations

[Timothy Keane]

I am a little unsure how to begin, I downloaded and unzipped the file, now I am planning on using a loop to search each month, to find the 10 hottest temperatures. I am just a beginner with programming so I am sort of stuck on how to carry this out.

[Timothy Keane]

I need to unzip the data file for each month seprately using a loop.