Downloading HYCOM data using Matlab and OPeNDAP

122 views (last 30 days)
I am trying to download HYCOM (www.hycom.org) data using the OPeNDAP access method and MATLAB. The data url can be generated by selecting desired variables from this link. I need to download the data using scripts to automate the download of many years of data.
My MATLAB code looks like below, which you should be able to run as well without any modification:
% This script download the Global HYCOM reanalysis data for a given time step
for i=0:2863
timestep = 1; % This value need to be adjusted for duration of the record needed
j=i+timestep;
url = ['http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_53.X/data/1994?depth[0:1:39],lat[0:1:3250],lon[0:1:4499],time['...
num2str(i) ':1:' num2str(j) ...
'],tau[0:1:2864],water_u[0:1:0][0:1:0][0:1:0][0:1:0],water_u_bottom[0:1:0][0:1:0][0:1:0],water_v[0:1:0][0:1:0][0:1:0][0:1:0],water_v_bottom[0:1:0][0:1:0][0:1:0],water_temp[0:1:0][0:1:0][0:1:0][0:1:0],water_temp_bottom[0:1:0][0:1:0][0:1:0],salinity[0:1:0][0:1:0][0:1:0][0:1:0],salinity_bottom[0:1:0][0:1:0][0:1:0],surf_el[0:1:0][0:1:0][0:1:0]'];
info = ncinfo(url);
% Successfully read these variables
depth = ncread(url, 'depth');
lat = ncread(url, 'lat');
lon = ncread(url, 'lon');
time = ncread(url, 'time');
% Trouble reading below variables
water_u = ncread(url, 'water_u');
water_v = ncread(url, 'water_v');
water_u_bottom = ncread(url, 'water_u_bottom');
water_v_bottom = ncread(url, 'water_v_bottom');
surf_el = ncread(url, 'surf_el');
water_temp = ncread(url, 'water_temp');
end
The scripts reads the depth, lat, lon, time fine, but gives error in reading the water_u variable on wards. The error message is:
Error using netcdflib
The NetCDF library encountered an error during execution of 'getVarShort' function - 'Index exceeds dimension bound
(NC_EINVALCOORDS)'.
Error in netcdf.getVar (line 136)
data = netcdflib(funcstr,ncid,varid,varargin{:});
Error in internal.matlab.imagesci.nc/read (line 605)
data = netcdf.getVar(gid, varid);
Error in ncread (line 58)
vardata = ncObj.read(varName, varargin{:});
I am not sure what is wrong with my code, and couldn't find any answer searching online so far. Appreciate your help.
  1 Comment
Belinda Finlay
Belinda Finlay on 31 Jul 2020
Did you solve this as I am having the same issue (although I got the surf_el values)?

Sign in to comment.

Accepted Answer

Amitava Guha
Amitava Guha on 4 Jan 2021
I found out that downloading data from HYCOM is easier using Powershell script. Below is my code for a single grid point and for one year. I have been successful in running 5 such scripts at the same time, downloading 5 years of data for 1 grid point in 4-6 hour timeframe.
# This script is used to download current hindcast data from www.hycom.org
# Result directory
# --------------------------------------------------------
$resultDirectory = $PSScriptRoot + "\Data"
If (!(test-path $resultDirectory))
{
md $resultDirectory
}
# Input
# -------------------------------------------------------
# Select coordinate for region of interest
# Location: Latitude 6° 20' S , Longitude: 11° 15' E
#East
$east = 11.30
#West
$west = 11.25
#South
$south = -6.33
#North
$north = -6.31
$date_start = '01-Jan-1994 12:00:00'
$date_end = '31-Dec-1994 23:00:00'
# --------------------------------------------------------
# Converting dates to datetime
$startDate = [datetime]::ParseExact($date_start,'dd-MMM-yyyy HH:mm:ss',$null)
$endDate = [datetime]::ParseExact($date_end,'dd-MMM-yyyy HH:mm:ss',$null)
Write-Host "Downloading data from " $startDate.ToString('yyyy-MM-ddTHH:mm:ssZ') " to " $endDate.ToString('yyyy-MM-ddTHH:mm:ssZ')
for ($time = $startDate; $time -le $endDate; $time=$time.AddHours(3)){
$error_flag = 1
while ($error_flag -eq 1){
# Example url. Do not delete. Used for date time format reference purposes
#$url = "http://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_53.X/data/2015?var=water_u&var=water_v&north=8.6&west=-57.3&east=-57.25&south=8.5&time=2015-01-01T18:00:00Z&accept=netcdf4"
# Download url
$url = "http://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_53.X/data/" + $time.ToString('yyyy') + "?var=water_u&var=water_v&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"
# Output file name
$fileName = $time.ToString('yyyyMMdd_HH') + ".nc"
$output = $PSScriptRoot + "\Data\" + $fileName
Try{
# Creating a web client which has the download file functionality
#WebProxy = New-Object System.Net.WebProxy("hoeprx01.na.xom.com:8080",$true)
$WebClient = New-Object System.Net.WebClient
#$WebClient.Proxy=$WebProxy
$WebClient.DownloadFile($url,$output)
Write-Host "Successfully downloaded file:" $fileName
$error_flag = 0
}
Catch {
Write-Host $_.Exception.Message`n
$error_flag = 1
Write-Host "Retrying downloading file:" $fileName " ...."
}
}
}
  5 Comments
Amitava Guha
Amitava Guha on 3 Jan 2023
Dear Greg,
Glad you found my script useful. I did encounter issues you mentioned above when there is no data available for a given timestamp and had to restart the code. For latest years, you can change the $url to anothe experiment number as shown below:
$url = "https://ncss.hycom.org/thredds/ncss/GLBy0.08/expt_93.0/uv3z?var=water_u&var=water_v&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"
Best,
Amitava
Gregory Pelletier
Gregory Pelletier on 11 Jan 2023
Dear Amitava,
Thank you very much. This worked perfectly for 2016-present.
I made the following modified version of the Powershell script that is more generalized and has all of the $url selections to use for any year between 1994 to the present. This modified version also lets the user select any of the hycom variables to download. I also added a short section with instructions to aid the user
Best
Greg
# ----------
# get_hycom_3hr.ps1
#
# Microsoft Powershell script to extract 3-hr hycom model output
# This script is used to download hycom hindcast data from www.hycom.org
# Original script by Amitava Guha
# Modified by Greg Pelletier (gjpelletier@gmail.com)
# ----------
# - - -
# INSTRUCTIONS
#
# 1) In the user input section above the while loop, specify the following:
# - specify the list of variables to be extracted from any combination of $var_list = "surf_el,water_temp,salinity,water_u,water_v"
# - specify the $west, $east, $south, and $north extent of the bounding box to be extracted
# - specify the name of the $resultDirectory where the hycom data will be saved as output
# - specify the $date_start and $date_end of the period to be extracted
# 2) In the user selection section within the while loop, select the appropriate $url for the time period and variables to be extracted
# 3) In the Windows search box, search for "Powershell" and open a Powershell window,
# and then copy and paste the entire contents of this script into the Powershell window to execute.
# If you do not have Windows, you may use this script in the new cross-platform version of PowerShell
# that is available for MacOS and Linux at the following link from Microsoft: https://aka.ms/pscore6
# 4) During execution you sould see the progress of each 3-hour file that is extracted during the period of interest
# from beginning to end. Each nc file name has the format yyyyMMdd_HH.nc to indicate the datetime stamp in UTC
#
# Occasionaly there may be times when the hycom server is not responsive
# and the script may try several times to download a particular 3-hour time.
# Each 3-hour time is saved in a separate netcdf file in the specified output folder.
# Sometimes if the hycom data are missing there may appear to be an endless loop of trials to download the missing data.
# In those cases it may be necessary to stop the script and change the $date_start to a value that resumes with non-missing data
# and then re-run the script with that new start time.
#
# Sometimes problems with endless error loops may be fixed by re-initialize a new powershell session with the following
# two commands at the Powershell prompt before submitting the rest of the script in the powershell window:
Clear-Host
Get-PSSession | Remove-PSSession
# - - -
# VERSION HISTORY
#
# v06 changed the default bounding box to Parker's LiveOcean open boundary box
# v05 added option for $var_list in user inputs
# v04 $url choices for all of the experiments in the same script to allow use for any time period from 1994-present
# - - -
# This main functional parts of this script were obtained from the following link:
# https://www.mathworks.com/matlabcentral/answers/542831-downloading-hycom-data-using-matlab-and-opendap
# by Amitava Guha on 04-Jan-2021
# modified by Greg Pelletier 10-Jan-2023
# - - -
# ----------
# ----------
# ----------
# USER INPUT SECTION
# NOTE: In addition to the user inputs in this section, the user also must select the appropriate url in the while loop
# by commenting/uncommenting the appropriate $url lines of code as needed from the choices that are provided below
# - - -
# uncomment one of the following choices for $var_list, or input any other subset of variables:
$var_list = "surf_el,water_temp,salinity,water_u,water_v"
# $var_list = "water_temp,salinity,water_u,water_v"
# $var_list = "water_temp,salinity"
# $var_list = "water_u,water_v"
# - - -
# input the bounding box west, east, south, and north for region to extract
# to download single grid points, use $east-$west and $south-$north inputs that are tiny differences, like +/- .04 difference
# Defaults below are Parker MacCready's LiveOcean Model boundary hycom extraction box:
$west = 360 - 131 # 0 to 360 degE
$east = 360 - 121 # 0 to 360 degE
$south = 39 # -80 to 80 degN
$north = 53 # -80 to 80 degN
# - - -
# input the directory where the extracted nc files are saved
# $resultDirectory = $PSScriptRoot + "\data"
$resultDirectory = "c:\data\hycom\3hr\2022\"
# $resultDirectory = "/mnt/c/data/hycom/3hr/2022/"
# -
# input the start and end datetime stamp (edit below as needed)
# Note: all years start on at 12:00 UTC on 01-Jan except as noted below regarding missing data
# -
# 1994-2015 (edit below as needed to extract up to one year at a time between 1994-2015)
# Use GLBv0.08 expt 53.X for 1994-2015:
# $date_start = '01-Jan-2015 12:00:00'
# $date_end = '31-Dec-2015 21:00:00'
# -
# 2016 (1/1/2016 to 4/30/2016)
# Use GLBv0.08/expt_56.3 for 1/1/2016 or 7/1/2014 to 4/30/2016, Base Time: 2014-07-01T12:00:00Z
# $date_start = '01-Jan-2016 12:00:00'
# $date_end = '30-Apr-2016 21:00:00'
# -
# 2016 (5/1/2016 to 12/31/2016) and 2017 (1/1/2017 to 1/31/2017)
# Use GLBv0.08/expt_57.2 for 5/1/2016 to 1/31/2017, Base Time: 2016-05-01T12:00:00Z
# NOTE: manually move the Jan-2017 data to the 2017 output folder after downloading it to the 2016 folder
# $date_start = '01-May-2016 12:00:00'
# $date_end = '31-Jan-2017 21:00:00'
# -
# 2017 (2/1/2017 to 5/31/2017)
# Use GLBv0.08/expt_92.8 for 2/1/2017 to 5/31/2017, Base Time: 2017-02-01T12:00:00Z
# $date_start = '01-Feb-2017 12:00:00'
# $date_end = '31-May-2017 21:00:00'
# -
# 2017 (6/1/2017 to 9/30/2017)
# Use GLBv0.08/expt_57.7 for 6/1/2017 to 9/30/2017, Base Time: 2017-06-01T12:00:00Z
# $date_start = '01-Jun-2017 12:00:00'
# $date_end = '30-Sep-2017 21:00:00'
# -
# 2017 (10/1/2017 to 12/31/2017)
# Use GLBv0.08/expt_92.9 for 10/1/2017 to 12/31/2017, Base Time: 2017-10-01T12:00:00Z
# $date_start = '01-Oct-2017 12:00:00'
# $date_end = '31-Dec-2017 21:00:00'
# -
# 2018
# Use GLBv0.08/expt_93.0 for 1/1/2018 to 12/31/2018 or 2/18/2020, Base Time: 2018-01-01T12:00:00Z
# $date_start = '01-Jan-2018 12:00:00'
# $date_end = '31-Dec-2018 21:00:00'
# -
# 2019-present (edit below as needed to extract up to one year at a time)
# Use GLBy0.08 expt 93.0 for 2019-present:
# 2019
# $date_start = '01-Jan-2019 12:00:00'
# $date_end = '31-Dec-2019 21:00:00'
# 2020
# $date_start = '01-Jan-2020 12:00:00'
# $date_end = '31-Dec-2020 21:00:00'
# 2021
# $date_start = '01-Jan-2021 12:00:00'
# $date_end = '31-Dec-2021 21:00:00'
# 2022
$date_start = '01-Jan-2022 12:00:00'
$date_end = '31-Dec-2022 21:00:00'
# END OF USER INPUTS
# ----------
# ----------
# ----------
# make a new directory to store the output results if it does not exist already
If (!(test-path $resultDirectory))
{
md $resultDirectory
}
# - - -
# Converting dates to datetime
$startDate = [datetime]::ParseExact($date_start,'dd-MMM-yyyy HH:mm:ss',$null)
$endDate = [datetime]::ParseExact($date_end,'dd-MMM-yyyy HH:mm:ss',$null)
Write-Host "Downloading data from " $startDate.ToString('yyyy-MM-ddTHH:mm:ssZ') " to " $endDate.ToString('yyyy-MM-ddTHH:mm:ssZ')
# - - -
# the following for loop does the extraction
for ($time = $startDate; $time -le $endDate; $time=$time.AddHours(3)){
$error_flag = 1
while ($error_flag -eq 1){
# - - -
# - - -
# - - -
# USER SELECTION OF $url (uncomment one of the $url choices below for the time period being downloaded)
# - - -
# # 1994-2015 (up to one year at a time)
# # Use GLBv0.08 expt 53.X for 1994-2015
# $url = "http://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_53.X/data/" + $time.ToString('yyyy') + "?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"
# - - -
# # 2016-2018 (note that 2016 and 2017 are split into several experiments, the user must choose the appropriate experiment for the dates being extracted)
# -
# # 2016 (1/1/2016 to 4/30/2016)
# # Use GLBv0.08/expt_56.3 for 7/1/2014 to 4/30/2016, Base Time: 2014-07-01T12:00:00Z
# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_56.3?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"
# -
# # 2016 (5/1/2016 to 12/31/2016) and 2017 (1/1/2017 to 1/31/2017)
# # Use GLBv0.08/expt_57.2 for 5/1/2016 to 1/31/2017, Base Time: 2016-05-01T12:00:00Z
# # NOTE: manually move the Jan-2017 data to the 2017 folder after downloading it to the 2016 folder
# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_57.2?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"
# -
# # 2017 (2/1/2017 to 5/31/2017)
# # Use GLBv0.08/expt_92.8 for 2/1/2017 to 5/31/2017, Base Time: 2017-02-01T12:00:00Z
# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_92.8?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"
# -
# # 2017 (6/1/2017 to 9/30/2017)
# # Use GLBv0.08/expt_57.7 for 6/1/2017 to 9/30/2017, Base Time: 2017-06-01T12:00:00Z
# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_57.7?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"
# -
# # 2017 (10/1/2017 to 12/31/2017)
# # Use GLBv0.08/expt_92.9 for 10/1/2017 to 12/31/2017, Base Time: 2017-10-01T12:00:00Z
# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_92.9?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"
# -
# # 2018
# # Use GLBv0.08/expt_93.0 for 1/1/2018 to 12/31/2018 or 2/18/2020, Base Time: 2018-01-01T12:00:00Z
# $url = "https://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_93.0?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"
# - - -
# # 2019-present (up to one year at a time)
# # Use GLBy0.08 expt 93.0 for 04-Dec-2018 to present
# # Native hycom .[ab] data converted to NetCDF at NRL, GLBy0.08 grid is 0.08 deg lon x 0.04 deg lat that covers 80 S to 90 N.
# # Note from Amitava Guha: For latest years, you can change the $url to another experiment number as shown below:
$url = "https://ncss.hycom.org/thredds/ncss/GLBy0.08/expt_93.0?var=" + $var_list + "&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"
# END OF USER SELECTION OF $url
# - - -
# - - -
# - - -
# Output file name
$fileName = $time.ToString('yyyyMMdd_HH') + ".nc"
$output = $resultDirectory + $fileName
Try{
# [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls -bor [Net.SecurityProtocolType]::Tls11 -bor [Net.SecurityProtocolType]::Tls12
# Creating a web client which has the download file functionality
#WebProxy = New-Object System.Net.WebProxy("hoeprx01.na.xom.com:8080",$true)
$WebClient = New-Object System.Net.WebClient
#$WebClient.Proxy=$WebProxy
$WebClient.DownloadFile($url,$output)
Write-Host "Successfully downloaded:" $fileName
$error_flag = 0
}
Catch {
Write-Host $_.Exception.Message`n
$error_flag = 1
Write-Host "Retrying download of file:" $fileName " ...."
}
}
}

Sign in to comment.

More Answers (2)

Zhiguo Mei
Zhiguo Mei on 26 Nov 2020
Maybe you can try this "water_u = ncread(url, 'water_u',[1,1,1,1],[200,200,40,1]);"
Reduce the size of the variable you want read at once,and it always works for me

Gabriel Ruiz-Martinez
Gabriel Ruiz-Martinez on 19 Apr 2021
Edited: Gabriel Ruiz-Martinez on 19 Apr 2021
HI Amitava, thanks for sharing your Powershell script. Recently, I faced off the same problem as you. Using the NCTOOLBOX (Schilining et al., 2009), I did a Matlab script to download the HYCOM data from opeNDAP.
The code can find in Matlab File Exchange:
  2 Comments
Muhammad Nadzrin Nazri
Muhammad Nadzrin Nazri on 19 Apr 2021
Dear Gabriel,
Can you guide me to download using MatLab as well? Instead of point interest I would like to download in form of area (south china sea):
North: 14
South: 0
East: 99
West: 117
I would like to download 1 hourly, surface elevation, U & V elevation data from jan 2019 - Aug 2020:
Thanks in advance.
Gabriel Ruiz-Martinez
Gabriel Ruiz-Martinez on 19 Apr 2021
Edited: Gabriel Ruiz-Martinez on 19 Apr 2021
Hi Muhammad,
I recognize you have to download points from an entire region. In this case, you should run my function for every point (or node). You should change the dataUrl and dataUrl1 variables with the Hindcast HYCOM dataset URL that you need to use.
My advice is to write a script where you save in vectors the inputs that function requires and then... call the function for every point.
Reading the OpeNDAP Quickstart doc, if I understood well, you can download a maximum of 50 MB of data. I think if you want to download an entire region of data, the data will exceed 50 MB, sending an error message. For this reason, maybe you need to download node by node. Taking into account this fact, I wrote the function.
Notice the URL you shared is NetcdfSubset URL, the function HYCOMnc2txtS requires the OPENDAP URL:
(I would think this the URL you need, but I don't sure)
Regards.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!