MATLAB Answers

Downloading HYCOM data using Matlab and OPeNDAP

142 views (last 30 days)
Amitava Guha
Amitava Guha on 5 Jun 2020
Edited: Gabriel Ruiz Martinez on 19 Apr 2021 at 17:54
I am trying to download HYCOM (www.hycom.org) data using the OPeNDAP access method and MATLAB. The data url can be generated by selecting desired variables from this link. I need to download the data using scripts to automate the download of many years of data.
My MATLAB code looks like below, which you should be able to run as well without any modification:
% This script download the Global HYCOM reanalysis data for a given time step
for i=0:2863
timestep = 1; % This value need to be adjusted for duration of the record needed
j=i+timestep;
url = ['http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_53.X/data/1994?depth[0:1:39],lat[0:1:3250],lon[0:1:4499],time['...
num2str(i) ':1:' num2str(j) ...
'],tau[0:1:2864],water_u[0:1:0][0:1:0][0:1:0][0:1:0],water_u_bottom[0:1:0][0:1:0][0:1:0],water_v[0:1:0][0:1:0][0:1:0][0:1:0],water_v_bottom[0:1:0][0:1:0][0:1:0],water_temp[0:1:0][0:1:0][0:1:0][0:1:0],water_temp_bottom[0:1:0][0:1:0][0:1:0],salinity[0:1:0][0:1:0][0:1:0][0:1:0],salinity_bottom[0:1:0][0:1:0][0:1:0],surf_el[0:1:0][0:1:0][0:1:0]'];
info = ncinfo(url);
% Successfully read these variables
depth = ncread(url, 'depth');
lat = ncread(url, 'lat');
lon = ncread(url, 'lon');
time = ncread(url, 'time');
% Trouble reading below variables
water_u = ncread(url, 'water_u');
water_v = ncread(url, 'water_v');
water_u_bottom = ncread(url, 'water_u_bottom');
water_v_bottom = ncread(url, 'water_v_bottom');
surf_el = ncread(url, 'surf_el');
water_temp = ncread(url, 'water_temp');
end
The scripts reads the depth, lat, lon, time fine, but gives error in reading the water_u variable on wards. The error message is:
Error using netcdflib
The NetCDF library encountered an error during execution of 'getVarShort' function - 'Index exceeds dimension bound
(NC_EINVALCOORDS)'.
Error in netcdf.getVar (line 136)
data = netcdflib(funcstr,ncid,varid,varargin{:});
Error in internal.matlab.imagesci.nc/read (line 605)
data = netcdf.getVar(gid, varid);
Error in ncread (line 58)
vardata = ncObj.read(varName, varargin{:});
I am not sure what is wrong with my code, and couldn't find any answer searching online so far. Appreciate your help.
  1 Comment
Belinda Finlay
Belinda Finlay on 31 Jul 2020
Did you solve this as I am having the same issue (although I got the surf_el values)?

Sign in to comment.

Accepted Answer

Amitava Guha
Amitava Guha on 4 Jan 2021
I found out that downloading data from HYCOM is easier using Powershell script. Below is my code for a single grid point and for one year. I have been successful in running 5 such scripts at the same time, downloading 5 years of data for 1 grid point in 4-6 hour timeframe.
# This script is used to download current hindcast data from www.hycom.org
# Result directory
# --------------------------------------------------------
$resultDirectory = $PSScriptRoot + "\Data"
If (!(test-path $resultDirectory))
{
md $resultDirectory
}
# Input
# -------------------------------------------------------
# Select coordinate for region of interest
# Location: Latitude 6° 20' S , Longitude: 11° 15' E
#East
$east = 11.30
#West
$west = 11.25
#South
$south = -6.33
#North
$north = -6.31
$date_start = '01-Jan-1994 12:00:00'
$date_end = '31-Dec-1994 23:00:00'
# --------------------------------------------------------
# Converting dates to datetime
$startDate = [datetime]::ParseExact($date_start,'dd-MMM-yyyy HH:mm:ss',$null)
$endDate = [datetime]::ParseExact($date_end,'dd-MMM-yyyy HH:mm:ss',$null)
Write-Host "Downloading data from " $startDate.ToString('yyyy-MM-ddTHH:mm:ssZ') " to " $endDate.ToString('yyyy-MM-ddTHH:mm:ssZ')
for ($time = $startDate; $time -le $endDate; $time=$time.AddHours(3)){
$error_flag = 1
while ($error_flag -eq 1){
# Example url. Do not delete. Used for date time format reference purposes
#$url = "http://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_53.X/data/2015?var=water_u&var=water_v&north=8.6&west=-57.3&east=-57.25&south=8.5&time=2015-01-01T18:00:00Z&accept=netcdf4"
# Download url
$url = "http://ncss.hycom.org/thredds/ncss/GLBv0.08/expt_53.X/data/" + $time.ToString('yyyy') + "?var=water_u&var=water_v&north=" + $north.ToString() + "&west=" + $west.ToString() + "&east=" + $east.ToString() + "&south=" + $south.ToString() + "&time=" + $time.ToString('yyyy-MM-ddTHH:mm:ssZ') + "&accept=netcdf4"
# Output file name
$fileName = $time.ToString('yyyyMMdd_HH') + ".nc"
$output = $PSScriptRoot + "\Data\" + $fileName
Try{
# Creating a web client which has the download file functionality
#WebProxy = New-Object System.Net.WebProxy("hoeprx01.na.xom.com:8080",$true)
$WebClient = New-Object System.Net.WebClient
#$WebClient.Proxy=$WebProxy
$WebClient.DownloadFile($url,$output)
Write-Host "Successfully downloaded file:" $fileName
$error_flag = 0
}
Catch {
Write-Host $_.Exception.Message`n
$error_flag = 1
Write-Host "Retrying downloading file:" $fileName " ...."
}
}
}
  1 Comment
Muhammad Nadzrin Nazri
Muhammad Nadzrin Nazri on 3 Mar 2021
Dear Amitava,
Can you guide me to download using power shell? My area (south china sea):
North: 14
South: 0
East: 99
West: 117
I want 1 hourly, surface data from jan 2019 - Aug 2020:

Sign in to comment.

More Answers (2)

Zhiguo Mei
Zhiguo Mei on 26 Nov 2020
Maybe you can try this "water_u = ncread(url, 'water_u',[1,1,1,1],[200,200,40,1]);"
Reduce the size of the variable you want read at once,and it always works for me

Gabriel Ruiz Martinez
Gabriel Ruiz Martinez on 19 Apr 2021 at 1:54
Edited: Gabriel Ruiz Martinez on 19 Apr 2021 at 17:53
HI Amitava, thanks for sharing your Powershell script. Recently, I faced off the same problem as you. Using the NCTOOLBOX (Schilining et al., 2009), I did a Matlab script to download the HYCOM data from opeNDAP.
The code can find in Matlab File Exchange:
  2 Comments
Gabriel Ruiz Martinez
Gabriel Ruiz Martinez on 19 Apr 2021 at 16:09
Hi Muhammad,
I recognize you have to download points from an entire region. In this case, you should run my function for every point (or node). You should change the dataUrl and dataUrl1 variables with the Hindcast HYCOM dataset URL that you need to use.
My advice is to write a script where you save in vectors the inputs that function requires and then... call the function for every point.
Reading the OpeNDAP Quickstart doc, if I understood well, you can download a maximum of 50 MB of data. I think if you want to download an entire region of data, the data will exceed 50 MB, sending an error message. For this reason, maybe you need to download node by node. Taking into account this fact, I wrote the function.
Notice the URL you shared is NetcdfSubset URL, the function HYCOMnc2txtS requires the OPENDAP URL:
(I would think this the URL you need, but I don't sure)
Regards.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!