Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
netCDF & Out of memory

Subject: netCDF & Out of memory

From: Jose Miguel

Date: 23 Nov, 2011 13:39:08

Message: 1 of 4

Dear users,

I'm having some problems with netCDF matlab functions.
I'm using a 32 bit Matlab in Windows XP 32 bit and raised virtual memory to 3GB.

I have a for loop reading about 30 .nc files (500MB each). I'm having the "Out of Memory error"

This is how I open the netCDF file and read variables:

for i:1:50
eval(['ncid1 = netcdf.open(''uwnd.',num2str(i),'.nc'',''NC_NOWRITE'');']);
datau=netcdf.getVar(ncid1,4);
latitud=netcdf.getVar(ncid1,1);
longitud=netcdf.getVar(ncid1,2);
presion=netcdf.getVar(ncid1,0);
.
.
.
netcdf.close(ncid1);
clear datau latitud longitud presion
end

Would it be possible to have some indication on what to do?
There must be other people having problems with netcdf

Thank you very much for your comments,
Best regards

Subject: netCDF & Out of memory

From: Ben

Date: 23 Nov, 2011 16:04:09

Message: 2 of 4

"Jose Miguel " <josemijauregui@hotmail.com> wrote in message <jait1s$kpr$1@newscl01ah.mathworks.com>...
> Dear users,
>
> I'm having some problems with netCDF matlab functions.
> I'm using a 32 bit Matlab in Windows XP 32 bit and raised virtual memory to 3GB.
>
> I have a for loop reading about 30 .nc files (500MB each). I'm having the "Out of Memory error"
>
> This is how I open the netCDF file and read variables:
>
> for i:1:50
> eval(['ncid1 = netcdf.open(''uwnd.',num2str(i),'.nc'',''NC_NOWRITE'');']);
> datau=netcdf.getVar(ncid1,4);
> latitud=netcdf.getVar(ncid1,1);
> longitud=netcdf.getVar(ncid1,2);
> presion=netcdf.getVar(ncid1,0);
> .
> .
> .
> netcdf.close(ncid1);
> clear datau latitud longitud presion
> end
>
> Would it be possible to have some indication on what to do?
> There must be other people having problems with netcdf
>
> Thank you very much for your comments,
> Best regards

Although I haven't worked with netcdf, I have had my share of out of memory issues. One problem I had in the past is similar to what you describe: I loaded in some files, did some data extraction/reorganization, kept a very small amount of data and cleared the variables that had loaded in the files. Even though I was well under my memory limit, I was still getting out of memory errors.

It appears that MATLAB was clearing the variables, but not clearing the memory that was allocated for the variables. Even though the variables are no longer accessible from the function, the memory wasn't freed, so I still had memory problems. I talked with a MATLAB representative and he seemed un-surprised by it. I don't know Java, but he said "clear" function is a request, not a command. The code will get to it when it can. Apparently it wasn't getting to it fast enough for me.

I ended up having to run the function for half my files, save the state of everything and then run another function that loaded the state back in and picked up where I left off.

Good luck!

Subject: netCDF & Out of memory

From: TideMan

Date: 23 Nov, 2011 19:17:35

Message: 3 of 4

On Nov 24, 2:39 am, "Jose Miguel " <josemijaure...@hotmail.com> wrote:
> Dear users,
>
> I'm having some problems with netCDF matlab functions.
> I'm using a 32 bit Matlab in Windows XP 32 bit and raised virtual memory to 3GB.
>
> I have a for loop reading about 30 .nc files (500MB each). I'm having the "Out of Memory error"
>
> This is how I open the netCDF file and read variables:
>
> for i:1:50
> eval(['ncid1 = netcdf.open(''uwnd.',num2str(i),'.nc'',''NC_NOWRITE'');']);
> datau=netcdf.getVar(ncid1,4);
> latitud=netcdf.getVar(ncid1,1);
> longitud=netcdf.getVar(ncid1,2);
> presion=netcdf.getVar(ncid1,0);
> .
> .
> .
> netcdf.close(ncid1);
> clear datau latitud longitud presion
> end
>
> Would it be possible to have some indication on what to do?
> There must be other people having problems with netcdf
>
> Thank you very much for your comments,
> Best regards

First of all, what possessed you to use such an ugly first line in the
loop?
Why not:
ncid1=netcdf.open(['uwnd.' num2str(i) '.nc'],'NC_NOWRITE');
or:
ncfile=['uwnd.' num2str(i) '.nc'];
ncid1=netcdf.open(ncfile,'NC_NOWRITE');

Second, you should avoid using i or j as indexes. By default they are
sqrt(-1).

Third, your problem has nothing to do with netcdf, as Ben has pointed
out.

Subject: netCDF & Out of memory

From: Jose Miguel

Date: 24 Nov, 2011 20:03:08

Message: 4 of 4

Thanks to both of you.
You're totally right about the problem, sorry about the eval statement, it was some old stuff I've just wipe out.
I'm trying to read netcdf in a more efficient way, it's almost finished, access the data instead of opening each netcdf.
REgards
Jose

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us