Error combining netcdf files

I'm having a problem when processing this netcdf data

When I run the commands, at the end the following message appears:,

Unrecognized field name "all".

Error in FUN_nc_merge (line 223)
       MV.all = [ MV.all ; tem(:) ];

The function I used is attached here with the name FUN_nc_merge.m and can be obtained through this link https://github.com/HappySpring/Easy_NetCDF

The script I used is just below and the link with google drive is the NetCDF files I used https://drive.google.com/drive/folders/1yts62Bbd5P1N_O3we3R2iQi0PgYbkJNT?usp=share_link

% input_dir: path for the folder containing the files
input_dir = '/home/augusto/Documentos/Dados_Mensais_CHIRPS';
% filelist
filelist  = dir(fullfile(input_dir,'Merge_Demo*.nc'));
% output filename
output_fn = 'Merged_Output.nc';
    
% name of the demension to be merged.
merge_dim_name = 'time';

% compatibility_mode:
%     compatibility_mode = 1: write netCDF in 'CLOBBER'; Compression would be disabled.
%     compatibility_mode = 0: write netCDF in 'NETCDF4'.
compatibility_mode = 0;

strvcat( filelist(:).name )
cd /home/augusto/Downloads/Easy_NetCDF-main
FUN_nc_merge( input_dir, filelist, output_fn, merge_dim_name, compatibility_mode )

Answer 1

[Augusto Gabriel da Costa Pereira]

see in detail at: https://la.mathworks.com/matlabcentral/answers/573892-combining-merging-output-for-multiple-netcdf-files?s_tid=srchtitle_combining%252Fmerging%20output%20for%20multiple%20netcdf%20files_1

the author: https://www.mathworks.com/matlabcentral/profile/authors/22061492

% input_dir: path for the folder containing the files
input_dir = './Data/';
% filelist
filelist  = dir(fullfile(input_dir,'data_chirps*.nc'));
% output filename
output_fn = 'Merged_Output.nc';
    
% name of the demension to be merged.
merge_dim_name = 'time';
% compatibility_mode:
%     compatibility_mode = 1: write netCDF in 'CLOBBER'; Compression would be disabled.
%     compatibility_mode = 0: write netCDF in 'NETCDF4'.
compatibility_mode = 0;
strvcat( filelist(:).name )
FUN_nc_merge( input_dir, filelist, output_fn, merge_dim_name, compatibility_mode )

Answer 2

[L Chi]

I guess I figured out what happened. I happened to introduce a bug in a recent commit on Dec 14, 2022, which does not define the variable MV.all correclty if is_overlap_allowed is false. I found it a few days later and fix it in a following commit (480d64d) on Dec 28, 2022. If you happened to download the latest toolbox between Dec 14 and Dec 28, this is probably the case. Please download the latest toolbox from github and try it again. It should work, you're welcomed to email me/open an issue on github/replay this post if you still have problems in using the toolbox.

[L Chi]

By the way, I always try to make sure the releases ( https://github.com/HappySpring/Easy_NetCDF/releases ) are free of bugs. The latest release currently is version 1.12 released on July 27, 2022. I add experimental features in commits after the latest release, however, this may also include bugs. My suggestion would be that the latest release would always be a better choice unless you need new features introduced after it.

Answer 3

[Walter Roberson]

You have the code structure

if is_overlap_allowed
    %stuff here
     MV.all = cell2mat( MV.val(:) );
else
   for ii = 1:length( filepath_list )
       fprintf('Checking: %s\n', filepath_list{ii} );
       tem = FUN_nc_varget( filepath_list{ii}, merge_dim_var_name ); 

       MV.all = [ MV.all ; tem(:) ];
       %more stuff

Notice that MV.all is not assigned to before the if and inside the if it is assigned to only if is_overlap_allowed; in the case that is_overlap_allowed is false, it is not assigned to before it is used inside the for ii loop.

[Augusto Gabriel da Costa Pereira]

I believe that this function does not FUN_nc_merge is not correct. I tried your modification and I didn't get success

[Walter Roberson]

I did not suggest any particular modification, so I do not know what you did.

Have you considered assigning

MV.all = [];

before the if is_overlap_allowed ?

[Augusto Gabriel da Costa Pereira]

before the tbefore the if is_overlap_allowed there is no MV.all

[Walter Roberson]

In the case that is_overlap_allowed is true, you assign to MV.all

In the case that is_overlap_allowed is false, you loop appending to MV.all . If that code worked, then at the end of the loop, you would have an MV.all .

So either way, after the end of the if else if the code worked, MV.all would be assigned to.

Is there any situation in which you deliberately want MV.all to not exist after the if/else ? For example if filepath_list was empty you would not execute the loop body: is it a deliberate choice on your part that after the end of the else that MV.all would not exist in that case?

Is there any situation in which MV.all could exist before the if/else and have data that you wanted to preserve ?

[Augusto Gabriel da Costa Pereira]

I just saw that you already answered the post here on mathworks and I used the code that you answered for the user

%%
clc;clear all

%%
cd /home/augusto/Documentos/Dados_Mensais_CHIRPS

ncfiles = dir('*.nc');      
Nfiles = length(ncfiles);   
all_times = cell(Nfiles,1);
for i = 1:Nfiles
    ncfiles(i).name        
    all_times{i} = ncread(ncfiles(i).name, 'time')          
end

How do I transform the structure generated in the workspace called ncfiles into NetCDF (.nc)

[Walter Roberson]

Suppose you took a census of everyone living on your block. You end up with a structure of data, one entry per person, with information listing their name, age, and address. The structure describes the people in the block.

What you are asking for is how to take the census of people and turn that into the people themselves. You cannot do that.

The variable ncfiles in the above code is a struct array, one entry for each file found. Each entry has information about the file, including its name and size and date and location. You would have no reason to convert the struct array into an .nc file itself.

If you want to know the names of the files, then they are accessible as ncfiles(INDEX).name . The code above illustrates reading one particular variable time from each variable; by adding more ncread commands inside the loop you could read additional variables.

If you want to know the names of the files ahead of time (there are various reasons to do that) then you can extract just the names as

filenames = fullfile({ncfiles.folder}, {ncfiles.name});

The result would be a cell array of character vectors, with each entry being a complete path to one .nc file.

[Augusto Gabriel da Costa Pereira]

in case I want to gather all the netcdf files I mentioned in my question into ONLY one netcdf file? if the answer is yes, which script?

[Walter Roberson]

Not enough information.

.nc files generally consist of a number of different "groups" with subgroups (essentially subdirectories), some of which might have multiple variables.

It is unclear what form you would want the combined results to be. For example should a new high-level group be formed for each input file, with the contents of the individual files placed under that top level? Or should the same group structures be preserved but one new variable be created for each input file, such as time_Qz7b18 time_Qz7b19 time_Qz7b20 variables where now you have variable time in Qz7b18.nc, Qz7b19.nc, Qz7b20.nc ? Or should nearly the same variables be preserved, but a new dimension should be added for each variable with an "index" element for the dimension being the filename? So for example instead of a 2D rainfall variable the same variable name rainfall would be used except it would become 3D instead of 2D with the third dimension being indexed by file name?

[Augusto Gabriel da Costa Pereira]

that is, I have these files: file_janeiro2000.nc file_february2000.nc file_march2000.nc the three files have days of each month from the 1st to the 31st (for February it is from the 1st to the 28th or 29th), longitude, latitude and temperature data. lat x lon x temperature and each one has a day.

I want to join the three monthly files to form just one. that is, when viewing all together they would have 90 days (January 31 days, February 28 days and March 31 days) that is, the new netcdf (.nc) file would join the three and get 90 days, latitude, longitude and respective daytime temperature, latitudes and longitude.

[Walter Roberson]

Can we assume that the latitude and longitude will be exactly the same for all three files?

[Augusto Gabriel da Costa Pereira]

yes, exactly that, latitude and longitude will be the same.

I think now you understand my question.

I've been trying to make this script for a while but I haven't been successful.