Problem handling files with 0 timesteps

[ccarouge]

I've ran this command:
nccompress -r -pa -p --nccopy -o /g/data3/hh5/tmp/LIS-WRF-CABLE/Jason and got a lot of long error messages:

/g/data3/hh5/tmp/LIS-WRF-CABLE/Jason/run2/wrfhrly_d02_2028-01-01_00:00:00 is not the same as /g/data3/hh5/tmp/LIS-WRF-CABLE/Jason/run2/tmp.nc_compress/wrfhrly_d02_2028-01-01_00:00:00
Problem comparing two netCDF files: cdo diffn /g/data3/hh5/tmp/LIS-WRF-CABLE/Jason/run2/wrfdly_d02_2028-01-01_00:00:00 /g/data3/hh5/tmp/LIS-WRF-CABLE/Jason/run2/tmp.nc_compress/wrfdly_d02_2028-01-01_00:00:00
Exception: b'Warning (cdf_check_vars) : Number of time steps undefined, skipped variable XLAT!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable XLONG!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable SUNSHINE!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable TSUNSHINE!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable PRMAX5!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable TPRMAX5!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable UV10MAX5!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable TUV10MAX5!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable PRMAX10!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable TPRMAX10!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable UV10MAX10!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable TUV10MAX10!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable PRMAX20!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable TPRMAX20!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable UV10MAX20!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable TUV10MAX20!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable PRMAX30!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable TPRMAX30!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable UV10MAX30!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable TUV10MAX30!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable PRMAX60!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable TPRMAX60!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable UV10MAX60!\nWarning (cdf_check_vars) : Number of time steps undefined, skipped variable TUV10MAX60!\ncdo diffn: Open failed on >/g/data3/hh5/tmp/LIS-WRF-CABLE/Jason/run2/wrfdly_d02_2028-01-01_00:00:00<\nUnsupported file structure\n\n'
Output file /g/data3/hh5/tmp/LIS-WRF-CABLE/Jason/run2/tmp.nc_compress/wrfdly_d02_2028-01-01_00:00:00 exists, but is not the same as the input /g/data3/hh5/tmp/LIS-WRF-CABLE/Jason/run2/wrfdly_d02_2028-01-01_00:00:00
Deleting output and recompressing

It turns out this happens when the netcdf file only has a header and not data (0 timesteps):

netcdf wrfhrly_d02_2028-01-01_00:00:00 {
dimensions:
Time = UNLIMITED ; // (0 currently)

It'd be good to add better handling of this case with a simple skipping message rather than tons of error outputs. And retrying to compress etc. And also, it shouldn't stop removing the tmp.nc_compress/ directory.

[aidanheerdegen]

I can't reproduce this:

$ cat tmp.cdl 
netcdf wrfhrly_d02_2028-01-01_00:00:00 {
dimensions:
Time = UNLIMITED ; // (0 currently)
}

ncgen -o tmp.nc tmp.cdl 

$ nccompress -o tmp.nc
Directory: ./
    Number files compressed: 1
    Total space saved: -736.00 B
    Average compression ratio: 0.06

[aidanheerdegen]

Same for --nccopy

$ nccompress --nccopy -o tmp.nc
Directory: ./
    Number files compressed: 1
    Total space saved: -672.00 B
    Average compression ratio: 0.06

[ccarouge]

You didn't use the paranoid option. The error is with the cdo diffn step.

[aidanheerdegen]

Oh right, thanks

[aidanheerdegen]

I don't think there is anything that can be done here. If the user asks for --paranoid and cdo diffn can't understand parts of the file it has to return that the files do not match, as far as cdo is concerned.

In that case, if the error output is suppressed then the user has no way of knowing why the files seemingly don't match.

[aidanheerdegen]

Could do more checks in the file input stage, but what checks? There are multiple ways a file could not be compatible, e.g. no variables, do we want to check them all?