CF Compliance of variable length strings written by xarray

[jalder-usgs]

Question

Hi all, I have a problem that has vexed me for some time that I hope we can pin down. I'm really having trouble writing string variables into NetCDF files and have them pass the cfchecks application. The documentation for strings is here. I've created a sample in Python using xarray that produces a similar scheme:

import numpy as np
import xarray as xr

out_file = 'nc_string_test.nc'

strings = np.arange(0, 30).astype("str")

ds = xr.Dataset(
    data_vars=dict(
        str_variable=("strings", strings, dict(long_name = "strings of type string"))
    ),
    attrs=dict(Conventions="CF-1.8")   
)

ds.to_netcdf(out_file)

Here is the ncdump

netcdf nc_string_test {
dimensions:
	strings = 30 ;
variables:
	string str_variable(strings) ;
		str_variable:long_name = "strings of type string" ;

// global attributes:
		:Conventions = "CF-1.8" ;
}

From a high level, it looks a lot like the documentation. However, when I run cfchecks I get:

CHECKING NetCDF FILE: nc_string_test.nc
=====================
Using CF Checker Version 4.1.0
Checking against CF Version CF-1.8
Using Standard Name Table Version 87 (2024-11-12T12:27:52Z)
Using Area Type Table Version 11 (06 July 2023)
Using Standardized Region Name Table Version 5 (12 November 2024)


------------------
Checking variable: str_variable
------------------
ERROR: (2.2): Invalid variable type: <class 'netCDF4._netCDF4.VLType'>: string type (vlen types not supported)
WARN: (2.2): Could not get typecode of variable.  Variable types supported are: char, byte, short, int, float, real, double

ERRORS detected: 1
WARNINGS given: 1
INFORMATION messages: 0

This has been plaguing me for some time now. I don't know if it is coming from xarray or the underlying netCDF4 python library, but this is a pretty simple example that you'd think should work. I've also tried xarray engine="h5netcdf" and get the same result. I'm ultimately trying to make CF-compliant Discrete Sampling Geometries time series NetCDF files, but I use string variables for the geometry ids. I also noticed the cfchecks application lists supported types as char, byte, short, int, float, real, double, but excluding strings, which seems to run counter to the CF-conventions documentation. Is this an issue with xarray or the cfchecker application itself?

Any help would be appreciated.

[DocOtak]

Hi @jalder-usgs to get xarray to do the right CF thing here, you need to set the dtype of the encoding property of the DataArray to "S1", details in the io string encoding section of xarray's docs. If my memory isn't bad, you cannot pass in the encoding as part of the constructors for a DataArray, you need to set it after the object is created.

Your example would become:

import numpy as np
import xarray as xr

out_file = 'nc_string_test.nc'

strings = np.arange(0, 30).astype("str")

ds = xr.Dataset(
    data_vars=dict(
        str_variable=("strings", strings, dict(long_name = "strings of type string"))
    ),
    attrs=dict(Conventions="CF-1.8")
)

ds.str_variable.encoding["dtype"] = "S1"

ds.to_netcdf(out_file)

Output of ncdump -h

netcdf nc_string_test {
dimensions:
	strings = 30 ;
	string2 = 2 ;
variables:
	char str_variable(strings, string2) ;
		str_variable:long_name = "strings of type string" ;
		str_variable:_Encoding = "utf-8" ;

// global attributes:
		:Conventions = "CF-1.8" ;
}

[DocOtak]

Some followup, you are correct in that there are some shortcomings in the state of the CF Checker and the IOOS checker doesn't seem to care about these data types at all.

In my own data, I prefer using the char dtype because netCDF compression can be used with char to reduce the file size on disk. I found compression not possible with the vlen string dtype.

[jalder-usgs]

Thanks @DocOtak, this is really useful. Converting to char arrays does allow the file to pass the CF Checker. However, I want to press on this a bit more. You said "get xarray to do the right CF thing", but wasn't the file correct in the first place? The CF spec documentation says NetCDF4 strings are a supported data type (in addition to char arrays). So was the file actually wrong or is it more a matter of the CF Checker just saying it's wrong because the checker itself doesn't support vlen strings? Strings as a NetCDF data type have been in NetCDF4 since release in 2008, so it seems a bit arcane to be forced into char arrays as if we still live in the NetCDF3 world.

So yes, it does work, but we shouldn't be happy about needing to do:

ds.sel(model=b'my_climate_model_id')

Rather than:

ds.sel(model="my_climate_model_id")

Part of the task here is to get novice users to be able to apply climate data in Python, so I worry about explaining they need to use a byte array of characters, rather than just a string.

[DocOtak]

I think for this one, the CF checker is not correct. String data types are fine for variables from CF 1.8+. But not attributes yet for CF defined attributes. My group tried using string dtypes at first a few years ago, but it became quite a pain when dealing with matlab and sharing data internally so we did end up going back to netCDF3 style char arrays. I think my compression comment still applies, if you want to compress internally in netCDF string data using e.g. zlib, you still need to use the char array method.

Unless you are passing a decode_cf=False into your open/load xarray call, this should only change the serialization of the data on disk and basically be entirely transparent to you within xarray.

[jalder-usgs]

Thanks, this does seem to do the trick. I was making the mistake of manually casting da.astype('str') to force a conversion from dtype=object to dtype=U. If the dtype=U then setting encoding["dtype"] = "S" to write chars, it wouldn't transparently serialize when read back in. If I leave the DataArrays of strings as dtype=object and force the encoding on write, I seem to be good to go.

Not to harp on this, but these seem like convoluted extra steps to ensure xarray writes CF compliant files when strings "should just work" per the CF spec. So add my vote to updating CF Checker to accept vlen strings.

[DocOtak]

I've run into some relatively minor issues with the some of the new ways that numpy 2.0 deals with strings, but the speedups have been worth it for me.

It looks like maybe the string dtype was added to the checker already, but it hasn't been "released" or deployed yet. I'm not involved with that so lack intimate knowledge.