'/' in variable name causes to_netcdf() to crash

[28raining]

What happened?

If variable name includes a '/' then to_netcdf crashes

What did you expect to happen?

No response

Minimal Complete Verifiable Example

import pandas as pd
df = pd.read_csv('slashCSV.csv')
index = ['temperature']
df = df.set_index(index)
xarr = df.to_xarray()
xarr.to_netcdf("new.nc")


- slashCSV.csv contents:
PASS/FAIL,temperature
PASS, 10
FAIL, 20

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

File "/opt/homebrew/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 56, in __init__
    array = self.get_array()
            ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 83, in get_array
    variable = ds.variables[self.variable_name]
               ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
KeyError: 'PASS/FAIL'

Anything else we need to know?

No response

Environment

Details

INSTALLED VERSIONS

commit: None
python: 3.11.2 (main, Mar 24 2023, 00:16:47) [Clang 14.0.0 (clang-1400.0.29.202)]
python-bits: 64
OS: Darwin
OS-release: 22.5.0
machine: arm64
processor: arm
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.9.1

xarray: 2023.2.0
pandas: 1.5.3
numpy: 1.24.2
scipy: 1.10.1
netCDF4: 1.6.3
pydap: None
h5netcdf: 1.1.0
h5py: 3.8.0
Nio: None
zarr: 2.14.2
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.3.6
cfgrib: 0.9.10.3
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2023.3.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 65.6.3
pip: 23.1.1
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None

[jhamman]

This example runs fine for me:

df = pd.DataFrame()
df["PASS/FAIL"] = ["PASS", "FAIL"]
df["temperature"] = [10, 20]
df = df.set_index("PASS/FAIL")

index = ['temperature']
df = df.set_index(index)
xarr = df.to_xarray()
xarr.to_netcdf("new.nc")

xarr.info()

xarray.Dataset {
dimensions:
	temperature = 2 ;

variables:
	int64 temperature(temperature) ;

// global attributes:
}

Can you include the full traceback of your error?

[28raining]

  File "REMOVED_PERSONAL_PATH/xarray_bug_slash.py", line 7, in <module>
    xarr.to_netcdf("new.nc")
  File "/opt/homebrew/lib/python3.11/site-packages/xarray/core/dataset.py", line 1911, in to_netcdf
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/xarray/backends/api.py", line 1217, in to_netcdf
    dump_to_store(
  File "/opt/homebrew/lib/python3.11/site-packages/xarray/backends/api.py", line 1264, in dump_to_store
    store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
  File "/opt/homebrew/lib/python3.11/site-packages/xarray/backends/common.py", line 271, in store
    self.set_variables(
  File "/opt/homebrew/lib/python3.11/site-packages/xarray/backends/common.py", line 309, in set_variables
    target, source = self.prepare_variable(
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 505, in prepare_variable
    target = NetCDF4ArrayWrapper(name, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 56, in __init__
    array = self.get_array()
            ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 83, in get_array
    variable = ds.variables[self.variable_name]
               ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
KeyError: 'PASS/FAIL'

[jhamman]

Thank you. Can you run the example I included and confirm you get the same output from xarr.info() that I show above?

[28raining]

Yeah it looks the same.
Code:

import pandas as pd
df = pd.DataFrame()
df["PASS/FAIL"] = ["PASS", "FAIL"]
df["temperature"] = [10, 20]
df = df.set_index("PASS/FAIL")

index = ['temperature']
df = df.set_index(index)
xarr = df.to_xarray()
xarr.to_netcdf("new.nc")

xarr.info()

result:

xarray.Dataset {
dimensions:
        temperature = 2 ;

variables:
        int64 temperature(temperature) ;

// global attributes:
}%

[jhamman]

I feel like we're missing something then. Your set_index(['temperature']) line removes the PASS/FAIL column from your data frame. I don't get an error in my example and it seems like you don't either. Can you provide a single copy-and-pastable example that reproduces your problem.

[28raining]

Can you copy and paste my original post?
Because I created the xarray from a .csv, you created it manaully

[28raining]

You can copy-paste this:

with open('newSlash.csv','w') as f:
  f.write('''PASS/FAIL,temperature
PASS, 10
FAIL, 20''')

import pandas as pd
df = pd.read_csv('newSlash.csv')
index = ['temperature']
df = df.set_index(index)
xarr = df.to_xarray()
xarr.to_netcdf("new.nc")

[kmuehlbauer]

The arr in @28raining's example looks like:

<xarray.Dataset>
Dimensions:      (temperature: 2)
Coordinates:
  * temperature  (temperature) int64 10 20
Data variables:
    PASS/FAIL    (temperature) object 'PASS' 'FAIL'

NetCDF4 and HDF5 do not allow forward slashes in variable/dataset names.

We also can take pandas out of the equation:

import xarray as xr
import numpy as np
xarr2 = xr.Dataset(
    data_vars={"PASS/FAIL": (["temperature"], np.array(["PASS", "FAIL"], dtype=object))},
    coords={"temperature": [10, 20]},
)
display(xarr2)
xarr2.to_netcdf("test.nc")

<xarray.Dataset>
Dimensions:      (temperature: 2)
Coordinates:
  * temperature  (temperature) int64 10 20
Data variables:
    PASS/FAIL    (temperature) object 'PASS' 'FAIL'

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[15], line 8
      3 xarr2 = xr.Dataset(
      4     data_vars={"PASS/FAIL": (["temperature"], np.array(["PASS", "FAIL"], dtype=object))},
      5     coords={"temperature": [10, 20]},
      6 )
      7 print(xarr2)
----> 8 xarr2.to_netcdf("test.nc")

File /home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/core/dataset.py:1957, in Dataset.to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
   1954     encoding = {}
   1955 from xarray.backends.api import to_netcdf
-> 1957 return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(
   1958     self,
   1959     path,
   1960     mode=mode,
   1961     format=format,
   1962     group=group,
   1963     engine=engine,
   1964     encoding=encoding,
   1965     unlimited_dims=unlimited_dims,
   1966     compute=compute,
   1967     multifile=False,
   1968     invalid_netcdf=invalid_netcdf,
   1969 )

File /home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/backends/api.py:1272, in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
   1267 # TODO: figure out how to refactor this logic (here and in save_mfdataset)
   1268 # to avoid this mess of conditionals
   1269 try:
   1270     # TODO: allow this work (setting up the file for writing array data)
   1271     # to be parallelized with dask
-> 1272     dump_to_store(
   1273         dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
   1274     )
   1275     if autoclose:
   1276         store.close()

File /home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/backends/api.py:1319, in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
   1316 if encoder:
   1317     variables, attrs = encoder(variables, attrs)
-> 1319 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)

File /home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/backends/common.py:278, in AbstractWritableDataStore.store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
    276 self.set_attributes(attributes)
    277 self.set_dimensions(variables, unlimited_dims=unlimited_dims)
--> 278 self.set_variables(
    279     variables, check_encoding_set, writer, unlimited_dims=unlimited_dims
    280 )

File /home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/backends/common.py:316, in AbstractWritableDataStore.set_variables(self, variables, check_encoding_set, writer, unlimited_dims)
    314 name = _encode_variable_name(vn)
    315 check = vn in check_encoding_set
--> 316 target, source = self.prepare_variable(
    317     name, v, check, unlimited_dims=unlimited_dims
    318 )
    320 writer.add(source, target)

File /home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:514, in NetCDF4DataStore.prepare_variable(self, name, variable, check_encoding, unlimited_dims)
    497     nc4_var = self.ds.createVariable(
    498         varname=name,
    499         datatype=datatype,
   (...)
    509         fill_value=fill_value,
    510     )
    512 nc4_var.setncatts(attrs)
--> 514 target = NetCDF4ArrayWrapper(name, self)
    516 return target, variable.data

File /home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:63, in BaseNetCDF4Array.__init__(self, variable_name, datastore)
     60 self.datastore = datastore
     61 self.variable_name = variable_name
---> 63 array = self.get_array()
     64 self.shape = array.shape
     66 dtype = array.dtype

File /home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/backends/netCDF4_.py:92, in NetCDF4ArrayWrapper.get_array(self, needs_lock)
     90 def get_array(self, needs_lock=True):
     91     ds = self.datastore._acquire(needs_lock)
---> 92     variable = ds.variables[self.variable_name]
     93     variable.set_auto_maskandscale(False)
     94     # only added in netCDF4-python v1.2.8

KeyError: 'PASS/FAIL'

The tricky thing is that xarray does currently not check this and the two backends (netcdf4, h5netcdf) will create sub-group PASS and variable FAIL in that sub-group in due course.

To make the confusion complete, the above works fine for backend scipy with NECTCDF3_CLASSIC format.

One solution would be to check for presence of / in variable names (/ only allowed as first character of variables in root-group) and raise a meaningful error when writing with netcdf4/h5netcdf engines.

Update: mentioned root-groop in last paragraph.