Implement DataArray.idxmax()

[shoyer]

Should match the pandas function: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.idxmax.html

[jcmgray]

Just as I am interested in having this functionality, and the new apply_ufunc is available, would something along these lines suffice?

from wherever import argmax, take  # numpy or dask

def gufunc_idxmax(x, y, axis=None):
    indx = argmax(x, axis)
    return take(y, indx)

def idxmax(obj, dim):
    sig = ([(dim,), (dim,)], [()])
    kwargs = {'axis': -1}
    return apply_ufunc(gufunc_idxmin, obj, obj[dim],
                       signature=sig, kwargs=kwargs,
                       dask_array='allowed')

[shoyer]

See http://stackoverflow.com/questions/40179593/how-to-get-the-coordinates-of-the-maximum-in-xarray for examples of how to do this with the current version of xarray. @MaximilianR's answer using where is pretty clean, but maybe not the most efficient or exactly what we want. (I think it breaks in a few edge cases, such as if the max value appears multiple times, or the array is all NaN.)

@jcmgray Your proposal looks pretty close to me. But to handle higher dimension arrays, instead of take(y, indx), I think you need to NumPy style fancy indexing, y[indx,]. That doesn't work with dask, so you'll need to write a function function that uses dask.array.map_blocks when necessary.

I think something like the following would work:

def _index_from_1d_array(array, indices):
    return array[indices,]

def gufunc_idxmax(x, y, axis=None):
    # note: y is always a numpy.ndarray, because IndexVariable objects
    # always have their data loaded into memory
    indx = argmax(x, axis)
    func = functools.partial(_index_from_1d_array, y)

    if isinstance(array, dask_array_type):
        import dask.array as da
        return da.map_blocks(func, indx, dtype=indx.dtype)
    else:
        return func(indx)

[jcmgray]

So I thought take was just the functional equivalent of fancy indexing - I spotted it in the dask api and assumed it would work but having tried it does indeed just raise a 'not implemented error'. Just as a note, with the map_blocks approach above take is working for some cases where x[inds, ] is not -- related to #1237?

Regarding edge cases: multiple maxes is presumably fine as long as user is aware it just takes the first.
However, nanargmax is probably the actual desired function here, but looks like it will raise on all-nan slices. Would dropping these and then re-aligning be too much overhead?

[shoyer]

take is numpy function that only handles scalar or 1d arguments: https://docs.scipy.org/doc/numpy/reference/generated/numpy.take.html#numpy.take

I just merged #1237 -- see if it works with that.

multiple maxes is presumably fine as long as user is aware it just takes the first.

Yeah, that's not a problem here, only for the where based implementation.

However, nanargmax is probably the actual desired function here, but looks like it will raise on all-nan slices. Would dropping these and then re-aligning be too much overhead?

This behavior for nanargmax is unfortunate. The "right" behavior for xarray is probably to use NaN or NaT to mark the index in such locations but numpy makes this tricky. I think this could be achieved, though, with some mix of where, isnull and other vectorized operations. Basically you need to replace all NaN slices with some placeholder value before calculating nanargmax, and then use the locations of all NaN slices again to replace the results of nanargmax with the appropriate fill value.

[jcmgray]

Ah yes both ways are working now, thanks. Just had a little play around with timings, and this seems like a reasonably quick way to achieve correct NaN behaviour:

def xr_idxmax(obj, dim):
    sig = ([(dim,), (dim,)], [()])
    kwargs = {'axis': -1}
    
    allna = obj.isnull().all(dim)
    
    return apply_ufunc(gufunc_idxmax, obj.fillna(-np.inf), obj[dim],
                       signature=sig, kwargs=kwargs,
                       dask_array='allowed').where(~allna).fillna(np.nan)

i.e. originally replace all NaN values with -Inf, use the usual argmax, and remask the all-NaN values afterwards.

[shoyer]

Yes, that looks pretty reasonable. Two minor concerns:

• obj.fillna(-np.inf) converts all dtypes to float. It would be better to stick to obj.fillna(0), though integers can't have NaNs anyways.
• I'm pretty sure .fillna(np.nan) is a no-op, filling in NaNs with NaN.

[jcmgray]

Would using obj.fillna(0) not mess with argmax if for instance all the data is negative? Could fill with the min value instead?

Ah yes true. I was slightly anticipating e.g. filling with NaT if the dim was time-like, though time types are not something I am familiar with.