[C++/Python] Kernel for SetItem(IntegerArray, values) ("replace_with_indices")

[asfimport]

We should have a kernel that allows overriding the values of an array using an integer array as the indexer and a scalar or array of equal length as the values.

Reporter: Uwe Korn / @xhochy

Related issues:

• [C++] String algorithm library for StringArray/BinaryArray (is a child of)
• [C++/Python] Kernel for SetItem(BooleanArray, values) (relates to)
• [C++] Add "replace" kernel (is related to)
• [R] Implement updating values via square brackets not implemented (is depended upon by)
• [C++/Python] Support necessary functionality to have an Arrow-string type in pandas (is depended upon by)

_{Note: This issue was originally created as ARROW-9431. Please see the migration documentation for further details.}

[asfimport]

Niranda Perera / @nirandaperera:
Identical issue

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@nirandaperera you linked ARROW-9430 as duplicate. Both are of course closely related, but there is a difference: using integer indices vs a boolean mask to select the values you want to set

[asfimport]

Niranda Perera / @nirandaperera:
Thank you @jorisvandenbossche. That was clearly my mistake :(

[asfimport]

Niranda Perera / @nirandaperera:
I'd like to take this one. @jorisvandenbossche  @xhochy @pitrou I'd like to get your feedback on the following.

Just to be clear, an example of this would be,

 

// code placeholder
arr1 = [a, b, c, d]
arr2 = [l, m, n, o, p]

to_replace = [0, 2, 3]

res = replace_by_index(to_replace, arr1, arr2)

# res --> [l, b, n, o]

 

What is a good name for the kernel? I'm not sure about replace_by_index because Arrays are immutable. Any suggestions?

Can we assume that 'to_replace' is a non-null Int64Array?

And should we enforce 'to_replace' being sorted (IMO this would be more efficient)?

What is the arity of this kernel? Unary or binary?

 

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I think one difference in your example is that I expect the arr2 to be of the same length as to_replace

Can we assume that 'to_replace' is a non-null Int64Array?

That might be the easiest for now. Or otherwise we can say nulls in to_replace give nulls in the output. The question is then how to interpret arr2, though (should it be of the same length as to_replace, or as the non-null values of to_replace ?)

And should we enforce 'to_replace' being sorted (IMO this would be more efficient)?

Hmm, not sure. Also for Take we generally don't require sortedness of the indices, I think?

What is the arity of this kernel? Unary or binary?

Ternary?

[asfimport]

Antoine Pitrou / @pitrou:
Indeed, we don't require sortedness of the indices for Take. Also, I'm not convinced it would be more efficient (why?).

[asfimport]

Antoine Pitrou / @pitrou:
The example doesn't quite look right to me. I would expect:

arr1 = [a, b, c, d]
arr2 = [l, m, n, o, p]

to_replace = [0, 2, 3]

res = replace_by_index(to_replace, arr1, arr2)

# res --> [l, b, m, n]

@xhochy What do you think?

[asfimport]

Niranda Perera / @nirandaperera:
@pitrou my idea was something like this.

def replace_by_index(to_replace: Array, arr1: Array, arr2: Array) -> Array:
  out = arr1 # copy array
  for i in to_replace:
    out[i] = arr2[i]
  return out 

 

Its like an if-else mask, but the mask's set bits are encoded in an array ARROW-10640

[asfimport]

Antoine Pitrou / @pitrou:
If this is meant to emulate Numpy's advanced indexing, then I don't think this is the right semantics.

[asfimport]

Niranda Perera / @nirandaperera:
I see.. Can you give an example?

[asfimport]

Antoine Pitrou / @pitrou:

>>> a = np.array(["a", "b", "c", "d"])
>>> b = np.array(["m", "n", "o"])
>>> a[[0,2,3]] = b
>>> a
array(['m', 'b', 'n', 'o'], dtype='<U1')

[asfimport]

Antoine Pitrou / @pitrou:
@xhochy What is the intended use case?

[asfimport]

Niranda Perera / @nirandaperera:
Got it. Think that makes sense. So, this will ultimately be another ternary kernel, isnt it?

[asfimport]

Antoine Pitrou / @pitrou:
What do you mean by "another ternary kernel"?

[asfimport]

Niranda Perera / @nirandaperera:
I'm currently working on the 'if-else' function ARROW-10640. And AFAIK it is the only ternary function that is there ATM. This will be another one.