GH-34882: [Python] Binding for FixedShapeTensorType

[AlenkaF]

Rationale for this change

In the C++ the fixed shape tensor canonical extension type is implementated #8510 so we can add bindings to the extension type in Python.

What changes are included in this PR?

Binding for fixed shape tensor canonical extension type.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

• Closes: [Python] Binding for FixedShapeTensorType #34882

[github-actions]

• Closes: [Python] Binding for FixedShapeTensorType #34882

[jorisvandenbossche]

(some quick comments)

[AlenkaF]

I suggest we then first use the 3rd option: "Return always the actual C-contiguous array and raise error if permutation is non-trivial" and as Joris mentioned, finetune it in a follow-up PR (after the work on to/from_tensor in C++ is finished).

The check for to_numpy_ndarray was added with the last commit: 1ebb829

@rok do you agree with the current state of this PR? If yes we could merge it today and get it into 12.0.0.

[rok]

LGTM!

[AlenkaF]

@jorisvandenbossche if u agree we can go ahead and merge this.

[jorisvandenbossche]

pyarrow->numpy conversion should (imo) use permutation data to create ndarray with correct strides while using the same physical layout (zero-copy) of the data buffer. I don't see any good reason to not do this in pyarrow but I might be missing something?

@rok see my comments above?
Potential reasons that someone might not want to have the permuted ndarray:

• For a specific use case, someone might not care about the permutation and just wants the physical C-contiguous array
• Because you want to do the handling of permutation / dim_names yourself, and therefore first want to get the numpy array as the physical array (see GH-34882: [Python] Binding for FixedShapeTensorType #34883 (comment) above for a possible reason for this, eg if you need to handle the potential combination of both permutation and/or dim_names, this might be easier to just all do yourself)

[rok]

• For a specific use case, someone might not care about the permutation and just wants the physical C-contiguous array

Wouldn't that not be possible with tensor_ext_arr.storage.to_numpy()? Or would that not be zero-copy?

• Because you want to do the handling of permutation / dim_names yourself, and therefore first want to get the numpy array as the physical array (see GH-34882: [Python] Binding for FixedShapeTensorType #34883 (comment) above for a possible reason for this, eg if you need to handle the potential combination of both permutation and/or dim_names, this might be easier to just all do yourself)

How about we have pyarrow pass the permutation/strides information by default and have users use lower level API for manual handling of permutation / dim_names? I'm assuming underlying buffer doesn't get actually permuted in either case of course.

[AlenkaF]

Short interruption of the continuation of the thread 😊

Thank you Joris and Rok for all the help and reviews! 🙏
The docs for this PR are ready for review: #34957

[jorisvandenbossche]

For a specific use case, someone might not care about the permutation and just wants the physical C-contiguous array

Wouldn't that not be possible with tensor_ext_arr.storage.to_numpy()? Or would that not be zero-copy?

to_numpy() gives a 1D object-dtype array of arrays, so that's not convenient to work with if you want the nD array (it can actually be "zero-copy", but you can't get the zero-copy nD array from it)

How about we have pyarrow pass the permutation/strides information by default and have users use lower level API for manual handling of permutation / dim_names?

What do you mean exactly with "pass the permutation/strides information"? (I suppose you mean returning the permuted/logical array by default?)
Users could indeed do this conversion to an nD array manually (the to_numpy_ndarray implementation isn't that long), but to me that feels a bit the point of providing this method ;) (it could be a keyword argument to the function, to indicate if you want the permuted logical array or not)

[rok]

What do you mean exactly with "pass the permutation/strides information"? (I suppose you mean returning the permuted/logical array by default?)

I think so yes :D.

Users could indeed do this conversion to an nD array manually (the to_numpy_ndarray implementation isn't that long), but to me that feels a bit the point of providing this method ;) (it could be a keyword argument to the function, to indicate if you want the permuted logical array or not)

I suppose having this as an opt-in behavior (with the keyword) would work yeah. Just want to make sure this is not on a default path as that would be surprising.

[ursabot]

Benchmark runs are scheduled for baseline = e488942 and contender = d21c1c7. d21c1c7 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Finished ⬇️0.26% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.06%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] d21c1c79 ec2-t3-xlarge-us-east-2
[Failed] d21c1c79 test-mac-arm
[Finished] d21c1c79 ursa-i9-9960x
[Finished] d21c1c79 ursa-thinkcentre-m75q
[Finished] e488942c ec2-t3-xlarge-us-east-2
[Failed] e488942c test-mac-arm
[Finished] e488942c ursa-i9-9960x
[Finished] e488942c ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java