ARROW-7168: [Python] Respect the specified dictionary type for pd.Categorical conversion #5866
Conversation
…rting pd.Categorical
| # (the type of the 'codes' in pandas is not part of the data type) | ||
| typ = pa.dictionary(index_type=pa.int16(), value_type=pa.string()) | ||
| result = pa.array(cat, type=typ) | ||
| assert result.type.equals(typ) |
There was a problem hiding this comment.
Shouldn't you check also the array value? (for example using result.to_pylist())
| typ = pa.dictionary(index_type=pa.int16(), value_type=pa.string()) | ||
| result = pa.array(cat, type=typ) | ||
| assert result.type.equals(typ) | ||
|
|
There was a problem hiding this comment.
Also, should you perhaps check passing a mask argument to pa.array()?
|
@pitrou thanks! Added some additional tests |
|
Also added some additional code to do it with a warning for now, falling back to the previous behaviour if the new behaviour fails. |
pitrou
left a comment
pitrou
left a comment
There was a problem hiding this comment.
I'm not sure the warning is necessary, and it's more maintenance work in the future, but I'll let you make the final choice. +1 otherwise.
|
We had some regressions with the changes in 0.15 related to following the types in the schema and the pandas index etc, so thought to be more cautious this time. But certainly not attached to it (I can't really judge if it will be something easy to run into or not). I think the extra maintenance work is limited, we will only need to remove the warnings and replace with an error at some point. |
|
Ok, merged as is :-) |
I guess this should be in #5866 Closes #5952 from mrkn/ARROW-7306 and squashes the following commits: e0793a3 <Kenta Murata> Add Result-returning version of FileSystemFromUri Authored-by: Kenta Murata <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>
…pe (int/string) Pandas dataframe to pyarrow Table
This PR homogenizes error messages for mixed-type `Pandas` inputs to `pa.Table`.
The message for `Pandas` column with `int` followed by `string` is now
```
In [2]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19, 'a']}))
(... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to int', 'Conversion failed for column a with type object')
```
the same as for `double` followed by `string`:
```
In [3]: table = pa.Table.from_pandas(pd.DataFrame({'a': [ 19.0, 'a']}))
(... traceback...)
ArrowInvalid: ('Could not convert a with type str: tried to convert to double', 'Conversion failed for column a with type object')
```
As a side effect, this snippet [xref #5866, ARROW-7168] now throws an `ArrowInvalid` (has been `FutureWarning` since 0.16):
```
In [8]: cat = pd.Categorical.from_codes(np.array([0, 1], dtype='int8'), np.array(['a', 'b'], dtype=object))
...: typ = pa.dictionary(index_type=pa.int8(), value_type=pa.int64())
...: result = pa.array(cat, type=typ)
(... traceback...)
ArrowInvalid: Could not convert a with type str: tried to convert to int
```
Finally, this *does* break a test [xref #4484, ARROW-4036] - see code comment
Closes #8044 from arw2019/ARROW-7663
Authored-by: arw2019 <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
2 participants
https://issues.apache.org/jira/browse/ARROW-7168
This change ensures that if you specify a
typeinpa.array, we ensure the output actually has this type when converting to dictionary array (as we also do for other types).The PR now implements this change, but we might want to do this with a deprecation first, as this can break people's code.