Differences between Ibis and Pandas dtypes?

[inigohidalgo]

Hiya,

I've run into a weird behavior when pulling data from an MSSQL DB to Pandas.

This is just a snippet of the columns but it illustrates the issue:

>>> ibis_table.schema()

 ESTADO_RB                 string
  CTD_REPARTO               decimal(20, 5)
  COD_CONTRAPARTE           string
  EXIST_TOTALES_MWH         decimal(20, 5)

>>> ibis_table.schema().to_pandas()

 ('ESTADO_RB', dtype('O')),
 ('CTD_REPARTO', dtype('O')),
 ('COD_CONTRAPARTE', dtype('O')),
 ('EXIST_TOTALES_MWH', dtype('O')),

>>> ibis_table.to_pandas().info()

 6   ESTADO_RB                 32207 non-null  object        
 7   CTD_REPARTO               33596 non-null  object        
 8   COD_CONTRAPARTE           0 non-null      object        
 9   EXIST_TOTALES_MWH         489 non-null    object       

Is it an issue with the source SQL datatypes?

EDIT: saw this issue #5316, not sure if it might be related, but just in case, my ibis version is 5.0.0

[gforsyth]

Hi @inigohidalgo !
We haven't yet moved over to the newer pandas dtypes when we return a pandas dataframe -- that's why you're seeing object for the string columns. I don't believe that pandas has a native decimal dtype, so those are going to remain object types.

If you are looking for better type fidelity, you can use to_pyarrow() instead:

[ins] In [14]: t.schema()
Out[14]: 
ibis.Schema {
  a  decimal(20, 5)
  b  string
}

[ins] In [15]: t.schema().to_pandas()
Out[15]: [('a', dtype('O')), ('b', dtype('O'))]

[ins] In [16]: t.schema().to_pyarrow()
Out[16]: 
a: decimal128(20, 5)
b: string