cannot read a file created from python

[maverickg]

The python code used was from pandas's doc:

import pandas as pd
import numpy as np

df = pd.DataFrame({'a' : list('abc'),
                   'b' : list(range(1, 4)),
                   'c' : np.arange(3, 6).astype('u1'),
                   'd' : np.arange(4.0, 7.0, dtype='float64'),
                   'e' : [True, False, True],
                   'f' : pd.Categorical(list('abc')),
                   'g' : pd.date_range('20130101', periods=3),
                   'h' : pd.date_range('20130101', periods=3, tz='US/Eastern'),
                   'i' : pd.date_range('20130101', periods=3, freq='ns')})
print(df)

df.to_feather('example.feather')

But I cannot read it in R:

$ R -q -f read.R
>
> library(feather)
>
> df <- read_feather('example.feather')
Error in coldataFeather(x, i) : Not an INT64
Calls: read_feather ... as_data_frame.feather -> [ -> [.feather -> coldataFeather -> .Call
In addition: Warning message:
Coercing int64 to double
Execution halted

Python environment: 3.6, feather 0.4.0

R environment: R 3.3.1, feather 0.3.1, Rcpp 0.12.11

[wesm]

Can someone check if this is a regression from 0.3.1 in Python?

[imadam]

It does appear to be a regression, checked with maverickg's example using python feather 0.4.0 vs 0.3.1.

I see a similar issue with a single timestamp column:

wt = pd.DataFrame({"T" : [pd.Timestamp.now(), pd.Timestamp.now(),pd.Timestamp.now()]})`
wt.to_feather('wt.fea')

The test file reads back fine in python, but in R I get:

read_feather('wt.fea')
Error in coldataFeather(x, i) : Not an INT64
In addition: Warning messages:
1: Coercing int64 to double 
2: Coercing int64 to double 

If I downgrade my python feather to 0.3.1 and repeat, I can read:

 > read_feather('wt.fea')
# A tibble: 3 x 1
                    T
               <dttm>
1 2017-05-25 11:11:42
2 2017-05-25 11:11:42
3 2017-05-25 11:11:42

My env is python 3.5.2, R 3.3.2 with feather 0.3.1.

[wesm]

I believe I found the problem, see https://issues.apache.org/jira/browse/ARROW-1070. We should set up some simple integration tests which will help until R is using the exact same code again as Python (if you know an R developer who wants to take ownership of Feather and Arrow bindings, please let me know)

[wesm]

This will be closed when apache/arrow#725 is merged and fix deployed to conda-forge. I verified locally that things check out

[quantitative-technologies]

This is an issue again. The original MRE still serves.

R: R version 3.6.3, Rcpp 1.0.4.6, feather 0.3.5
python: Python 3.8, feather 0.4.1

[nealrichardson]

Could you please try with arrow::read_feather()?

[quantitative-technologies]

Yes, it works fine, thanks.

If I now understand, the R feather package is superceeded by arrow.