[R] R API for reading and writing Parquet files

[asfimport]

To start, this would be at the level of complexity of pyarrow.parquet.read_table and write_table

Reporter: Wes McKinney / @wesm
Assignee: Jeffrey Wong / @jeffwong-nflx

PRs and other links:

• GitHub Pull Request #3230

_{Note: This issue was originally created as ARROW-3731. Please see the migration documentation for further details.}

[asfimport]

Jeffrey Wong / @jeffwong-nflx:
Is this a matter of implementing the ParquetReader as shown in 

arrow/cpp/examples/parquet/low-level-api/reader-writer.cc

Line 181 in 723a437

std::unique_ptr<parquet::ParquetFileReader> parquet_reader =

into the list of IO types for the R package?

https://github.com/apache/arrow/blob/master/r/src/io.cpp#L110. I think if the ParquetReader is implemented, it could create a arrow::Table type just like RecordBatchFileReader, and then we would be able to reuse the table to tibble conversion function that already exists

[asfimport]

Wes McKinney / @wesm:
It means implementing an Rcpp version of

https://github.com/apache/arrow/blob/master/python/pyarrow/_parquet.pyx

and

https://github.com/apache/arrow/blob/master/python/pyarrow/parquet.py

[asfimport]

Jeffrey Wong / @jeffwong-nflx:
Hello, I have an MVP of this working. Being able to read parquet has decreased the latency in my project by 60%. I would love to be able to contribute back.

 

I can contribute the Rcpp code to load parquet into a tibble, but I have little experience with producing a configure file and pkg-config. One thing I need help with is producing the configure file in https://github.com/apache/arrow/blob/master/r/configure It references pkg-config to figure out how to link to the c++ arrow library. I believe it references information in https://github.com/apache/arrow/blob/master/cpp/src/arrow/arrow.pc.in. However, in order for the R library to read parquet we need to link to both -lparquet and -larrow. To get this to work on my application I hardcoded the configure file, but I would like to learn how to do this properly. I sense that arrow and parquet are organized as separate projects in the arrow repository, how should I produce a configure that will link to both?

[asfimport]

Wes McKinney / @wesm:
Can you post a link to a WIP branch with your code? I think @romainfrancois should be able to help you. Though, given the time of year, no guarantees until after the new year

[asfimport]

Jeffrey Wong / @jeffwong-nflx:
Hi @romainfrancois, I've posted a PR here #3230. It's already functional, but I had to hard code something into the makefile. Would you or someone else be able to help me generalize the make?

[asfimport]

Romain Francois / @romainfrancois:
Some updates to your PR @jeffwong-nflx, sent as a PR to your fork: https://github.com/jeffwong-nflx/arrow/pull/2

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 3230
#3230