[R] Read parquet files from raw vector

[asfimport]

read_parquet currently handles a path to a local file or an Arrow input stream.  Would it be possible to add support for a raw vector containing the contents of a parquet file?

Apologies if there is already a way to do this.  I have tried populating a buffer and passing that as input, but that is unsupported as well.  An example of how to work using an input stream would be useful as well.

Reporter: Brendan Hogan
Assignee: Neal Richardson / @nealrichardson

PRs and other links:

• GitHub Pull Request #5141

_{Note: This issue was originally created as ARROW-6278. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
Could you give an example of the code you have that you'd expect to work?

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
There's the BufferReader in C++

https://github.com/apache/arrow/blob/master/cpp/src/arrow/io/memory.h#L131-L168

which seems to be referenced/reachable in R bindinds:

https://github.com/apache/arrow/blob/master/r/src/io.cpp#L137-L141
https://github.com/apache/arrow/blob/master/r/R/io.R#L233-L245

[asfimport]

Brendan Hogan:
@nealrichardson, here is an example of what I'm trying to do:

> test_raw <- readBin(system.file("v0.7.1.parquet", package="arrow"), what = "raw", n = 5000) 
> test_df <- read_parquet(test_raw) 
Error in UseMethod("parquet_file_reader") :
 no applicable method for 'parquet_file_reader' applied to an object of class "raw"

[asfimport]

Brendan Hogan:
@fsaintjacques, yes BufferReader appears to work fine.  Thank you.

> test_br <- BufferReader(test_raw) 
> test_df <- read_parquet(test_br) 
>

 

[asfimport]

Neal Richardson / @nealrichardson:
Thanks. Out of curiosity, why are you trying to do this? Why do you have a Parquet file in memory as a raw vector? I'm wondering if there's a better solution to your actual problem than extending read_parquet to read raw vectors.

[asfimport]

Brendan Hogan:
Fair question.  I have parquet files in HDFS.  I can, of course, open a spark session and spark_read_parquet, but I am exploring options for lighter-weight read access.  I can grab the data into a raw vector via WebHDFS (e.g. https://mitre.github.io/webhdfs/).  Hence my interest in read_parquet on that.  I'm open to other suggestions here.

[asfimport]

Neal Richardson / @nealrichardson:
I know there is some support for HDFS in the Arrow C++ library but I don't see any R bindings to it yet. Would you mind if I rewrote this ticket to be for adding HDFS support to the R package?

It looks like François's suggestion unblocks you for now. You may also consider syncing the files from HDFS to your local file system and passing the file path to read_parquet; if the files are large that will be much more efficient with memory.

[asfimport]

Brendan Hogan:
Sure, feel free to translate as adding HDFS support.  That would be interesting to try out.

I will add that the real value for any of this parquet access will only be unlocked once arrow properly supports nested fields, i.e. ARROW-1644.  Although I am happy to put a plug in for HDFS support in the meantime.  Thanks.

 

[asfimport]

Neal Richardson / @nealrichardson:
Noted. Feel free to comment on that ticket and check on its status. I'm not sure where that work currently stands.

[asfimport]

Wes McKinney / @wesm:
Automatically recognizing raw binary in functions like read_parquet seems like it would be useful, and maybe not that much work

[asfimport]

Neal Richardson / @nealrichardson:
It's easy enough to add. PR coming.

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
Issue resolved by pull request 5141
#5141