-
Notifications
You must be signed in to change notification settings - Fork 4k
ARROW-16879: [R][CI] Test R GCS bindings with testbench #13542
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-16879: [R][CI] Test R GCS bindings with testbench #13542
Conversation
|
|
|
Tests seem to be failing right now because of a difference in how GCS and S3 handle paths: library(arrow)
#>
#> Attaching package: 'arrow'
#> The following object is masked from 'package:utils':
#>
#> timestamp
example_data <- arrow_table(x = Array$create(c(1, 2, 3)))
testbench_port <- Sys.getenv("TESTBENCH_PORT", "9001")
fs <- GcsFileSystem$create(
endpoint_override = sprintf("localhost:%s", testbench_port),
retry_limit_seconds = 1,
scheme = "http",
anonymous = TRUE # Will fail to resolve host name if anonymous isn't TRUE
)
fs$CreateDir("test")
write_parquet(example_data, fs$path("test/test.parquet"))
write_parquet(example_data, fs$path("test/test.parquet/"))
# GCS seems to handle them as separate paths
fs$ls("test")
#> [1] "test/test.parquet" "test/test.parquet"
minio_key <- Sys.getenv("MINIO_ACCESS_KEY", "minioadmin")
minio_secret <- Sys.getenv("MINIO_SECRET_KEY", "minioadmin")
minio_port <- Sys.getenv("MINIO_PORT", "9000")
fs <- S3FileSystem$create(
access_key = minio_key,
secret_key = minio_secret,
scheme = "http",
endpoint_override = paste0("localhost:", minio_port),
allow_bucket_creation = TRUE,
allow_bucket_deletion = TRUE
)
fs$CreateDir("test")
write_parquet(example_data, fs$path("test/test.parquet"))
write_parquet(example_data, fs$path("test/test.parquet/"))
# S3 implementation seems to remove the last slash
fs$ls("test")
#> [1] "test/test.parquet"Created on 2022-07-11 by the reprex package (v2.0.1) |
7c3bc5d to
8710059
Compare
|
I've included the change set from #13577. The C++ changes will go away when I rebase after merging that PR. |
f059ed4 to
c4fd663
Compare
nealrichardson
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is really cool, thanks for doing this.
paleolimbot
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few notes! Looks great!
609a166 to
66c3c84
Compare
66c3c84 to
f107de9
Compare
8fe2c67 to
b42042a
Compare
nealrichardson
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CI failure is unrelated, will merge
|
Benchmark runs are scheduled for baseline = 4660180 and contender = b7f9dfc. b7f9dfc is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
This PR:
GcsFileSystem$create()(andgs_bucket()introducted in ARROW-16887: [R][Docs] Update Filesystem Vignette for GCS #13601), but not URIs.