-
Notifications
You must be signed in to change notification settings - Fork 186
Closed
Labels
Milestone
Description
I've been running a query that fails trying convert between the R data type and the BQ data type:
## Create test table
library(bigrquery)
library(stringr)
library(DBI)
library(dplyr)
project_id <- "my-project-id
dataset_id <- "my-test_dataset"
ds <- bq_dataset(project = project_id,
dataset = dataset_id)
if(!bq_dataset_exists(ds))
bigrquery::bq_dataset_create(ds)
con <- dbConnect(bigrquery::bigquery(),
project = project_id,
use_legacy_sql = FALSE,
dataset = dataset_id)
dbExecute(
con,
str_interp(
"
CREATE TABLE IF NOT EXISTS `${project_id}.${dataset_id}.iris`
(
Sepal_Length NUMERIC,
Sepal_Width NUMERIC,
Petal_Length NUMERIC,
Petal_Width STRING, # Intentionally incorrect
Species STRING
)
"
)
)
## Attempt to write data with mismatched types
iris_dat <- iris
colnames(iris_dat) <- gsub("\\.", "_", colnames(iris_dat)) # rename columns to match db table
try(
dbWriteTable(con, str_interp("${project_id}.${dataset_id}.iris"), iris_dat, append=TRUE)
)
#
# Running job 'XXXXXXXX.job_0bjYLFBQgvgF2EyvP7Suu6CN-3vu.US' [|] 1s
# Error: Error while reading data, error message: JSON table encountered too many errors,
# giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more
# details. [invalid]
#
The displayed error message is not very helpful, since it does not describe how to access the "errors[] collection".
Ideally, the error handler in `bigrquery` would extract and display the "errors[] collection" for the user, a-là:
job_meta <- bq_job(
project = project_id,
job = "job_0bjYLFBQgvgF2EyvP7Suu6CN-3vu",
location = "US") %>%
bq_job_meta()
bind_rows(job_meta$status$errors)
#
# # A tibble: 3 x 2
# reason message
# <chr> <chr>
# 1 invalid Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] c…
# 2 invalid Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent:…
# 3 invalid Error while reading data, error message: JSON parsing error in row starting at position 0: Could not convert value to string. Field: Petal_Widt…
#
OS Version:
Ubuntu 18.04.2 LTS
R version:
platform x86_64-pc-linux-gnu
arch x86_64
os linux-gnu
system x86_64, linux-gnu
status
major 3
minor 6.0
year 2019
month 04
day 26
svn rev 76424
language R
version.string R version 3.6.0 (2019-04-26)
nickname Planting of a Tree
Package versions:
DBI DBI 1.0.0 /usr/local/lib/R/site-library/DBI 4
stringr stringr 1.4.0 /usr/lib/R/site-library/stringr 5
dplyr dplyr 0.8.1 /usr/lib/R/site-library/dplyr 15
bigrquery bigrquery 1.2.0.9000 /usr/local/lib/R/site-library/bigrquery 17
loomalaine, codr86 and YingyingPeng22