Skip to content

Display all errors, not just first #355

@warnes

Description

@warnes

I've been running a query that fails trying convert between the R data type and the BQ data type:

## Create test table
library(bigrquery)
library(stringr)
library(DBI)
library(dplyr)

project_id <- "my-project-id
dataset_id <- "my-test_dataset"

ds <- bq_dataset(project = project_id,
                 dataset = dataset_id)

if(!bq_dataset_exists(ds))
  bigrquery::bq_dataset_create(ds)

con <- dbConnect(bigrquery::bigquery(), 
                 project = project_id,
                 use_legacy_sql = FALSE, 
                 dataset = dataset_id)

dbExecute(
  con, 
  str_interp(
    "
  CREATE TABLE IF NOT EXISTS `${project_id}.${dataset_id}.iris`
        (
      	Sepal_Length NUMERIC,
	      Sepal_Width NUMERIC,
	      Petal_Length NUMERIC,
	      Petal_Width STRING,  # Intentionally incorrect
	      Species STRING
        )
      "
  )
)

## Attempt to write data with mismatched types

iris_dat <- iris
colnames(iris_dat) <- gsub("\\.", "_", colnames(iris_dat)) # rename columns to match db table

try(
  dbWriteTable(con, str_interp("${project_id}.${dataset_id}.iris"), iris_dat, append=TRUE)
)
#
# Running job 'XXXXXXXX.job_0bjYLFBQgvgF2EyvP7Suu6CN-3vu.US' [|]  1s
# Error: Error while reading data, error message: JSON table encountered too many errors, 
#   giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more 
#   details. [invalid]
#

The displayed error message is not very helpful, since it does not describe how to access the "errors[] collection".

Ideally, the error handler in `bigrquery` would extract and display the "errors[] collection" for the user, a-là:
job_meta <- bq_job(
  project = project_id,
  job = "job_0bjYLFBQgvgF2EyvP7Suu6CN-3vu", 
  location = "US") %>% 
  bq_job_meta()

bind_rows(job_meta$status$errors)
#
# # A tibble: 3 x 2
# reason  message                                                                                                                                         
# <chr>   <chr>                                                                                                                                           
#   1 invalid Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] c…
#   2 invalid Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent:…
#   3 invalid Error while reading data, error message: JSON parsing error in row starting at position 0: Could not convert value to string. Field: Petal_Widt…
#

OS Version:

Ubuntu 18.04.2 LTS

R version:

platform       x86_64-pc-linux-gnu         
arch           x86_64                      
os             linux-gnu                   
system         x86_64, linux-gnu           
status                                     
major          3                           
minor          6.0                         
year           2019                        
month          04                          
day            26                          
svn rev        76424                       
language       R                           
version.string R version 3.6.0 (2019-04-26)
nickname       Planting of a Tree          

Package versions:

DBI                 DBI      1.0.0                                  /usr/local/lib/R/site-library/DBI          4
stringr         stringr      1.4.0                                    /usr/lib/R/site-library/stringr          5
dplyr             dplyr      0.8.1                                      /usr/lib/R/site-library/dplyr         15
bigrquery     bigrquery 1.2.0.9000                            /usr/local/lib/R/site-library/bigrquery         17

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions