Skip to content

BigQuery Not Expanding Source Variables #92

@bassrock

Description

@bassrock

Describe the bug

Using a variable or Jinja syntax in the variable for: snowplow__database or snowplow__atomic_schema does not expand when used with BigQuery.

Steps to reproduce

Using bigquery, within the DBT config put:

  snowplow_unified:
    snowplow__database: "{{ env_var('DBT_DATBASE', 'default') }}"
    snowplow__atomic_schema:  "{{ env_var('DBT_SCHEMA', 'default') }}"

NOTE: This does work for Snowflake, but not BigQuery.

Expected results

The models to execute successfully using the env_var that is used.

Actual results

I receive the following error:

22:18:45 6 of 11 START sql table model default.dbt_scratch.snowplow_unified_base_events_this_run  [RUN]
22:18:46 Unhandled error while executing 
400 GET https://bigquery.googleapis.com/bigquery/v2/projects/%7B%7B%20env_var('DBT_DATABASE',%20'default')%20%7D%7D/datasets/%7B%7B%20env_var('DBT_SCHEMA',%20'default')%20%7D%7D/tables/events?prettyPrint=false: Invalid resource name projects/{{ env_var('DBT_DATABASE', 'default') }}; Project id: {{ env_var('DBT_DATABASE', 'default') }}
22:18:46 6 of 11 ERROR creating sql table model default.dbt_scratch.snowplow_unified_base_events_this_run  [ERROR in 1.42s]

Upon investigation, the base macro is returniogn the right sql. It looks like the error is within the get_context field macros when they call source('atomic', 'events`) . Because the underlying source file is not expanding the snowplow variables.

System information

The contents of your packages.yml file:

packages:
  - package: snowplow/snowplow_unified
    version: 0.4.5
  - package: snowplow/snowplow_utils
    version: 0.16.8
  - package: snowplow/snowplow_normalize
    version: 0.3.5

Which database are you using dbt with?

  • postgres
  • redshift
  • bigquery
  • snowflake
  • databricks
  • other (specify: ____________)

The output of dbt --version:

dbt Cloud CLI - 0.38.22 (1183c2abdb6003083b0fa91fcd89cd5feb25f9f7 2024-11-20T15:49:01Z)

The operating system you're using:
Mac OS Sequoia

The output of python --version:
Python 3.8.18

Metadata

Metadata

Assignees

No one assigned

    Labels

    status:needs_triageNeeds maintainer triage.type:bugBugs or weaknesses. The issue has to contain steps to reproduce.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions