Skip to content

[CT-201] Reconcile configs + properties for sources #3662

@jtcohen6

Description

@jtcohen6

Follow-up to #2401, #3616

Let's convert some source properties into configs that can be set inside config: blocks and within dbt_project.yml:

# models/src_whatever.yml
version: 2
sources:
  - name: my_source
    description: ... # not a config
    config:
      enabled: ...
      quoting: {dict}
      freshness: {dict}
      loader: ...
      loaded_at_field: ...
      database: ... # or 'project' in dbt-bigquery
      schema: ... # or 'dataset' in dbt-bigquery
      identifier: ... # this is like an alias for alias
      meta: {dict}
      tags: ...
    tables:
      - name: my_src_table
        config: # all the same stuff as above. these take precedence for specific tables
        description: ... # not a config
        tests: ... # not a config
        columns: ... # not a config
# dbt_project.yml
sources:
  project_name:
    subdirectory:
      +database: raw
      +loader: fivetran
      another_subdirectory:
        +enabled: false

For backwards compatibility, we should still support setting these as top-level properties:

sources:
  - name: my_source
    loaded_at_field: updated_at

But raise an error if the same config is set in both places (even if it's with the same value):

sources:
  - name: my_source
    loaded_at_field: updated_at
    config:
      loaded_at_field: updated_at

Notes

  • We're thinking that descriptions shouldn't be configs. They're rendered with a different context (docs), and they don't make sense to set hierarchically. The same goes for tests and columns—this would be quite tricky to figure out, since tests actually generate new nodes, rather than adding properties to the existing node.

  • As far as the manifest / graph.sources context, I'm open to suggestions. For backwards compatibility, we'd want to store things like loaded_at_field in both node.config and node-level keys. But I think there's a valid argument for removing this as a top-level key and only storing it in node.config, so long as we communicate clearly that such a move is taking place.

  • For better state:modified comparisons, we'd want to store the un-rendered version of these configs in node.unrendered_config, regardless of whether they're set in dbt_project.yml or models/src_whatever.yml. Original issue for this is Store the unresolved form of source database representations #2744.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestjirastaleIssues that have gone stale

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions