-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Description
Let's convert some source properties into configs that can be set inside config: blocks and within dbt_project.yml:
# models/src_whatever.yml
version: 2
sources:
- name: my_source
description: ... # not a config
config:
enabled: ...
quoting: {dict}
freshness: {dict}
loader: ...
loaded_at_field: ...
database: ... # or 'project' in dbt-bigquery
schema: ... # or 'dataset' in dbt-bigquery
identifier: ... # this is like an alias for alias
meta: {dict}
tags: ...
tables:
- name: my_src_table
config: # all the same stuff as above. these take precedence for specific tables
description: ... # not a config
tests: ... # not a config
columns: ... # not a config# dbt_project.yml
sources:
project_name:
subdirectory:
+database: raw
+loader: fivetran
another_subdirectory:
+enabled: falseFor backwards compatibility, we should still support setting these as top-level properties:
sources:
- name: my_source
loaded_at_field: updated_atBut raise an error if the same config is set in both places (even if it's with the same value):
sources:
- name: my_source
loaded_at_field: updated_at
config:
loaded_at_field: updated_atNotes
-
We're thinking that descriptions shouldn't be configs. They're rendered with a different context (docs), and they don't make sense to set hierarchically. The same goes for
testsandcolumns—this would be quite tricky to figure out, sincetestsactually generate new nodes, rather than adding properties to the existing node. -
As far as the manifest /
graph.sourcescontext, I'm open to suggestions. For backwards compatibility, we'd want to store things likeloaded_at_fieldin bothnode.configand node-level keys. But I think there's a valid argument for removing this as a top-level key and only storing it innode.config, so long as we communicate clearly that such a move is taking place. -
For better
state:modifiedcomparisons, we'd want to store the un-rendered version of these configs innode.unrendered_config, regardless of whether they're set indbt_project.ymlormodels/src_whatever.yml. Original issue for this is Store the unresolved form of source database representations #2744.