Store the unresolved form of source database representations

[jtcohen6]

extension of #2713

Describe the feature

Let's say a project has a source defined like:

sources:
  - name: my_postgres_db
    database: "{{ 'raw' if target.name == 'prod' else 'raw_sampled' }}"

Today, if running in dev and comparing to a manifest generated by a prod target, state:modified+ will include all models that depend on source:my_postgres_db because the database differs. Ideally, we'd be able to tell that the unresolved database (Jinja statement) is identical.

I'm most interested in sources and database representations (database, schema, identifier). If it made sense, though, we could broaden this issue to the matter of storing unresolved:

• database representations
• resource properties (will the distinction between node configs and resource properties even make sense in a post-Set configs in schema.yml files #2401 world?)

Describe alternatives you've considered

• The proposal for state:modified subselectors gives decent user-side answers by letting people "switch off" the intersection of state:modified.database_representations and specific sources.

Who will this benefit?

• Users who develop, test, and deploy against genuinely different source datasets. While this isn't something we recommend in the general case, it's the norm at organizations of a certain size + maturity.

[fabrice-etanchaud]

You are totally right, @jtcohen6 . That 's exactly our use case, here @maif-vie (life insurance). We have a kind of environment notion (we call that a 'track') that groups data sources for a given project/feature. I ended up adding a extra credential parameter called 'environment' to use as a prefix for source's databases. That way I can configure separately the source version and the target database. This enhancement would be of great help here, and in as you already said 'mature' organizations.
Best regards from french west coast ;-)

[jtcohen6]

From #3201: It's possible to define sources that are implicitly different across environments, e.g. on Snowflake by excluding the database:

sources:
  - name: etl
    tables:
      - name: etl_sample_table
        identifier: sample_table

Is really the same as:

sources:
  - name: etl
    database: "{{ target.database }}"
    tables:
      - name: etl_sample_table
        identifier: sample_table

So running with a different target.database will result in the source being marked modified. Any resolution to this issue should also seek to resolve that case.

[jtcohen6]

• This will be improved by the introduction of sub-selectors (Subselectors for state:modified #2704)
• Let's revisit after Set configs in schema.yml files #2401, since that may have implications for how source database/schema/identifier are stored on the source node (node.config vs. node-level properties)

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.