Load and Save checks from file in Unity Catalog Volume (#512)

## Changes

* Added support for a new storage type for checks: Unity Catalog Volume.
* Unified checks location into a single configuration field:
`checks_location`. This replaces the previous `checks_file` and
`checks_table` fields, removing ambiguity by ensuring only one storage
location can be defined per run configuration.

BREAKING CHANGES!

* The `checks_file` and `checks_table` fields have been removed from the
installation run configuration. They are now consolidated into the
single `checks_location` field. This change simplifies the configuration
and clearly defines where checks are stored.

### Linked issues
This PR addresses the [FEATURE]: Load and Save quality checks from/in UC
Volume #386

### Tests

- [X] manually tested
- [x] added unit tests
- [X] added integration tests

---------

Co-authored-by: Marcin Wojtyczka <[email protected]>
Co-authored-by: Copilot <[email protected]>

Author: 3 people

README.md

@@ -13,7 +13,7 @@ Simplified Data Quality checking at Scale for PySpark Workloads on streaming and
 [![codecov](https://codecov.io/github/databrickslabs/dqx/graph/badge.svg)](https://codecov.io/github/databrickslabs/dqx) 
 ![linesofcode](https://aschey.tech/tokei/github/databrickslabs/dqx?category=code)
 [![PyPI](https://img.shields.io/pypi/v/databricks-labs-dqx?label=pypi%20package&cacheSeconds=3600)](https://pypi.org/project/databricks-labs-dqx/) 
-![PyPI - Downloads](https://img.shields.io/pypi/dm/databricks-labs-dqx?cacheSeconds=3600)
+![PyPI Downloads](https://static.pepy.tech/personalized-badge/databricks-labs-dqx?period=month&units=international_system&left_color=grey&right_color=orange&left_text=PyPI%20downloads&cacheSeconds=3600)
 
 # Documentation
 

demos/dqx_demo_tool.py

@@ -34,7 +34,7 @@
 # MAGIC   quarantine_config:
 # MAGIC     location: main.nytaxi.quarantine
 # MAGIC     mode: overwrite
-# MAGIC   checks_file: checks.yml
+# MAGIC   checks_location: checks.yml
 # MAGIC   profiler_config:
 # MAGIC     summary_stats_file: profile_summary_stats.yml
 # MAGIC   warehouse_id: your-warehouse-id

docs/dqx/docs/guide/data_profiling.mdx

@@ -219,9 +219,9 @@ You can use `options` parameter to pass a dictionary with custom options when pr
 
     The following DQX configuration from 'config.yml' is used by the profiler workflow:
     - 'input_config': configuration for the input data.
-    - 'checks_file': relative location of the generated quality rule candidates as `yaml` or `json` file inside the installation folder (default: `checks.yml`).
+    - 'checks_location': relative location of the generated quality rule candidates as `yaml` or `json` file inside the workspace installation folder (default: `checks.yml`).
     - 'profiler_config': configuration for the profiler containing:
-        - 'summary_stats_file': relative location of the summary statistics (default: `profile_summary.yml`) inside the installation folder
+        - 'summary_stats_file': relative location of the summary statistics (default: `profile_summary.yml`) inside the workspace installation folder
         - 'sample_fraction': fraction of data to sample for profiling.
         - 'sample_seed': seed for reproducible sampling.
         - 'limit': maximum number of records to analyze.
@@ -348,7 +348,7 @@ The DLT generator creates Lakeflow Pipelines expectation statements from profile
 
 ## Storing Quality Checks
 
-You can save checks defined in code or generated by the profiler to a table or file as `yaml` or `json` in the local path, workspace or installation folder.
+You can save checks defined in code or generated by the profiler to a table or file as `yaml` or `json` in the local path, workspace, installation folder or Unity Catalog Volume file.
 
 <Tabs>
   <TabItem value="Python" label="Python" default>
@@ -358,7 +358,8 @@ You can save checks defined in code or generated by the profiler to a table or f
       FileChecksStorageConfig,
       WorkspaceFileChecksStorageConfig,
       InstallationChecksStorageConfig,
-      TableChecksStorageConfig
+      TableChecksStorageConfig,
+      VolumeFileChecksStorageConfig
     )
     from databricks.sdk import WorkspaceClient
 
@@ -383,11 +384,6 @@ You can save checks defined in code or generated by the profiler to a table or f
     # always overwrite the file
     dq_engine.save_checks(checks, config=WorkspaceFileChecksStorageConfig(location="/Shared/App1/checks.yml"))
 
-    # save checks in file defined in 'checks_file' in the run config
-    # always overwrite the file
-    # only works if DQX is installed in the workspace
-    dq_engine.save_checks(checks, config=InstallationChecksStorageConfig(assume_user=True, run_config_name="default"))
-
     # save checks in a Delta table with default run config for filtering
     # append checks in the table for the default run config
     dq_engine.save_checks(checks, config=TableChecksStorageConfig(location="dq.config.checks_table", mode="append"))
@@ -396,8 +392,11 @@ You can save checks defined in code or generated by the profiler to a table or f
     # overwrite checks in the table for the given run config
     dq_engine.save_checks(checks, config=TableChecksStorageConfig(location="dq.config.checks_table", run_config_name="workflow_001", mode="overwrite"))
 
-    # save checks in table defined in 'checks_table' in the run config
-    # always overwrite checks in the table for the given run config
+    # save checks in a Unity Catalog volume location
+    # always overwrite the file
+    dq_engine.save_checks(checks, config=VolumeFileChecksStorageConfig(location="/Volumes/dq/config/checks_volume/App1/checks.yml"))
+
+    # save checks in file or table defined in 'checks_location' in the run config
     # only works if DQX is installed in the workspace
     dq_engine.save_checks(checks, config=InstallationChecksStorageConfig(assume_user=True, run_config_name="default"))
     ```

docs/dqx/docs/guide/quality_checks.mdx

@@ -21,7 +21,7 @@ There are several ways to define and apply quality checks in DQX:
 
 ## Quality Rules defined in a File
 
-Quality rules can be defined declaratively as part of `yaml` or `json` file and stored in the installation folder, workspace, or local file system.
+Quality rules can be defined declaratively as part of `yaml` or `json` file and stored in the installation folder, workspace, local file system or Unity Catalog Volume file.
 
 Below is an example `yaml` file ('checks.yml') defining several checks:
 ```yaml
@@ -175,7 +175,7 @@ In addition, you can also perform a standalone syntax validation of the checks a
 
     dq_engine = DQEngine(WorkspaceClient())
 
-    # Load check from the installation (from file defined in 'checks_file' in the run config)
+    # Load check from the installation (from file or table defined in 'checks_location' in the run config)
     # Only works if DQX is installed in the workspace
     default_checks = dq_engine.load_checks(config=InstallationChecksStorageConfig(assume_user=True, run_config_name="default"))
     workflow_checks = dq_engine.load_checks(config=InstallationChecksStorageConfig(assume_user=True, run_config_name="workflow_001"))
@@ -271,6 +271,46 @@ In addition, you can also perform a standalone syntax validation of the checks a
   </TabItem>
 </Tabs>
 
+#### Method 4: Loading and Applying Checks Defined in a Unity Catalog Volume File
+
+<Tabs>
+  <TabItem value="Python" label="Python" default>
+    ```python
+    from databricks.labs.dqx.config import OutputConfig
+    from databricks.labs.dqx.engine import DQEngine
+    from databricks.labs.dqx.config import VolumeFileChecksStorageConfig
+    from databricks.sdk import WorkspaceClient
+
+    dq_engine = DQEngine(WorkspaceClient())
+
+    # Load checks from multiple files and merge
+    default_checks = dq_engine.load_checks(config=VolumeFileChecksStorageConfig(location="/Volumes/catalog/schema/App1/default_checks.yml"))
+    workflow_checks = dq_engine.load_checks(config=VolumeFileChecksStorageConfig(location="/Volumes/catalog/schema/App1/workflow_checks.yml"))
+    checks = default_checks + workflow_checks
+
+    input_df = spark.read.table("catalog1.schema1.table1")
+
+    # Option 1: apply quality rules on the dataframe and provide valid and invalid (quarantined) dataframes
+    valid_df, quarantine_df = dq_engine.apply_checks_by_metadata_and_split(input_df, checks)
+    dq_engine.save_results_in_table(
+      output_df=valid_df, 
+      quarantine_df=quarantine_df, 
+      output_config=OutputConfig("catalog.schema.valid_data"), 
+      quarantine_config=OutputConfig("catalog.schema.quarantine_data")
+    )
+
+    # Option 2: apply quality rules on the dataframe and report issues as additional columns (`_warning` and `_error`)
+    valid_and_quarantine_df = dq_engine.apply_checks_by_metadata(input_df, checks)
+    dq_engine.save_results_in_table(
+      output_df=valid_df, 
+      quarantine_df=quarantine_df, 
+      output_config=OutputConfig("catalog.schema.valid_and_quarantine_data")
+    )
+    ```
+  </TabItem>
+</Tabs>
+
+
 ## Quality Rules defined in a Delta Table
 
 Quality rules can be stored in a Delta table in Unity Catalog. Each row represents a check with column values for the `name`, `check`, `criticality`, `filter`, and `run_config_name`.
@@ -452,7 +492,7 @@ In addition, you can also perform a standalone syntax validation of the checks a
     # Load checks from the "workflow_001" run config
     workflow_checks = dq_engine.load_checks(config=TableChecksStorageConfig(location="dq.config.checks_table", run_config_name="workflow_001"))
 
-    # Load checks from the installation (from a table defined in 'checks_table' in the run config)
+    # Load checks from the installation (from a table defined in 'checks_location' in the run config)
     # Only works if DQX is installed in the workspace
     workflow_checks = dq_engine.load_checks(config=InstallationChecksStorageConfig(assume_user=True, run_config_name="workflow_001"))
 
@@ -1025,7 +1065,7 @@ The validation cannot be used for checks defined programmatically using [DQX cla
     ```
 
     The following DQX configuration from 'config.yml' will be used by default:
-    - 'checks_file': relative location of the quality rules defined declaratively as `yaml` or `json` inside the installation folder (default: `checks.yml`).
+    - 'checks_location': relative location of the quality rules defined declaratively as `yaml` or `json` inside the workspace installation folder (default: `checks.yml`).
   </TabItem>
 </Tabs>
 

docs/dqx/docs/installation.mdx

@@ -140,18 +140,17 @@ run_configs:
     trigger:                        # <- streaming trigger, only applicable if input_config.is_streaming is enabled
       availableNow: true
 
-  quarantine_config:                     # <- quarantine data configuration, if specified, bad data is written to quarantine table
+  quarantine_config:                     # <- optional quarantine data configuration, if specified, bad data is written to quarantine table
     location: main.iot.silver_quarantine # <- quarantine location (table), used as input for quality dashboard
-    format: delta                        # <- format of the quarantine table
-    mode: append                         # <- write mode for the quarantine table (append or overwrite)
+    format: delta                        # <- format of the quarantine table (default: delta)
+    mode: append                         # <- write mode for the quarantine table (append or overwrite, default: append)
     options:                             # <- additional options for writing to the quarantine table (optional)
       mergeSchema: 'true'
       #checkpointLocation: /Volumes/catalog1/schema1/checkpoint  # <- only applicable if input_config.is_streaming is enabled
-    trigger:                             # <- streaming trigger, only applicable if input_config.is_streaming is enabled
+    trigger:                             # <- optional streaming trigger, only applicable if input_config.is_streaming is enabled
       availableNow: true
 
-  checks_file: iot_checks.yml            # <- relative location of the quality rules (checks) defined in json or yaml file
-  checks_table: main.iot.checks          # <- table storing the quality rules (checks)
+  checks_location: iot_checks.yml        # <- Quality rules (checks) can be stored in a table or defined in JSON or YAML files, located at a relative workspace file path or volume file path.
 
   profiler_config:                            # <- profiler configuration
     summary_stats_file: iot_summary_stats.yml # <- relative location of profiling summary stats