|
1 | 1 | # Expert Configuration of LLM API
|
2 | 2 |
|
3 |
| -For expert TensorRT-LLM users, we also expose the full set of [`LlmArgs`](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.py) |
| 3 | +For expert TensorRT-LLM users, we also expose the full set of `tensorrt_llm._torch.auto_deploy.llm_args.LlmArgs` |
4 | 4 | *at your own risk* (the argument list diverges from TRT-LLM's argument list):
|
5 | 5 |
|
6 | 6 | - All config fields that are used by the AutoDeploy core pipeline (i.e. the `InferenceOptimizer`) are
|
7 |
| - _exclusively_ exposed in the [`AutoDeployConfig` class](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.py). |
| 7 | + _exclusively_ exposed in the `AutoDeployConfig` in `tensorrt_llm._torch.auto_deploy.llm_args`. |
8 | 8 | Please make sure to refer to those first.
|
9 |
| -- For expert users we expose the full set of [`LlmArgs`](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.py) |
10 |
| - that can be used to configure the [AutoDeploy `LLM` API](../../../../../tensorrt_llm/_torch/auto_deploy/llm.py) including runtime options. |
11 |
| -- Note that some fields in the full [`LlmArgs`](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.py) |
| 9 | +- For expert users we expose the full set of `LlmArgs` in `tensorrt_llm._torch.auto_deploy.llm_args` |
| 10 | + that can be used to configure the AutoDeploy `LLM` API including runtime options. |
| 11 | +- Note that some fields in the full `LlmArgs` |
12 | 12 | object are overlapping, duplicated, and/or _ignored_ in AutoDeploy, particularly arguments
|
13 | 13 | pertaining to configuring the model itself since AutoDeploy's model ingestion+optimize pipeline
|
14 | 14 | significantly differs from the default manual workflow in TensorRT-LLM.
|
15 |
| -- However, with the proper care the full [`LlmArgs`](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.py) |
| 15 | +- However, with the proper care the full `LlmArgs` |
16 | 16 | objects can be used to configure advanced runtime options in TensorRT-LLM.
|
17 |
| -- Note that any valid field can be simply provided as keyword argument ("`**kwargs`") to the |
18 |
| - [AutoDeploy `LLM` API](../../../../../tensorrt_llm/_torch/auto_deploy/llm.py). |
| 17 | +- Note that any valid field can be simply provided as keyword argument ("`**kwargs`") to the AutoDeploy `LLM` API. |
19 | 18 |
|
20 | 19 | # Expert Configuration of `build_and_run_ad.py`
|
21 | 20 |
|
22 | 21 | For expert users, `build_and_run_ad.py` provides advanced configuration capabilities through a flexible argument parser powered by PyDantic Settings and OmegaConf. You can use dot notation for CLI arguments, provide multiple YAML configuration files, and leverage sophisticated configuration precedence rules to create complex deployment configurations.
|
23 | 22 |
|
24 | 23 | ## CLI Arguments with Dot Notation
|
25 | 24 |
|
26 |
| -The script supports flexible CLI argument parsing using dot notation to modify nested configurations dynamically. You can target any field in both the [`ExperimentConfig`](../../../../../examples/auto_deploy/build_and_run_ad.py) and nested [`AutoDeployConfig`](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.py)/[`LlmArgs`](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.) objects: |
| 25 | +The script supports flexible CLI argument parsing using dot notation to modify nested configurations dynamically. You can target any field in both the `ExperimentConfig` in `examples/auto_deploy/build_and_run_ad.py` and nested `AutoDeployConfig`/`LlmArgs` objects in `tensorrt_llm._torch.auto_deploy.llm_args`: |
27 | 26 |
|
28 | 27 | ```bash
|
29 | 28 | # Configure model parameters
|
@@ -56,7 +55,7 @@ python build_and_run_ad.py \
|
56 | 55 |
|
57 | 56 | ## YAML Configuration Files
|
58 | 57 |
|
59 |
| -Both [`ExperimentConfig`](../../../../../examples/auto_deploy/build_and_run_ad.py) and [`AutoDeployConfig`](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.py)/[`LlmArgs`](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.py) inherit from [`DynamicYamlMixInForSettings`](../../../../../tensorrt_llm/_torch/auto_deploy/utils/_config.py), enabling you to provide multiple YAML configuration files that are automatically deep-merged at runtime. |
| 58 | +Both `ExperimentConfig` and `AutoDeployConfig`/`LlmArgs` inherit from `DynamicYamlMixInForSettings`, enabling you to provide multiple YAML configuration files that are automatically deep-merged at runtime. |
60 | 59 |
|
61 | 60 | Create a YAML configuration file (e.g., `my_config.yaml`):
|
62 | 61 |
|
@@ -167,7 +166,7 @@ python build_and_run_ad.py \
|
167 | 166 |
|
168 | 167 | ## Built-in Default Configuration
|
169 | 168 |
|
170 |
| -Both [`AutoDeployConfig`](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.py) and [`LlmArgs`](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.py) classes automatically load a built-in [`default.yaml`](../../../../../tensorrt_llm/_torch/auto_deploy/config/default.yaml) configuration file that provides sensible defaults for the AutoDeploy inference optimizer pipeline. This file is specified in the [`_get_config_dict()`](../../../../../tensorrt_llm/_torch/auto_deploy/llm_args.py) function and defines default transform configurations for graph optimization stages. |
| 169 | +Both `AutoDeployConfig` and `LlmArgs` classes automatically load a built-in `default.yaml` configuration file that provides sensible defaults for the AutoDeploy inference optimizer pipeline. This file is specified in the `_get_config_dict()` function in `tensorrt_llm._torch.auto_deploy.llm_args` and defines default transform configurations for graph optimization stages. |
171 | 170 |
|
172 | 171 | The built-in defaults are automatically merged with your configurations at the lowest priority level, ensuring that your custom settings always override the defaults. You can inspect the current default configuration to understand the baseline transform pipeline:
|
173 | 172 |
|
|
0 commit comments