Enable native ModelOpt quantization support (3/3)

[Edwardf0t1]

This is the third PR in a three-part series to enable native ModelOpt quantization in SGLang. It includes changes from the first PR (#7149) and second PR (#9991) and will be rebased once the first two PRs are merged.

Motivation

We aim to enhance SGLang's quantization capabilities, making ModelOpt integration more robust and user-friendly while providing checkpoint persistence for better performance in production environments.

Modifications

• Integrated modelopt quantized model export functionalities.
• Added modelopt_export_path parameter to _setup_modelopt_quantization() in ModelOptModelLoader.
• Implemented _export_modelopt_checkpoint() method using modelopt's unified hf export API.
• Added modelopt_export_path parameter in ModelConfig and added --modelopt-export-path command-line argument in ServerArgs.
• Export happens automatically after quantization (or when restoring from checkpoint).
• Added unit tests for the export functionalities.
• Unified quantization flags in quantize + export and deployment phases.
• Added an example script to run modelopt quantize + export + deployment.
• TODO: Enable a quantize-and-serve mode for quantize + export + deployment with a single command.

Accuracy Tests

Production Workflow:

# Step 1: Quantize + Export
python examples/usage/modelopt_quantize_and_export.py quantize \
    --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --export-dir ./quantized_tinyllama_fp8 \
    --quantization-method modelopt_fp8

# Step 2: Deploy
python -m sglang.launch_server \
    --model-path ./quantized_tinyllama_fp8 \
    --quantization modelopt

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

Summary by CodeRabbit

• New Features

  • Added NVIDIA ModelOpt quantization support (FP8/FP4 auto-detection), export to Hugging Face format, and serving of exported models.
  • Introduced CLI options to export after quantization and to quantize-and-serve.
  • Added quantization choice: modelopt_fp8.
  • Included an example script demonstrating quantize, export, and deploy.

• Documentation

  • New guide “Using NVIDIA ModelOpt” covering installation, workflow, Python usage, deployment, and advanced features; reference updated.

• Tests

  • Expanded coverage for ModelOpt workflows and additional model/attention components.

• Chores

  • Added optional dependency group for ModelOpt.

[Edwardf0t1]

@zhyncs @Qiaolin-Yu Please help or find someone review this PR as well when you get a chance. Thank you!

[Edwardf0t1]

@zhyncs @Qiaolin-Yu Please let us know if it's okay to add modelopt as an optional dependency, or required dependency?

cc @Ying1123 @merrymercy

[JustinTong0323]

Why do we have the exact same duplicate codes?

[JustinTong0323]

It basically detects whether "hf_quant_config.json" exists in model file. Iirc there are similar helper functions. May you try to find and reuse? (If not, make one and put into "utils.py" and call here

[JustinTong0323]

DRY: move to a helper function, like

def _maybe_export(self):
    if export_path:
                    try:
                        # Get the original model path from the model config
                        original_model_path = getattr(self, "_original_model_path", None)
                        self._export_modelopt_checkpoint(
                            model, export_path, original_model_path
                        )
                        rank0_log(
                            f"Quantized model exported to HuggingFace format at {export_path}"
                        )
                    except Exception as e:
                        rank0_log(
                            f"Warning: Failed to export quantized model to {export_path}: {e}"
                        )

[JustinTong0323]

Not sure whether we could reuse loader to export the model.... cc @merrymercy

[Edwardf0t1]

I don't quite understand this comment, but let's discuss in our call tomorrow.