Support Deepseek v32

README.md

@@ -142,6 +142,8 @@ LMDeploy is a toolkit for compressing, deploying, and serving LLM, developed by
   <li>DeepSeek-MoE (16B)</li>
   <li>DeepSeek-V2 (16B, 236B)</li>
   <li>DeepSeek-V2.5 (236B)</li>
+  <li>DeepSeek-V3 (685B)</li>
+  <li>DeepSeek-V3.2 (685B)</li>
   <li>Mixtral (8x7B, 8x22B)</li>
   <li>Gemma (2B - 7B)</li>
   <li>StarCoder2 (3B - 15B)</li>

README_ja.md

@@ -129,6 +129,8 @@ LMDeploy TurboMindエンジンは卓越した推論能力を持ち、さまざ
   <li>DeepSeek-MoE (16B)</li>
   <li>DeepSeek-V2 (16B, 236B)</li>
   <li>DeepSeek-V2.5 (236B)</li>
+  <li>DeepSeek-V3 (685B)</li>
+  <li>DeepSeek-V3.2 (685B)</li>
   <li>Mixtral (8x7B, 8x22B)</li>
   <li>Gemma (2B - 7B)</li>
   <li>StarCoder2 (3B - 15B)</li>

README_zh-CN.md

@@ -143,6 +143,8 @@ LMDeploy TurboMind 引擎拥有卓越的推理能力，在各种规模的模型
   <li>DeepSeek-MoE (16B)</li>
   <li>DeepSeek-V2 (16B, 236B)</li>
   <li>DeepSeek-V2.5 (236B)</li>
+  <li>DeepSeek-V3 (685B)</li>
+  <li>DeepSeek-V3.2 (685B)</li>
   <li>Mixtral (8x7B, 8x22B)</li>
   <li>Gemma (2B - 7B)</li>
   <li>StarCoder2 (3B - 15B)</li>

docs/en/supported_models/supported_models.md

@@ -90,6 +90,8 @@ The following tables detail the models supported by LMDeploy's TurboMind engine
 |          DeepSeek-MoE          |       16B       | LLM  |    Yes    |   No    |   No    |  No  |  No   |
 |          DeepSeek-V2           |    16B, 236B    | LLM  |    Yes    |   No    |   No    |  No  |  No   |
 |         DeepSeek-V2.5          |      236B       | LLM  |    Yes    |   No    |   No    |  No  |  No   |
+|          DeepSeek-V3           |      685B       | LLM  |    Yes    |   No    |   No    |  No  |  No   |
+|         DeepSeek-V3.2          |      685B       | LLM  |    Yes    |   No    |   No    |  No  |  No   |
 |          DeepSeek-VL2          |    3B - 27B     | MLLM |    Yes    |   No    |   No    |  No  |  No   |
 |            MiniCPM3            |       4B        | LLM  |    Yes    |   Yes   |   Yes   |  No  |  No   |
 |         MiniCPM-V-2_6          |       8B        | LLM  |    Yes    |   No    |   No    |  No  |  Yes  |

docs/zh_cn/supported_models/supported_models.md

@@ -90,6 +90,8 @@
 |          DeepSeek-MoE          |       16B       | LLM  |    Yes    |   No    |   No    |  No  |  No   |
 |          DeepSeek-V2           |    16B, 236B    | LLM  |    Yes    |   No    |   No    |  No  |  No   |
 |         DeepSeek-V2.5          |      236B       | LLM  |    Yes    |   No    |   No    |  No  |  No   |
+|          DeepSeek-V3           |      685B       | LLM  |    Yes    |   No    |   No    |  No  |  No   |
+|         DeepSeek-V3.2          |      685B       | LLM  |    Yes    |   No    |   No    |  No  |  No   |
 |          DeepSeek-VL2          |    3B - 27B     | MLLM |    Yes    |   No    |   No    |  No  |  No   |
 |            MiniCPM3            |       4B        | LLM  |    Yes    |   Yes   |   Yes   |  No  |  No   |
 |         MiniCPM-V-2_6          |       8B        | LLM  |    Yes    |   No    |   No    |  No  |  Yes  |

lmdeploy/pytorch/backends/attention.py

@@ -15,6 +15,8 @@ class AttentionMetadata:
     q_seqlens: torch.Tensor = None
     kv_seqlens: torch.Tensor = None
     fill_seqlens: torch.Tensor = None
+    cu_seqlens_q: torch.Tensor = None
+    cu_seqlens_k: torch.Tensor = None
     quant_policy: Literal[0, 4, 8] = 0
 
 
@@ -70,6 +72,7 @@ def forward(
         k_scales_zeros: torch.Tensor = None,
         v_scales_zeros: torch.Tensor = None,
         learnable_sink: torch.Tensor = None,
+        nsa_indices: torch.Tensor = None,
         inplace: bool = False,
     ) -> torch.Tensor:
         """forward."""

lmdeploy/pytorch/backends/base.py

@@ -31,6 +31,7 @@ class OpType(Enum):
     FusedMoEW8A8 = auto()
     LinearBlockedF8 = auto()
     FusedMoEBlockedF8 = auto()
+    NSAIndexFP8 = auto()
 
 
 class OpsBackend(ABC):