model_init_kwargs not passed correctly #24
Description
In your demo:
from dripper.api import Dripper
Initialize Dripper with model configuration
dripper = Dripper(
config={
'model_path': '/path/to/your/model',
'use_fall_back': True,
'raise_errors': False,
'inference_backend': "vllm",
"model_init_kwargs": {
'tensor_parallel_size': 1, # Tensor parallel size
},
"model_gen_kwargs": {
'temperature': 0.0,
},
}
)
The model_init_kwargs parameter here should include the serving parameters of vllm LLM api. However, the model_init_kwargs is not passed into the LLM initialization of the get_llm() function
def get_llm(self) -> LLM:
"""
Get LLM instance (lazy-loaded).
Returns:
VLLM LLM instance
Raises:
DripperLoadModelError: When model loading fails
"""
if self._llm is None:
check_vllm_environment(self.state_machine)
try:
logger.info(f'Loading model: {self.model_path}')
self._llm = LLM(
model=self.model_path, tensor_parallel_size=self.tp,
)
logger.info('Model loading completed')
except Exception as e:
raise DripperLoadModelError(
f'Model loading failed: {str(e)}'
) from e