Is there anyway to run 1+ vllm models in one process?

[justinphan3110cais]

When init 2 vllm models I got the following error:

    assert _TENSOR_MODEL_PARALLEL_GROUP is None, (
AssertionError: tensor model parallel group is already initialized

Is there anyway to create multiple vllm model as my project require different conversation outputs from different models

[simon-mo]

Wo do not support this, considering LLM inference typically takes up the whole GPU and is also CPU intensive. I would recommend using ray to multiplex vLLM processes and orchestrate them. Or initiate multiple vLLM api server running on different ports.

[bibekyess]

Hi @justinphan3110cais, Did you figure out the efficient way to do it? If yes, can you share your approach? Thank you! :)