Welcome to UltraRAG Discussions!

[gdw439]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[daxue9]

OpenBMB/MiniCPM-V-2_6加载在cuda:0 cuda:1
OpenBMB/VisRAG-Ret 加载在cuda:3

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution)
Traceback:

File "/home/ps/007/UltraRAG/ultrarag/webui/pages/chat/pipeline/visrag.py", line 186, in chat_component
response_with_extra_info, pure_response = asyncio.run(listen(query,container_a))
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home/ps/007/UltraRAG/ultrarag/webui/pages/chat/pipeline/visrag.py", line 148, in listen
async for chunk in response:
File "/home/ps/007/UltraRAG/ultrarag/workflow/ultraragflow/visrag_flow.py", line 101, in aquery
for item in resp:
File "/home/ps/007/UltraRAG/ultrarag/modules/llm/huggingface_like.py", line 145, in _stream_response
answer = self._model.chat(
File "/home/ps/.cache/huggingface/modules/transformers_modules/MiniCPMV26/modeling_minicpmv.py", line 387, in chat
res = self.generate(
File "/home/ps/.cache/huggingface/modules/transformers_modules/MiniCPMV26/modeling_minicpmv.py", line 263, in generate
) = self.get_vllm_embedding(model_inputs)
File "/home/ps/.cache/huggingface/modules/transformers_modules/MiniCPMV26/modeling_minicpmv.py", line 118, in get_vllm_embedding
vision_embedding = self.vpm(all_pixel_values, patch_attention_mask=patch_attn_mask, tgt_sizes=tgt_sizes).last_hidden_state
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/site-packages/accelerate/hooks.py", line 175, in new_forward
output = module._old_forward(*args, **kwargs)
File "/home/ps/.cache/huggingface/modules/transformers_modules/MiniCPMV26/modeling_navit_siglip.py", line 904, in forward
hidden_states = self.embeddings(pixel_values=pixel_values, patch_attention_mask=patch_attention_mask, tgt_sizes=tgt_sizes)
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/site-packages/accelerate/hooks.py", line 175, in new_forward
output = module._old_forward(*args, **kwargs)
File "/home/ps/.cache/huggingface/modules/transformers_modules/MiniCPMV26/modeling_navit_siglip.py", line 320, in forward
patch_embeds = self.patch_embedding(pixel_values)
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/site-packages/accelerate/hooks.py", line 175, in new_forward
output = module._old_forward(*args, **kwargs)
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 554, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/ps/.conda/envs/ultrarag/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 549, in _conv_forward
return F.conv2d(

[gdw439]

这个我记得MiniCPM-V跑不了多卡推理

[gdw439]

OpenBMB/MiniCPM-V-2_6好像有个int4版本的，可以考虑单卡跑一下看看

[daxue9]

嗯，是的，现在跑的就是MiniCPMV26int4，单块4090上跑，loading后，占7G左右显存。

[daxue9]

但是模型只加载到了cuda:0,手动改为：

[daxue9]

"ultrarag/webui/utils/model_config.json" 没有这个文件，每次重启streamlit，model manage之前加载好的模型，都需要重新添加信息加载一次

[gdw439]

嗯嗯，按理说应该会生成一个默认的，我给你一个吧

{
    "last_model_name": "gpt-3.5-turbo",
    "embedding_model_selectbox": "Custom",
    "language_selectbox": "en",
    "last_reranker_model_path": "/data/private/evorag/resource/bge-large-reranker",
    "last_base_url": "https://localhost:8000/v1",
    "llm_model_selectbox": "API",
    "last_model_path": "",
    "reranker_model_selectbox": "Custom",
    "config": {
        "language": "en",
        "model_type": "API",
        "selected_devices_llm": [],
        "embedding_model_type": "Custom",
        "selected_devices_embedding": [],
        "embedding_model_path": "/data/private/evorag/resource/bge-large-zh-v1.5",
        "reranker_model_type": "Custom",
        "selected_devices_reranker": [],
        "reranker_model_path": "/data/private/evorag/resource/bge-large-reranker",
        "model_path": "",
        "model_name": "gpt-3.5-turbo",
        "api_key": "sk-",
        "base_url": "https://localhost:8000/v1"
    }
}

[daxue9]

谢谢

[daxue9]

不能同时选择多个collection,每次只能选一个，是否可以支持选择多个collection,