qwen1.5-32b-chat 使用vllm推理很慢

[zhangfan-algo]

已经运行了5分钟,一条数据也没有跑出来

[zhangfan-algo]

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python examples/pytorch/llm/llm_infer.py
--infer_backend vllm
--model_type qwen1half-32b-chat
--model_id_or_path /mnt/pfs/zhangfan/models/Qwen/Qwen1.5-32B-Chat
--ckpt_dir /mnt/pfs/zhangfan/study_info/LLaMA-Factory_0506/output/user_sft_prompt_test_classfiy_all_21W_0521_qwen1half5_32b_sft_0521/checkpoint-406
--dataset test.json
--dataset_test_ratio 1
--max_model_len 19500
--use_flash_attn true
--max_new_tokens 1500
--temperature 0.01
--top_p 0.99
--use_flash_attn true
--verbose false
--do_sample true
--show_dataset_sample 1000
--tensor_parallel_size 8
--merge_lora true

[Jintao-Huang]

vllm版本是什么呢

[zhangfan-algo]

最新版的 获取 Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ 发件人: Jintao ***@***.***> 发送时间: Wednesday, May 22, 2024 6:36:44 PM 收件人: modelscope/swift ***@***.***> 抄送: zhangfan-algo ***@***.***>; Author ***@***.***> 主题: Re: [modelscope/swift] qwen1.5-32b-chat 使用vllm推理很慢 (Issue #986) vllm版本是什么呢 ― Reply to this email directly, view it on GitHub<#986 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALMJFNDW73B6S6LO7EZBQ7TZDRYLZAVCNFSM6AAAAABIDGDU6OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRUGQ3DQMZWGQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[Jintao-Huang]

再试试呢 减少了vllm中调用磁盘IO的次数

[Jintao-Huang]

#1035

[zhangfan-algo]

可以了目前很快 获取 Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ 发件人: Jintao ***@***.***> 发送时间: Saturday, June 1, 2024 1:21:25 PM 收件人: modelscope/swift ***@***.***> 抄送: zhangfan-algo ***@***.***>; Author ***@***.***> 主题: Re: [modelscope/swift] qwen1.5-32b-chat 使用vllm推理很慢 (Issue #986) #1035<#1035> ― Reply to this email directly, view it on GitHub<#986 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALMJFNEH4IMWJ2SAJBP2ZYTZFFK5LAVCNFSM6AAAAABIDGDU6OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBTGMYDCNJTGQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>