Best docker image for ktransformer

[artheru]

It seems everyone hits the wall when installing and running the sys. Here're my trials:

Window 11 24H2 @ EPYC 9V74x2 1.1T 4800M, baseline llama.cpp 4720: 5~7 t/s

native compilation success using CUDA12.4 + torch 2.4 + python3.11, not OK to run, NaN

python ./ktransformers/local_chat.py --model_path D:\models\DeepSeek-R1\DeepSeek-R1 --gguf_path D:\models\DeepSeek-R1\weights\deepseek-r1-q4\DeepSeek-R1-Q4_K_M\ --cpu_infer 100 --max_new_tokens 1000

WSL2:
(I really don't want to compile again...)
use docker approachingai/ktransformers 0.2.1
runs, but super slow, at about 0.1t/s.

python ./ktransformers/local_chat.py --model_path /ssd-avril/models/DeepSeek-R1/DeepSeek-R1 --gguf_path /ssd-avril/models/DeepSeek-R1/weights/deepseek-r1-q4/DeepSeek-R1-Q4_K_M/ --cpu_infer 100 --max_new_tokens 1000

What're your configurations, plz share.

[blockcloud]

try docker pull approachingai/ktransformers:v0.2.2rc1-AVX512
i run it successfully
you should notice the suffix, that is related to your cpu
use lscpu to get your cpu info, if you find avx512, then use this version; if you find avx2, then use docker pull approachingai/ktransformers:v0.2.2rc1-AVX2
and you can find details in the related issues

[cyhasuka]

If u want to build this repo yourself, u can try this docker image provide by modelscope:

docker run --gpus all  -it --net=host --ipc=host --name ktransformer -v /workspace:/workspace  modelscope-registry.cn-beijing.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.1.0-py310-torch2.3.0-tf2.16.1-1.18.0 /bin/bash

Then,

docker exec -it ktransformers bash
git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule init
git submodule update
cd ./ktransformers
export USE_NUMA=1
bash install.sh --verbose

Important

When compiling, please make sure that the code for the two repositories in third_party has been pulled down correctly.

And Enjoy!

[cyhasuka]

Be noticed, the properly --cpu_infer is physical cores +1 ，so the best is --cpu_infer 161 in your sys.