[Feature Request]: Add System-level optimization for CPU inference to wiki

[LynxPDA]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

The work on the CPU can be quite long.
Using some system optimizations, borrowed from HuggingFace, it turned out to increase the speed of work from 1.25x to 1.5x.

For my inference:

• Xeon E3 1265L v3 (16Gb, 4 core) speed up from 10s/it to 8s/it
• Ryzen 9 7950X (32Gb, 16 core) speed up from 2.54s/it to 1.7s/it

Proposed workflow

I added the following lines to the end of the webui-user.sh file:

export OMP_NUM_THREADS=16
export MKL_NUM_THREADS=16
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so:$LD_PRELOAD
export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms: 60000,muzzy_decay_ms:60000"
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libiomp5.so:$LD_PRELOAD

Having previously installed

• sudo apt-get install -y libjemalloc-dev
• sudo apt-get install intel-mkl

Additional information

Other system informations:

COMMANDLINE_ARGS="--precision autocast --use-cpu all --no-half --opt-channelslast --skip-torch-cuda-test --enable-insecure-extension-access"

python: 3.10.6  •  torch: 2.1.0.dev20230506+cpu  •  xformers: N/A  •  gradio: 3.28.1  •  commit: 5ab7f213  •  checkpoint: b4391b7978

OS Ubuntu 22.04

[catboxanon]

The wiki is editable by anyone. #10180

[devingfx]

Hi @LynxPDA !

Could you please explain a bit what options does and how to tweak for other CPU capabilities?

Thx

[LynxPDA]

Hi @LynxPDA !

Could you please explain a bit what options does and how to tweak for other CPU capabilities?

Thx

Hi, @devingfx
Certainly,

---

jmalloc is a memory allocator.
The weights of the model when working on the CPU are in RAM and the work can be accelerated by a little tuning.

background_thread

Enabling jemalloc background threads generally improves the tail latency for application threads, since unused memory purging is shifted to the dedicated background threads. In addition, unintended purging delay caused by application inactivity is avoided with background threads.

Suggested: background_thread:true when jemalloc managed threads can be allowed.

metadata_thp

Allowing jemalloc to utilize transparent huge pages for its internal metadata usually reduces TLB misses significantly, especially for programs with large memory footprint and frequent allocation / deallocation activities. Metadata memory usage may increase due to the use of huge pages.

Suggested for allocation intensive programs: metadata_thp:auto or metadata_thp:always, which is expected to improve CPU utilization at a small memory cost.

dirty_decay_ms and muzzy_decay_ms

Decay time determines how fast jemalloc returns unused pages back to the operating system, and therefore provides a fairly straightforward trade-off between CPU and memory usage. Shorter decay time purges unused pages faster to reduces memory usage (usually at the cost of more CPU cycles spent on purging), and vice versa.

Suggested: tune the values based on the desired trade-offs.

More details on tuning and each of the parameters can be found at https://github.com/jemalloc/jemalloc/blob/dev/TUNING.md

And it is this setting that gives the maximum performance increase on the CPU.

---

As for libiomp5.so, it allows you to optimize parallel processing on the CPU. And this optimization also gives a small gain, but much less than with jmalloc.

I think these same settings will work for most other processors.
The only condition is probably to install jmalloc and work in linux.

[devingfx]

Hi, thx for this quick reply!

After more in depth reading, there is a typo on line ending:

dirty_decay_ms: 60000,muzzy_decay_ms:>

... looks like a nano copy/paste with line truncated ^^;

---

I meant also for NUM_THREADS part... I don't know how to figure out my potato PC capabilities... I know I don't have a GPU...

System:    Kernel: 5.15.0-58-generic x86_64 bits: 64 compiler: N/A Desktop: Cinnamon 5.2.7 
           wm: muffin dm: LightDM Distro: Linux Mint 20.3 Una base: Ubuntu 20.04 focal 
Machine:   Type: Desktop Mobo: Gigabyte model: B450M S2H serial: <filter> 
           UEFI: American Megatrends LLC. v: F62d date: 10/13/2021 
CPU:       Topology: 6-Core model: AMD Ryzen 5 1600 bits: 64 type: MT MCP arch: Zen rev: 1 
           L2 cache: 3072 KiB 
           flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm bogomips: 76654 
           Speed: 1481 MHz min/max: 1550/3200 MHz Core speeds (MHz): 1: 1594 2: 1326 3: 1398 
           4: 1375 5: 1352 6: 1374 7: 1382 8: 1374 9: 1742 10: 1339 11: 1461 12: 1378 
Graphics:  Device-1: NVIDIA GK208B [GeForce GT 710] vendor: ASUSTeK GT710-SL-1GD5 driver: nvidia https://www.asus.com/motherboards-components/graphics-cards/asus/gt710-sl-1gd5/techspec/
           v: 470.161.03 bus ID: 0a:00.0 chip ID: 10de:128b 
           Display: x11 server: X.Org 1.20.13 driver: nvidia 
           unloaded: fbdev,modesetting,nouveau,vesa resolution: 1920x1080~60Hz, 1920x1080~60Hz 
           OpenGL: renderer: NVIDIA GeForce GT 710/PCIe/SSE2 v: 4.6.0 NVIDIA 470.161.03 
           direct render: Yes 

[devingfx]

Is the intel-mkl package needed for non-intel CPU?

[devingfx]

I get :

################################################################
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libiomp5.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
<jemalloc>: Invalid conf value: muzzy_decay_ms:>

I tried with muzzy_decay_ms:60000, same errors...

[LynxPDA]

Hi, thx for this quick reply!

After more in depth reading, there is a typo on line ending:

dirty_decay_ms: 60000,muzzy_decay_ms:>

... looks like a nano copy/paste with line truncated ^^;

I meant also for NUM_THREADS part... I don't know how to figure out my potato PC capabilities... I know I don't have a GPU...

System:    Kernel: 5.15.0-58-generic x86_64 bits: 64 compiler: N/A Desktop: Cinnamon 5.2.7 
           wm: muffin dm: LightDM Distro: Linux Mint 20.3 Una base: Ubuntu 20.04 focal 
Machine:   Type: Desktop Mobo: Gigabyte model: B450M S2H serial: <filter> 
           UEFI: American Megatrends LLC. v: F62d date: 10/13/2021 
CPU:       Topology: 6-Core model: AMD Ryzen 5 1600 bits: 64 type: MT MCP arch: Zen rev: 1 
           L2 cache: 3072 KiB 
           flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm bogomips: 76654 
           Speed: 1481 MHz min/max: 1550/3200 MHz Core speeds (MHz): 1: 1594 2: 1326 3: 1398 
           4: 1375 5: 1352 6: 1374 7: 1382 8: 1374 9: 1742 10: 1339 11: 1461 12: 1378 
Graphics:  Device-1: NVIDIA GK208B [GeForce GT 710] vendor: ASUSTeK GT710-SL-1GD5 driver: nvidia https://www.asus.com/motherboards-components/graphics-cards/asus/gt710-sl-1gd5/techspec/
           v: 470.161.03 bus ID: 0a:00.0 chip ID: 10de:128b 
           Display: x11 server: X.Org 1.20.13 driver: nvidia 
           unloaded: fbdev,modesetting,nouveau,vesa resolution: 1920x1080~60Hz, 1920x1080~60Hz 
           OpenGL: renderer: NVIDIA GeForce GT 710/PCIe/SSE2 v: 4.6.0 NVIDIA 470.161.03 
           direct render: Yes 

I'm sorry, you're absolutely right, the line is truncated at the end when copying from the nano editor.

Corrected. The correct option is below.

export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms: 60000,muzzy_decay_ms:60000"

[LynxPDA]

Is the intel-mkl package needed for non-intel CPU?

Yes, intel-mkl can work with AMD processor too.
However, it is quite large. You can try installing it separately.
If I'm not mistaken, it is also part of Clang and llvm.

sudo apt-get install libomp-dev

[devingfx]

I get :

################################################################
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libiomp5.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
<jemalloc>: Invalid conf value: muzzy_decay_ms:>

I tried with muzzy_decay_ms:60000, same errors...

You didn't answer this...

Looks like setting muzzy/dirty to 30000 bypass the Invalid conf error, but still I have the ls.so error above...
I didn't (yet) installed intel package, is it related?

[devingfx]

Another subject:

Did you get ControlNet to work on CPU? Looks like preprocessors are using GPU (I have a cuda out of memory error) despite the --use-cpu all param...

[LynxPDA]

Another subject:

Did you get ControlNet to work on CPU? Looks like preprocessors are using GPU (I have a cuda out of memory error) despite the --use-cpu all param...

Yes, the ControlNet on the CPU worked for me without problems with the command line arguments:
--use-cpu all --no-half

If you are not using CUDA at all, you can install a nightly build of Pytorch for CPU only in venv.
You're right, with command line arguments "--use-cpu all" should work.

[LynxPDA]

ERROR: ld.so: object

Regarding:
ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/libiomp5.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Yes, please make sure that the library is available in the specified path, otherwise you need to install it, as I wrote earlier.

As I wrote earlier, you can try to install the missing library separately, without installing intel-mkl using the command below

sudo apt-get install libomp-dev

[hollowshiroyuki]

Hi, thanks for your optimizations !
I went from 30s/it to about 13s/it on my i7-1260P.
But I have the feeling there is still space for improvement on my setup : btop tells me python is using about 40% cpu and I still have about 4Gb of ram free.
Could you give me advice about how I should tweak your settings or what could I add to get even better performances ?

[devingfx]

Hi all!

On my side I could not get everthing working on CPU, I have got CUDA out of memory errors really often (even if it should not use CUDA on CPU only isn't it?) on image generation (that I restart and it work the 2nd time), I can not use (what I analysed) the "side models" like RealESRGAN, ControlNet preprocessing or faceswap by example...

I installed A1111 with default config the 1st time, then I tweaked webui-user.sh afterward (nottably with --use-cpu all) , so my question: Is there a special install process/config for all venv stuff (like pyTorch) to be used on CPU only?

Also, maybe an issue with my hardware, I do have a GPU but a poor 2Go RAM so I want to use CPU, but maybe there are some automatic detections that found a GPU and want to use it ?

PS: also after a recent "update all but you don't know what going on" button click in extension tab, image generation stopped working :( , I planned to do fresh reinstall, it is why I asked if I should go to a special process this time?