[Bug]: Restoring mean average of conditioning can cause extreme scaling

[Filexor]

Checklist

• The issue exists after disabling all extensions
• The issue exists on a clean installation of webui
• The issue is caused by an extension, but I believe it is caused by a bug in the webui
• The issue exists in the current version of the webui
• The issue has not been reported before recently
• The issue has been reported before but has not been fixed yet

What happened?

In modules\sd_hijack_clip.py at line 282,
There is a code that restores mean average of conditioning
and when new_mean that much closer to zero than original_mean is given to that code,
conditioning will be scaled extremely and breaks result.

Steps to reproduce the problem

1. Load animagineXLV3_v30.safetensors .
2. Put (worst quality, low quality:1.5), nsfw, as negative prompt. (positive prompt can by any.)
3. Generate.

What should have happened?

Result should not be like below:

Following is image without the restortion, with same setting:

What browsers do you use to access the UI ?

Microsoft Edge

Sysinfo

sysinfo-2024-02-08-13-58.json

Console logs

Creating venv in directory H:\stable-diffusion-webui-dev\venv using python "C:\Users\File_xor\AppData\Local\Programs\Python\Python310\python.exe"
venv "H:\stable-diffusion-webui-dev\venv\Scripts\Python.exe"
Python 3.10.7 (tags/v3.10.7:6cc6b13, Sep  5 2022, 14:08:36) [MSC v.1933 64 bit (AMD64)]
Version: v1.7.0
Commit hash: cf2772fab0af5573da775e7437e6acdca424f26e
Installing torch and torchvision
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu118
Collecting torch==2.0.1
  Downloading https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-win_amd64.whl (2619.1 MB)
     ---------------------------------------- 2.6/2.6 GB 1.0 MB/s eta 0:00:00
Collecting torchvision==0.15.2
  Downloading https://download.pytorch.org/whl/cu118/torchvision-0.15.2%2Bcu118-cp310-cp310-win_amd64.whl (4.9 MB)
     ---------------------------------------- 4.9/4.9 MB 35.0 MB/s eta 0:00:00
Collecting sympy
  Using cached https://download.pytorch.org/whl/sympy-1.12-py3-none-any.whl (5.7 MB)
Collecting jinja2
  Using cached Jinja2-3.1.3-py3-none-any.whl (133 kB)
Collecting filelock
  Using cached filelock-3.13.1-py3-none-any.whl (11 kB)
Collecting networkx
  Using cached https://download.pytorch.org/whl/networkx-3.2.1-py3-none-any.whl (1.6 MB)
Collecting typing-extensions
  Using cached typing_extensions-4.9.0-py3-none-any.whl (32 kB)
Collecting requests
  Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Collecting pillow!=8.3.*,>=5.3.0
  Using cached https://download.pytorch.org/whl/pillow-10.2.0-cp310-cp310-win_amd64.whl (2.6 MB)
Collecting numpy
  Using cached numpy-1.26.4-cp310-cp310-win_amd64.whl (15.8 MB)
Collecting MarkupSafe>=2.0
  Using cached MarkupSafe-2.1.5-cp310-cp310-win_amd64.whl (17 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.6-py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2024.2.2-py3-none-any.whl (163 kB)
Collecting urllib3<3,>=1.21.1
  Using cached urllib3-2.2.0-py3-none-any.whl (120 kB)
Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.3.2-cp310-cp310-win_amd64.whl (100 kB)
Collecting mpmath>=0.19
  Using cached https://download.pytorch.org/whl/mpmath-1.3.0-py3-none-any.whl (536 kB)
Installing collected packages: mpmath, urllib3, typing-extensions, sympy, pillow, numpy, networkx, MarkupSafe, idna, filelock, charset-normalizer, certifi, requests, jinja2, torch, torchvision
Successfully installed MarkupSafe-2.1.5 certifi-2024.2.2 charset-normalizer-3.3.2 filelock-3.13.1 idna-3.6 jinja2-3.1.3 mpmath-1.3.0 networkx-3.2.1 numpy-1.26.4 pillow-10.2.0 requests-2.31.0 sympy-1.12 torch-2.0.1+cu118 torchvision-0.15.2+cu118 typing-extensions-4.9.0 urllib3-2.2.0

[notice] A new release of pip available: 22.2.2 -> 24.0
[notice] To update, run: H:\stable-diffusion-webui-dev\venv\Scripts\python.exe -m pip install --upgrade pip
Installing clip
Installing open_clip
Cloning Stable Diffusion into H:\stable-diffusion-webui-dev\repositories\stable-diffusion-stability-ai...
Cloning into 'H:\stable-diffusion-webui-dev\repositories\stable-diffusion-stability-ai'...
remote: Enumerating objects: 580, done.
remote: Counting objects: 100% (357/357), done.
remote: Compressing objects: 100% (126/126), done.
Receiving objects:  95% (551/580), 66.04 MiB | 21.98 MiB/sk-reused 223
Receiving objects: 100% (580/580), 73.44 MiB | 21.84 MiB/s, done.
Resolving deltas: 100% (279/279), done.
Cloning Stable Diffusion XL into H:\stable-diffusion-webui-dev\repositories\generative-models...
Cloning into 'H:\stable-diffusion-webui-dev\repositories\generative-models'...
remote: Enumerating objects: 860, done.
remote: Counting objects: 100% (481/481), done.
remote: Compressing objects: 100% (212/212), done.
remote: Total 860 (delta 363), reused 269 (delta 269), pack-reused 379
Receiving objects: 100% (860/860), 42.67 MiB | 19.54 MiB/s, done.
Resolving deltas: 100% (444/444), done.
Cloning K-diffusion into H:\stable-diffusion-webui-dev\repositories\k-diffusion...
Cloning into 'H:\stable-diffusion-webui-dev\repositories\k-diffusion'...
remote: Enumerating objects: 1340, done.
remote: Counting objects: 100% (738/738), done.
remote: Compressing objects: 100% (91/91), done.
Receiving objects:  98% (1314/1340)sed 654 (delta 647), pack-reused 602
Receiving objects: 100% (1340/1340), 236.03 KiB | 10.73 MiB/s, done.
Resolving deltas: 100% (941/941), done.
Cloning CodeFormer into H:\stable-diffusion-webui-dev\repositories\CodeFormer...
Cloning into 'H:\stable-diffusion-webui-dev\repositories\CodeFormer'...
remote: Enumerating objects: 594, done.
remote: Counting objects: 100% (594/594), done.
remote: Compressing objects: 100% (317/317), done.
remote: Total 594 (delta 288), reused 491 (delta 268), pack-reused 0Receiving objects:  99% (589/594), 12.57 MiB | 25.13 MiB/s
Receiving objects: 100% (594/594), 17.31 MiB | 25.39 MiB/s, done.
Resolving deltas: 100% (288/288), done.
Cloning BLIP into H:\stable-diffusion-webui-dev\repositories\BLIP...
Cloning into 'H:\stable-diffusion-webui-dev\repositories\BLIP'...
remote: Enumerating objects: 277, done.
remote: Counting objects: 100% (165/165), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 277 (delta 137), reused 136 (delta 135), pack-reused 112
Receiving objects: 100% (277/277), 7.03 MiB | 26.68 MiB/s, done.
Resolving deltas: 100% (152/152), done.
Installing requirements for CodeFormer
Installing requirements
Launching Web UI with arguments:
no module 'xformers'. Processing without...
no module 'xformers'. Processing without...
No module 'xformers'. Proceeding without it.
Style database not found: H:\stable-diffusion-webui-dev\styles.csv
Downloading: "https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.safetensors" to H:\stable-diffusion-webui-dev\models\Stable-diffusion\v1-5-pruned-emaonly.safetensors

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.97G/3.97G [01:48<00:00, 39.2MB/s]
Calculating sha256 for H:\stable-diffusion-webui-dev\models\Stable-diffusion\v1-5-pruned-emaonly.safetensors: Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 483.6s (prepare environment: 359.6s, import torch: 4.5s, import gradio: 2.0s, setup paths: 2.3s, initialize shared: 0.6s, other imports: 2.4s, setup codeformer: 0.3s, list SD models: 109.5s, load scripts: 1.4s, create ui: 0.5s, gradio launch: 0.4s).
6ce0161689b3853acaa03779ec93eafe75a02f4ced659bee03f50797806fa2fa
Loading weights [6ce0161689] from H:\stable-diffusion-webui-dev\models\Stable-diffusion\v1-5-pruned-emaonly.safetensors
Creating model from config: H:\stable-diffusion-webui-dev\configs\v1-inference.yaml
Applying attention optimization: Doggettx... done.
Model loaded in 8.7s (calculate hash: 4.7s, load weights from disk: 0.2s, create model: 0.3s, apply weights to model: 3.1s, calculate empty prompt: 0.1s).
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00,  8.51it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 10.52it/s]
Reusing loaded model v1-5-pruned-emaonly.safetensors [6ce0161689] to load sd_xl_base_1.0_0.9vae.safetensors████████████████████| 20/20 [00:01<00:00, 11.22it/s]
Calculating sha256 for H:\stable-diffusion-webui-dev\models\Stable-diffusion\sd_xl_base_1.0_0.9vae.safetensors: e6bb9ea85bbf7bf6478a7c6d18b71246f22e95d41bcdd80ed40aa212c33cfeff
Loading weights [e6bb9ea85b] from H:\stable-diffusion-webui-dev\models\Stable-diffusion\sd_xl_base_1.0_0.9vae.safetensors
Creating model from config: H:\stable-diffusion-webui-dev\repositories\generative-models\configs\inference\sd_xl_base.yaml
Applying attention optimization: Doggettx... done.
Model loaded in 4.8s (create model: 0.5s, apply weights to model: 3.9s, move model to device: 0.1s, calculate empty prompt: 0.1s).
Downloading VAEApprox model to: H:\stable-diffusion-webui-dev\models\VAE-approx\vaeapprox-sdxl.pt
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 209k/209k [00:00<00:00, 13.3MB/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.76it/s]
==========================================================================================█████████████████████████████████████| 20/20 [00:09<00:00,  1.95it/s]
A tensor with all NaNs was produced in VAE.
Web UI will now convert VAE into 32-bit float and retry.
To disable this behavior, disable the 'Automatically revert VAE to 32-bit floats' setting.
To always start with 32-bit VAE, use --no-half-vae commandline flag.
==========================================================================================
Total progress: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:20<00:00,  1.05s/it]
Reusing loaded model sd_xl_base_1.0_0.9vae.safetensors [e6bb9ea85b] to load animagineXLV3_v30.safetensors██████████████████████| 20/20 [00:20<00:00,  1.95it/s]
Calculating sha256 for H:\stable-diffusion-webui-dev\models\Stable-diffusion\animagineXLV3_v30.safetensors: 1449e5b0b9de87b0f414c5f29cb11ce3b3dc61fa2b320e784c9441720bf7b766
Loading weights [1449e5b0b9] from H:\stable-diffusion-webui-dev\models\Stable-diffusion\animagineXLV3_v30.safetensors
Applying attention optimization: Doggettx... done.
Weights loaded in 34.3s (send model to cpu: 2.9s, calculate hash: 27.4s, load weights from disk: 0.8s, apply weights to model: 1.1s, move model to device: 2.0s).
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:10<00:00,  1.98it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:15<00:00,  1.25it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:15<00:00,  2.03it/s]

Additional information

No response

[catboxanon]

This was addressed in #14874 and e2b1990. Use the new emphasis option of No Norm to resolve it.