Confliction of A1111 extension of PwW+Control to the original extension of ControlNet

[lwchen6309]

The current implementation conflict with the original extension of ControlNet, as reported by @mykeehu, here.

This is caused by the same argument defined in preload.py in the extension.

def preload(parser):
    parser.add_argument("--controlnet-dir", type=str, help="Path to directory with ControlNet models", default=None)
    parser.add_argument("--no-half-controlnet", action='store_true', help="do not switch the ControlNet models to 16-bit floats (only needed without --no-half)", default=None)

I'm trying to fix this bug, and open this issue for further discussion if there is any.

[lwchen6309]

Just to update the progress.
Currently this extension is not compatible with the original extension of ControlNet dues to the conflict of arguments

Although this error can be fixed by changing the script to

def preload(parser):
    if not any(arg.dest == 'controlnet_dir' for arg in parser._actions):
        parser.add_argument("--controlnet-dir", type=str, help="Path to directory with ControlNet models", default=None)
    if not any(arg.dest == 'no_half_controlnet' for arg in parser._actions):
        parser.add_argument("--no-half-controlnet", action='store_true', help="do not switch the ControlNet models to 16-bit floats (only needed without --no-half)", default=None)

while it takes an unreasonably long time to load UI instead.

[nistvan86]

What is the reason why it loads so slowly when I apply this fix to preload.py? Is it reloading things multiple times?
Mod: Actually it won't load for me at all leaving it running for multiple minutes, it fills up my 16GB RAM completely.

[lwchen6309]

I had update the preload.py to avoid the confliction of arguments.

On the other hand, it turns out the program is stuck at updating ui at the line 845 of scripts/controlnet.py
when both repo are loaded.
Commenting this line solve the problem while making UI much more complicated.
The loaded UI would look something like this

Please see the explanation here

[mykeehu]

Just a hint: if there is a sd-webui-controlnet folder inside the extensions folder, you don't fill in the 845 line. It would only be better because if I manually comment it, git pull update is not possible.

After disabling line 845 and Controlnet, the UI comes up, but eventually it doesn't load, it's completely broken:

It may also be conflict with one of the other extensions. I have these installed:

[lwchen6309]

It seems to be another conflict since commenting line 845 works for my case, which I test it on Windows, Linux, and MACOS.

Just to confirm, do these extensions compatible with Mikubill/sd-webui-controlnet?
I would expect the same conflict of sd-webui-controlnet to other extension you installed since the extension of ControlNet+PwW simply adapt it by adding PwW.

[mykeehu]

Yes, Controlnet works with these and they do not cause problems with each other.

[lwchen6309]

Hi @mykeehu , thanks a lot for your information.

It's kinda weird to me since the two repo basically is doing the same thing.
I'm not able to check which extension ControlNet+PwW exactly conflict with dues to a large amount of extension in your case.

However, I guess this bug might come from duplicated UI, which is created when I inherit the UI of controlnet to create the UI for ControlNet+PwW.
To validate this, I create a simplified code of ControlNet+PwW that does not create a new UI for PwW but directly add it to the original controlnet UI in scripts/control.py. Also, I make it compatible with Mikubill/sd-webui-controlnet
Please try this version

git clone -b merge_control git@github.com:lwchen6309/sd-webui-controlnet-pww.git

and let me know if it works or not.

If it still doesn't work, please disable Mikubill/sd-webui-controlnet and uncomment line 845, which now becomes line 965 of controlnet.py
script_callbacks.on_after_component(img2img_tab_tracker.on_after_component_callback)

Thank you so much for your report :)

[mykeehu]

Very interesting...
I moved all the extensions to another folder, leaving yours alone. I got this on load, which I didn't get before:

H:\Stable-Diffusion-Automatic\stable-diffusion-webui\venv\lib\site-packages\gradio\deprecation.py:43: UserWarning: You have unused kwarg parameters in Image, please remove them: {'image': array([[[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]],

       [[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255]]], dtype=uint8)}
  warnings.warn(
H:\Stable-Diffusion-Automatic\stable-diffusion-webui\venv\lib\site-packages\gradio\deprecation.py:43: UserWarning: You have unused kwarg parameters in Tab, please remove them: {'open': False}
  warnings.warn(

and the UI still does not load...

Now I deleted the folder and reinstalled it with this command in extension folder, but no helped:
git clone https://github.com/lwchen6309/sd-webui-controlnet-pww

[lwchen6309]

Hmm... it's indeed interesting but also makes things clearer.
At least we don't have to deal with other extensions at a moment.
The warning of gradio is not clear to me but we can still start by gradually disabling the PwW function.

(a) Let's try a bit simpler version,

git clone -b dummy git@github.com:lwchen6309/sd-webui-controlnet-pww.git

where I add a dummy flag to disable all the PwW modules and UI.

https://github.com/lwchen6309/sd-webui-controlnet-pww/blob/647febe0f89ba5e7d58a8f9a11f91814d8423d2e/scripts/controlnet.py#L27

The PwW is disable when the flag is set to True. In this case, the function of this repo should be exactly the same as Mikubill/sd-webui-controlnet.

(b) If that does not works for you, then try this version

git clone -b totally_dummy git@github.com:lwchen6309/sd-webui-controlnet-pww.git

where I comment all the code associated with PwW and exactly match the code of Mikubill/sd-webui-controlnet; it should work for you as you mentioned previously.

Note that the Mikubill/sd-webui-controlnet I fork is the latest version (commit dbdd6b1), please also check if this veriosn works for you. Also, please check if you got the latest A1111 webui installed and let me know the gradio version you got.

I'd assume at least (b) should work as it is exactly the code of Mikubill/sd-webui-controlnet.
If (b) works while (a) does not work, then the only three candidates to bring bug is scripts/pww_utils.py, scripts/hook_pww.py, and scripts/controlney.py.
Validation of the first two is simply commenting all the code of these files, respectively, and see if UI works or not.
If none of these works, then the problem comes from scripts/controlney.py, which is likely to be the PwW ui from the warning message you showed. If that happens, we probably have to split the PwW ui later.

Lastly, thank you so much for your patience in figuring out the bug with me :)

[mykeehu]

The UI worked fine on the first version, but I got an interesting error message on the console after loaded embeddings:

Model loaded in 10.9s (load weights from disk: 2.2s, create model: 1.3s, apply weights to model: 0.9s, apply half(): 3.2s, load VAE: 0.5s, move model to device: 1.1s, load textual inversion embeddings: 1.6s).
H:\Stable-Diffusion-Automatic\stable-diffusion-webui\venv\lib\site-packages\gradio\deprecation.py:43: UserWarning: You have unused kwarg parameters in Tab, please remove them: {'open': False}
  warnings.warn(
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 47.7s (import gradio: 3.4s, import ldm: 4.1s, other imports: 2.4s, list extensions: 0.2s, list SD models: 0.2s, setup codeformer: 0.1s, load scripts: 3.1s, load SD checkpoint: 11.1s, create ui: 22.5s, gradio launch: 0.6s).

I'll try version b in a moment, but this is how I retrieve it from git, because I get an error on your command:

H:\Stable-Diffusion-Automatic\stable-diffusion-webui\extensions>git clone -b dummy git@github.com:lwchen6309/sd-webui-controlnet-pww.git
Cloning into 'sd-webui-controlnet-pww'...
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

H:\Stable-Diffusion-Automatic\stable-diffusion-webui\extensions>git clone -b dummy https://github.com/lwchen6309/sd-webui-controlnet-pww.git

The second version is also tested, the UI works there too. No other extensions were installed when I tested it.
Without the debug command, the previously described problem remains.

[lwchen6309]

It's great at least the dummy version works, except for the error message on the console after loaded embeddings (we can come back to this issue after making PwW UI work for you.

The code of PwW in "scripts/controlney.py" is basically doing 3 things, which will be checked gradually in the rest of debugging steps:
(1) Import the PwW core function from scripts/pww_utils.py and the hook that hijacks diffusion UNet in scripts/hook_pww.py.
(2) Create UI
(3) Apply the hook and Inject CrossAttention forward function

Steps:
(a) Please try this version that enable (1) importing functions from scripts/pww_utils.py and in scripts/hook_pww.py

git clone -b dummy_importFunction https://github.com/lwchen6309/sd-webui-controlnet-pww.git

(b) If (a) works, please try this version that enable (1) and (2)

git clone -b dummy_createUI https://github.com/lwchen6309/sd-webui-controlnet-pww.git

I'd expect some error at this stage from the gradio warning message you showed a moment ago.
Also, from the gradio error message, the problem might come from line 419 of scripts/controlnet.py

def create_canvas(h, w):
    return np.zeros(shape=(h, w, 3), dtype=np.uint8) + 255

If you encounter the same gradio warning, try this version

git clone -b dummy_createUI_pil https://github.com/lwchen6309/sd-webui-controlnet-pww.git

that I disable the some of UI initialization utilizing "create_canvas".

Thanks you so much for your effort and information :)

[mykeehu]

Here are the final results:

• the first version worked fine
• the second version caused the usual error in the console
• the third version did not give any error in the console, but the UI stopped loading at the Loading prompt, as before.

[lwchen6309]

Thanks a lot for the results.

(a) Please try this version that I further comment the UI of "Color context option" that might leads to problem.

git clone -b dummy_createUI_pil https://github.com/lwchen6309/sd-webui-controlnet-pww.git

The rest of UI is simple gradio component, as usual as in the Controlnet UI, and should be safe.

(b) If this version work, please set the dummy flag to false to enable PwW function. This would not change UI but activate (3) Apply the hook and Inject CrossAttention forward function.
You can then use PwW following the instruction of repo, hopefully.

I hope we are reaching the end of debugging, thanks again for your patience and help :)

[mykeehu]

Good news! Everything is now loaded, including Controlnet and PWW before deactivated dummy!
The canvas is very slow to colour, it crashed on the first try. It is impossible to draw.

It still works after turning off the dummy, so that's fine. :) Apart from the slowness of drawing, Extract and Generate do nothing for me.
I loaded a picture, the two buttons still don't work. What do you mean? What should I do?
"Apply the hook and Inject CrossAttention forward function."
Is this the one in Settings? "Allow other script to control this extension"