[Feature Request] Control model offsets and some others

[lllyasviel]

Hi great plugin. I played with it a bit and the memory optimization is really good. Some considerations should be put with higher priority:

1. The requirements for some mode like the segmentation is broken.

2. We really need a button to offset the weight inside the controlnet: that will immediately solve the distorted faces in pose mode and distorted edges in canny mode. This can be done by offsetting the weight copy inside the controlnet with user models and sd15. Right now all controls are targeted to SD15. These should be retargeted to user models.

[lllyasviel]

or at least we should put some control_any3_X.pth files for anime models. I cannot release these officially because of many considerations. But this can be done by 3rd projects.
But a button to directly compute it from user model is better.

[lllyasviel]

the formulation is

Any3.control_model.weights 
= SD15.control_model.weights + Any3.model.diffusion_model.weights - SD15.model.diffusion_model.weights

[lllyasviel]

or we can directly store

SD15.control_model.weights - SD15.model.diffusion_model.weights

and any time user load the model, it add the base weight from user model

[lllyasviel]

Updated lllyasviel/ControlNet#12

[lllyasviel]

This should be considered early since it seems that this plugin is super popular in both english and asian community. so the early this is fixed, the less things we will need to ask people to try download new files.

[Mikubill]

Thanks for pointing out. Will implement some fix immediately

[Mikubill]

Fixed in b9efb60, should work but still need some tests.

[lllyasviel]

seems broken, not working anymore

[catboxanon]

Seems to be working for me with this most recent change. I got Offset cloned: 298 values in my console.

[lllyasviel]

Great. Now working. But how is the current offset computed? I do not even have sd15 in my webui.

[catboxanon]

This is what it looks like when unchecking Apply transfer control when loading models in settings by the way, just to confirm there is a difference. Same seed.