Add DoRA (weight-decompose) support for LoRA/LoHa/LoKr

[KohakuBlueleaf]

Description

Implementation for inference with DoRA
The key name is based on the implementation in LyCORIS.

And since weight-decompose is a general idea on top of all low-rank method. I implement it in the NetworkModule instead of each algo's module.

And here is a quick sanity check:

Checklist:

• I have read contributing wiki page
• I have performed a self-review of my own code
• My code follows the style guidelines
• My code passes tests

[pessimal]

@KohakuBlueleaf I don't know if it's for every Lora but with this PR I've constantly been getting warnings/errors like this.

loading network /srv/linuxdata/stable-diffusion-webui_dev/models/Lora/SDXL_Lora/add-detail-xl.safetensors: AttributeError
Traceback (most recent call last):
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/networks.py", line 280, in load_networks
net = load_network(name, network_on_disk)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/networks.py", line 219, in load_network
net_module = nettype.create_module(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network_lora.py", line 11, in create_module
return NetworkModuleLora(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network_lora.py", line 18, in init
super().init(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network.py", line 150, in init
self.dora_mean_dim = tuple(i for i in range(len(self.shape)) if i != 1)
AttributeError: 'NetworkModuleLora' object has no attribute 'shape'

[KohakuBlueleaf]

@KohakuBlueleaf I don't know if it's for every Lora but with this PR I've constantly been getting warnings/errors like this. Other than that I feel Loras now have much more effect on the result of the generation

loading network /srv/linuxdata/stable-diffusion-webui_dev/models/Lora/SDXL_Lora/add-detail-xl.safetensors: AttributeError
Traceback (most recent call last):
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/networks.py", line 280, in load_networks
net = load_network(name, network_on_disk)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/networks.py", line 219, in load_network
net_module = nettype.create_module(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network_lora.py", line 11, in create_module
return NetworkModuleLora(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network_lora.py", line 18, in init
super().init(net, weights)
File "/srv/linuxdata/stable-diffusion-webui_dev/extensions-builtin/Lora/network.py", line 150, in init
self.dora_mean_dim = tuple(i for i in range(len(self.shape)) if i != 1)
AttributeError: 'NetworkModuleLora' object has no attribute 'shape'

Thx for this info
I will fix it later

[clayne]

Also seeing it:

Using already loaded model sdxl\sd_xl_base_1.0_0.9vae.safetensors [e6bb9ea85b]: done in 2.8s (send model to cpu: 1.8s, send model to device: 1.
0s)
loading network F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\models\Lora\sdxl\Syd Mead Style.safetensors: AttributeError
Traceback (most recent call last):
  File "F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\extensions-builtin\Lora\networks.py", line 280, in load_networks
    net = load_network(name, network_on_disk)
  File "F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\extensions-builtin\Lora\networks.py", line 219, in load_network
    net_module = nettype.create_module(net, weights)
  File "F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\extensions-builtin\Lora\network_lora.py", line 11, in create_module
    return NetworkModuleLora(net, weights)
  File "F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\extensions-builtin\Lora\network_lora.py", line 18, in __init__
    super().__init__(net, weights)
  File "F:\stablediffusion\stable-diffusion-webui-1-8-0-rc\extensions-builtin\Lora\network.py", line 150, in __init__
    self.dora_mean_dim = tuple(i for i in range(len(self.shape)) if i != 1)
AttributeError: 'NetworkModuleLora' object has no attribute 'shape'. Did you mean: 'scale'?

[pessimal]

Until now I assumed this was more a warning but any Lora that shows this error is just discarded and not used during generation. (doesn't get listed under "Lora hashes:" in the generation output)

[SlZeroth]

someone tested dora in this code?

I got this error

merged_scale1 / merged_scale1(dim=self.dora_mean_dim, keepdim=True) * self.dora_scale

TypeError: 'Tensor' object is not callable

[SlZeroth]

merged_scale1 = updown + orig_weight

merged_scale1 type is a tensor that is not callable

[KohakuBlueleaf]

merged_scale1 = updown + orig_weight

merged_scale1 type is a tensor that is not callable

I said
I will fix it later