Fix model prediction detection

[catboxanon]

Addresses #1109

For newer SDXL-based models such as https://civitai.com/models/833294?modelVersionId=962003, authors will need to include a v_pred key in the state_dict with the model in order for it to be detected correctly.

Related: AUTOMATIC1111/stable-diffusion-webui#16567

[rabidcopy]

If models need to have prediction type hard coded into them, what does the end user do for models that don't have this? Example being the model linked.

[catboxanon]

SD2 models should hopefully be autodetected by this: https://github.com/lllyasviel/huggingface_guess/blob/84826248b49bb7ca754c73293299c4d4e23a548d/huggingface_guess/model_list.py#L173

The model I linked is now confirmed to have it included into the next release, which happens via the detection here: https://github.com/lllyasviel/huggingface_guess/blob/84826248b49bb7ca754c73293299c4d4e23a548d/huggingface_guess/model_list.py#L287-L288
(I added a download link for the current version with the v_pred key in the related PR linked above)

All of this code actually comes upstream from ComfyUI. https://github.com/comfyanonymous/ComfyUI/blob/7390ff3b1ec2e15017ba4a52d6eaabc4aa4636e3/comfy/supported_models.py

[rabidcopy]

And for old v-prediction SD1 models? From my understanding Forge no longer looks for yamls next to the model in Stable-diffusion/. Unless I'm mistaken?

[catboxanon]

I can try taking a look at that, but it might be challenging to get working since my guess is that was using the code from upstream A1111 (which can load .yaml configs next to models), and when lllyasviel reworked the entire backend a few months ago, he gutted that out. Forge is completely ComfyUI-based more or less for it's backend now, and I don't believe ComfyUI has support to load .yaml configs next to models like that currently.

[rabidcopy]

That's where I'm sort of seeing the disconnect in functionality. reForge for example has this extension that mimics (if not effectively ports) ComfyUI's ModelSamplingDiscrete node. Which allows the user to set prediction type on the fly as well as ZSNR and other specific architecture-specific sampling options. https://github.com/Panchovix/stable-diffusion-webui-reForge/tree/dev_upstream/extensions-builtin/reForge-advanced_model_sampling
ComfyUI hasn't depended on YAMLs ever since the checkpoint loader that allowed specifying a YAML was deprecated and as such it's been possible to set the prediction for models like this for a while now. ComfyUI does automatically detect prediction for base models and whatnot, but there's been a way to manually set the prediction when the auto-detect doesn't meet user needs when unconventional models arise. I feel like expecting the user to alter their models or hoping model authors will be aware of how to make their models' prediction type auto-detected by various frontends and forks is an inconvenient solution compared to just allowing the user to manually set the prediction.

Edit: Though I'm not saying the auto-detection and your PR is bad, it's actually still a step in the right direction. I am just pondering what can be done in situations where it can fall short.

[catboxanon]

Oh, I did just realize you're the one that wrote the patch to add it in settings. If you make a PR for that I'll take a second look at it and see about getting it merged. Having it default to "Automatic" with other options overwriting it seems ideal to me.

[internetakias]

I tried to add the v_pred key to a SD 1.5 v_pred model but it didn't seem to work. Does the code not look for they key at all in 1.5 models?

[catboxanon]

Fixed .yaml loading now. #2224

[turbo-boo]

noobai xl does not produce a normal image. Can't I just load the checkpoints?

[catboxanon]

@Bocchi-Chan2023 Assuming you mean the v-pred test version, that one does not have the necessary keys in the state_dict to be detected automatically. You would need to add a .yaml file next to the checkpoint with the same filename (i.e. file.safetensors and file.yaml) that contains this:

model:
  params:
    parameterization: "v"

You would also be expected to enable zero terminal SNR in settings.

Future v-pred releases for NoobAI will contain the necessary state_dict keys for auto-detection and make it function as normal without any extra work.

[turbo-boo]

Thank you, it was different from A1111