Some problems about the finetuning animate-anything

[Shuweis]

When I finetune the animate-anything, I found that the gradient of unfrozen layer of Unet (eg. conv_in) is None. And I print the requires_grad of conv_in, the result is True. It means that the fine-tuning on the animate-anything is not work. What makes this phenomenon? Maybe It has some problems in the training process.

Hope for your reply!

[sculmh]

Within the unet model, conv_in and conv_in2 are defined, but one of them is not involved in the computation during training, resulting in its gradient being None. This is illustrated in the following code from models.unet_3d_blocks_mask:

# 2. pre-process
if self.motion_mask and mask is not None:
    mask = repeat(mask, 'b 1 1 h w -> (t b) 1 f h w', t=sample.shape[0] // mask.shape[0], f=sample.shape[2])
    sample = torch.cat([mask, sample], dim=1)
    sample = sample.permute(0, 2, 1, 3, 4).reshape((sample.shape[0] * num_frames, -1) + sample.shape[3:])
    sample = self.conv_in2(sample)
else:
    sample = sample.permute(0, 2, 1, 3, 4).reshape((sample.shape[0] * num_frames, -1) + sample.shape[3:])
    sample = self.conv_in(sample)

[Shuweis]

Thanks for your reply!
And I found another question about it. If I directly print the "unet.conv_in2.weight.grad" after "accelerator.backward(loss)"
It will return None when I use accelerate launch for distributed training. And If I do not use it, just use "python train.py -- ..."
It will not return None. Does the accelerate launch affect the gradient of this layer?

[Shuweis]

And I have another question. Why do you use conv_in2 instead of conv_in to handle the [mask concat latent], I saw you conduct zero init when perform the load function:

`def load_primary_models(pretrained_model_path, in_channels=-1, motion_strength=False):
noise_scheduler = DDPMScheduler.from_pretrained(pretrained_model_path, subfolder="scheduler")
tokenizer = CLIPTokenizer.from_pretrained(pretrained_model_path, subfolder="tokenizer")
text_encoder = CLIPTextModel.from_pretrained(pretrained_model_path, subfolder="text_encoder")
vae = AutoencoderKL.from_pretrained(pretrained_model_path, subfolder="vae")
unet = UNet3DConditionModel.from_pretrained(pretrained_model_path, subfolder="unet")
if in_channels>0 and unet.config.in_channels != in_channels:
#first time init, modify unet conv in
unet2 = unet
unet = UNet3DConditionModel.from_pretrained(pretrained_model_path, subfolder="unet",
in_channels=in_channels,
low_cpu_mem_usage=False, device_map=None, ignore_mismatched_sizes=True,
motion_strength=motion_strength)
unet.conv_in.bias.data = copy.deepcopy(unet2.conv_in.bias)
torch.nn.init.zeros_(unet.conv_in.weight)
load_in_channel = unet2.conv_in.weight.data.shape[1]
unet.conv_in.weight.data[:,in_channels-load_in_channel:]= copy.deepcopy(unet2.conv_in.weight.data)
del unet2

return noise_scheduler, tokenizer, text_encoder, vae, unet`

It seems that the zero init with mask in conv is more reasonable.