Official implementation of the paper "Region-to-Region: Enhancing Generative Image Harmonization with Adaptive Regional Injection".

Note:

1. This repository is released solely for anonymous review purposes. The paper is currently under review.
2. To ensure a double-blind review process, the issue section has been disabled during the review period.
3. All download links are anonymous.

cd Region_to_Region
conda env create -f environment.yml -n r2r
conda activate r2r

Download LDM checkpoint: here
Download Clear-VAE checkpoint: here

Place them under inference/checkpoints. forming the following structure:

inference/
└── checkpoints/
    ├── clear_vae/
    ├── stable-diffusion-inpainting/

Please refer to the DiffHarmony to prepare data. Place the iHarmony4 dataset inside the data folder, forming the following structure:

data/iHarmony4
|- HAdobe5k/
    |- composite_images
    |- masks
    |- real_images
    |- ...
|- HCOCO/
|- HFlickr/
|- Hday2night/
|- train.jsonl
|- test.jsonl

Please download it from here and place it under the data folder. Make sure the structure is just like that:

data/RPHarmony
|- R-ADE20K
    |- composite_images
    |- masks
    |- real_images
    |- ...
|- R-DUTS
|- train.jsonl
|- test.jsonl
|- train.txt
|- test.txt

The RPHarmony dataset is built using our proposed Random Poisson Blending method.
The implementation of Poisson Blending is based on pytorch-poisson-image-editing.

cd inference

For using pretrained checkpoints, see the Download Checkpoints section for instructions.

Test the model performance on the iHarmony4 dataset:

sh scripts/inference.sh

To test on other datasets, simply change the parameters in the inference.sh. You can customize the testing settings by editing the inference.sh script.

The weights of our model are derived from DiffHarmony.
We offer converted weights compatible with ControlNet-style training.

Please download the pretrained model weights and place them in the ./ckpt directory. Alternatively, you can manually convert the weights using the tool_add_control.py

Please modify the configuration files in the ./configs, such as datasets.yaml, and then:

sh train.sh

For detailed training options and configurations, see train_control.py.

The VAE is initialized with weights from Stable Diffusion. Please download the pretrained VAE weights from here and place them in the ./ckpt directory.

python train_VAE.py

For detailed training options and configurations, see train_VAE.py.

If you use the weights obtained from training, please run inference/convert.py to convert them from Stable Diffusion (SD) format to Diffusers format.
For parameter settings, please refer to the convert.py.

cd inference
python convert.py  --ckpt_path *** --save_path *** --vae_path *** --vae_save_path ***

The convert.py provides weight conversion for the core components (e.g., ControlNet, clear-VAE). However, you still need to download the remaining weights (e.g., safety_checker) from here.

Download our pretrained Clear-VAE checkpoint (Stable Diffusion format) from here, modify the ckpt_path in configs/vae.yaml, and then run:

python infer_VAE.py

This project is developped on the codebase of ControlNet, AnyDoor, DiffHarmony, FinetuneVAE-SD and pytorch-poisson-image-editing. We appreciate these great work!