support sd3.5 for controlnet example #9860
Conversation
|
@sayakpaul Hi, in this PR, I add sd3.5 to the controlnet example, with only a few lines added! |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
@DavyMorgan can you fix the test? |
Yeah, I have updated the test. |
|
@DavyMorgan |
Sure, I can fix that. |
|
@yiyixuxu @sayakpaul Could you please run the test workflows again? |
|
@sayakpaul I updated the codes. This time it should be ok. :) |
There was a problem hiding this comment.
Thanks for the changes!
Could you provide some example results as well?
Additionally, I hosted the tiny pipe to our testing org:
https://huggingface.co/hf-internal-testing/tiny-sd35-pipe
Could you please change the path accordingly in test_controlnet.py?
I will let @yiyixuxu review the other changes introduced.
| @@ -986,7 +989,7 @@ def main(args): | |||
| controlnet = SD3ControlNetModel.from_pretrained(args.controlnet_model_name_or_path) | |||
| else: | |||
| logger.info("Initializing controlnet weights from transformer") | |||
| controlnet = SD3ControlNetModel.from_transformer(transformer) | |||
| controlnet = SD3ControlNetModel.from_transformer(transformer, num_extra_conditioning_channels=0) | |||
There was a problem hiding this comment.
Should this be hard-coded like this? Can we use a CLI arg?
There was a problem hiding this comment.
Thanks, I have added a CLI arg.
|
@sayakpaul I have updated the codes to use hf-internal-testing/tiny-sd35-pipe. Meanwhile, the example results can be found at the end of examples/controlnet/README_sd3.md. |
| @@ -344,7 +343,7 @@ def custom_forward(*inputs): | |||
|
|
|||
| # controlnet residual | |||
| if block_controlnet_hidden_states is not None and block.context_pre_only is False: | |||
| interval_control = len(self.transformer_blocks) // len(block_controlnet_hidden_states) | |||
| interval_control = int(math.ceil(len(self.transformer_blocks) / len(block_controlnet_hidden_states))) | |||
There was a problem hiding this comment.
sorry I missed this change - any reason we make this change here?
|
related ( a little bit overlap with) #9758 |
|
@yiyixuxu @sayakpaul Can you run the tests again to check whether it works well with the current HEAD? |
|
@DavyMorgan sorry for the late reply. Could you consider the changes introduced in #9758 and repurpose this PR accordingly? |
| @@ -56,6 +56,10 @@ def __init__( | |||
| pooled_projection_dim: int = 2048, | |||
| out_channels: int = 16, | |||
| pos_embed_max_size: int = 96, | |||
| dual_attention_layers: Tuple[ | |||
There was a problem hiding this comment.
we already have this change
can you rebase?
|
@yiyixuxu @sayakpaul Hi, I have fixed the codes to pass the quality test. Please run the tests again! |
|
Sorry for missing the typo😂Now fixed. @yiyixuxu @sayakpaul |
* support sd3.5 in controlnet --------- Co-authored-by: YiYi Xu <[email protected]>
What does this PR do?
Fixes # (issue)
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.