Skip to content

【Hackathon 5th No.83】PaddleMIX ppdiffusers models模块功能升级同步HF -part #322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 23 commits into from
Dec 14, 2023
Merged

【Hackathon 5th No.83】PaddleMIX ppdiffusers models模块功能升级同步HF -part #322

merged 23 commits into from
Dec 14, 2023

Conversation

@co63oc
Copy link
Contributor

@co63oc co63oc commented Nov 24, 2023 •
edited
Loading

PaddleMIX ppdiffusers models模块功能升级同步HF

https://github.com/PaddlePaddle/community/blob/master/hackathon/hackathon_5th/%E3%80%90PaddlePaddle%20Hackathon%205th%E3%80%91%E5%BC%80%E6%BA%90%E8%B4%A1%E7%8C%AE%E4%B8%AA%E4%BA%BA%E6%8C%91%E6%88%98%E8%B5%9B%E5%A5%97%E4%BB%B6%E5%BC%80%E5%8F%91%E4%BB%BB%E5%8A%A1%E5%90%88%E9%9B%86.md#no83paddlemix-ppdiffusers-models%E6%A8%A1%E5%9D%97%E5%8A%9F%E8%83%BD%E5%8D%87%E7%BA%A7%E5%90%8C%E6%AD%A5hf

#259
#260
#262
#264
#265
#266
#263
#261

升级 models
增加scale
增加get_processor
diffusers上LoRA类逐渐取消，改为使用lora_layer
部分测试误差增加，测试修改前代码同样有误差
增加tests/lora目录
flax模型paddle没有使用，没有迁移

test_attention_processor.py
image

test_models_unet_2d.py
image

test_lora_layers.py
image

test_models_unet_motion.py
image

@paddle-bot
Copy link

paddle-bot bot commented Nov 24, 2023

Thanks for your contribution!

@JunnYu
Copy link
Member

JunnYu commented Nov 24, 2023

你好，总的来看当前代码没有什么问题。但是当前无法直接合入这里的PR，因为还需要等待我们将当前develop的CI、CE给修复通过后，你这里没有太大的影响后我们才会合入

@co63oc
Copy link
Contributor Author

co63oc commented Nov 24, 2023

好的

This was referenced Nov 26, 2023
Closed
Closed
@guoshengCS
Copy link
Collaborator

这个可否再进一步升级到diffusers最新的0.23版本呢，0.23下会包含当前社区热门的LCM-LoRA等模型，会挺重要的

@co63oc
Copy link
Contributor Author

co63oc commented Nov 29, 2023

这个可否再进一步升级到diffusers最新的0.23版本呢，0.23下会包含当前社区热门的LCM-LoRA等模型，会挺重要的

已升级到0.23

test_lora_layers.py
image

test_models_unet_2d_condition.py
image

test_models_unet_3d_condition.py
image

@shiyutang shiyutang mentioned this pull request Nov 29, 2023
Closed
@co63oc
Copy link
Contributor Author

co63oc commented Nov 30, 2023

image
测试网络连接错误

@co63oc
Copy link
Contributor Author

co63oc commented Dec 2, 2023 •
edited
Loading

CI中 test_stable_diffusion_adapter.py, test_stable_diffusion_pix2pix_zero.py 报错，除了网络问题，
已修改在V100 32G上测试test_stable_diffusion_adapter.py, test_stable_diffusion_pix2pix_zero.py通过
image
image

unet_2d_condition.py 中比pytorch多一行不大理解，按同样已增加
image

@co63oc
Copy link
Contributor Author

co63oc commented Dec 4, 2023

@guoshengCS 麻烦看下还有什么修改，使用网络可以访问的环境测试是可以

JunnYu
JunnYu reviewed Dec 5, 2023
View reviewed changes
JunnYu
JunnYu reviewed Dec 5, 2023
View reviewed changes
JunnYu
JunnYu reviewed Dec 5, 2023
View reviewed changes
@co63oc
Copy link
Contributor Author

co63oc commented Dec 12, 2023

现在看应该没啥太大问题了，等本周这里发个0.19.4的小版本后，你把现有的冲突解决后就可以合入了

好的

@JunnYu
Copy link
Member

JunnYu commented Dec 13, 2023

@co63oc 当前已经发布了0.19.4了，你这里重新将develop的内容合入到你当前分支，然后等所有ci通过后就可以合入了

@co63oc
Copy link
Contributor Author

co63oc commented Dec 14, 2023

@co63oc 当前已经发布了0.19.4了，你这里重新将develop的内容合入到你当前分支，然后等所有ci通过后就可以合入了

已合并develop分支CI完成

JunnYu
JunnYu approved these changes Dec 14, 2023
View reviewed changes
Copy link
Member

@JunnYu JunnYu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

JunnYu
JunnYu reviewed Dec 14, 2023
View reviewed changes
ppdiffusers/ppdiffusers/utils/paddle_utils.py

# Non-power of 2 images must be float32
if (W & (W - 1)) != 0 or (H & (H - 1)) != 0:
x = x.to(dtype=paddle.float32)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

paddle的tensor没有to方法，要用cast或者astype

JunnYu
JunnYu reviewed Dec 14, 2023
View reviewed changes
ppdiffusers/ppdiffusers/utils/paddle_utils.py
x_freq = ifftshift(x_freq, axes=(-2, -1))
x_filtered = ifftn(x_freq, axes=(-2, -1)).real

return x_filtered.to(dtype=x_in.dtype)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

paddle的tensor没有to方法，要用cast或者astype

@JunnYu JunnYu merged commit c4eb91e into PaddlePaddle:develop Dec 14, 2023
This was referenced Dec 15, 2023
Closed
Closed
Closed
@shiyutang shiyutang added ContributorPRisMerged add this when a contributor's PR is merged. HappyOpenSource 快乐开源活动issue与PR labels Dec 18, 2023
@JunnYu
Copy link
Member

JunnYu commented Dec 19, 2023 •
edited
Loading

@co63oc
你好，当前发现这个PR对我们的监控存在了如下影响，需要进行回滚操作。你这里需要解决一下【问题1】，并且找出【问题2】中的sd recompute粒度的变化，例如：之前哪些模块没有加recompute，现在加了recompute，从而导致存在的diff。

  1. fastdeploy的sd和sdxl 推理造成了错误。无法顺利加载vae的decoder模型，可能当前升级有部分修改，导致这个发生错误。
    image
    详细的话，需要你这里使用这里的脚本，参考这里的文档 顺利跑出结果。
  2. sd的预训练和lora微调存在性能下降10%，可能怀疑是recompute粒度发生了修改，部分模块之前没有加recompute现在加上了recompute导致的。

@shiyutang tangshiyu你这里也知悉一下，当前需要回滚一下

JunnYu added a commit that referenced this pull request Dec 19, 2023
@JunnYu
Revert "【Hackathon 5th No.83】PaddleMIX ppdiffusers models模块功能升级同步HF (#…
65694da 
…322)"

This reverts commit c4eb91e.
@JunnYu JunnYu mentioned this pull request Dec 19, 2023
Merged
JunnYu added a commit that referenced this pull request Dec 19, 2023
@JunnYu
Revert "【Hackathon 5th No.83】PaddleMIX ppdiffusers models模块功能升级同步HF" (#…
449bb34 
…354)

Reverts #322
回滚该PR，等之后修复后再合入，下面是对应的问题。
1. fastdeploy的sd和sdxl 推理造成了错误。无法顺利加载vae的decoder模型，可能当前升级有部分修改，导致这个发生错误。
2.
sd的预训练和lora微调存在性能下降10%，可能怀疑是recompute粒度发生了修改，部分模块之前没有加recompute现在加上了recompute导致的。
@Ligoml Ligoml changed the title 【Hackathon 5th No.83】PaddleMIX ppdiffusers models模块功能升级同步HF 【Hackathon 5th No.83】PaddleMIX ppdiffusers models模块功能升级同步HF -part Dec 20, 2023
@co63oc co63oc mentioned this pull request Dec 20, 2023
Closed
@guoshengCS
Copy link
Collaborator

guoshengCS commented Jan 3, 2024 •
edited
Loading

@co63oc 基于之前的工作我们这边在做ppdiffuers 0.24版本的升级，有拉一个分支出来 https://github.com/PaddlePaddle/PaddleMIX/tree/upgrade_ppdiffusers0240 ，也会在快乐开源放一些新题目出来，请问您那边有兴趣一起不

@co63oc
Copy link
Contributor Author

co63oc commented Jan 3, 2024

@co63oc 基于之前的工作我们这边在做ppdiffuers 0.24版本的升级，有拉一个分支出来 https://github.com/PaddlePaddle/PaddleMIX/tree/upgrade_ppdiffusers0240 ，也会在快乐开源放一些新题目出来，请问您那边有兴趣一起不

好的参加任务

@guoshengCS
Copy link
Collaborator

@co63oc 基于之前的工作我们这边在做ppdiffuers 0.24版本的升级，有拉一个分支出来 https://github.com/PaddlePaddle/PaddleMIX/tree/upgrade_ppdiffusers0240 ，也会在快乐开源放一些新题目出来，请问您那边有兴趣一起不

好的参加任务

赞，我们后面会同步出来一些任务，感谢支持和贡献，一起共建

@JunnYu
Copy link
Member

JunnYu commented Jan 4, 2024

@co63oc 关于fd推理的问题不需要再看了，你这里需要对比一下旧版0.19.4的代码跟你当前升级后的代码里面recompute粒度在哪里发生了变化

@co63oc
Copy link
Contributor Author

co63oc commented Jan 4, 2024 •
edited
Loading

@co63oc 关于fd推理的问题不需要再看了，你这里需要对比一下旧版0.19.4的代码跟你当前升级后的代码里面recompute粒度在哪里发生了变化

@JunnYu
#374 这是去掉recompute版本，看有没有问题

@co63oc
Copy link
Contributor Author

co63oc commented Jan 6, 2024

@JunnYu 现在问题是
1 在哪个分支上修改，develop 还是 upgrade_ppdiffusers0240，但是upgrade_ppdiffusers0240的分支测试用例不能运行，看到有增加use_old_recompute，这个可以迁移到develop分支，是不是迁移到develop分支
2 recompute粒度是运行哪个脚本比较

@guoshengCS
Copy link
Collaborator

@co63oc 基于之前的工作我们这边在做ppdiffuers 0.24版本的升级，有拉一个分支出来 https://github.com/PaddlePaddle/PaddleMIX/tree/upgrade_ppdiffusers0240 ，也会在快乐开源放一些新题目出来，请问您那边有兴趣一起不

好的参加任务

#377 这里列了一些新的任务 @co63oc 可以看下哈

@co63oc
Copy link
Contributor Author

co63oc commented Jan 10, 2024

#377 这里列了一些新的任务 @co63oc 可以看下哈

好的

westfish pushed a commit to westfish/PaddleMIX that referenced this pull request Sep 25, 2024
@co63oc
【Hackathon 5th No.83】PaddleMIX ppdiffusers models模块功能升级同步HF (PaddlePa…
85ac514 
…ddle#322)

PaddleMIX ppdiffusers models模块功能升级同步HF


https://github.com/PaddlePaddle/community/blob/master/hackathon/hackathon_5th/%E3%80%90PaddlePaddle%20Hackathon%205th%E3%80%91%E5%BC%80%E6%BA%90%E8%B4%A1%E7%8C%AE%E4%B8%AA%E4%BA%BA%E6%8C%91%E6%88%98%E8%B5%9B%E5%A5%97%E4%BB%B6%E5%BC%80%E5%8F%91%E4%BB%BB%E5%8A%A1%E5%90%88%E9%9B%86.md#no83paddlemix-ppdiffusers-models%E6%A8%A1%E5%9D%97%E5%8A%9F%E8%83%BD%E5%8D%87%E7%BA%A7%E5%90%8C%E6%AD%A5hf

PaddlePaddle#259
PaddlePaddle#260
PaddlePaddle#262
PaddlePaddle#264
PaddlePaddle#265

升级 models
增加scale
增加get_processor
diffusers上LoRA类逐渐取消，改为使用lora_layer
部分测试误差增加，测试修改前代码同样有误差
增加tests/lora目录
flax模型paddle没有使用，没有迁移

test_attention_processor.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/3a7cdfcf-6313-42f7-8bb3-9fd7d46b36cf)

test_models_unet_2d.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/51f9ec37-e0f2-4010-b621-966d596bf451)

test_lora_layers.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/d4761fde-1eda-4329-8f94-79d4ed6535b3)

test_models_unet_motion.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/f3ff8f8c-89f2-46e2-a733-4b6c442e36b5)
westfish pushed a commit to westfish/PaddleMIX that referenced this pull request Sep 25, 2024
@JunnYu
Revert "【Hackathon 5th No.83】PaddleMIX ppdiffusers models模块功能升级同步HF" (P…
8339d14 
…addlePaddle#354)

Reverts PaddlePaddle#322
回滚该PR，等之后修复后再合入，下面是对应的问题。
1. fastdeploy的sd和sdxl 推理造成了错误。无法顺利加载vae的decoder模型，可能当前升级有部分修改，导致这个发生错误。
2.
sd的预训练和lora微调存在性能下降10%，可能怀疑是recompute粒度发生了修改，部分模块之前没有加recompute现在加上了recompute导致的。
lyuwenyu added a commit that referenced this pull request Feb 20, 2025
@lyuwenyu
[wip] datacopilot analysis, filter and visualize (#1060)
0d362e3 
## 算子目录

- [1. 转换算子](#1-转换算子)
  - [1.1 llava转换算子](#11-llava转换算子)
    - [1.1.1 llava_convert](#111-llava_convert)
- [2. 过滤算子](#2-过滤算子)
  - [2.1 基础过滤算子](#21-基础过滤算子)
    - [2.1.1 valid_data_filter](#211-valid_data_filter)
- [2.1.1.1 image_compliance_operator](#2111-image_compliance_operator)
- [2.1.1.2
conversation_compliance_operator](#2112-conversation_compliance_operator)
  - [2.2 文本过滤算子](#22-文本过滤算子)
- [2.2.1 conversation_length_filter](#221-conversation_length_filter)
- [2.2.2 average_line_length_filter](#222-average_line_length_filter)
- [2.2.3 maximum_line_length_filter](#223-maximum_line_length_filter)
- [2.2.4
conversation_percentage_filter](#224-conversation_percentage_filter)
    - [2.2.5 token_num_filter](#225-token_num_filter)
    - [2.2.6 alphanumeric_ratio_filter](#226-alphanumeric_ratio_filter)
    - [2.2.7 stopwords_ratio_filter](#227-stopwords_ratio_filter)
    - [2.2.8 special_characters_filter](#228-special_characters_filter)
    - [2.2.9 language_id_filter](#229-language_id_filter)
    - [2.2.10 text_action_filter](#2210-text_action_filter)
- [2.2.11
text_entity_dependency_filter](#2211-text_entity_dependency_filter)
- [2.2.12
char_ngram_repetition_filter](#2212-char_ngram_repetition_filter)
- [2.2.13
word_ngram_repetition_filter](#2213-word_ngram_repetition_filter)
    - [2.2.14 conversation_hash_filter](#2214-conversation_hash_filter)
- [2.2.14.1
simhash_duplicate_operator](#22141-simhash_duplicate_operator)
- [2.2.14.2
minhash_duplicate_operator](#22142-minhash_duplicate_operator)
    - [2.2.15 llm_judge_filter](#2215-llm_judge_filter)
  - [2.3 图像过滤算子](#23-图像过滤算子)
    - [2.3.1 image_filesize_filter](#231-image_filesize_filter)
    - [2.3.2 image_ration_filter](#232-image_ration_filter)
    - [2.3.3 image_resolution_filter](#233-image_resolution_filter)
    - [2.3.4 image_hash_filter](#234-image_hash_filter)
  - [2.4 图文过滤算子](#24-图文过滤算子)
    - [2.4.1 image_clip_filter](#241-image_clip_filter)
- [3. 分析算子](#3-分析算子)
  - [3.1 基础分析算子](#31-基础分析算子)
    - [3.1.1 base_analysis_pipeline](#311-base_analysis_pipeline)
- [3.1.1.1 analyze_dataset_statistics](#3111-analyze_dataset_statistics)
- [3.1.1.2
analyze_language_distribution](#3112-analyze_language_distribution)
      - [3.1.1.3 analyze_image_paths](#3113-analyze_image_paths)
      - [3.1.1.4 analyze_data_anomalies](#3114-analyze_data_anomalies)
- [3.1.1.5
analyze_conversation_tokens](#3115-analyze_conversation_tokens)
  - [3.2 进阶分析算子](#32-进阶分析算子)
    - [3.2.1 description_analysis](#321-description_analysis)
    - [3.2.2 quality_analysis](#322-quality_analysis)
- [4. 可视化算子](#4-可视化算子)
  - [4.1 lda可视化算子](#41-lda可视化算子)
    - [4.1.1 lda_topic_clustering](#411-lda_topic_clustering)
- [5. 生成算子](#5-生成算子)
  - [5.1 多模态生成算子](#51-多模态生成算子)
    - [5.1.1 generate_qna_for_images](#511-generate_qna_for_images)



--- 
- #1055
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JunnYu JunnYu JunnYu approved these changes

+1 more reviewer

@CrazyBoyM CrazyBoyM CrazyBoyM approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

contributor ContributorPRisMerged add this when a contributor's PR is merged. HappyOpenSource 快乐开源活动issue与PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@co63oc @JunnYu @guoshengCS @shiyutang @CrazyBoyM