add blip2 #4

wjm202 · 2023-07-11T09:17:43Z

No description provided.

LokeZhou · 2023-07-12T03:16:21Z

paddlevlp/examples/blip2/run_predict.py

绝对路径要去掉

LokeZhou · 2023-07-12T03:16:42Z

paddlevlp/examples/blip2/run_pretrain_stage2.py

绝对路径去掉

已经删掉了

jerrywgz · 2023-07-12T05:49:32Z

paddlevlp/examples/blip2/merge_weight.py

+    for n, p in state_dict.items():
+        if n.startswith("vision_model") or n.startswith("qformer") or n == "query_tokens":
+            model_dict[n] = p
+    print("[1/3] load ViT, qformer and query_tokens from blip2-flan-t5-xxl done!")


可以用logger打印

jerrywgz · 2023-07-12T05:51:46Z

paddlevlp/examples/blip2/merge_weight.py

+    parser = argparse.ArgumentParser()
+
+    parser.add_argument("--blip2_path", default="/blip2/dirname", type=str, help="The dir name of blip2-flan-t5-xxl.")
+    parser.add_argument("--vicuna_path", default="/vicuna/dirname", type=str, help="The dir name of vicuna.")


这是minigpt4的merge权重脚本吧，可以让用户指定llm类型

jerrywgz · 2023-07-12T05:54:17Z

paddlevlp/examples/blip2/run_predict.py

 from paddlevlp.processors.blip_processing import Blip2Processor
 from paddlevlp.utils.log import logger

+from paddlevlp.examples.blip2.Logger import MetricLogger, SmoothedValue


logger是公共部分，统一在utils/log.py下

jerrywgz · 2023-07-12T05:56:23Z

paddlevlp/examples/blip2/run_pretrain_stage2.py

-    processor = Blip2Processor.from_pretrained(model_args.model_name_or_path)
-    blip_collator = BlipCollator(processor)
+
+    image_processor = BlipImageProcessor(image_mean=[0.48145466, 0.4578275, 0.40821073],


参数通过processor自动下载的配置文件获取，训练推理可以在processor内做区分

jerrywgz · 2023-07-12T05:59:06Z

paddlevlp/examples/blip2/run_pretrain_stage2.py

        optimizers=(optimizer, lr_sched),
+        processor=processor,
+        eval_processor=eval_processor,
+        tokenizer=AutoTokenizer.from_pretrained("facebook/opt-2.7b", use_fast=False)


tokenizer已经在collator中实现，是否还需要

jerrywgz · 2023-07-12T06:01:23Z

paddlevlp/models/blip2/modeling.py

        image_attention_mask = paddle.ones(image_embeds.shape[:-1], dtype="int64")

        query_tokens = self.query_tokens.expand([image_embeds.shape[0], -1, -1])
        # print('DEBUG!! Blip2ForCond query_tokens: ', query_tokens.shape, np.abs(query_tokens.numpy()).mean())


删除debug代码

jerrywgz · 2023-07-12T06:02:19Z

paddlevlp/trainer/blip2_trainer.py

+__all__ = ["BLIP2_Trainer"]
+
+
+class BLIP2_Trainer(Trainer):


命名不带_, BLIP2Trainer

jerrywgz · 2023-07-12T06:03:12Z

paddlevlp/trainer/blip2_trainer.py

+
+class BLIP2_Trainer(Trainer):
+    """
+    BLIP2_Trainer is a simple but feature-complete training and eval loop for PaddlePaddle, optimized for PaddleNLP.


注释说明下和Trainer的差异

remove if in PatchEmbed(

add blip2

## 算子目录 - [1. 转换算子](#1-转换算子) - [1.1 llava转换算子](#11-llava转换算子) - [1.1.1 llava_convert](#111-llava_convert) - [2. 过滤算子](#2-过滤算子) - [2.1 基础过滤算子](#21-基础过滤算子) - [2.1.1 valid_data_filter](#211-valid_data_filter) - [2.1.1.1 image_compliance_operator](#2111-image_compliance_operator) - [2.1.1.2 conversation_compliance_operator](#2112-conversation_compliance_operator) - [2.2 文本过滤算子](#22-文本过滤算子) - [2.2.1 conversation_length_filter](#221-conversation_length_filter) - [2.2.2 average_line_length_filter](#222-average_line_length_filter) - [2.2.3 maximum_line_length_filter](#223-maximum_line_length_filter) - [2.2.4 conversation_percentage_filter](#224-conversation_percentage_filter) - [2.2.5 token_num_filter](#225-token_num_filter) - [2.2.6 alphanumeric_ratio_filter](#226-alphanumeric_ratio_filter) - [2.2.7 stopwords_ratio_filter](#227-stopwords_ratio_filter) - [2.2.8 special_characters_filter](#228-special_characters_filter) - [2.2.9 language_id_filter](#229-language_id_filter) - [2.2.10 text_action_filter](#2210-text_action_filter) - [2.2.11 text_entity_dependency_filter](#2211-text_entity_dependency_filter) - [2.2.12 char_ngram_repetition_filter](#2212-char_ngram_repetition_filter) - [2.2.13 word_ngram_repetition_filter](#2213-word_ngram_repetition_filter) - [2.2.14 conversation_hash_filter](#2214-conversation_hash_filter) - [2.2.14.1 simhash_duplicate_operator](#22141-simhash_duplicate_operator) - [2.2.14.2 minhash_duplicate_operator](#22142-minhash_duplicate_operator) - [2.2.15 llm_judge_filter](#2215-llm_judge_filter) - [2.3 图像过滤算子](#23-图像过滤算子) - [2.3.1 image_filesize_filter](#231-image_filesize_filter) - [2.3.2 image_ration_filter](#232-image_ration_filter) - [2.3.3 image_resolution_filter](#233-image_resolution_filter) - [2.3.4 image_hash_filter](#234-image_hash_filter) - [2.4 图文过滤算子](#24-图文过滤算子) - [2.4.1 image_clip_filter](#241-image_clip_filter) - [3. 分析算子](#3-分析算子) - [3.1 基础分析算子](#31-基础分析算子) - [3.1.1 base_analysis_pipeline](#311-base_analysis_pipeline) - [3.1.1.1 analyze_dataset_statistics](#3111-analyze_dataset_statistics) - [3.1.1.2 analyze_language_distribution](#3112-analyze_language_distribution) - [3.1.1.3 analyze_image_paths](#3113-analyze_image_paths) - [3.1.1.4 analyze_data_anomalies](#3114-analyze_data_anomalies) - [3.1.1.5 analyze_conversation_tokens](#3115-analyze_conversation_tokens) - [3.2 进阶分析算子](#32-进阶分析算子) - [3.2.1 description_analysis](#321-description_analysis) - [3.2.2 quality_analysis](#322-quality_analysis) - [4. 可视化算子](#4-可视化算子) - [4.1 lda可视化算子](#41-lda可视化算子) - [4.1.1 lda_topic_clustering](#411-lda_topic_clustering) - [5. 生成算子](#5-生成算子) - [5.1 多模态生成算子](#51-多模态生成算子) - [5.1.1 generate_qna_for_images](#511-generate_qna_for_images) --- - #1055

add blip2

a90aa49

LokeZhou reviewed Jul 12, 2023

View reviewed changes

paddlevlp/examples/blip2/run_predict.py

Copy link

Collaborator

LokeZhou Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

绝对路径要去掉

LokeZhou reviewed Jul 12, 2023

View reviewed changes

add blip2

43b703d

jerrywgz reviewed Jul 12, 2023

View reviewed changes

wjm202 added 5 commits July 12, 2023 10:58

add description

1471ba3

recompute

802d260

add blip2 export dist

29729e3

add recompute

fca6f49

add mp

f5b9835

lyuwenyu previously approved these changes Jul 24, 2023

View reviewed changes

wjm202 dismissed lyuwenyu’s stale review via 9b79d2f July 25, 2023 02:10

wjm202 added 3 commits July 25, 2023 02:10

complete train

9b79d2f

del debug

23d4c4e

blip2

38ecdbd

wjm202 force-pushed the add_blip2 branch from 036c356 to 38ecdbd Compare July 25, 2023 10:02

lyuwenyu approved these changes Jul 25, 2023

View reviewed changes

lyuwenyu merged commit c770b16 into PaddlePaddle:develop Jul 25, 2023

zhoutianzi666 pushed a commit to zhoutianzi666/PaddleMIX that referenced this pull request Sep 19, 2023

Merge pull request PaddlePaddle#4 from zhoutianzi666/appflow1

3bf9f1e

remove if in PatchEmbed(

westfish pushed a commit to westfish/PaddleMIX that referenced this pull request Sep 25, 2024

Merge pull request PaddlePaddle#4 from wjm202/add_blip2

8c3fabc

add blip2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add blip2 #4

add blip2 #4

Uh oh!

wjm202 commented Jul 11, 2023

Uh oh!

LokeZhou Jul 12, 2023

Uh oh!

LokeZhou Jul 12, 2023

Uh oh!

wjm202 Jul 12, 2023

Uh oh!

jerrywgz Jul 12, 2023

Uh oh!

jerrywgz Jul 12, 2023

Uh oh!

jerrywgz Jul 12, 2023

Uh oh!

jerrywgz Jul 12, 2023

Uh oh!

jerrywgz Jul 12, 2023

Uh oh!

jerrywgz Jul 12, 2023

Uh oh!

jerrywgz Jul 12, 2023

Uh oh!

jerrywgz Jul 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		__all__ = ["BLIP2_Trainer"]


		class BLIP2_Trainer(Trainer):

add blip2 #4

add blip2 #4

Uh oh!

Conversation

wjm202 commented Jul 11, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants