|
| 1 | + |
| 2 | +# GLM4V 最佳实践 |
| 3 | + |
| 4 | +## 目录 |
| 5 | +- [环境准备](#环境准备) |
| 6 | +- [推理](#推理) |
| 7 | +- [微调](#微调) |
| 8 | +- [微调后推理](#微调后推理) |
| 9 | + |
| 10 | + |
| 11 | +## 环境准备 |
| 12 | +```shell |
| 13 | +git clone https://github.com/modelscope/swift.git |
| 14 | +cd swift |
| 15 | +pip install -e '.[llm]' |
| 16 | +``` |
| 17 | + |
| 18 | +模型链接: |
| 19 | +- glm4v-9b-chat: [https://modelscope.cn/models/ZhipuAI/glm-4v-9b/summary](https://modelscope.cn/models/ZhipuAI/glm-4v-9b/summary) |
| 20 | + |
| 21 | +## 推理 |
| 22 | + |
| 23 | +推理glm4v-9b-chat: |
| 24 | +```shell |
| 25 | +# Experimental environment: A100 |
| 26 | +# 30GB GPU memory |
| 27 | +CUDA_VISIBLE_DEVICES=0 swift infer --model_type glm4v-9b-chat |
| 28 | +``` |
| 29 | + |
| 30 | +输出: (支持传入本地路径或URL) |
| 31 | +```python |
| 32 | +""" |
| 33 | +<<< 描述这张图片 |
| 34 | +Input a media path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png |
| 35 | +这是一张特写照片，展示了一只毛茸茸的小猫。小猫的眼睛大而圆，呈深蓝色，眼珠呈金黄色，非常明亮。它的鼻子短而小巧，是粉色的。小猫的嘴巴紧闭，胡须细长。它的耳朵竖立着，耳朵内侧是白色的，外侧是棕色的。小猫的毛发看起来柔软而浓密，主要是白色和棕色相间的条纹图案。背景模糊不清，但似乎是一个室内环境。 |
| 36 | +-------------------------------------------------- |
| 37 | +<<< clear |
| 38 | +<<< 图中有几只羊 |
| 39 | +Input a media path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png |
| 40 | +图中共有四只羊。其中最左边的羊身体较小，后边三只羊体型逐渐变大，且最右边的两只羊体型大小一致。 |
| 41 | +-------------------------------------------------- |
| 42 | +<<< clear |
| 43 | +<<< 计算结果是多少? |
| 44 | +Input a media path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/math.png |
| 45 | +1452+45304=46756 |
| 46 | +-------------------------------------------------- |
| 47 | +<<< clear |
| 48 | +<<< 根据图片中的内容写首诗 |
| 49 | +Input a media path or URL <<< http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/poem.png |
| 50 | +湖光山色映小船， |
| 51 | +
|
| 52 | +星辉点点伴旅程。 |
| 53 | +
|
| 54 | +人在画中寻诗意， |
| 55 | +
|
| 56 | +心随景迁忘忧愁。 |
| 57 | +""" |
| 58 | +``` |
| 59 | + |
| 60 | +示例图片如下: |
| 61 | + |
| 62 | +cat: |
| 63 | + |
| 64 | +<img src="http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png" width="250" style="display: inline-block;"> |
| 65 | + |
| 66 | +animal: |
| 67 | + |
| 68 | +<img src="http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png" width="250" style="display: inline-block;"> |
| 69 | + |
| 70 | +math: |
| 71 | + |
| 72 | +<img src="http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/math.png" width="250" style="display: inline-block;"> |
| 73 | + |
| 74 | +poem: |
| 75 | + |
| 76 | +<img src="http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/poem.png" width="250" style="display: inline-block;"> |
| 77 | + |
| 78 | +**单样本推理** |
| 79 | + |
| 80 | +```python |
| 81 | +import os |
| 82 | +os.environ['CUDA_VISIBLE_DEVICES'] = '0' |
| 83 | + |
| 84 | +from swift.llm import ( |
| 85 | + get_model_tokenizer, get_template, inference, ModelType, |
| 86 | + get_default_template_type, inference_stream |
| 87 | +) |
| 88 | +from swift.utils import seed_everything |
| 89 | +import torch |
| 90 | + |
| 91 | +model_type = ModelType.glm4v_9b_chat |
| 92 | +template_type = get_default_template_type(model_type) |
| 93 | +print(f'template_type: {template_type}') |
| 94 | + |
| 95 | +model, tokenizer = get_model_tokenizer(model_type, torch.float16, |
| 96 | + model_kwargs={'device_map': 'auto'}) |
| 97 | +model.generation_config.max_new_tokens = 256 |
| 98 | +template = get_template(template_type, tokenizer) |
| 99 | +seed_everything(42) |
| 100 | + |
| 101 | +images = ['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/road.png'] |
| 102 | +query = '距离各城市多远？' |
| 103 | +response, history = inference(model, template, query, images=images) |
| 104 | +print(f'query: {query}') |
| 105 | +print(f'response: {response}') |
| 106 | + |
| 107 | +# 流式 |
| 108 | +query = '距离最远的城市是哪？' |
| 109 | +images = images |
| 110 | +gen = inference_stream(model, template, query, history, images=images) |
| 111 | +print_idx = 0 |
| 112 | +print(f'query: {query}\nresponse: ', end='') |
| 113 | +for response, _ in gen: |
| 114 | + delta = response[print_idx:] |
| 115 | + print(delta, end='', flush=True) |
| 116 | + print_idx = len(response) |
| 117 | +print() |
| 118 | + |
| 119 | +""" |
| 120 | +query: 距离各城市多远？ |
| 121 | +response: 距离马踏还有14Km，距离阳江还有62Km，距离广州还有293Km。 |
| 122 | +query: 距离最远的城市是哪？ |
| 123 | +response: 距离最远的城市是广州，有293Km。 |
| 124 | +""" |
| 125 | +``` |
| 126 | + |
| 127 | +示例图片如下: |
| 128 | + |
| 129 | +road: |
| 130 | + |
| 131 | +<img src="http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/road.png" width="250" style="display: inline-block;"> |
| 132 | + |
| 133 | + |
| 134 | +## 微调 |
| 135 | +多模态大模型微调通常使用**自定义数据集**进行微调. 这里展示可直接运行的demo: |
| 136 | + |
| 137 | +(默认对语言和视觉模型的qkv进行lora微调. 如果你想对所有linear都进行微调, 可以指定`--lora_target_modules ALL`) |
| 138 | +```shell |
| 139 | +# Experimental environment: A100 |
| 140 | +# 40GB GPU memory |
| 141 | +CUDA_VISIBLE_DEVICES=0 swift sft \ |
| 142 | + --model_type glm4v-9b-chat \ |
| 143 | + --dataset coco-en-2-mini \ |
| 144 | + |
| 145 | +# DDP |
| 146 | +NPROC_PER_NODE=2 \ |
| 147 | +CUDA_VISIBLE_DEVICES=0,1 swift sft \ |
| 148 | + --model_type glm4v-9b-chat \ |
| 149 | + --dataset coco-en-2-mini#10000 \ |
| 150 | + --ddp_find_unused_parameters true \ |
| 151 | +``` |
| 152 | + |
| 153 | +[自定义数据集](../LLM/自定义与拓展.md#-推荐命令行参数的形式)支持json, jsonl样式, 以下是自定义数据集的例子: |
| 154 | + |
| 155 | +(支持多轮对话, 但总的轮次对话只能包含一张图片, 支持传入本地路径或URL) |
| 156 | + |
| 157 | +```jsonl |
| 158 | +{"query": "55555", "response": "66666", "images": ["image_path"]} |
| 159 | +{"query": "eeeee", "response": "fffff", "history": [], "images": ["image_path"]} |
| 160 | +{"query": "EEEEE", "response": "FFFFF", "history": [["AAAAA", "BBBBB"], ["CCCCC", "DDDDD"]], "images": ["image_path"]} |
| 161 | +``` |
| 162 | + |
| 163 | + |
| 164 | +## 微调后推理 |
| 165 | +直接推理: |
| 166 | +```shell |
| 167 | +CUDA_VISIBLE_DEVICES=0 swift infer \ |
| 168 | + --ckpt_dir output/glm4v-9b-chat/vx-xxx/checkpoint-xxx \ |
| 169 | + --load_dataset_config true \ |
| 170 | +``` |
| 171 | + |
| 172 | +**merge-lora**并推理: |
| 173 | +```shell |
| 174 | +CUDA_VISIBLE_DEVICES=0 swift export \ |
| 175 | + --ckpt_dir output/glm4v-9b-chat/vx-xxx/checkpoint-xxx \ |
| 176 | + --merge_lora true |
| 177 | + |
| 178 | +CUDA_VISIBLE_DEVICES=0 swift infer \ |
| 179 | + --ckpt_dir output/glm4v-9b-chat/vx-xxx/checkpoint-xxx-merged \ |
| 180 | + --load_dataset_config true |
| 181 | +``` |
0 commit comments