|
| 1 | + |
| 2 | +# CogVLM2 Video 最佳实践 |
| 3 | + |
| 4 | +## 目录 |
| 5 | +- [环境准备](#环境准备) |
| 6 | +- [推理](#推理) |
| 7 | +- [微调](#微调) |
| 8 | +- [微调后推理](#微调后推理) |
| 9 | + |
| 10 | + |
| 11 | +## 环境准备 |
| 12 | +```shell |
| 13 | +git clone https://github.com/modelscope/swift.git |
| 14 | +cd swift |
| 15 | +pip install -e '.[llm]' |
| 16 | + |
| 17 | +# https://github.com/facebookresearch/pytorchvideo/issues/258 |
| 18 | +# https://github.com/dmlc/decord/issues/177 |
| 19 | +pip install decord pytorchvideo |
| 20 | +``` |
| 21 | + |
| 22 | +模型链接: |
| 23 | +- cogvlm2-video-13b-chat: [https://modelscope.cn/models/ZhipuAI/cogvlm2-video-llama3-chat](https://modelscope.cn/models/ZhipuAI/cogvlm2-video-llama3-chat) |
| 24 | + |
| 25 | + |
| 26 | +## 推理 |
| 27 | + |
| 28 | +推理cogvlm2-video-13b-chat: |
| 29 | +```shell |
| 30 | +# Experimental environment: A100 |
| 31 | +# 28GB GPU memory |
| 32 | +CUDA_VISIBLE_DEVICES=0 swift infer --model_type cogvlm2-video-13b-chat |
| 33 | +``` |
| 34 | + |
| 35 | +输出: (支持传入本地路径或URL) |
| 36 | +```python |
| 37 | +""" |
| 38 | +<<< 描述这段视频 |
| 39 | +Input a video path or URL <<< https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4 |
| 40 | +In the video, a young child is seen sitting on a bed and reading a book. The child is wearing glasses and is dressed in a light blue top and pink pants. The room appears to be a bedroom with a crib in the background. The child is engrossed in the book, and the scene is captured in a series of frames showing the child's interaction with the book. |
| 41 | +-------------------------------------------------- |
| 42 | +<<< clear |
| 43 | +<<< Describe this video. |
| 44 | +Input a video path or URL <<< https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/fire.mp4 |
| 45 | +In the video, a person is seen lighting a fire in a backyard setting. They start by holding a piece of food and then proceed to light a match to the food. The fire is then ignited, and the person continues to light more pieces of food, including a bag of chips and a piece of wood. The fire is seen burning brightly, and the person is seen standing over the fire, possibly enjoying the warmth. The video captures the process of starting a fire and the person's interaction with the flames, creating a cozy and inviting atmosphere. |
| 46 | +-------------------------------------------------- |
| 47 | +<<< clear |
| 48 | +<<< who are you |
| 49 | +Input a video path or URL <<< |
| 50 | +I am a person named John. |
| 51 | +""" |
| 52 | +``` |
| 53 | + |
| 54 | +**单样本推理** |
| 55 | + |
| 56 | +```python |
| 57 | +import os |
| 58 | +os.environ['CUDA_VISIBLE_DEVICES'] = '0' |
| 59 | + |
| 60 | +from swift.llm import ( |
| 61 | + get_model_tokenizer, get_template, inference, ModelType, |
| 62 | + get_default_template_type, inference_stream |
| 63 | +) |
| 64 | +from swift.utils import seed_everything |
| 65 | +import torch |
| 66 | + |
| 67 | +model_type = ModelType.cogvlm2_video_13b_chat |
| 68 | +template_type = get_default_template_type(model_type) |
| 69 | +print(f'template_type: {template_type}') |
| 70 | + |
| 71 | +model, tokenizer = get_model_tokenizer(model_type, torch.float16, |
| 72 | + model_kwargs={'device_map': 'auto'}) |
| 73 | +model.generation_config.max_new_tokens = 256 |
| 74 | +template = get_template(template_type, tokenizer) |
| 75 | +seed_everything(42) |
| 76 | + |
| 77 | +videos = ['https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/baby.mp4'] |
| 78 | +query = '描述这段视频' |
| 79 | +response, history = inference(model, template, query, videos=videos) |
| 80 | +print(f'query: {query}') |
| 81 | +print(f'response: {response}') |
| 82 | + |
| 83 | +# 流式 |
| 84 | +query = 'Describe this video.' |
| 85 | +videos = ['https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/fire.mp4'] |
| 86 | +gen = inference_stream(model, template, query, history, videos=videos) |
| 87 | +print_idx = 0 |
| 88 | +print(f'query: {query}\nresponse: ', end='') |
| 89 | +for response, _ in gen: |
| 90 | + delta = response[print_idx:] |
| 91 | + print(delta, end='', flush=True) |
| 92 | + print_idx = len(response) |
| 93 | +print() |
| 94 | + |
| 95 | +""" |
| 96 | +query: 描述这段视频 |
| 97 | +response: The video depicts a young child sitting on a bed and reading a book. The child is wearing glasses and is seen in various positions, such as sitting on the bed, sitting on a couch, and sitting on a bed with a blanket. The child's attire changes from a light blue top and pink pants to a light blue top and pink leggings. The room has a cozy and warm atmosphere with soft lighting, and there are personal items scattered around, such as a crib, a television, and a white garment. |
| 98 | +query: Describe this video. |
| 99 | +response: The video shows a person lighting a fire in a backyard setting. The person is seen holding a piece of food and a lighter, and then lighting the food on fire. The fire is then used to light other pieces of wood, and the person is seen standing over the fire, holding a bag of food. The video captures the process of starting a fire and the person's interaction with the fire. |
| 100 | +""" |
| 101 | +``` |
| 102 | + |
| 103 | + |
| 104 | +## 微调 |
| 105 | +多模态大模型微调通常使用**自定义数据集**进行微调. 这里展示可直接运行的demo: |
| 106 | + |
| 107 | +(默认对LLM的qkv进行lora微调. 如果你想对所有linear都进行微调, 可以指定`--lora_target_modules ALL`) |
| 108 | +```shell |
| 109 | +# Experimental environment: A100 |
| 110 | +# 40GB GPU memory |
| 111 | +CUDA_VISIBLE_DEVICES=0 swift sft \ |
| 112 | + --model_type cogvlm2-video-13b-chat \ |
| 113 | + --dataset video-chatgpt |
| 114 | +``` |
| 115 | + |
| 116 | +[自定义数据集](../LLM/自定义与拓展.md#-推荐命令行参数的形式)支持json, jsonl样式, 以下是自定义数据集的例子: |
| 117 | + |
| 118 | +(支持多轮对话, 但总的轮次对话只能包含一张图片, 支持传入本地路径或URL) |
| 119 | + |
| 120 | +```jsonl |
| 121 | +{"query": "55555", "response": "66666", "videos": ["video_path"]} |
| 122 | +{"query": "eeeee", "response": "fffff", "history": [], "videos": ["video_path"]} |
| 123 | +{"query": "EEEEE", "response": "FFFFF", "history": [["AAAAA", "BBBBB"], ["CCCCC", "DDDDD"]], "videos": ["video_path"]} |
| 124 | +``` |
| 125 | + |
| 126 | + |
| 127 | +## 微调后推理 |
| 128 | +直接推理: |
| 129 | +```shell |
| 130 | +CUDA_VISIBLE_DEVICES=0 swift infer \ |
| 131 | + --ckpt_dir output/cogvlm2-video-13b-chat/vx-xxx/checkpoint-xxx \ |
| 132 | + --load_dataset_config true \ |
| 133 | +``` |
| 134 | + |
| 135 | +**merge-lora**并推理: |
| 136 | +```shell |
| 137 | +CUDA_VISIBLE_DEVICES=0 swift export \ |
| 138 | + --ckpt_dir output/cogvlm2-video-13b-chat/vx-xxx/checkpoint-xxx \ |
| 139 | + --merge_lora true |
| 140 | + |
| 141 | +CUDA_VISIBLE_DEVICES=0 swift infer \ |
| 142 | + --ckpt_dir output/cogvlm2-video-13b-chat/vx-xxx/checkpoint-xxx-merged \ |
| 143 | + --load_dataset_config true |
| 144 | +``` |
0 commit comments