中文羊驼大模型二期 v1.1

[ymcui]

本次以功能性更新为主，包括添加了FlashAttention-2支持脚本、vLLM支持、长回复模板等。

🚀 已添加FlashAttention-2训练支持（#38）

• 针对模型训练添加了FlashAttention-2补丁，可有效提高模型训练速度并减少显存占用
• 可通过参数flash_attn控制是否开启FlashAttention-2加速训练，具体可查看预训练脚本Wiki或指令精调脚本Wiki

🚀 vLLM加速支持（#35、#48)

• vLLM 是一个LLM推理和服务部署库，其具有高速运行、灵活易用等特点
• 在gradio_demo.py和inference_hf.py中添加了对vLLM部署服务的支持。此外，也提供了vLLM作为后端的仿OpenAI API风格的服务器DEMO（openai_api_server_vllm.py）
• 经测试，V100-16G GPU上使用7B模型推理，vLLM的平均推理速度相比transformers提升了470%，结合批量处理后vLLM的速度进一步达到了3800%的提升

🤖 生成长回复的系统提示语模板（#31）

• 默认系统提示语较为简单，模型生成的回复略短于一期Alpaca-Pro系列模型
• 提供了alpaca-2-long.txt系统提示语，可进一步提升回复长度（也可自行尝试修改）

其他更新

• 添加了自定义系统提示语的选项（add arguments for setting system prompts #32）
• 添加了Alpaca-2-7B模型的输出示例（Add Alpaca-2-7B output examples #33）
• 修正了部分脚本中的提示信息（Fix system prompt and improve messages #42）
• 添加了基于Colab的Gradio demo，可以自行架设demo体验模型了（Add Colab-based Gradio web demo #46）

For English release note, please refer to Discussion.

---

This discussion was created from the release 中文羊驼大模型二期 v1.1.

[ymcui]

Release Note for v1.1

This update is primarily focused on functional enhancements, including the addition of scripts supporting FlashAttention-2, vLLM support, and templates for long responses.

🚀 Added FlashAttention-2 training support (#38)

• Added FlashAttention-2 patch for model training, which can effectively increase the training speed and reduce the memory usage
• It is possible to control whether FlashAttention-2 acceleration training is enabled through the parameter flash_attn. For more details, you can refer to the Pretraining Scripts Wiki or the Fine-tuning Scripts Wiki

🚀 vLLM acceleration support (#35, #48)

• vLLM is an LLM inference and service deployment library, which is known for its high-speed operation and flexibility
• Added support for the vLLM deployment service in gradio_demo.py and inference_hf.py. In addition, a DEMO server in the style of the OpenAI API with vLLM as the backend is also provided (openai_api_server_vllm.py)
• After testing, on a V100-16G GPU, using a 7B model for inference, the average inference speed of vLLM is 470% faster than transformers. When combined with batch processing, the speed of vLLM can increase by 3800%

🤖 Template for system prompts for generating long responses (#31)

• The default system prompts are quite simple, and the responses generated by the model are slightly shorter than those of the first Alpaca-Pro series model
• Provided the alpaca-2-long.txt system prompt, which can further increase the length of the response (you can also try to modify it yourself)

Other updates

• Added the option of custom system prompts (add arguments for setting system prompts #32)
• Added output examples of the Alpaca-2-7B model (Add Alpaca-2-7B output examples #33)
• Corrected some information prompts in certain scripts (Fix system prompt and improve messages #42)
• Added a Gradio demo based on Colab, so you can set up your own demo to experience the model (Add Colab-based Gradio web demo #46)