Chinese-Vicuna: A Chinese Instruction-following LLaMA-based Model —— 一个中文低资源的llama+lora方案

| English | 中文 | NOTE&FAQ(Please take a look before using)

This is the repo for the Chinese-Vicuna project, which aims to build and share instruction-following Chinese LLaMA model tuning methods which can be trained on a single Nvidia RTX-2080TI, multi-round chatbot which can be trained on a single Nvidia RTX-3090 with the context len 2048.

Why is it called Vicuna: In view of the successful development of alpaca models such as llama,alpaca,guanaco，We want to train a Chinese small alpaca like Vicuna, small but strong enough !

The advantages of our solution are high parameter efficiency, graphics card friendliness, and easy deployment:

• Llama-7B instruction tuning is possible on a 2080Ti (11G) (7b-instruct)
• Llama-13B instruction tuning is possible on a 3090 (24G) (13b-instruct)
• Llama 7B can be fine-tuned on 3090 even for conversations of 2048 length; Use 50,000 pieces of data to get good results (chatv1)
• Llama 7B fine-tuning example on medical and legal domains
• Support qlora-4bit which can train Llama 13B on 2080Ti.
• Easily deployable on 2080Ti/3090, support multiple-gpu inference, which can reduce VRAM more.

The repo contains:

• code for finetune the model
• code for generation based on trained model
• code for run on CPU (fp16 or int4 is support, in purely C++)
• tools to download/convert/quantify original facebook llama.ckpt

This is our instruction demo (with beam-size=4, so you will see 4 process output in the meantime):

https://user-images.githubusercontent.com/72137647/228496412-60043912-f491-430b-848a-599e6edfa5ef.mp4

This is our multi-turn instruction demo (with beam-size=4, so you will see 4 process output in the meantime):

https://user-images.githubusercontent.com/72137647/229739363-1b48f3a9-02a1-46ab-81ee-8c62dc1399b2.mp4

NOTICE!

Before asking questions, take a look at this FAQ first! In the FAQ, you can find how to solve problems may be encountered when installing and using this project.

What‘s New

• June, 12, 2023: Release Chinese-Vicuna-4bit andChinese-Vicuna-4bit-11600 which can be continue-finetuned
• June, 1, 2023: support for 4bit training + inference, providing a multi-GPU inference interface (NOTICE THAT the environment is different from the original 8bit! Also provides test_tokenizers.py to further check EOS token)
• May 17, 2023: Llama 7B fine-tuning example on legal domains, The performance is in here
• May 10, 2023: Released chatv1 which have better conversational ability. The performance is in here
• May 10, 2023: Released instruct_chat_50k.jsonl which is composed of 30k Chinese sharegpt dataset and 20k alpaca-instruction-Chinese-dataset
• April 11, 2023: Released our continuous-finetune on the vertical corpus of Chinese medical quizzes Chinese-Vicuna-medical.Provides examples of vertical corpus training
• April 4, 2023: Add performance for 13B, which trains on a single 3090.
• April 1, 2023: Add better support for multi-turn chat in chat.py ( Now support 4 generation mode in stream mode/typewriter style: beam search, greedy, sample, beam sample ; We also add cancel button for regeneration )
• March 29, 2023: Add more detailed test samples. performance
• March 29, 2023: Added breakpoint retraining interface to support continued training of other datasets from our checkpoint
• March 29, 2023: Released our new 13B-based lora model
• March 28, 2023: Released our model on huggingface
• March 27, 2023: Released checkpoint-final for training 3 epochs on belle+guanaco
• March 27, 2023: Added multi-round interactive dialog script with alpaca-lora-serve service
• March 29, 2023: Added gradio typewriter-like output with beam search, better user interaction support.
• March 26, 2023: Provides a quantitative approach
• March 24, 2023: Released checkpoint-8000 for training about 1.5 epochs on belle+guanaco（100w data）
• March 23, 2023: Released checkpoint-4000 with 50w data training
• March 23, 2023: Deploy the code for fine-tuning and inferencing in colab
• March 23, 2023: Provides code that can be used for inference in pure c++

Table of Contents

Vicuna

• what's new
• what is the meaning
• try on colab
• performance
  • Checkpoint-4000(Facico/Chinese-Vicuna-lora-7b-0.75epoch-belle-and-guanaco)
  • Checkpoint-8000(Facico/Chinese-Vicuna-lora-7b-1.5epoch-belle-and-guanaco)
  • Checkpoint-final(Facico/Chinese-Vicuna-lora-7b-3epoch-belle-and-guanaco) and it is used for multiple rounds of dialogue
• What we need?
  • code、data、Large Language Model、LORA model、Device
• How to use
  • Installing、Multi-gpu training、Single-gpu training、Inference and use gradio to generate a web page(Streaming mode+beam search)、 multi-round interaction and use gradio to generate a web page(Streaming mode+beam search)、Streaming mode base on alpaca-lora-serve
• inference on CPU with pure C++
• More tools，for more details, see tool readme
  • ways for faster weight download ( 8MB/s )download_llama.sh
  • convert tools between the original facebook checkpoint and huggingface formatconvert_llama.py
  • a quantitative approach that requires less than 4G graphics memory for inference
• Possible problems encountered
• todo
• citation

Overview

• LLaMA paper: https://arxiv.org/abs/2302.13971v1
• Self-Instruct paper: https://arxiv.org/abs/2212.10560
• data generation: https://github.com/LianjiaTech/BELLE and https://guanaco-model.github.io/
• the first work: https://github.com/tatsu-lab/stanford_alpaca

We currently select the combination of BELLE and Guanaco data as our main training dataset. We will train on multi-turn instruction data.

What is the meaning?

Similar to the explosion of the stable diffusion model, platforms like civitai have emerged, consisting of a base model + various LORA models in an open source community.

The repo hopes to help you to train these LORA models.

what is LORA?: Simply, it's a plugin used to help adapt large models to your dataset, technical details can be found inLoRA: Low-Rank Adaptation of Large Language Models。Its advantage is that the finetune is very fast when it comes to getting a small model, about 30M, and the key is that it supports Plug and Play. As can be expected, this is a very suitable architecture for open source ecology.

Here, we will help you train through a very low-demand environment, with only one 2080 (11G) to achieve certain results.

Try on colab

colab link	Descriptions
	Use the specified lora model to inference and generate a webui
	Finetune with the data we collected
	Provide a webui for multi-round dialogue interaction

Performance

7B performance can be seen here, 13B is here (which train on a single 3090). There are also people reporting that our method works on 33B and gains good result.

history-checkpoint performance can be seen here