[Readme ENG][Readme ZH]

CuteGPT is an open-source conversational language model that supports both Chinese and English, developed by Fudan University KnowledgeWorks Laboratory. It is based on the original Llama model structure, and has a scale of 13B (13 billion) parameters. It can perform int8 precision inference on a single 3090 graphics card. CuteGPT expands the Chinese vocabulary and performs pre-training, improving its ability to understand Chinese. Subsequently, it is fine-tuned with conversational instructions to enhance the model's ability to understand instructions.

Based on the KW-CuteGPT-7b version, KW-CuteGPT-13b has improved accuracy in knowledge, understanding of complex instructions, ability to comprehend long texts, reasoning ability, faithful question answering, and other capabilities. Currently, the KW-CuteGPT-13b version model outperforms the majority of models of similar scale in certain evaluation tasks.

We have open-sourced the following versions of the parameters:

We selected a subset of typical Chinese-English datasets for evaluation, and the following are the evaluation results of the CuteGPT model on C-eval and XieZhi.

Xiezhi (獬豸) by the team of Yanghua Xiao at Fudan University is a domain evaluation benchmark for Language Models (LMs). It consists of 249587 multi-choice questions spanning 516 different disciplines and three different levels of granularity. The discipline classification and all first-level disciplines are shown in the figure below, with the number of second-level disciplines within each first-level discipline indicated in parentheses.

All tested LLMs need to select the most suitable answer from 50 options. Each question has three confusing options in addition to the correct answer, as well as 46 completely unrelated options randomly selected from all the options of all questions. It is worth noting that we have pre-filtered out questions that require selecting the incorrect answer.

The probability of each model generating a specific answer is used to calculate the probability of each model selecting each option. Then, the options are ranked accordingly based on their probabilities, and the Mean Reciprocal Rank (MRR) is used as the evaluation metric. MRR indicates at which position, on average, the correct answer is ranked. A higher MRR value close to 1 indicates that the model is more capable of ranking the correct answer at the top, while a lower MRR value close to 0 suggests that the model tends to rank the correct answer at the bottom.

The following usage examples are taken from our work CELLO, "Can Large Language Models Understand Real-World Complex Instructions?". The aim of this work is to investigate the ability of large language models to comprehend complex instructions. Stay tuned for more information.

conda create -n cutegpt python=3.7
conda activate cutegpt
pip install -r requirements.txt 

from transformers import LlamaForCausalLM, LlamaTokenizer
from peft import PeftModel
import torch

• The prompt template for inference

overall_instruction = "你是复旦大学知识工场实验室训练出来的语言模型CuteGPT。给定任务描述，请给出对应请求的回答。\n"
def generate_prompt(query, history, input=None):
    prompt = overall_instruction
    for i, (old_query, response) in enumerate(history):
        prompt  = "问：{}\n答：\n{}\n".format(old_query, response)
    prompt  = "问：{}\n答：\n".format(query)
    return prompt

• Inference Code

  You can run the following script directly for inference:

  • Full-finetuning version

    CUDA_VISIBLE_DEVICES=0,1 python inference_ft.py

  • LoRA version

    CUDA_VISIBLE_DEVICES=0,1 python inference_lora.py

The data is stored in .pkl format, and you can use the following code to view the data:

import pickle
import random
file = 'data/test/ift_data.pkl'
with open(file, "rb") as f:
    datas = pickle.load(f)
print(len(datas))
print(random.sample(datas,3))

The data is stored in a two-dimensional array, where len(datas) represents the number of samples, and len(data[i]) represents the number of dialogue turns for the i-th sample. Each dialogue turn consists of three key-value pairs: instruction, input, and output. Here is an example:

[[{'instruction': '请帮我生成一首诗。', 'input': '', 'output': '好的，请给我一些关键词来启发诗歌的创作。'},
  {'instruction': '月亮、海浪、夜晚。',
   'input': '',
   'output': '心中的孤独由天空掩盖，\n夜幕笼罩下，海浪轻扬，\n月光照亮着沙滩，\n让我不再孤单。'}],
 [{'instruction': '根据给定的数据集，训练一个能够对垃圾邮件进行分类的模型。\n提供垃圾邮件和非垃圾邮件两类的数据集。',
   'input': '',
   'output': '要训练一个能...'}]]

We designed sampling strategies based on td-idf and instruction, output length, which improved the diversity of the instruction fine-tuning data.

Preprocess the data, concatenate it into the format of multi-turn dialogues, and encode it to obtain input_ids, then save it.

python code/convert_data.py \
    --tokenizer XuYipei/kw-cutegpt-13b-base \
    --max_length 2048 \
    --out_data_path data/test/

Note: If you continue fine-tuning based on XuYipei/kw-cutegpt-13b-ift, you need to replace the tokenizer with XuYipei/kw-cutegpt-13b-ift because the latter has added special tokens.

Train the model

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 deepspeed --master_port 12932 code/finetune.py \
    --save_steps 2000 \
    --max_epoches 4 \
    --save_name llama_lora \
    --model_path XuYipei/kw-cutegpt-13b-base \
    --dataset_type DatasetIds \
    --data_path data/test/llama_ift_data_ids.pkl \
    --max_length 2048 \
    --use_lora \
    --use_flash_attention

Parameter Explanation

• model_path: Path to the base model.
• dataset_type: Defines the dataset class used for data encapsulation, defined in code/dataset.py.
• use_flash_attention: Whether to use flash attention to speed up training and reduce GPU memory consumption.
• use_lora: Whether to use LoRA fine-tuning. If set to false, it defaults to full fine-tuning.
• load_lora: Whether to load the Lora checkpoint for continued training. If load_lora==True, define the path to the Lora checkpoint in load_lora_path.

Note: If performing full fine-tuning, we have added a special token (<end>) to help the model better learn the conversational patterns of multi-turn dialogue.

Refer to code/config.py for specific deepspeed parameters (e.g., learning rate, batch size) and Lora parameters (e.g., Lora rank).

You can directly run the following command to start training:

• Full-finetuning version

  bash finetune_ft.sh
  

• LoRA version

  bash finetune_lora.sh
  

The LLaMA model officially released by Facebook explicitly prohibits commercial use. The models and code open-sourced by this project are for academic use only. We are training commercial versions of equivalent or even better models in parallel to provide higher quality and more widely applicable solutions.

The content generated by this project may be affected by various factors such as model calculations, randomness, and loss of quantization precision, so we cannot guarantee its accuracy. We solemnly declare that this project is not responsible for any consequences arising from the use of any content generated by this project. This project does not assume any legal responsibility for any losses that may arise from the use of the resources and output results related to this project. Users bear full responsibility for any inappropriate statements resulting from user-induced misconduct.

If you find the model, data, code in our project useful, please consider citing our work as follows:

@misc{CuteGPT,
  author = {Jiaqing Liang, Qianyu He, Yipei Xu, Yikai Zhang, Yuncheng Huang, Wenhao Huang, Xintao Wang, Lida Chen, Haixia Han, Jie Shi, Tinghui Zhu, Yidan Xu, Shisong Chen, Zhouhong Gu, Yanghua Xiao},
  title = {CuteGPT: Towards a Useful Chinese Large Language Model},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Abbey4799/cuteGPT}},
}