This work (GaC) allows multiple heterogeneous LLMs to ensemble at each generation step and collectively decide the next token. We take the union of the vocabularies of all LLMs participating in the ensemble and, at each generation step, map the probability vectors generated by each LLM to the union vocab. We then compute the (weighted) average to determine the next token. Experiments show that this simple method can break the ceiling of the open-source LLM community, allowing the ensemble result to outperform any single state-of-the-art LLM. [Paper]

We provide two types of ensemble methods:

1. Every Step Ensemble: All LLMs ensemble at each generation step.
2. Thresholded Ensemble: A primary(gate) LLM is specified, and ensemble is performed only if the maximum confidence token of the primary LLM at a step is below a threshold, thereby saving computational resources.

We support parallel execution of the LLMs involved in the ensemble to save time. If each LLM is allocated to a different GPU, the latency of ensembling is almost the same as using a single LLM, all managed by Ray.

Note: Ensemble of available SOTA LLMs from different periods. The top part lists the individual models, while the bottom part shows the ensemble results (model names abbreviated). ↑ indicates the percentage improvement over the individual models.

System Requirements

Get Starteds

• Create a New Environment
• Install GaC Required Packages
• Launch GaC Server
• Run GaC Ensemble

• Operating System: Ubuntu 20.04. We have not tested on Windows.
• Python Version: 3.11
• GPU: Ensure that your GPU has enough RAM to load all the models you want to ensemble.
• Environment Management Tool: Anaconda or any other suitable tool

1. Open your terminal.
2. Create and activate a new conda environment:

conda create -n gac_env python=3.11
conda activate gac_env

cd [root-of-this-repo]/GaC
pip install -r requirements.txt

We have integrated our work into an API server, which can be configured with a YAML file at startup to determine which LLMs to use for ensembling. An example is shown below:

NORM_TYPE_API_SERVER: 'average' # 'average' or 'score'
THRESHOLD_API_SERVER: 1.0
CONFIG_API_SERVER:
  - weight: '[Please replace with the path to the local model weight]' # or 'upstage/SOLAR-10.7B-Instruct-v1.0'
    max_memory:
      0: '24GiB'
    num_gpus: 0.5
    name: 'SOLAR-10.7B-Instruct-v1.0'
    score: 100
    priority: 'supportive' # 'primary' or 'supportive'
  
  - weight: '[Please replace with the path to the local model weight]' # or 'openchat/openchat-3.5-0106'
    max_memory:
      0: '24GiB'
    num_gpus: 0.5
    name: 'openchat-3.5-0106'
    score: 100
    priority: 'supportive' # 'primary' or 'supportive'

Note: Please ensure that the number of GPUs on your computer is greater than the sum of all num_gpus values, and that the max_memory index for each model always starts from 0 (you can assume each model runs on an independent machine managed by Ray).

To start the GaC server, use the following command in your terminal:

python gac_api_server.py --config-path [path-to-your-config-file.yaml] --host 0.0.0.0 --port 8000

After setting up the API, you can directly execute the GaC ensemble by making calls as demonstrated in call.py. Here’s an explanation of the key parameters:

• messages_list: A list of messages, where each messages represent a conversation. Each messages can contain multiple sub-messages in the format {"role": "..", "content": ".."}. The value of "role" can be system, user, or assistant. The list containing n messages indicates a batch size of n. However, for thresholded ensembles, only a batch size of 1 is currently supported.
• max_new_tokens: An integer that determines the maximum number of tokens that can be generated.
• apply_chat_template: A boolean. If True, each model will assemble the messages into a specific template according to the "chat_template" Jinja template defined in its tokenizer_config.json.

Here’s an example code:

import requests

url = "http://0.0.0.0:8000/api/generate/"

data = {
    "messages_list": [
	# Conversation 1
        [{"role": "user", "content": "9.11 and 9.9, which is bigger?"},
         {"role": "assistant", "content": "..."},
         {"role": "user", "content": "..."}],
        # Conversation 2
        [{"role": "user", "content": "How are you?"}]                     
    ],
    "max_new_tokens": 1024,
    "apply_chat_template": True,
}

response = requests.post(url, json=data)
print(response.json())

@misc{yu2024breaking,
  title={Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling},
  author={Yu, Yao-Ching and Kuo, Chun-Chih and Ye, Ziqi and Chang, Yu-Cheng and Li, Yueh-Se},
  year={2024},
  eprint={2406.12585},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}