Evaluating the capabilities of large language models is very difficult. There are already many public leaderboards that do this work. However, public leaderboards are often prone to malicious manipulation and some evaluation benchmarks are not suitable for real application scenarios. So I decided to create my own assessment benchmark and evaluate of my favourite models.

Note:

• Due to limitations of my GPU (24G VRAM), I can only run quantized models, so the performance should be lower that the original models.

• Nous-Hermes-2-Mixtral-8x7B-DPO.Q4_K_M.gguf
• miqu-1-70b-iq2_xs.gguf
• Smaug-34B-v0.1_Q4_K_M.gguf
• senku-70b-iq2_xxs.gguf
• openchat-3.5-0106.Q8_0.gguf
• Starling-LM-7B-beta-Q8_0.gguf
• nous-capybara-34b.Q4_K_M.gguf
• gemma-7b-it.Q8_0.gguf
• gemma-2b-it.Q8_0.gguf
• phi-2.Q8_0.gguf
• qwen1_5-1_8b-chat-q8_0.gguf

I collected 61 test questions from the Internet, it includes:

• Knowledge (7)
• Coding (8)
• Censorship (6)
• Instruction (6)
• Math (7)
• Extraction (7)
• Reasoning (10)
• Summarizing (6)
• Writing (4)

• Nous-Hermes-2-Mixtral-8x7B-DPO.Q4_K_M.gguf
• miqu-1-70b-iq2_xs.gguf
• Smaug-34B-v0.1_Q4_K_M.gguf
• Starling-LM-7B-beta-Q8_0.gguf
• openchat-3.5-0106.Q8_0.gguf
• senku-70b-iq2_xxs.gguf
• nous-capybara-34b.Q4_K_M.gguf
• gemma-7b-it.Q8_0.gguf
• gemma-2b-it.Q8_0.gguf
• qwen1_5-1_8b-chat-q8_0.gguf
• phi-2.Q8_0.gguf

• GeForce RTX 4090 (24G VRAM)
• Intel I9-14900K
• 64G RAM
• Ubuntu 22.04
• Python 3.10
• llama-cpp-python

CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install -r requirements.txt

Download models from huggingface and put gguf files to models folder.

Create model config file in models folder, here is an example:

{
  "name": "gemma-2b",
  "chatFormat": "gemma",
  "modelPath": "gemma-2b-it.Q8_0.gguf",
  "context": 8192
}

python evaluate.py -m models/gemma-7b-it.json