δΈζ Β ο½ Β English Β
π Chinese Documentation Β ο½ Β π English Documentation
β If you like this project, please click the "Star" button in the upper right corner to support us. Your support is our motivation to move forward!
EvalScope is a powerful and easily extensible model evaluation framework created by the ModelScope Community, aiming to provide a one-stop evaluation solution for large model developers.
Whether you want to evaluate the general capabilities of models, conduct multi-model performance comparisons, or need to stress test models, EvalScope can meet your needs.
- π Comprehensive Evaluation Benchmarks: Built-in multiple industry-recognized evaluation benchmarks including MMLU, C-Eval, GSM8K, and more.
- 𧩠Multi-modal and Multi-domain Support: Supports evaluation of various model types including Large Language Models (LLM), Vision Language Models (VLM), Embedding, Reranker, AIGC, and more.
- π Multi-backend Integration: Seamlessly integrates multiple evaluation backends including OpenCompass, VLMEvalKit, RAGEval to meet different evaluation needs.
- β‘ Inference Performance Testing: Provides powerful model service stress testing tools, supporting multiple performance metrics such as TTFT, TPOT.
- π Interactive Reports: Provides WebUI visualization interface, supporting multi-dimensional model comparison, report overview and detailed inspection.
- βοΈ Arena Mode: Supports multi-model battles (Pairwise Battle), intuitively ranking and evaluating models.
- π§ Highly Extensible: Developers can easily add custom datasets, models and evaluation metrics.
ποΈ Overall Architecture
EvalScope Overall Architecture.
-
Input Layer
- Model Sources: API models (OpenAI API), Local models (ModelScope)
- Datasets: Standard evaluation benchmarks (MMLU/GSM8k etc.), Custom data (MCQ/QA)
-
Core Functions
- Multi-backend Evaluation: Native backend, OpenCompass, MTEB, VLMEvalKit, RAGAS
- Performance Monitoring: Supports multiple model service APIs and data formats, tracking TTFT/TPOP and other metrics
- Tool Extensions: Integrates Tool-Bench, Needle-in-a-Haystack, etc.
-
Output Layer
- Structured Reports: Supports JSON, Table, Logs
- Visualization Platform: Supports Gradio, Wandb, SwanLab
Important
Version 1.0 Refactoring
Version 1.0 introduces a major overhaul of the evaluation framework, establishing a new, more modular and extensible API layer under evalscope/api. Key improvements include standardized data models for benchmarks, samples, and results; a registry-based design for components such as benchmarks and metrics; and a rewritten core evaluator that orchestrates the new architecture. Existing benchmark adapters have been migrated to this API, resulting in cleaner, more consistent, and easier-to-maintain implementations.
- π₯ [2025.10.27] Added support for LogiQA, HaluEval, MathQA, MRI-QA, PIQA, QASC, CommonsenseQA and other evaluation benchmarks. Thanks to @penguinwang96825 for the code implementation.
- π₯ [2025.10.26] Added support for Conll-2003, CrossNER, Copious, GeniaNER, HarveyNER, MIT-Movie-Trivia, MIT-Restaurant, OntoNotes5, WNUT2017 and other Named Entity Recognition evaluation benchmarks. Thanks to @penguinwang96825 for the code implementation.
- π₯ [2025.10.21] Optimized sandbox environment usage in code evaluation, supporting both local and remote operation modes. For details, refer to the documentation.
- π₯ [2025.10.20] Added support for evaluation benchmarks including PolyMath, SimpleVQA, MathVerse, MathVision, AA-LCR; optimized evalscope perf performance to align with vLLM Bench. For details, refer to the documentation.
- π₯ [2025.10.14] Added support for OCRBench, OCRBench-v2, DocVQA, InfoVQA, ChartQA, and BLINK multimodal image-text evaluation benchmarks.
- π₯ [2025.09.22] Code evaluation benchmarks (HumanEval, LiveCodeBench) now support running in a sandbox environment. To use this feature, please install ms-enclave first.
- π₯ [2025.09.19] Added support for multimodal image-text evaluation benchmarks including RealWorldQA, AI2D, MMStar, MMBench, and OmniBench, as well as pure text evaluation benchmarks such as Multi-IF, HealthBench, and AMC.
- π₯ [2025.09.05] Added support for vision-language multimodal model evaluation tasks, such as MathVista and MMMU. For more supported datasets, please refer to the documentation.
- π₯ [2025.09.04] Added support for image editing task evaluation, including the GEdit-Bench benchmark. For usage instructions, refer to the documentation.
- π₯ [2025.08.22] Version 1.0 Refactoring. Break changes, please refer to.
- π₯ [2025.07.18] The model stress testing now supports randomly generating image-text data for multimodal model evaluation. For usage instructions, refer to the documentation.
- π₯ [2025.07.16] Support for Ο-bench has been added, enabling the evaluation of AI Agent performance and reliability in real-world scenarios involving dynamic user and tool interactions. For usage instructions, please refer to the documentation.
- π₯ [2025.07.14] Support for "Humanity's Last Exam" (Humanity's-Last-Exam), a highly challenging evaluation benchmark. For usage instructions, refer to the documentation.
- π₯ [2025.07.03] Refactored Arena Mode: now supports custom model battles, outputs a model leaderboard, and provides battle result visualization. See reference for details.
More
- π₯ [2025.06.28] Optimized custom dataset evaluation: now supports evaluation without reference answers. Enhanced LLM judge usage, with built-in modes for "scoring directly without reference answers" and "checking answer consistency with reference answers". See reference for details.
- π₯ [2025.06.19] Added support for the BFCL-v3 benchmark, designed to evaluate model function-calling capabilities across various scenarios. For more information, refer to the documentation.
- π₯ [2025.06.02] Added support for the Needle-in-a-Haystack test. Simply specify
needle_haystackto conduct the test, and a corresponding heatmap will be generated in theoutputs/reportsfolder, providing a visual representation of the model's performance. Refer to the documentation for more details. - π₯ [2025.05.29] Added support for two long document evaluation benchmarks: DocMath and FRAMES. For usage guidelines, please refer to the documentation.
- π₯ [2025.05.16] Model service performance stress testing now supports setting various levels of concurrency and outputs a performance test report. Reference example.
- π₯ [2025.05.13] Added support for the ToolBench-Static dataset to evaluate model's tool-calling capabilities. Refer to the documentation for usage instructions. Also added support for the DROP and Winogrande benchmarks to assess the reasoning capabilities of models.
- π₯ [2025.04.29] Added Qwen3 Evaluation Best Practices, welcome to read π
- π₯ [2025.04.27] Support for text-to-image evaluation: Supports 8 metrics including MPS, HPSv2.1Score, etc., and evaluation benchmarks such as EvalMuse, GenAI-Bench. Refer to the user documentation for more details.
- π₯ [2025.04.10] Model service stress testing tool now supports the
/v1/completionsendpoint (the default endpoint for vLLM benchmarking) - π₯ [2025.04.08] Support for evaluating embedding model services compatible with the OpenAI API has been added. For more details, check the user guide.
- π₯ [2025.03.27] Added support for AlpacaEval and ArenaHard evaluation benchmarks. For usage notes, please refer to the documentation
- π₯ [2025.03.20] The model inference service stress testing now supports generating prompts of specified length using random values. Refer to the user guide for more details.
- π₯ [2025.03.13] Added support for the LiveCodeBench code evaluation benchmark, which can be used by specifying
live_code_bench. Supports evaluating QwQ-32B on LiveCodeBench, refer to the best practices. - π₯ [2025.03.11] Added support for the SimpleQA and Chinese SimpleQA evaluation benchmarks. These are used to assess the factual accuracy of models, and you can specify
simple_qaandchinese_simpleqafor use. Support for specifying a judge model is also available. For more details, refer to the relevant parameter documentation. - π₯ [2025.03.07] Added support for the QwQ-32B model, evaluate the model's reasoning ability and reasoning efficiency, refer to π Best Practices for QwQ-32B Evaluation for more details.
- π₯ [2025.03.04] Added support for the SuperGPQA dataset, which covers 13 categories, 72 first-level disciplines, and 285 second-level disciplines, totaling 26,529 questions. You can use it by specifying
super_gpqa. - π₯ [2025.03.03] Added support for evaluating the IQ and EQ of models. Refer to π Best Practices for IQ and EQ Evaluation to find out how smart your AI is!
- π₯ [2025.02.27] Added support for evaluating the reasoning efficiency of models. Refer to π Best Practices for Evaluating Thinking Efficiency. This implementation is inspired by the works Overthinking and Underthinking.
- π₯ [2025.02.25] Added support for two model inference-related evaluation benchmarks: MuSR and ProcessBench. To use them, simply specify
musrandprocess_benchrespectively in the datasets parameter. - π₯ [2025.02.18] Supports the AIME25 dataset, which contains 15 questions (Grok3 scored 93 on this dataset).
- π₯ [2025.02.13] Added support for evaluating DeepSeek distilled models, including AIME24, MATH-500, and GPQA-Diamond datasetsοΌrefer to best practice; Added support for specifying the
eval_batch_sizeparameter to accelerate model evaluation. - π₯ [2025.01.20] Support for visualizing evaluation results, including single model evaluation results and multi-model comparison, refer to the π Visualizing Evaluation Results for more details; Added
iquizevaluation example, evaluating the IQ and EQ of the model. - π₯ [2025.01.07] Native backend: Support for model API evaluation is now available. Refer to the π Model API Evaluation Guide for more details. Additionally, support for the
ifevalevaluation benchmark has been added. - π₯π₯ [2024.12.31] Support for adding benchmark evaluations, refer to the π Benchmark Evaluation Addition Guide; support for custom mixed dataset evaluations, allowing for more comprehensive model evaluations with less data, refer to the π Mixed Dataset Evaluation Guide.
- π₯ [2024.12.13] Model evaluation optimization: no need to pass the
--template-typeparameter anymore; supports starting evaluation withevalscope eval --args. Refer to the π User Guide for more details. - π₯ [2024.11.26] The model inference service performance evaluator has been completely refactored: it now supports local inference service startup and Speed Benchmark; asynchronous call error handling has been optimized. For more details, refer to the π User Guide.
- π₯ [2024.10.31] The best practice for evaluating Multimodal-RAG has been updated, please check the π Blog for more details.
- π₯ [2024.10.23] Supports multimodal RAG evaluation, including the assessment of image-text retrieval using CLIP_Benchmark, and extends RAGAS to support end-to-end multimodal metrics evaluation.
- π₯ [2024.10.8] Support for RAG evaluation, including independent evaluation of embedding models and rerankers using MTEB/CMTEB, as well as end-to-end evaluation using RAGAS.
- π₯ [2024.09.18] Our documentation has been updated to include a blog module, featuring some technical research and discussions related to evaluations. We invite you to π read it.
- π₯ [2024.09.12] Support for LongWriter evaluation, which supports 10,000+ word generation. You can use the benchmark LongBench-Write to measure the long output quality as well as the output length.
- π₯ [2024.08.30] Support for custom dataset evaluations, including text datasets and multimodal image-text datasets.
- π₯ [2024.08.20] Updated the official documentation, including getting started guides, best practices, and FAQs. Feel free to πread it here!
- π₯ [2024.08.09] Simplified the installation process, allowing for pypi installation of vlmeval dependencies; optimized the multimodal model evaluation experience, achieving up to 10x acceleration based on the OpenAI API evaluation chain.
- π₯ [2024.07.31] Important change: The package name
llmuseshas been changed toevalscope. Please update your code accordingly. - π₯ [2024.07.26] Support for VLMEvalKit as a third-party evaluation framework to initiate multimodal model evaluation tasks.
- π₯ [2024.06.29] Support for OpenCompass as a third-party evaluation framework, which we have encapsulated at a higher level, supporting pip installation and simplifying evaluation task configuration.
- π₯ [2024.06.13] EvalScope seamlessly integrates with the fine-tuning framework SWIFT, providing full-chain support from LLM training to evaluation.
- π₯ [2024.06.13] Integrated the Agent evaluation dataset ToolBench.
Welcome to join our community to communicate with other developers and get help.
| Discord Group | WeChat Group | DingTalk Group |
|---|---|---|
 |
 |
 |
We recommend using conda to create a virtual environment and install with pip.
-
Create and Activate Conda Environment (Python 3.10 recommended)
conda create -n evalscope python=3.10 conda activate evalscope
-
Install EvalScope
-
Method 1: Install via PyPI (Recommended)
pip install evalscope
-
Method 2: Install from Source (For Development)
git clone https://github.com/modelscope/evalscope.git cd evalscope pip install -e .
-
-
Install Additional Dependencies (Optional) Install corresponding feature extensions according to your needs:
# Performance testing pip install 'evalscope[perf]' # Visualization App pip install 'evalscope[app]' # Other evaluation backends pip install 'evalscope[opencompass]' pip install 'evalscope[vlmeval]' pip install 'evalscope[rag]' # Install all dependencies pip install 'evalscope[all]'
If you installed from source, please replace
evalscopewith., for examplepip install '.[perf]'.
Note
This project was formerly known as llmuses. If you need to use v0.4.3 or earlier versions, please run pip install llmuses<=0.4.3 and use from llmuses import ... for imports.
You can start evaluation tasks in two ways: command line or Python code.
Execute the evalscope eval command in any path to start evaluation. The following command will evaluate the Qwen/Qwen2.5-0.5B-Instruct model on gsm8k and arc datasets, taking only 5 samples from each dataset.
evalscope eval \
--model Qwen/Qwen2.5-0.5B-Instruct \
--datasets gsm8k arc \
--limit 5Use the run_task function and TaskConfig object to configure and start evaluation tasks.
from evalscope import run_task, TaskConfig
# Configure evaluation task
task_cfg = TaskConfig(
model='Qwen/Qwen2.5-0.5B-Instruct',
datasets=['gsm8k', 'arc'],
limit=5
)
# Start evaluation
run_task(task_cfg)π‘ Tip: `run_task` also supports dictionaries, YAML or JSON files as configuration.
Using Python Dictionary
from evalscope.run import run_task
task_cfg = {
'model': 'Qwen/Qwen2.5-0.5B-Instruct',
'datasets': ['gsm8k', 'arc'],
'limit': 5
}
run_task(task_cfg=task_cfg)Using YAML File (config.yaml)
model: Qwen/Qwen2.5-0.5B-Instruct
datasets:
- gsm8k
- arc
limit: 5from evalscope.run import run_task
run_task(task_cfg="config.yaml")After evaluation completion, you will see a report in the terminal in the following format:
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
| Model Name | Dataset Name | Metric Name | Category Name | Subset Name | Num | Score |
+=======================+================+=================+=================+===============+=======+=========+
| Qwen2.5-0.5B-Instruct | gsm8k | AverageAccuracy | default | main | 5 | 0.4 |
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
| Qwen2.5-0.5B-Instruct | ai2_arc | AverageAccuracy | default | ARC-Easy | 5 | 0.8 |
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
| Qwen2.5-0.5B-Instruct | ai2_arc | AverageAccuracy | default | ARC-Challenge | 5 | 0.4 |
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
You can fine-tune model loading, inference, and dataset configuration through command line parameters.
evalscope eval \
--model Qwen/Qwen3-0.6B \
--model-args '{"revision": "master", "precision": "torch.float16", "device_map": "auto"}' \
--generation-config '{"do_sample":true,"temperature":0.6,"max_tokens":512}' \
--dataset-args '{"gsm8k": {"few_shot_num": 0, "few_shot_random": false}}' \
--datasets gsm8k \
--limit 10--model-args: Model loading parameters such asrevision,precision, etc.--generation-config: Model generation parameters such astemperature,max_tokens, etc.--dataset-args: Dataset configuration parameters such asfew_shot_num, etc.
For details, please refer to π Complete Parameter Guide.
EvalScope supports evaluating model services deployed via APIs (such as services deployed with vLLM). Simply specify the service address and API Key.
-
Start Model Service (using vLLM as example)
export VLLM_USE_MODELSCOPE=True python -m vllm.entrypoints.openai.api_server \ --model Qwen/Qwen2.5-0.5B-Instruct \ --served-model-name qwen2.5 \ --port 8801 -
Run Evaluation
evalscope eval \ --model qwen2.5 \ --eval-type service \ --api-url http://127.0.0.1:8801/v1 \ --api-key EMPTY \ --datasets gsm8k \ --limit 10
Arena mode evaluates model performance through pairwise battles between models, providing win rates and rankings, perfect for horizontal comparison of multiple models.
# Example evaluation results
Model WinRate (%) CI (%)
------------ ------------- ---------------
qwen2.5-72b 69.3 (-13.3 / +12.2)
qwen2.5-7b 50 (+0.0 / +0.0)
qwen2.5-0.5b 4.7 (-2.5 / +4.4)
For details, please refer to π Arena Mode Usage Guide.
EvalScope allows you to easily add and evaluate your own datasets. For details, please refer to π Custom Dataset Evaluation Guide.
EvalScope supports launching evaluation tasks through third-party evaluation frameworks (we call them "backends") to meet diverse evaluation needs.
- Native: EvalScope's default evaluation framework with comprehensive functionality.
- OpenCompass: Focuses on text-only evaluation. π Usage Guide
- VLMEvalKit: Focuses on multi-modal evaluation. π Usage Guide
- RAGEval: Focuses on RAG evaluation, supporting Embedding and Reranker models. π Usage Guide
- Third-party Evaluation Tools: Supports evaluation tasks like ToolBench.
EvalScope provides a powerful stress testing tool for evaluating the performance of large language model services.
- Key Metrics: Supports throughput (Tokens/s), first token latency (TTFT), token generation latency (TPOT), etc.
- Result Recording: Supports recording results to
wandbandswanlab. - Speed Benchmarks: Can generate speed benchmark results similar to official reports.
For details, please refer to π Performance Testing Usage Guide.
Example output is shown below:
EvalScope provides a Gradio-based WebUI for interactive analysis and comparison of evaluation results.
-
Install Dependencies
pip install 'evalscope[app]' -
Start Service
evalscope app
Visit
http://127.0.0.1:7861to open the visualization interface.
Settings Interface |
Model Comparison |
Report Overview |
Report Details |
For details, please refer to π Visualizing Evaluation Results.
We welcome any contributions from the community! If you want to add new evaluation benchmarks, models, or features, please refer to our Contributing Guide.
Thanks to all developers who have contributed to EvalScope!
|
|
|---|
If you use EvalScope in your research, please cite our work:
@misc{evalscope_2024,
title={{EvalScope}: Evaluation Framework for Large Models},
author={ModelScope Team},
year={2024},
url={https://github.com/modelscope/evalscope}
}