Skip to content

Commit 3538411

Browse files
vllm-ascend-ciAngazenn
authored andcommitted
[Doc] Update accuracy reports for v0.10.1rc1 (vllm-project#2755)
The accuracy results running on NPU Altlas A2 have changed, updating reports for: All models (Qwen3-30B-A3B, Qwen2.5-VL-7B-Instruct, Qwen3-8B-Base, DeepSeek-V2-Lite) - [Workflow run][1] [1]: https://github.com/vllm-project/vllm-ascend/actions/runs/17459225764 - vLLM version: v0.10.1.1 - vLLM main: vllm-project/vllm@2b30afa Signed-off-by: vllm-ascend-ci <[email protected]> Co-authored-by: vllm-ascend-ci <[email protected]>
1 parent 4aaa388 commit 3538411

File tree

5 files changed

+85
-0
lines changed

5 files changed

+85
-0
lines changed
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# deepseek-ai/DeepSeek-V2-Lite
2+
3+
- **vLLM Version**: vLLM: 0.10.1.1 ([1da94e6](https://github.com/vllm-project/vllm/commit/1da94e6)), **vLLM Ascend Version**: v0.10.1rc1 ([7e16b4a](https://github.com/vllm-project/vllm-ascend/commit/7e16b4a))
4+
- **Software Environment**: **CANN**: 8.2.RC1, **PyTorch**: 2.7.1, **torch-npu**: 2.7.1.dev20250724
5+
- **Hardware Environment**: Atlas A2 Series
6+
- **Parallel mode**: TP2
7+
- **Execution mode**: ACLGraph
8+
9+
**Command**:
10+
11+
```bash
12+
export MODEL_ARGS='pretrained=deepseek-ai/DeepSeek-V2-Lite,tensor_parallel_size=2,dtype=auto,trust_remote_code=True,max_model_len=4096,enforce_eager=True'
13+
lm_eval --model vllm --model_args $MODEL_ARGS --tasks gsm8k \
14+
--batch_size auto
15+
```
16+
17+
| Task | Metric | Value | Stderr |
18+
|-----------------------|-------------|----------:|-------:|
19+
| gsm8k | exact_match,strict-match | ✅0.3813 | ± 0.0134 |
20+
| gsm8k | exact_match,flexible-extract | ✅0.3836 | ± 0.0134 |
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Qwen/Qwen2.5-VL-7B-Instruct
2+
3+
- **vLLM Version**: vLLM: 0.10.1.1 ([1da94e6](https://github.com/vllm-project/vllm/commit/1da94e6)), **vLLM Ascend Version**: v0.10.1rc1 ([7e16b4a](https://github.com/vllm-project/vllm-ascend/commit/7e16b4a))
4+
- **Software Environment**: **CANN**: 8.2.RC1, **PyTorch**: 2.7.1, **torch-npu**: 2.7.1.dev20250724
5+
- **Hardware Environment**: Atlas A2 Series
6+
- **Parallel mode**: TP1
7+
- **Execution mode**: ACLGraph
8+
9+
**Command**:
10+
11+
```bash
12+
export MODEL_ARGS='pretrained=Qwen/Qwen2.5-VL-7B-Instruct,tensor_parallel_size=1,dtype=auto,trust_remote_code=False,max_model_len=8192'
13+
lm_eval --model vllm-vlm --model_args $MODEL_ARGS --tasks mmmu_val \
14+
--apply_chat_template True --fewshot_as_multiturn True --batch_size auto
15+
```
16+
17+
| Task | Metric | Value | Stderr |
18+
|-----------------------|-------------|----------:|-------:|
19+
| mmmu_val | acc,none | ✅0.52 | ± 0.0162 |
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Qwen/Qwen3-30B-A3B
2+
3+
- **vLLM Version**: vLLM: 0.10.1.1 ([1da94e6](https://github.com/vllm-project/vllm/commit/1da94e6)), **vLLM Ascend Version**: v0.10.1rc1 ([7e16b4a](https://github.com/vllm-project/vllm-ascend/commit/7e16b4a))
4+
- **Software Environment**: **CANN**: 8.2.RC1, **PyTorch**: 2.7.1, **torch-npu**: 2.7.1.dev20250724
5+
- **Hardware Environment**: Atlas A2 Series
6+
- **Parallel mode**: TP2 + EP
7+
- **Execution mode**: ACLGraph
8+
9+
**Command**:
10+
11+
```bash
12+
export MODEL_ARGS='pretrained=Qwen/Qwen3-30B-A3B,tensor_parallel_size=2,dtype=auto,trust_remote_code=False,max_model_len=4096,gpu_memory_utilization=0.6,enable_expert_parallel=True'
13+
lm_eval --model vllm --model_args $MODEL_ARGS --tasks gsm8k,ceval-valid \
14+
--num_fewshot 5 --batch_size auto
15+
```
16+
17+
| Task | Metric | Value | Stderr |
18+
|-----------------------|-------------|----------:|-------:|
19+
| gsm8k | exact_match,strict-match | ✅0.8923 | ± 0.0085 |
20+
| gsm8k | exact_match,flexible-extract | ✅0.8506 | ± 0.0098 |
21+
| ceval-valid | acc,none | ✅0.8358 | ± 0.0099 |
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Qwen/Qwen3-8B-Base
2+
3+
- **vLLM Version**: vLLM: 0.10.1.1 ([1da94e6](https://github.com/vllm-project/vllm/commit/1da94e6)), **vLLM Ascend Version**: v0.10.1rc1 ([7e16b4a](https://github.com/vllm-project/vllm-ascend/commit/7e16b4a))
4+
- **Software Environment**: **CANN**: 8.2.RC1, **PyTorch**: 2.7.1, **torch-npu**: 2.7.1.dev20250724
5+
- **Hardware Environment**: Atlas A2 Series
6+
- **Parallel mode**: TP1
7+
- **Execution mode**: ACLGraph
8+
9+
**Command**:
10+
11+
```bash
12+
export MODEL_ARGS='pretrained=Qwen/Qwen3-8B-Base,tensor_parallel_size=1,dtype=auto,trust_remote_code=False,max_model_len=4096'
13+
lm_eval --model vllm --model_args $MODEL_ARGS --tasks gsm8k,ceval-valid \
14+
--apply_chat_template True --fewshot_as_multiturn True --num_fewshot 5 --batch_size auto
15+
```
16+
17+
| Task | Metric | Value | Stderr |
18+
|-----------------------|-------------|----------:|-------:|
19+
| gsm8k | exact_match,strict-match | ✅0.8271 | ± 0.0104 |
20+
| gsm8k | exact_match,flexible-extract | ✅0.8294 | ± 0.0104 |
21+
| ceval-valid | acc,none | ✅0.815 | ± 0.0103 |

docs/source/developer_guide/evaluation/accuracy_report/index.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,8 @@
33
:::{toctree}
44
:caption: Accuracy Report
55
:maxdepth: 1
6+
DeepSeek-V2-Lite
7+
Qwen2.5-VL-7B-Instruct
8+
Qwen3-30B-A3B
9+
Qwen3-8B-Base
610
:::

0 commit comments

Comments
 (0)