What max_tokens to use for Qwq evaluation?

[sophiapeng90]

Since QwQ support max 32768 tokens, however, the default max_tokens is 81920 in scripts/run_web_thinker.py, I am wondering what length is used for evaluating QwQ model for difference benchmarks?

python scripts/run_web_thinker_report.py \
    --dataset_name glaive \
    --split test \
    --concurrent_limit 32 \
    --search_engine "serper" \
    --serper_api_key "YOUR_GOOGLE_SERPER_API" \
    --api_base_url "YOUR_API_BASE_URL" \
    --model_name "QwQ-32B" \
    --aux_api_base_url "YOUR_AUX_API_BASE_URL" \
    --aux_model_name "Qwen2.5-32B-Instruct" \
    --tokenizer_path "PATH_TO_YOUR_TOKENIZER" \
    --aux_tokenizer_path "PATH_TO_YOUR_AUX_TOKENIZER"

[sunnynexus]

Hello, thank you for your interest in our work!
We followed the official Qwen recipe for extending the model’s context window, described in the “5. Handle Long Inputs” section of https://huggingface.co/Qwen/QwQ-32B. This recipe is applicable to the Qwen family.

Alternatively, you can leave the model’s config file untouched and simply launch the vLLM server with the --rope-scaling flag:

vllm serve YOUR_MODEL_PATH \
    --served-model-name QwQ-32B \
    --port 8081 \
    --host 0.0.0.0 \
    --tensor-parallel-size 8 \
    --gpu-memory-utilization 0.92 \
    --trust-remote-code \
    --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}' \
    --max-model-len 131072