Weekly release: 0.19.0.dev2025040800

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we have updated a weekly release 0.19.0.dev2025040800, and pushed an update to the Triton backend this April 8, 2025.

The 0.19.0.dev2025040800 dev release includes:

• Model Support
  • Added support for Phi-4-mini (feat: Add support for Phi-4-mini #2990).
• Features
  • fp8_blockscale_gemm is now open-sourced. (feat: Open source fp8_blockscale_gemm #3071).
  • Added AWQ support for ModelOpt checkpoints (feat: [NVBUG 5200562] AWQ support Modelopt ckpts. #3258).
  • Added Linear block scale layout support in FP4 quantization (Feat: Support Linear block scale layout in FP4 quantization #3045).
  • Added pre-quantized FP8 checkpoint support for Nemotron-mini-4b-instruct (feat: Support prequantized fp8 ckpt for nemotron-mini-4b-instruct #3046).
  • Added no-cache attention support for PyTorch workflow (feat: no-cache attention in PyTorch workflow #3085).
  • Added PeftCacheManager support for PyTorch workflow (feat: Support PeftCacheManager in Torch #3186).
  • Added Variable-Beam-Width-Search (VBWS) support (part2) (feat: Variable-Beam-Width-Search (VBWS) Part2 #3133).
  • Added LoRA support for Gemma (feat: Add LoRA support for gemma #3068).
  • Refactored scaffolding worker, added OpenAI API worker support (feat: refactor scaffolding worker and support openai api worker #3166).
  • Optionally split MoE inputs into chunks to reduce GPU memory usage (feat: Optionally split MoE inputs into chunks to reduce GPU memory usage #3104).
  • Added UCX IP interface support (feat: user can specify UCX ip interface #3306).
  • Dependent CUTLASS version is updated (feat: Update cutlass #2981).
  • [BREAKING CHANGE] Added output of first token to additional generation outputs ([BREAKING CHANGE][TRTLLM-4354] chore: Add output of first token to additional generation outputs #3205).
• API
  • Exposed DecoderState via bindings and integrated it in decoder (refactor: Expose DecoderState via bindings and integrate in TRTLLMDecoder #3139).
  • Refactored the LlmArgs with Pydantic and migrated remaining pybinding configurations to Python (chore: refactor the LlmArgs with Pydantic and migrate remaining pybinding configs to python #3025).
  • Refactored disaggregated serving scripts (chore: Refactor disaggregated serving scripts #3073).
• Bug fixes
  • Fixed deepseek multi-GPU tests timeout (fix deepseek multi gpu tests timeout #3285).
  • Fixed Vila test (fix: [NVBUG 5087143] Fix vila test #3042).
  • Fixed bot check error when triggered by pull request (infra: Fix bot check error when triggered by pull request #3268).
  • Fixed minor issues in test_autotuner.py and loosened the cache check for test GEMMs (fix: Fix minor issues in test_autotuner.py #3261).
  • Fixed the acceptance rate of the PyTorch workflow in trtllm-bench (fix: fix the acceptance rate of pytorch workflow in trtllm-bench #3240).
  • Fixed a warning message (Fix: Fix warning for fused_moe.py #3254).
  • Fixed an error related to dummy requests when MTP was used (fix: Fix an error related to dummy request when MTP is used #3146).
  • Raised an error for PP and MTP (chore: Raise error for PP + MTP #3244).
  • Fixed deepseek failure with pipeline parallelism (fix deepseek failure with pipeline parallelism #3225).
  • Removed a test that relied on timing (fix: remove test relies on timing #3228).
  • Fixed GPT-Next convert failure (fix: GPT-Next convert failure #3220).
  • Fixed the conditional disaggregation test name (fix: conditional disagg test name #3161).
  • Fixed a segfault in cudaDriverWrapper (fix: segfault in cudaDriverWrapper #3017).
  • Fixed streaming issues in dist-serving (fix: dist-serving streaming mode returns http 500 #3087).
  • Updated FP8 SF layout for Blackwell and relaxed blockwise GEMM assertions (fix: Update FP8 sf layout for Blackwell and relax blockwise GEMM assertions #3144).
  • Fixed a bug in glm-4-9b CI (nvbug_5196515) (fix: fix bug of nvbug 5196515 which is related to path of glm-4-9b in ci #3184).
  • Fixed illegal memory access when MTP was greater than or equal to 2 (fix：fix illeagel memory access when mtp >= 2 #3006).
• Benchmark
  • Added a total token throughput metric (perf: Add total token throughput metric. #3212).
• Performance
  • Used pinned H2D to reduce bubbles (perf: Use pinned H2D to reduce bubbles #3147).
  • Added optimizations for deepseek in minimum latency mode (perf: Add optimizations for deepseek in min latency mode #3093).
• Documentation
  • Added L4 latest news to the README (L4 added to readme #3301).
  • Refactored trtllm-serve examples and documentation (doc: refactor trtllm-serve examples and doc #3187).
  • Added a serving section for DeepSeek V3 documentation (doc: Add serving section for DS V3 document #3262).
  • Updated DeepSeek-v3 MTP documentation (doc: fix deepseek-v3 mtp cmd #3272).
  • Added supported models to the PyTorch example (doc: add supported-models on PyTorch example #3179).
  • Refined the integration test guide (doc: refine integration test guide #3215).
  • Added documentation about development on cloud or Runpod (doc: add doc about development on cloud or runpod #3194).
  • Added a directory for scaffolding contributors (doc: add a directory for scaffolding contributors #3224).
  • Updated the README for AutoDeploy (fix: [AutoDeploy] Update README.md #3072).
  • Added examples for trtllm-llmapi-launch on Slurm or other MPI-based system (doc: add MGMN doc #3135).

The cut-off commit to this release is 5aeef6d. The code changes can be seen here: 7549573...5aeef6d.

Thanks,
The TensorRT-LLM Engineering Team