Update on the development branch

[Shixiaowei02]

Hi,

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch (and the Triton backend) this March 26, 2024.

This update includes:

• Model Support
  • Support Smaug-72B-v0.1
• Features
  • Medusa IFB support
  • [Experimental] Support FP8 FMHA, note that the performance is not optimal, and we will keep optimizing it
  • [BREAKING CHANGE] Support embedding sharing for Gemma
• API
  • [BREAKING CHANGE] Remove LoRA related parameters from convert checkpoint scripts
  • [BREAKING CHANGE] Simplify Qwen convert checkpoint script
  • Add advanced and multi-GPU examples for Python binding of executor C++ API, see examples/bindings/README.md
  • High-level API, please refer to examples/high-level-api/README.md for guide
    • [BREAKING CHANGE] Reuse the QuantConfig used in trtllm-build tool, support broader quantization features
    • Add support for TensorRT-LLM checkpoint as model input
    • Refine SamplingConfig used in LLM.generate or LLM.generate_async APIs, with the support of beam search, a variety of penalties, and more features
    • Add support for the StreamingLLM feature, enable it by setting LLM(streaming_llm=...)
• Bug fixes
  • Fix the issue that ModelRunnerCpp does not transfer SamplingConfig tensor fields correctly ModelRunnerCpp does not transfer SamplingConfig Tensor fields correctly #1183
• Benchmark
  • Add percentile latency report to gptManagerBenchmark
• Performance
  • Improve custom all-reduce kernel

Thanks,
The TensorRT-LLM Engineering Team