Skip to content

Conversation

@littledgg
Copy link
Contributor

@littledgg littledgg commented Nov 3, 2025

Motivation

Achieving batch invariance in the PaddlePaddle framework.
Batch invariance:https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

想要跑通需要安装如下内容,paddle必须是比较新的(建议用最新的)

pip install --pre paddlepaddle-gpu -i https://www.paddlepaddle.org.cn/packages/nightly/cu129/
pip install triton

python tests/batch_invariant/test_batch_invariance.py

如果能看见Batch-Invariant Mode下均为0就代表正确
image
目前只有log_softmax算子尽管精心构造了输入数据,但是在原版实现似乎就已经具备批处理不变性了。

TODO:严格对齐API目前(mm和log_softmax还存在问题),可以考虑把test case整合进一个文件,文件中列出的若干TODO

Modifications

Usage or Command

Accuracy Tests

Checklist

  • Add at least a tag in the PR title.
    • Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
    • You can add new tags based on the PR content, but the semantics must be clear.
  • Format your code, run pre-commit before commit.
  • Add unit tests. Please write the reason in this PR if no unit tests.
  • Provide accuracy results.
  • If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

@paddle-bot
Copy link

paddle-bot bot commented Nov 3, 2025

Thanks for your contribution!

@paddle-bot paddle-bot bot added the contributor External developers label Nov 3, 2025
@gongshaotian gongshaotian marked this pull request as ready for review November 3, 2025 07:08
@littledgg littledgg changed the title [Deterministic] Move batch paddle version invariant pkg to Fastdeploy [Deterministic] Move paddle version batch invariant pkg to Fastdeploy Nov 3, 2025
@gongshaotian
Copy link
Collaborator

please format you code

@littledgg
Copy link
Contributor Author

please format you code

done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

2 participants