支持RangeSpecifiedFieldSelector使用指定字段的值域进行数据选择 #432
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2108038773
had a problem deploying
to
Testing
GitHub Actions
Failure
|
请参考单测日志对照失败样例尝试修复代码或者更新单测样例 link |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
目前的RangeSpecifiedFieldSelector类只支持通过百分位数和rank进行select，这并不符合直觉，最简单且最常用的方法应该是根据某个字段的值域进行选择，比如相似度大于某个阈值，PPL小于某个阈值等，本次PR支持这一功能。
此外，原来的process函数在逻辑判断时存在一定的问题（不支持某些情况下的缺省），lower_percentile和lower_rank不能同时为None，upper_percentile和upper_rank也不能同时为None，否则就不会进行select，这不适用于只有上界或者只有下界的情况，本次PR针对这一逻辑进行了优化。