Skip to content

Lisal.fix batch trie #2368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 53 commits into
base: main
Choose a base branch
from
Draft

Lisal.fix batch trie #2368

wants to merge 53 commits into from

Conversation

lisaliu1
Copy link
Contributor

Trie cache works with extensive tests for single beam.

@lisaliu1
fix non-deterministic issue seen in multiple beams
c9fa7d2
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.multi-beams
9568032
@lisaliu1
clean up
868aa52
@lisaliu1
use original topk
f0689eb
@lisaliu1
remove commented code
89f73a5
@lisaliu1
Merge branch 'lisal.multi-beams' into lisal.fix-kv-cache-issues
0187b7a
@lisaliu1
fix issues related with base attention cache
ce7725a
@lisaliu1
single batch and single beam trie == base
ecb437a
@lisaliu1
do not free pages in other batch req
0a1166c
@lisaliu1
fix page leak in trie
79e0897
@lisaliu1
cleanup
00486a4
@lisaliu1
trie works with batch > 1
9cb9d34
@lisaliu1
resolve conflict
54914ba
@lisaliu1
fix issues detected by unit tests
5ec78e5
@lisaliu1
multi beam multi batch and multi work with Trie works
d17bb9a
@lisaliu1
cleanup
fc36d0b
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.fix-batch-trie
327648c
@lisaliu1
move the duplicated pages to shared allocated_pages for handling same…
e494be3 
… requests in different batch
@lisaliu1
re-org cache APIs
3cf0f83
@lisaliu1
cleanup message.py
d767eed
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.fix-batch-trie
ca9fad0
@lisaliu1
remove commentted lines
7d8b285
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.fix-batch-trie
f90b7af
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.fix-batch-trie
26e603e
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.fix-batch-trie
9bb6d86
@lisaliu1
remove duplicated_code
99cf478
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.fix-batch-trie
f9a46f6
@lisaliu1
remove unused argument in cache API
da8994a
@lisaliu1
remove unnecessary locking in decoder
b85fcfa
@lisaliu1
remove locking in PageManager
5523c75
@lisaliu1
fix creating duplicated trie node and the page leakage issue caused b…
8dda840 
…y that
@lisaliu1
remove lookup from allocate in Trie
4b43ede
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.fix-batch-trie
27ac25d
@lisaliu1
cleanup comments
195fef8
@lisaliu1
cleanup APIs
c5a49e0
@lisaliu1
cleanup
486c496
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.fix-batch-trie
05e0f9b
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.fix-batch-trie
a387265
@lisaliu1
cleanup a bit
ef4e0ef
@lisaliu1
publish pages after decode request finishes
4d124c1
@lisaliu1
evict pages work, but after eviction mismatch happens
9216766
@lisaliu1
solved duplicated pages issue caused by idential requests, but evict …
5b7d3ff 
…pages still cause issue
@lisaliu1
only evict during creating prefill request, got inconsistent results …
c4f2d68 
…when evict pages are involved.
@lisaliu1
update decoder
3b0cdae
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.fix-batch-trie
f23bf5a
@lisaliu1
fix ref_count issue with alllocate nodes along the same branch of the…
e4e9f6d 
… Trie tree
@lisaliu1
Merge remote-tracking branch 'origin/main' into lisal.fix-batch-trie
b1697d5
@lisaliu1
remove evict_pages from lookup
886ac18
@lisaliu1
Trie cached works with all tests including one tests for evicting pages
c7b36d9
@lisaliu1
remove debugging print
1791fe8
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 28, 2025 •
edited
Loading

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)		Lines missing
  sharktank/sharktank/ops
  default_impls.py
  signatures.py
  sharktank/sharktank/types
  tensors.py
Project Total  

This report was generated by python-coverage-comment-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@lisaliu1