Feat: add toc extraction method supported by llm #10353
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dosubot
bot
added
the
size:L
dosubot
bot
added
the
💞 feature
dosubot
bot
added
size:XL
Magicbook1108
changed the title
|
All issues solved |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Feat, add toc extraction method NOT FINISHED!!!
Type of change
Current issues:
1. Excessive Token Usage with img_url Input (Line 540)
When passing img_url formatted input, the token count explodes to an extremely large range (approximately 250,000–600,000 tokens). This suggests that the way image URLs are being processed leads to massive and unnecessary tokenization, which severely impacts performance and cost.
2. Task Repetition with pending >= 1
When pending >= 1, the task keeps being re-executed in a loop but never progresses to the next stage. In other words, the system detects a pending task and repeatedly retries execution, but it doesn’t move forward in the pipeline, causing a deadlock-like state.