-
Notifications
You must be signed in to change notification settings - Fork 181
feat: Expand protobuf block read capacity #21457
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Matt Hess <[email protected]>
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #21457 +/- ##
============================================
- Coverage 70.66% 70.60% -0.07%
- Complexity 24316 24338 +22
============================================
Files 2674 2674
Lines 104044 104217 +173
Branches 10912 10935 +23
============================================
+ Hits 73523 73581 +58
- Misses 26473 26583 +110
- Partials 4048 4053 +5
... and 32 files with indirect coverage changes 🚀 New features to boost your workflow:
|
Coverage summary from CodacySee diff coverage on Codacy
Coverage variation details
Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: Diff coverage details
Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: See your quality gate settings Change summary preferences |
hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockBufferConfig.java
Outdated
Show resolved
Hide resolved
...a-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockBufferIO.java
Outdated
Show resolved
Hide resolved
hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockBufferConfig.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Matt Hess <[email protected]>
Signed-off-by: Matt Hess <[email protected]>
hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
Outdated
Show resolved
Hide resolved
hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Matt Hess <[email protected]>
Signed-off-by: Matt Hess <[email protected]>
Signed-off-by: Petar Tonev <[email protected]>
Signed-off-by: Anthony Petrov <[email protected]> Signed-off-by: Jendrik Johannes <[email protected]> Co-authored-by: Jendrik Johannes <[email protected]> Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Tim Farber-Newman <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
...a-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockBufferIO.java
Outdated
Show resolved
Hide resolved
hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Petar Tonev <[email protected]>
| final Bytes bytes = Bytes.wrap(payload); | ||
|
|
||
| return BufferedBlock.PROTOBUF.parse(bytes); | ||
| return BufferedBlock.PROTOBUF.parse( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there is an error caught by the caller, it should be at the error log level as these are unacknowledged blocks being unable to be read back in from disk.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated 👍 I also changed the log level to error when we try to parse pending blocks as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of using the configuration maxReadBytesSize, why not just use the length variable as the max size? We already know upfront what the size of the proto is going to be. Ignoring that value and just using the config value that may very well cause a failure if the actual proto is larger than the config seems like an unnecessary risk, no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The maxSize is a security-related parameter. It is wrong to calculate its value from the input that you're about to parse because this completely defeats its purpose. As I suggested previously, please define the maximum reasonable size in the configuration of your application and use it as the maxSize value.
Signed-off-by: Petar Tonev <[email protected]>
Signed-off-by: Petar Tonev <[email protected]>
# Conflicts: # hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockBufferService.java # hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
Signed-off-by: Petar Tonev <[email protected]>
Signed-off-by: Matt Hess <[email protected]> Signed-off-by: Petar Tonev <[email protected]> Signed-off-by: Anthony Petrov <[email protected]> Signed-off-by: Jendrik Johannes <[email protected]> Signed-off-by: Derek Riley <[email protected]> Signed-off-by: Tim Farber-Newman <[email protected]> Co-authored-by: Petar Tonev <[email protected]> Co-authored-by: anthony-swirldslabs <[email protected]> Co-authored-by: Jendrik Johannes <[email protected]> Co-authored-by: Derek Riley <[email protected]> Co-authored-by: Tim Farber-Newman <[email protected]> Signed-off-by: Petar Tonev <[email protected]>
Signed-off-by: Matt Hess <[email protected]> Signed-off-by: Petar Tonev <[email protected]> Signed-off-by: Anthony Petrov <[email protected]> Signed-off-by: Jendrik Johannes <[email protected]> Signed-off-by: Derek Riley <[email protected]> Signed-off-by: Tim Farber-Newman <[email protected]> Co-authored-by: Petar Tonev <[email protected]> Co-authored-by: anthony-swirldslabs <[email protected]> Co-authored-by: Jendrik Johannes <[email protected]> Co-authored-by: Derek Riley <[email protected]> Co-authored-by: Tim Farber-Newman <[email protected]>
This PR uses the new PBJ version from #21395 to read block protobufs bigger than the default hardcoded limit. Closes #21430