Skip to content

Conversation

@mhess-swl
Copy link
Contributor

This PR uses the new PBJ version from #21395 to read block protobufs bigger than the default hardcoded limit. Closes #21430

@mhess-swl mhess-swl added this to the v0.67 milestone Oct 7, 2025
@mhess-swl mhess-swl requested a review from derektriley October 7, 2025 20:11
@mhess-swl mhess-swl self-assigned this Oct 7, 2025
@mhess-swl mhess-swl requested a review from a team as a code owner October 7, 2025 20:11
@lfdt-bot
Copy link

lfdt-bot commented Oct 7, 2025

Snyk checks have passed. No issues have been found so far.

Status Scanner Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@codecov
Copy link

codecov bot commented Oct 7, 2025

Codecov Report

❌ Patch coverage is 41.66667% with 14 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...a/node/app/blocks/impl/BlockStreamManagerImpl.java 0.00% 8 Missing ⚠️
...app/blocks/impl/streaming/FileBlockItemWriter.java 0.00% 6 Missing ⚠️

Impacted file tree graph

@@             Coverage Diff              @@
##               main   #21457      +/-   ##
============================================
- Coverage     70.66%   70.60%   -0.07%     
- Complexity    24316    24338      +22     
============================================
  Files          2674     2674              
  Lines        104044   104217     +173     
  Branches      10912    10935      +23     
============================================
+ Hits          73523    73581      +58     
- Misses        26473    26583     +110     
- Partials       4048     4053       +5     
Files with missing lines Coverage Δ Complexity Δ
.../node/app/blocks/impl/streaming/BlockBufferIO.java 87.06% <100.00%> (+1.98%) 4.00 <1.00> (ø)
.../app/blocks/impl/streaming/BlockBufferService.java 76.75% <100.00%> (+0.25%) 80.00 <1.00> (+1.00)
...com/hedera/node/config/data/BlockStreamConfig.java 100.00% <ø> (ø) 3.00 <0.00> (ø)
...app/blocks/impl/streaming/FileBlockItemWriter.java 40.98% <0.00%> (-0.46%) 21.00 <0.00> (ø)
...a/node/app/blocks/impl/BlockStreamManagerImpl.java 73.38% <0.00%> (-1.32%) 56.00 <0.00> (-1.00)

... and 32 files with indirect coverage changes

Impacted file tree graph

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codacy-production
Copy link

codacy-production bot commented Oct 7, 2025

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
-0.06% (target: -1.00%) 41.67%
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (9ac9d16) 103949 77525 74.58%
Head commit (a4ed417) 104122 (+173) 77588 (+63) 74.52% (-0.06%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#21457) 24 10 41.67%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

Signed-off-by: Matt Hess <[email protected]>
Signed-off-by: Matt Hess <[email protected]>
derektriley
derektriley previously approved these changes Oct 7, 2025
petreze
petreze previously approved these changes Oct 8, 2025
petreze and others added 9 commits October 8, 2025 13:35
Signed-off-by: Anthony Petrov <[email protected]>
Signed-off-by: Jendrik Johannes <[email protected]>
Co-authored-by: Jendrik Johannes <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Petar Tonev <[email protected]>
final Bytes bytes = Bytes.wrap(payload);

return BufferedBlock.PROTOBUF.parse(bytes);
return BufferedBlock.PROTOBUF.parse(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is an error caught by the caller, it should be at the error log level as these are unacknowledged blocks being unable to be read back in from disk.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated 👍 I also changed the log level to error when we try to parse pending blocks as well

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using the configuration maxReadBytesSize, why not just use the length variable as the max size? We already know upfront what the size of the proto is going to be. Ignoring that value and just using the config value that may very well cause a failure if the actual proto is larger than the config seems like an unnecessary risk, no?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The maxSize is a security-related parameter. It is wrong to calculate its value from the input that you're about to parse because this completely defeats its purpose. As I suggested previously, please define the maximum reasonable size in the configuration of your application and use it as the maxSize value.

Signed-off-by: Petar Tonev <[email protected]>
# Conflicts:
#	hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/BlockBufferService.java
#	hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
Signed-off-by: Petar Tonev <[email protected]>
@petreze petreze requested review from akdev and derektriley October 23, 2025 19:38
derektriley
derektriley previously approved these changes Oct 28, 2025
@petreze petreze merged commit 2119a0e into main Oct 29, 2025
79 of 80 checks passed
@petreze petreze deleted the 21430-expand-proto-read-max branch October 29, 2025 15:10
petreze added a commit that referenced this pull request Nov 3, 2025
Signed-off-by: Matt Hess <[email protected]>
Signed-off-by: Petar Tonev <[email protected]>
Signed-off-by: Anthony Petrov <[email protected]>
Signed-off-by: Jendrik Johannes <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Tim Farber-Newman <[email protected]>
Co-authored-by: Petar Tonev <[email protected]>
Co-authored-by: anthony-swirldslabs <[email protected]>
Co-authored-by: Jendrik Johannes <[email protected]>
Co-authored-by: Derek Riley <[email protected]>
Co-authored-by: Tim Farber-Newman <[email protected]>
Signed-off-by: Petar Tonev <[email protected]>
mxtartaglia-sl pushed a commit that referenced this pull request Nov 3, 2025
Signed-off-by: Matt Hess <[email protected]>
Signed-off-by: Petar Tonev <[email protected]>
Signed-off-by: Anthony Petrov <[email protected]>
Signed-off-by: Jendrik Johannes <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Tim Farber-Newman <[email protected]>
Co-authored-by: Petar Tonev <[email protected]>
Co-authored-by: anthony-swirldslabs <[email protected]>
Co-authored-by: Jendrik Johannes <[email protected]>
Co-authored-by: Derek Riley <[email protected]>
Co-authored-by: Tim Farber-Newman <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BlockBufferIO update parsing with new PBJ version

9 participants