Skip to content

fix: reduce log spam when remote node is falling behind (backport) #21828

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

fix: reduce log spam when remote node is falling behind (backport) #21828

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 15 additions & 12 deletions ...s-platform-core/src/main/java/com/swirlds/platform/gossip/shadowgraph/RpcPeerHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

import com.hedera.hapi.platform.event.GossipEvent;
import com.swirlds.base.time.Time;
import com.swirlds.common.utility.throttle.RateLimiter;
import com.swirlds.logging.legacy.LogMarker;
import com.swirlds.platform.gossip.IntakeEventCounter;
import com.swirlds.platform.gossip.permits.SyncGuard;
Expand Down Expand Up @@ -87,6 +88,11 @@ public class RpcPeerHandler implements GossipRpcReceiver {
*/
private final RpcPeerState state = new RpcPeerState();

/**
* Limiter to not spam with logs about falling behind compared to other nodes
*/
private final RateLimiter fallBehindRateLimiter;
Copy link
Contributor

@lpetrovic05 lpetrovic05 Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also a RateLimitedLogger, might be more appropriate

Copy link
Contributor

@poulok poulok Oct 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's true. We can use that class instead in main


/**
* How many events were sent out to peer node during latest sync
*/
Expand Down Expand Up @@ -134,6 +140,7 @@ public RpcPeerHandler(
this.intakeEventCounter = Objects.requireNonNull(intakeEventCounter);
this.eventHandler = Objects.requireNonNull(eventHandler);
this.syncGuard = syncGuard;
this.fallBehindRateLimiter = new RateLimiter(time, Duration.ofMinutes(1));
}

/**
Expand Down Expand Up @@ -289,13 +296,14 @@ private void maybeBothSentSyncData() {
final SyncFallenBehindStatus behindStatus = sharedShadowgraphSynchronizer.hasFallenBehind(
state.mySyncData.eventWindow(), state.remoteSyncData.eventWindow(), peerId);
if (behindStatus != SyncFallenBehindStatus.NONE_FALLEN_BEHIND) {
logger.info(
LogMarker.RECONNECT.getMarker(),
"{} local ev={} remote ev={}",
behindStatus,
state.mySyncData.eventWindow(),
state.remoteSyncData.eventWindow());

if (fallBehindRateLimiter.requestAndTrigger()) {
logger.info(
LogMarker.RECONNECT.getMarker(),
"{} local ev={} remote ev={}",
behindStatus,
state.mySyncData.eventWindow(),
state.remoteSyncData.eventWindow());
}
clearInternalState();
if (behindStatus == SyncFallenBehindStatus.OTHER_FALLEN_BEHIND) {
this.syncMetrics.reportSyncPhase(peerId, SyncPhase.OTHER_FALLEN_BEHIND);
Expand Down Expand Up @@ -323,11 +331,6 @@ private boolean tryFixSelfFallBehind(final EventWindow remoteEventWindow) {
latestShadowWindow.getEventWindow(), remoteEventWindow, peerId);
if (behindStatus != SyncFallenBehindStatus.SELF_FALLEN_BEHIND) {
// we seem to be ok after all, let's wait for another sync to happen
logger.info(
LogMarker.RECONNECT.getMarker(),
"Latest event window is not really falling behind, will retry sync local ev={} remote ev={}",
latestShadowWindow.getEventWindow(),
remoteEventWindow);
return true;
}

Expand Down
Loading