Skip to content

Various improvements/fixes on long-running stability #176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 20 commits into from
Sep 15, 2025
Merged

Various improvements/fixes on long-running stability #176

merged 20 commits into from
Sep 15, 2025

Conversation

popduke
Copy link
Contributor

@popduke popduke commented Sep 15, 2025

  1. Improved the stability of builtin balancers
  2. Reduced memory-overhead when downstream back pressure
  3. Reduced the message-overhead during membership anti-entropy
  4. Improved logging and metrics

@popduke
add shortcut to prevent NPE when landscape doesn't contain local desc…
7f639aa 
…riptor
@popduke
1. fixed an identified config change failure process by: 1)make wal c…
e1e628b 
…ompaction whenever fsm applied config entry, 2. send EnsureRequest only from leader during applying RequestConfigChange

2. quit zombie only when local replica is not in current config
@popduke
Reduce memory head caused by inefficient argument formatter
22f9f4c
@popduke
Improve the backpressure mechanism when the downstream is stalled
2138a7e
@popduke
1. Reducing AGENT_HOST_MAP sync overhead when failed member detected
c89355e 
2. Remove deprecated field in internal proto
@popduke
1. Correct the log context for CRDT
e68de01 
2. MDCLogger support lambda
@popduke
fixed a race condition which may cause pipeline graceful retargeting …
2fdf789 
…stuck
@popduke
1. reduce HostMemberList sync overhead during failure broadcast
a575ce5 
2. expose refute signal to upper layer so it could be used to trigger CRDT refresh to speedup convergence.
@popduke
1. fixed an issue in RedundantRangeRemovalBalancer which may cause th…
a2a7f71 
…e removal of functioning range mistakenly

2. improve the stability of RangeSplitBalancer and ReplicaCntBalancer
3. enhance balancer controller to support accept partial load rules from API
@popduke
1. Optimize bootstrap and config change workflow
0abe945 
2. Enhance ReplicaCntBalancer to consider more edge cases
@popduke
1. correctly handle the duplicate matchinfo in inbox ingestion package
23f12e9 
2. correct the code format of Setting's file
@popduke
Fixed an unbalanced issue of ReplicaCntBalancer
2fc4018
@popduke
CRDT AntiEntropy Improvements:
4e65e0f 
1. Reset `resendCount` on ACK to avoid inflated backoff and spurious probe resets
2. Continue anti-entropy after ACK when needed to drain remaining deltas.
3. Leverage late or unmatched ACK whenever possible
@popduke
Improve stale member cleanup logic
1c6e896
@popduke
base-crdt: metering delta send rate and throughput correctly during a…
fab1267 
…nti-entropy
@popduke
optimize balancer's log output
8ff32b3
@popduke
Improved housekeeping logic for crdt-based metadata service
91dea9b
@popduke
Improved the built-in balancers efficiency
cfc1e03
@popduke
Exclude ranges in terminated states from effective route
7fbf1a8
@popduke
Enable manually triggered cov build
91723a2
@coveralls
Copy link

Pull Request Test Coverage Report for Build 17727125372

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 675 unchanged lines in 52 files lost coverage.
  • Overall coverage increased (+0.1%) to 63.439%
Files with Coverage Reduction New Missed Lines %
org/apache/bifromq/basecluster/memberlist/agent/AgentMember.java 1 98.08%
org/apache/bifromq/basecrdt/core/internal/AWORSetInflater.java 1 83.33%
org/apache/bifromq/basecrdt/core/internal/CCounterInflater.java 1 83.33%
org/apache/bifromq/basecrdt/core/internal/DWFlagInflater.java 1 83.33%
org/apache/bifromq/basecrdt/core/internal/MVRegInflater.java 1 83.33%
org/apache/bifromq/basecrdt/core/internal/ORMapInflater.java 1 83.33%
org/apache/bifromq/basecrdt/core/internal/RWORSetInflater.java 1 83.33%
org/apache/bifromq/basecrdt/service/CRDTService.java 1 88.16%
org/apache/bifromq/basescheduler/EMALong.java 1 83.02%
org/apache/bifromq/dist/worker/DeliverExecutorGroup.java 1 95.15%
Totals Coverage Status
Change from base Build 17638390533: 0.1%
Covered Lines: 24024
Relevant Lines: 36557
💛 - Coveralls

@Gujiawei-Edinburgh Gujiawei-Edinburgh merged commit 18e33d6 into apache:main Sep 15, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Gujiawei-Edinburgh Gujiawei-Edinburgh Gujiawei-Edinburgh approved these changes

@ZHANGSHUNLIN ZHANGSHUNLIN Awaiting requested review from ZHANGSHUNLIN

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@popduke @coveralls @Gujiawei-Edinburgh