fix(cluster): resolve four steady-state distributed cluster regressions#133
Merged
Conversation
Fix a set of independent bugs causing rebalance counter churn, phantom key creation, and incarnation inflation in multi-node clusters: - Clone key string in applySet before storing as map key. HTTP frameworks (Fiber) back path-parameter strings with pooled request buffers reused on the next request; storing the raw pointer let map keys silently mutate, producing phantom shard entries and a persistent rebalance loop (~60 bumps/s on a 5-node cluster). - Add applyForwardedSet with a receiver-side ownership guard for all transport-receiver paths (InProcessTransport.ForwardSet, HTTP /internal/set). Writes forwarded from peers with a divergent ring view are silently dropped, breaking the migrate→fan-out-back→stuck cycle. Exposes new dist.write.apply_refused metric. - Release local copy immediately in migrateIfNeeded when removalGracePeriod == 0. Previously the local item was never removed, so the rebalance scanner re-flagged the same lost-ownership keys on every tick. Each stuck key now produces exactly one migration. - Make Membership.Mark a no-op on same-state calls: incarnation, version vector, and observers all stay quiet during steady-state heartbeat success. Add Membership.Refute as the dedicated SWIM self-refute primitive that unconditionally bumps incarnation. Update refuteIfSuspected to use Refute(). Adds integration tests pinning idle-cluster silence, traffic-under-load silence, one-shot drain semantics, and the ownership-guard contract. Adds five unit tests covering Mark no-op and Refute always-bumps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix a set of independent bugs causing rebalance counter churn, phantom key creation, and incarnation inflation in multi-node clusters:
Clone key string in applySet before storing as map key. HTTP frameworks (Fiber) back path-parameter strings with pooled request buffers reused on the next request; storing the raw pointer let map keys silently mutate, producing phantom shard entries and a persistent rebalance loop (~60 bumps/s on a 5-node cluster).
Add applyForwardedSet with a receiver-side ownership guard for all transport-receiver paths (InProcessTransport.ForwardSet, HTTP /internal/set). Writes forwarded from peers with a divergent ring view are silently dropped, breaking the migrate→fan-out-back→stuck cycle. Exposes new dist.write.apply_refused metric.
Release local copy immediately in migrateIfNeeded when removalGracePeriod == 0. Previously the local item was never removed, so the rebalance scanner re-flagged the same lost-ownership keys on every tick. Each stuck key now produces exactly one migration.
Make Membership.Mark a no-op on same-state calls: incarnation, version vector, and observers all stay quiet during steady-state heartbeat success. Add Membership.Refute as the dedicated SWIM self-refute primitive that unconditionally bumps incarnation. Update refuteIfSuspected to use Refute().
Adds integration tests pinning idle-cluster silence, traffic-under-load silence, one-shot drain semantics, and the ownership-guard contract. Adds five unit tests covering Mark no-op and Refute always-bumps.