Implements SatelliteReplicationStrategy read/write/paxos plans with
quorum-of-quorums semantics, built on a new CompositeTracker (replacing
PerDcResponseTracker) that requires N of M child quorums to succeed.
Adds SatelliteFailoverState, tracking per-range NORMAL / TRANSITION_ACK /
TRANSITION state so replica layout and consistency requirements can vary
by failover phase. Paxos runs within the primary DC but includes
satellite/secondary summary nodes on prepare and commit via
SatellitePaxosParticipants; PaxosCommitRemoteMutationVerbHandler rejects
remote paxos commit mutations during TRANSITION_ACK so stale commits
can't leak into the other DCs during failover.
Patch by Blake Eggleston; reviewed by Francisco Guerrero and Ariel Weisberg for CASSANDRA-21106
Extracts coordination completion logic out of the read/write response
handlers into a ResponseTracker abstraction (simple, write, per-DC), and
pairs it with the ReplicaPlan in a CoordinationPlan so replica selection
and quorum requirements are always produced together by the replication
strategy from a single view of state.
Replication strategies now own construction of both halves. Write/read
paths, paxos, batchlog, read repair, and range reads are converted to
consume CoordinationPlan; ReplicaPlanIterator/Merger are renamed
accordingly. Tests included for each tracker implementation.
Patch by Blake Eggleston; reviewed by Francisco Guerrero and Ariel Weisberg for CASSANDRA-21106
Randomized tests covering response/failure orderings and timeouts for
write response handlers, ReadCallback, paxos prepare/propose/commit, and
tracked read reconciliation. Adds the small test hooks needed to drive
them (isComplete, overridable isFromLocalDc, @VisibleForTesting ctors),
and fixes PaxosPrepare to set haveTrackedDataResponseIfNeeded in the
constructor rather than at start.
Patch by Blake Eggleston; reviewed by Francisco Guerrero and Ariel Weisberg for CASSANDRA-21106
Implement mutation tracking repair as a new repair task that replaces
Merkle tree validation and streaming for tracked keyspaces. Instead of
building hash trees and comparing data, MutationTrackingSyncCoordinator
sends MT_SYNC_REQ to all participants to collect their witnessed offsets,
then waits for background offset broadcasts to confirm all replicas have
reconciled to those target offsets.
Key changes:
- Add MutationTrackingIncrementalRepairTask that creates per-range
MutationTrackingSyncCoordinator instances and blocks until all
shards reach the target reconciled offsets or timeout.
- Add MT_SYNC_REQ/MT_SYNC_RSP verbs and MutationTrackingSyncRequest/
MutationTrackingSyncResponse messages for the repair protocol to
establish a happens-before relationship between repair start and
offset collection.
- RepairCoordinator.create() factory method snapshots TCM state to
decide whether to use mutation tracking repair and flips incremental
to false (skipping anti-compaction) when MT is active without
migration.
- Support mutation tracking migration: during untracked->tracked
migration, run incremental repair first (for pre-migration data),
then MT sync. KeyspaceMigrationInfo validates that repair ranges
don't partially overlap pending migration ranges and routes
streaming/SSTable finalization through the correct tracked vs
untracked path.
- Temporarily route read repair mutations through the untracked write
path during migration by adding isReadRepair flag to Mutation and
bypassing migration routing checks in ReadRepairVerbHandler and
CassandraKeyspaceWriteHandler. This is a stopgap; CASSANDRA-21252
will roll back this approach and handle read repair properly.
- Add offset collection APIs to CoordinatorLog, Node2OffsetsMap, and
Shard for computing union and intersection of witnessed offsets
scoped to specific participant host IDs (supporting --force with
dead node exclusion).
- Add configurable mutation_tracking_sync_timeout (default 2m) with
JMX get/set on StorageServiceMBean.
- Fix ActiveRepairService to use tryFailure() instead of setFailure()
to avoid double-completion exceptions during concurrent repair
failures.
Co-Authored-By: Ariel Weisberg <aweisberg@apple.com>
patch by Caleb Rackliffe; reviewed by Blake Eggleston and Sam Tunnicliffe for CASSANDRA-20926
Co-authored-by: Blake Eggleston <blake@ultrablake.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Tracked keyspaces cannot accept new data without first registering it in
the log. Any unreconciled data that isn't present in the log will break
read monotonicity, since mutation tracking uses a single data read and
can only read reconcile mutation IDs that are present in the log.
Full repair sync tasks also deliver data to replicas, and require
integration with the log just like imports do. The general design of this
integration is to give repair SyncTasks the same two-phase commit as
import transfers, where we stream SSTables to a pending directory, then
once sufficient streams complete successfully, we "activate" those
streams and move them out of the pending directory and into the live set.
patch by Caleb Rackliffe; reviewed by Abe Ratnofsky for CASSANDRA-21066
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Abe Ratnofsky <abe@aber.io>
Blake Eggleston