Implements SatelliteReplicationStrategy read/write/paxos plans with
quorum-of-quorums semantics, built on a new CompositeTracker (replacing
PerDcResponseTracker) that requires N of M child quorums to succeed.
Adds SatelliteFailoverState, tracking per-range NORMAL / TRANSITION_ACK /
TRANSITION state so replica layout and consistency requirements can vary
by failover phase. Paxos runs within the primary DC but includes
satellite/secondary summary nodes on prepare and commit via
SatellitePaxosParticipants; PaxosCommitRemoteMutationVerbHandler rejects
remote paxos commit mutations during TRANSITION_ACK so stale commits
can't leak into the other DCs during failover.
Patch by Blake Eggleston; reviewed by Francisco Guerrero and Ariel Weisberg for CASSANDRA-21106
Extracts coordination completion logic out of the read/write response
handlers into a ResponseTracker abstraction (simple, write, per-DC), and
pairs it with the ReplicaPlan in a CoordinationPlan so replica selection
and quorum requirements are always produced together by the replication
strategy from a single view of state.
Replication strategies now own construction of both halves. Write/read
paths, paxos, batchlog, read repair, and range reads are converted to
consume CoordinationPlan; ReplicaPlanIterator/Merger are renamed
accordingly. Tests included for each tracker implementation.
Patch by Blake Eggleston; reviewed by Francisco Guerrero and Ariel Weisberg for CASSANDRA-21106
Randomized tests covering response/failure orderings and timeouts for
write response handlers, ReadCallback, paxos prepare/propose/commit, and
tracked read reconciliation. Adds the small test hooks needed to drive
them (isComplete, overridable isFromLocalDc, @VisibleForTesting ctors),
and fixes PaxosPrepare to set haveTrackedDataResponseIfNeeded in the
constructor rather than at start.
Patch by Blake Eggleston; reviewed by Francisco Guerrero and Ariel Weisberg for CASSANDRA-21106
Implement mutation tracking repair as a new repair task that replaces
Merkle tree validation and streaming for tracked keyspaces. Instead of
building hash trees and comparing data, MutationTrackingSyncCoordinator
sends MT_SYNC_REQ to all participants to collect their witnessed offsets,
then waits for background offset broadcasts to confirm all replicas have
reconciled to those target offsets.
Key changes:
- Add MutationTrackingIncrementalRepairTask that creates per-range
MutationTrackingSyncCoordinator instances and blocks until all
shards reach the target reconciled offsets or timeout.
- Add MT_SYNC_REQ/MT_SYNC_RSP verbs and MutationTrackingSyncRequest/
MutationTrackingSyncResponse messages for the repair protocol to
establish a happens-before relationship between repair start and
offset collection.
- RepairCoordinator.create() factory method snapshots TCM state to
decide whether to use mutation tracking repair and flips incremental
to false (skipping anti-compaction) when MT is active without
migration.
- Support mutation tracking migration: during untracked->tracked
migration, run incremental repair first (for pre-migration data),
then MT sync. KeyspaceMigrationInfo validates that repair ranges
don't partially overlap pending migration ranges and routes
streaming/SSTable finalization through the correct tracked vs
untracked path.
- Temporarily route read repair mutations through the untracked write
path during migration by adding isReadRepair flag to Mutation and
bypassing migration routing checks in ReadRepairVerbHandler and
CassandraKeyspaceWriteHandler. This is a stopgap; CASSANDRA-21252
will roll back this approach and handle read repair properly.
- Add offset collection APIs to CoordinatorLog, Node2OffsetsMap, and
Shard for computing union and intersection of witnessed offsets
scoped to specific participant host IDs (supporting --force with
dead node exclusion).
- Add configurable mutation_tracking_sync_timeout (default 2m) with
JMX get/set on StorageServiceMBean.
- Fix ActiveRepairService to use tryFailure() instead of setFailure()
to avoid double-completion exceptions during concurrent repair
failures.
Co-Authored-By: Ariel Weisberg <aweisberg@apple.com>
patch by Caleb Rackliffe; reviewed by Blake Eggleston and Sam Tunnicliffe for CASSANDRA-20926
Co-authored-by: Blake Eggleston <blake@ultrablake.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Tracked keyspaces cannot accept new data without first registering it in
the log. Any unreconciled data that isn't present in the log will break
read monotonicity, since mutation tracking uses a single data read and
can only read reconcile mutation IDs that are present in the log.
Full repair sync tasks also deliver data to replicas, and require
integration with the log just like imports do. The general design of this
integration is to give repair SyncTasks the same two-phase commit as
import transfers, where we stream SSTables to a pending directory, then
once sufficient streams complete successfully, we "activate" those
streams and move them out of the pending directory and into the live set.
patch by Caleb Rackliffe; reviewed by Abe Ratnofsky for CASSANDRA-21066
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Abe Ratnofsky <abe@aber.io>
This patch enables mutation tracking for queries using secondary indexes (both
legacy 2i and SAI). Key changes include the addition of the MultiStepSearcher
interface, which allows existing index implementations to iterate over matches
and filter augmented data, proper Memtable snapshotting, and the integration of
these in PartialTrackedIndexRead.
patch by Blake Eggleston; reviewed by Aleksey Yeschenko and Caleb Rackliffe for CASSANDRA-20374
Co-authored-by: Blake Eggleston <blake@ultrablake.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Aleksey Yeschenko <aleksey@apache.org>
* Tracking of Memtable -> SSTable flushes in Mutation Journal segments
* Replay of Mutation Journal segments that were not fully flushed to SSTables
* Distinguish between CommitLog and MutationJournal offsets
* Effectively adds support for node bounces
Patch by Alex Petrov; reviewed by Aleksey Yeschenko for CASSANDRA-20919
- filter augmenting mutations and add comment explaining why we don't exclude reconciled mutations when augmenting
- propagate exceptions during acknowledgeReconcile
- fix UnreconciledMutations handling of single key bounds
- fix single offset remove bug
patch by Blake Eggleston; reviewed by Caleb Rackliffe for CASSANDRA-20830
Includes bug fixes and features:
- Improved observability in AutoRepair (CASSANDRA-20581)
- Stop repair scheduler if two major versions detected (CASSANDRA-20048)
- Safeguard Full repair against disk protection (CASSANDRA-20045)
- Stop AutoRepair monitoring thread upon shutdown (CASSANDRA-20623)
- Fix race condition in auto-repair scheduler (CASSANDRA-20265)
- Minimum repair task duration setting (CASSANDRA-20160)
- Preview_repaired auto-repair type (CASSANDRA-20046)
- Gate auto-repair behind cassandra.autorepair.enable JVM property
- Add cassandra.autorepair.check_min_version to gate minimum version enforcement
- Prevent auto-repair from running if any node is below 5.0.7
- Make system_distributed auto-repair schema conditional on feature being enabled
- Add user-friendly errors for disabled auto-repair and schema incompatibility
patch by Paulo Motta; reviewed by Andy Tolbert, Jaydeepkumar Chovatia for CASSANDRA-21138
Co-Authored-By: Andy Tolbert <andy_tolbert@apple.com>
Co-Authored-By: Chris Lohfink <clohfink@netflix.com>
Co-Authored-By: Francisco Guerrero <frankgh@apache.org>
Co-Authored-By: Himanshu Jindal <himanshj@amazon.com>
Co-Authored-By: Jaydeepkumar Chovatia <jchovati@uber.com>
Co-Authored-By: Kristijonas Zalys <kzalys@uber.com>
Co-Authored-By: jaydeepkumar1984 <chovatia.jaydeep@gmail.com>
this diff was supposed to be included in 5f9de8ca13
Rate limit password changes
Introduces property cassandra.role_password_update_interval_in_ms, with default value of 5000.
Logging added when passwords on roles are changed or are rate-limited.
patch by Mick Semb Wever; reviewed by Doug Rohrer, Jeremiah Jordan for CASSANDRA-21202
Blake Eggleston