- Filter interval-tree false positives: skip SSTables that intersect a
shard range but contain no partitions within it, precomputing and
reusing partition positions per SSTable
- Register PendingLocalTransfer once in finished() instead of per-SSTable
in received(), fixing an assertion failure when streaming >1 SSTable
- Guard against null currentWriter in RangeAwareSSTableWriter.setOpenResult
- Schedule local cleanup before notifyFailure so cleanup always runs on
the failure path, and remove completed transfers from the local map
- Adds dtests for the interval-tree false positive and multi-SSTable import
cases.
Patch by Alan Wang; Reviewed by Blake Eggleston for CASSANDRA-21470
Don't truncate journal segments until witnessed offsets they contain are
flushed. Also moves MutationTrackingService startup to after the commit
log is replayed
Patch by Blake Eggleston; Reviewed by Francisco Guerrero for CASSANDRA-21443
During mutation tracking migration (tracked <-> untracked), the per-range
migration state must be consulted for every routing decision. Before this
change, all Paxos V1 and V2 paths used the static keyspace-level
replicationType().isTracked() check, which does not reflect per-range
migration state and produces incorrect routing decisions during migration.
Coordinator-side routing: Replace all static isTracked() checks with
MigrationRouter calls across StorageProxy (commitPaxos, sendCommit,
isTrackedKeyspaceRequiringPaxosCommitForwarding, checkAndForwardCasIfNeeded,
checkAndForwardConsensusReadIfNeeded), PaxosCommit (constructor,
isTrackedKeyspaceRequiringForwarding), PaxosCommitAndPrepare,
PaxosPrepare (start + isTracked field), PaxosPrepareRefresh, and
PaxosState truncation acknowledgment.
Handler-side validation: Add migration state validation to four Paxos
replica handlers that receive messages carrying mutations or tracked reads:
PaxosCommit.RequestHandler (direct V1/V2 commits), PaxosPrepare.RequestHandler
(V2 prepare with tracked read), PaxosCommitAndPrepare.RequestHandler
(combined commit+prepare), and PaxosPrepareRefresh.RequestHandler (refresh
commits). Each uses the conditional-fetch pattern from
AbstractMutationVerbHandler.checkReplicationMigration: compare the
coordinator routing decision against the handler MigrationRouter result,
fetch only on mismatch when the coordinator epoch is ahead (handler is
behind and needs to catch up), throw CoordinatorBehindException when the
coordinator epoch is behind.
Coordinator-side commit retry: Add commit-level COORDINATOR_BEHIND retry
in Paxos.cas() (V2) and commitPaxos() (V1). When replicas reject a commit
due to migration state mismatch, ResponseVerbHandler.maybeFetchLogs()
catches up the coordinator synchronously before delivering the failure.
The retry re-creates the commit with fresh MigrationRouter routing. This
retries only the commit phase, not the entire prepare+propose protocol.
Stale mutation ID reconciliation: Commits saved in system.paxos may have
a mutation ID from when the keyspace was tracked. When replayed after
migration to untracked (via PaxosPrepareRefresh, PaxosCommitAndPrepare,
sendCommit, or commitPaxos), the stale ID must be stripped to avoid
Keyspace.apply() rejecting the mutation. Uses Commit.withMutationId() to
reconcile in all four replay paths.
Forward handlers: Forwarding is harmless -- the receiving replica
re-executes the full CAS/read with its own fresh routing decisions, so no
migration validation is needed at the forward boundary itself. Removed the
"reject if keyspace not tracked" guards from CasForwardHandler and
ConsensusReadForwardHandler (forwarding is now valid in either direction).
Replaced unconditional fetchLogFromPeerOrCMS with the conditional-fetch
pattern in Paxos2CommitForwardHandler, PaxosCommitForwardHandler,
PrepareRefreshForwardHandler, and PaxosCommitAndPrepare.RequestHandler
(unconditional fetch added unnecessary latency on the no-mismatch case).
PaxosCommit failure tracking: Added super.onFailure() call to
PaxosCommit.onFailure() so FailureRecordingCallback.failureResponses is
populated, enabling failureReasonsAsMap() to return actual failure reasons.
This was required for the V2 commit retry to detect COORDINATOR_BEHIND
in MaybeFailure.failures.
PaxosCommit hint suppression: Tracked mutations must not be written as
hints because hint replay routes through Keyspace.applyInternalTracked()
based on the mutation ID presence, which fails after migration to untracked.
Guard submitHint with !isTracked() -- tracked mutations use
MutationTrackingService for retries, not the hint system.
MigrationRouter null safety: Replace getKeyspaceMetadata() (throws
NoSuchElementException on missing keyspace) with
maybeGetKeyspaceMetadata().orElse(null) at four call sites so the existing
null guards actually protect against concurrent keyspace drops.
patch by Blake Eggleston and Aleksey Yeschenko; reviewed by Aleksey
Yeschenko and Blake Eggleston for CASSANDRA-21328
Co-authored-by: Aleksey Yeschenko <aleksey@yeschenko.com>
Co-authored-by: Blake Eggleston <blake@ultrablake.com>
During migration from untracked to tracked replication, blocking read
repair (BRR) continues to run because mutation tracking lacks sufficient
initialized state to provide monotonic reads for pending ranges. However,
BRR was previously bypassing mutation tracking entirely — read repair
mutations were sent as untracked writes via READ_REPAIR_REQ, which meant
they were invisible to the mutation journal. This is unsafe because the
migration repair that completes the transition assumes all writes after it
started are tracked, so that tracked reads can safely begin afterward.
Route BRR writes through TrackedWriteRequest.perform() when
MigrationRouter.shouldUseTrackedForWrites() indicates the token is
migrating. This gives each read repair mutation a proper MutationId and
records it in the mutation journal, ensuring the migration invariant holds.
Key changes:
- Remove the isReadRepair flag from Mutation and all special-case bypasses
that let read repair skip mutation tracking validation and routing
- Add repairViaTrackedWrite() to BlockingReadRepair which sends each
per-replica mutation as an independent tracked write with retry logic
for migration races (RetryOnDifferentSystemException/CoordinatorBehindException)
- Introduce WriteCallback interface (replacing Runnable) on write response
handlers so tracked write completion can be observed without races
- Make tracked→untracked instant in AlterSchema by skipping migration
state entirely — tracked writes are already quorum writes so untracked
reads are strictly less demanding. Reverting an in-progress
untracked→tracked migration cleans up the now-unnecessary state.
- Add RepairedBlockingViaTrackedWrite metric to observe BRR-via-tracked path
- Add pauseRegularPriorityForTesting() to ActiveLogReconciler to allow
tests to suppress background reconciliation while keeping high-priority
(tracked read) reconciliation active
- Add MutationTrackingReadRepairTest covering BRR behavior across all
migration phases with both point reads and range scans
Implement mutation tracking repair as a new repair task that replaces
Merkle tree validation and streaming for tracked keyspaces. Instead of
building hash trees and comparing data, MutationTrackingSyncCoordinator
sends MT_SYNC_REQ to all participants to collect their witnessed offsets,
then waits for background offset broadcasts to confirm all replicas have
reconciled to those target offsets.
Key changes:
- Add MutationTrackingIncrementalRepairTask that creates per-range
MutationTrackingSyncCoordinator instances and blocks until all
shards reach the target reconciled offsets or timeout.
- Add MT_SYNC_REQ/MT_SYNC_RSP verbs and MutationTrackingSyncRequest/
MutationTrackingSyncResponse messages for the repair protocol to
establish a happens-before relationship between repair start and
offset collection.
- RepairCoordinator.create() factory method snapshots TCM state to
decide whether to use mutation tracking repair and flips incremental
to false (skipping anti-compaction) when MT is active without
migration.
- Support mutation tracking migration: during untracked->tracked
migration, run incremental repair first (for pre-migration data),
then MT sync. KeyspaceMigrationInfo validates that repair ranges
don't partially overlap pending migration ranges and routes
streaming/SSTable finalization through the correct tracked vs
untracked path.
- Temporarily route read repair mutations through the untracked write
path during migration by adding isReadRepair flag to Mutation and
bypassing migration routing checks in ReadRepairVerbHandler and
CassandraKeyspaceWriteHandler. This is a stopgap; CASSANDRA-21252
will roll back this approach and handle read repair properly.
- Add offset collection APIs to CoordinatorLog, Node2OffsetsMap, and
Shard for computing union and intersection of witnessed offsets
scoped to specific participant host IDs (supporting --force with
dead node exclusion).
- Add configurable mutation_tracking_sync_timeout (default 2m) with
JMX get/set on StorageServiceMBean.
- Fix ActiveRepairService to use tryFailure() instead of setFailure()
to avoid double-completion exceptions during concurrent repair
failures.
Co-Authored-By: Ariel Weisberg <aweisberg@apple.com>
patch by Caleb Rackliffe; reviewed by Blake Eggleston and Sam Tunnicliffe for CASSANDRA-20926
Co-authored-by: Blake Eggleston <blake@ultrablake.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Tracked keyspaces cannot accept new data without first registering it in
the log. Any unreconciled data that isn't present in the log will break
read monotonicity, since mutation tracking uses a single data read and
can only read reconcile mutation IDs that are present in the log.
Full repair sync tasks also deliver data to replicas, and require
integration with the log just like imports do. The general design of this
integration is to give repair SyncTasks the same two-phase commit as
import transfers, where we stream SSTables to a pending directory, then
once sufficient streams complete successfully, we "activate" those
streams and move them out of the pending directory and into the live set.
patch by Caleb Rackliffe; reviewed by Abe Ratnofsky for CASSANDRA-21066
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Abe Ratnofsky <abe@aber.io>
This patch enables mutation tracking for queries using secondary indexes (both
legacy 2i and SAI). Key changes include the addition of the MultiStepSearcher
interface, which allows existing index implementations to iterate over matches
and filter augmented data, proper Memtable snapshotting, and the integration of
these in PartialTrackedIndexRead.
patch by Blake Eggleston; reviewed by Aleksey Yeschenko and Caleb Rackliffe for CASSANDRA-20374
Co-authored-by: Blake Eggleston <blake@ultrablake.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Aleksey Yeschenko <aleksey@apache.org>
* Tracking of Memtable -> SSTable flushes in Mutation Journal segments
* Replay of Mutation Journal segments that were not fully flushed to SSTables
* Distinguish between CommitLog and MutationJournal offsets
* Effectively adds support for node bounces
Patch by Alex Petrov; reviewed by Aleksey Yeschenko for CASSANDRA-20919
- filter augmenting mutations and add comment explaining why we don't exclude reconciled mutations when augmenting
- propagate exceptions during acknowledgeReconcile
- fix UnreconciledMutations handling of single key bounds
- fix single offset remove bug
patch by Blake Eggleston; reviewed by Caleb Rackliffe for CASSANDRA-20830
Includes bug fixes and features:
- Improved observability in AutoRepair (CASSANDRA-20581)
- Stop repair scheduler if two major versions detected (CASSANDRA-20048)
- Safeguard Full repair against disk protection (CASSANDRA-20045)
- Stop AutoRepair monitoring thread upon shutdown (CASSANDRA-20623)
- Fix race condition in auto-repair scheduler (CASSANDRA-20265)
- Minimum repair task duration setting (CASSANDRA-20160)
- Preview_repaired auto-repair type (CASSANDRA-20046)
- Gate auto-repair behind cassandra.autorepair.enable JVM property
- Add cassandra.autorepair.check_min_version to gate minimum version enforcement
- Prevent auto-repair from running if any node is below 5.0.7
- Make system_distributed auto-repair schema conditional on feature being enabled
- Add user-friendly errors for disabled auto-repair and schema incompatibility
patch by Paulo Motta; reviewed by Andy Tolbert, Jaydeepkumar Chovatia for CASSANDRA-21138
Co-Authored-By: Andy Tolbert <andy_tolbert@apple.com>
Co-Authored-By: Chris Lohfink <clohfink@netflix.com>
Co-Authored-By: Francisco Guerrero <frankgh@apache.org>
Co-Authored-By: Himanshu Jindal <himanshj@amazon.com>
Co-Authored-By: Jaydeepkumar Chovatia <jchovati@uber.com>
Co-Authored-By: Kristijonas Zalys <kzalys@uber.com>
Co-Authored-By: jaydeepkumar1984 <chovatia.jaydeep@gmail.com>
Alan Wang