natas_hw
/
cassandra
mirror of https://github.com/apache/cassandra
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Paxos support for live migration to mutation tracking 

During mutation tracking migration (tracked <-> untracked), the per-range
migration state must be consulted for every routing decision. Before this
change, all Paxos V1 and V2 paths used the static keyspace-level
replicationType().isTracked() check, which does not reflect per-range
migration state and produces incorrect routing decisions during migration.

Coordinator-side routing: Replace all static isTracked() checks with
MigrationRouter calls across StorageProxy (commitPaxos, sendCommit,
isTrackedKeyspaceRequiringPaxosCommitForwarding, checkAndForwardCasIfNeeded,
checkAndForwardConsensusReadIfNeeded), PaxosCommit (constructor,
isTrackedKeyspaceRequiringForwarding), PaxosCommitAndPrepare,
PaxosPrepare (start + isTracked field), PaxosPrepareRefresh, and
PaxosState truncation acknowledgment.

Handler-side validation: Add migration state validation to four Paxos
replica handlers that receive messages carrying mutations or tracked reads:
PaxosCommit.RequestHandler (direct V1/V2 commits), PaxosPrepare.RequestHandler
(V2 prepare with tracked read), PaxosCommitAndPrepare.RequestHandler
(combined commit+prepare), and PaxosPrepareRefresh.RequestHandler (refresh
commits). Each uses the conditional-fetch pattern from
AbstractMutationVerbHandler.checkReplicationMigration: compare the
coordinator routing decision against the handler MigrationRouter result,
fetch only on mismatch when the coordinator epoch is ahead (handler is
behind and needs to catch up), throw CoordinatorBehindException when the
coordinator epoch is behind.

Coordinator-side commit retry: Add commit-level COORDINATOR_BEHIND retry
in Paxos.cas() (V2) and commitPaxos() (V1). When replicas reject a commit
due to migration state mismatch, ResponseVerbHandler.maybeFetchLogs()
catches up the coordinator synchronously before delivering the failure.
The retry re-creates the commit with fresh MigrationRouter routing. This
retries only the commit phase, not the entire prepare+propose protocol.

Stale mutation ID reconciliation: Commits saved in system.paxos may have
a mutation ID from when the keyspace was tracked. When replayed after
migration to untracked (via PaxosPrepareRefresh, PaxosCommitAndPrepare,
sendCommit, or commitPaxos), the stale ID must be stripped to avoid
Keyspace.apply() rejecting the mutation. Uses Commit.withMutationId() to
reconcile in all four replay paths.

Forward handlers: Forwarding is harmless -- the receiving replica
re-executes the full CAS/read with its own fresh routing decisions, so no
migration validation is needed at the forward boundary itself. Removed the
"reject if keyspace not tracked" guards from CasForwardHandler and
ConsensusReadForwardHandler (forwarding is now valid in either direction).
Replaced unconditional fetchLogFromPeerOrCMS with the conditional-fetch
pattern in Paxos2CommitForwardHandler, PaxosCommitForwardHandler,
PrepareRefreshForwardHandler, and PaxosCommitAndPrepare.RequestHandler
(unconditional fetch added unnecessary latency on the no-mismatch case).

PaxosCommit failure tracking: Added super.onFailure() call to
PaxosCommit.onFailure() so FailureRecordingCallback.failureResponses is
populated, enabling failureReasonsAsMap() to return actual failure reasons.
This was required for the V2 commit retry to detect COORDINATOR_BEHIND
in MaybeFailure.failures.

PaxosCommit hint suppression: Tracked mutations must not be written as
hints because hint replay routes through Keyspace.applyInternalTracked()
based on the mutation ID presence, which fails after migration to untracked.
Guard submitHint with !isTracked() -- tracked mutations use
MutationTrackingService for retries, not the hint system.

MigrationRouter null safety: Replace getKeyspaceMetadata() (throws
NoSuchElementException on missing keyspace) with
maybeGetKeyspaceMetadata().orElse(null) at four call sites so the existing
null guards actually protect against concurrent keyspace drops.
This commit is contained in:
Ariel Weisberg 2026-04-21 12:43:47 -04:00 committed by Blake Eggleston
parent e1ae7d654a
commit 595b158958
32 changed files with 4723 additions and 444 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
src/java/org/apache/cassandra/db/Keyspace.java
View File

@ -50,6 +50,7 @@ import org.apache.cassandra.db.view.ViewManager;
import org.apache.cassandra.db.virtual.VirtualKeyspaceRegistry;
import org.apache.cassandra.dht.Range;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.exceptions.CoordinatorBehindException;
import org.apache.cassandra.exceptions.WriteTimeoutException;
import org.apache.cassandra.index.Index;
import org.apache.cassandra.index.SecondaryIndexManager;
@ -617,7 +618,10 @@ public class Keyspace
private Future<?> applyInternalTracked(Mutation mutation, Promise<?> future)
{
MutationTrackingService.ensureEnabled();
Preconditions.checkState(MigrationRouter.isFullyTracked(mutation) && !mutation.id().isNone());
if (!MigrationRouter.isFullyTracked(mutation) || mutation.id().isNone())
throw new CoordinatorBehindException("Mutation routing mismatch in applyInternalTracked: isFullyTracked=" +
MigrationRouter.isFullyTracked(mutation) + ", id.isNone=" + mutation.id().isNone() +
", keyspace=" + mutation.getKeyspaceName());
ClusterMetadata cm = ClusterMetadata.current();
if (TEST_FAIL_WRITES && getMetadata().name.equals(TEST_FAIL_WRITES_KS))

2
src/java/org/apache/cassandra/exceptions/CoordinatorBehindException.java
View File

@ -25,7 +25,7 @@ public class CoordinatorBehindException extends RuntimeException
super(msg);
}
public CoordinatorBehindException(String msg, UnknownTableException cause)
public CoordinatorBehindException(String msg, Throwable cause)
{
super(msg, cause);
}

6
src/java/org/apache/cassandra/exceptions/WriteTimeoutException.java
View File

@ -42,6 +42,12 @@ public class WriteTimeoutException extends RequestTimeoutException
this.writeType = writeType;
}
public WriteTimeoutException(WriteType writeType, ConsistencyLevel consistency, int received, int blockFor, Throwable cause)
{
super(ExceptionCode.WRITE_TIMEOUT, consistency, received, blockFor, cause);
this.writeType = writeType;
}
@Override
protected void serializeSpecificFields(DataOutputPlus out, int version) throws IOException
{

11
src/java/org/apache/cassandra/hints/HintsService.java
View File

@ -73,6 +73,14 @@ import static org.apache.cassandra.config.CassandraRelevantProperties.HINT_DISPA
*/
public final class HintsService implements HintsServiceMBean
{
private static long REJECT_HINTS_BEFORE_NANOS;
@VisibleForTesting
public static void setRejectHintsBeforeNanos(long nanos)
{
REJECT_HINTS_BEFORE_NANOS = nanos;
}
// Dummy address to use for storing metrics for hints that will be retried on a different transaction system
// and aren't being sent to a specific node
public static final InetAddressAndPort RETRY_ON_DIFFERENT_SYSTEM_ADDRESS;
@ -189,6 +197,9 @@ public final class HintsService implements HintsServiceMBean
if (isShutDown)
throw new IllegalStateException("HintsService is shut down and can't accept new hints");
if (hint.mutation.getApproxCreatedAtNanos() < REJECT_HINTS_BEFORE_NANOS)
return;
// we have to make sure that the HintsStore instances get properly initialized - otherwise dispatch will not trigger
catalog.maybeLoadStores(hostIds);

35
src/java/org/apache/cassandra/replication/ActiveLogReconciler.java
View File

@ -59,7 +59,40 @@ public final class ActiveLogReconciler implements Shutdownable
{
private static final Logger logger = LoggerFactory.getLogger(ActiveLogReconciler.class);
public enum Priority { HIGH, REGULAR }
public enum Priority
{
HIGH(0),
REGULAR(1);
public final int id;
private static final Priority[] idMapping;
static
{
int maxId = -1;
for (Priority p : values())
maxId = Math.max(maxId, p.id);
idMapping = new Priority[maxId + 1];
for (Priority p : values())
{
if (idMapping[p.id] != null)
throw new IllegalStateException("Duplicate Priority id " + p.id);
idMapping[p.id] = p;
}
}
Priority(int id)
{
this.id = id;
}
public static Priority fromId(int id)
{
if (id < 0 || id >= idMapping.length || idMapping[id] == null)
throw new IllegalArgumentException("Unknown Priority id: " + id);
return idMapping[id];
}
}
// prioritised delivery of mutations that are needed by reads;
private final ManyToOneConcurrentLinkedQueue<Task> highPriorityTasks;

6
src/java/org/apache/cassandra/replication/MutationTrackingService.java
View File

@ -787,9 +787,9 @@ public class MutationTrackingService implements MutationTrackingServiceMBean
}
}
public void requestMissingMutations(Offsets offsets, InetAddressAndPort forHost)
public void requestMissingMutations(Offsets offsets, InetAddressAndPort forHost, ActiveLogReconciler.Priority priority)
{
activeReconciler.schedule(offsets, forHost, ActiveLogReconciler.Priority.HIGH);
activeReconciler.schedule(offsets, forHost, priority);
}
@Nullable
@ -1496,7 +1496,7 @@ public class MutationTrackingService implements MutationTrackingServiceMBean
}
// TODO (expected): backoff, rate limits, per host and total
PullMutationsRequest request = new PullMutationsRequest(offsets);
PullMutationsRequest request = new PullMutationsRequest(offsets, ActiveLogReconciler.Priority.REGULAR);
logger.trace("Requesting pull mutation request from replica {} for missing offset {}", pullFrom, offsets);
MessagingService.instance().send(Message.out(Verb.MT_PULL_MUTATIONS_REQ, request), pullFrom);
}

17
src/java/org/apache/cassandra/replication/PullMutationsRequest.java
View File

@ -33,10 +33,12 @@ public final class PullMutationsRequest
private static final Logger logger = LoggerFactory.getLogger(PullMutationsRequest.class);
private final Offsets.Immutable offsets;
private final ActiveLogReconciler.Priority priority;
public PullMutationsRequest(Offsets.Immutable offsets)
public PullMutationsRequest(Offsets.Immutable offsets, ActiveLogReconciler.Priority priority)
{
this.offsets = offsets;
this.priority = priority;
}
public static final UnversionedSerializer<PullMutationsRequest> serializer = new UnversionedSerializer<>()
@ -45,18 +47,21 @@ public final class PullMutationsRequest
public void serialize(PullMutationsRequest pull, DataOutputPlus out) throws IOException
{
Offsets.serializer.serialize(pull.offsets, out);
out.writeByte(pull.priority.id);
}
@Override
public PullMutationsRequest deserialize(DataInputPlus in) throws IOException
{
return new PullMutationsRequest(Offsets.serializer.deserialize(in));
Offsets.Immutable offsets = Offsets.serializer.deserialize(in);
ActiveLogReconciler.Priority priority = ActiveLogReconciler.Priority.fromId(in.readUnsignedByte());
return new PullMutationsRequest(offsets, priority);
}
@Override
public long serializedSize(PullMutationsRequest pull)
{
return Offsets.serializer.serializedSize(pull.offsets);
return Offsets.serializer.serializedSize(pull.offsets) + 1;
}
};
@ -64,8 +69,9 @@ public final class PullMutationsRequest
MutationTrackingService.ensureEnabled();
InetAddressAndPort forHost = message.from();
Offsets offsets = message.payload.offsets;
logger.trace("Received pull mutations request from {} for {}", forHost, offsets);
MutationTrackingService.instance().requestMissingMutations(offsets, forHost);
ActiveLogReconciler.Priority priority = message.payload.priority;
logger.trace("Received pull mutations request from {} for {} with priority {}", forHost, offsets, priority);
MutationTrackingService.instance().requestMissingMutations(offsets, forHost, priority);
};
@Override
@ -73,6 +79,7 @@ public final class PullMutationsRequest
{
return "PullMutationsRequest{" +
"offsets=" + offsets +
", priority=" + priority +
'}';
}
}

190
src/java/org/apache/cassandra/service/StorageProxy.java
View File

@ -111,6 +111,7 @@ import org.apache.cassandra.exceptions.ReadFailureException;
import org.apache.cassandra.exceptions.ReadTimeoutException;
import org.apache.cassandra.exceptions.RequestFailure;
import org.apache.cassandra.exceptions.RequestFailureException;
import org.apache.cassandra.exceptions.RequestFailureReason;
import org.apache.cassandra.exceptions.RequestTimeoutException;
import org.apache.cassandra.exceptions.RetryOnDifferentSystemException;
import org.apache.cassandra.exceptions.UnavailableException;
@ -140,11 +141,11 @@ import org.apache.cassandra.net.MessageFlag;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.net.NoPayload;
import org.apache.cassandra.net.RequestCallback;
import org.apache.cassandra.net.RequestCallbackWithFailure;
import org.apache.cassandra.net.Verb;
import org.apache.cassandra.replication.MutationId;
import org.apache.cassandra.replication.MutationTrackingService;
import org.apache.cassandra.replication.TrackedWriteRequest;
import org.apache.cassandra.schema.KeyspaceMetadata;
import org.apache.cassandra.schema.PartitionDenylist;
import org.apache.cassandra.schema.Schema;
import org.apache.cassandra.schema.SchemaConstants;
@ -189,6 +190,7 @@ import org.apache.cassandra.service.reads.ReadCoordinator;
import org.apache.cassandra.service.reads.ReadExecutor;
import org.apache.cassandra.service.reads.range.RangeCommands;
import org.apache.cassandra.service.reads.repair.ReadRepair;
import org.apache.cassandra.service.replication.migration.MigrationRouter;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.membership.NodeState;
import org.apache.cassandra.tcm.ownership.VersionedEndpoints;
@ -800,7 +802,7 @@ public class StorageProxy implements StorageProxyMBean
{
Tracing.trace("Finishing incomplete paxos round {}", inProgress);
casMetrics.unfinishedCommit.inc();
Commit refreshedInProgress = Commit.newProposal(ballot, inProgress.update);
Commit refreshedInProgress = Commit.newProposal(ballot, inProgress.mutation);
if (proposePaxos(refreshedInProgress, paxosPlan, false, requestTime))
{
commitPaxos(refreshedInProgress, consistencyForCommit, false, requestTime);
@ -844,16 +846,29 @@ public class StorageProxy implements StorageProxyMBean
}
/**
* Unlike commitPaxos, this does not wait for replies.
* For tracked keyspaces, ensures proper mutation ID generation and tracking.
* Unlike commitPaxos, this does not wait for commit application replies.
* For tracked keyspaces where the local node is not a replica, waits for the
* forwarding coordinator to acknowledge dispatch.
*/
private static void sendCommit(Commit commit, Iterable<InetAddressAndPort> targetReplicas, ReplicaPlan.ForPaxosWrite replicaPlan)
{
String ksName = commit.metadata().keyspace;
KeyspaceMetadata ksMetadata = Schema.instance.getKeyspaceMetadata(ksName);
ClusterMetadata cm = ClusterMetadata.current();
boolean shouldBeTracked = MigrationRouter.shouldUseTrackedForWrites(cm, ksName, commit.metadata().id, commit.partitionKey().getToken());
// Reconcile the mutation's ID with the current migration state.
// The commit may have been saved to system.paxos under a different replication type.
if (!shouldBeTracked && !commit.mutation.id().isNone())
{
logger.warn("Stripping mutation ID {} from V2 paxos sendCommit for {}.{} partition {} - keyspace migrated to untracked",
commit.mutation.id(), ksName, commit.metadata().name, commit.partitionKey());
Tracing.trace("Stripping mutation ID {} from V2 paxos sendCommit for {}.{} partition {} - keyspace migrated to untracked",
commit.mutation.id(), ksName, commit.metadata().name, commit.partitionKey());
commit = commit.withMutationId(MutationId.none());
}
// Non-tracked keyspaces OR already has mutation ID: fire and forget to target replicas
if (ksMetadata == null || !ksMetadata.params.replicationType.isTracked() || !commit.mutation.id().isNone())
if (!shouldBeTracked || !commit.mutation.id().isNone())
{
Message<Commit> message = Message.out(PAXOS_COMMIT_REQ, commit);
for (InetAddressAndPort target : targetReplicas)
@ -887,30 +902,23 @@ public class StorageProxy implements StorageProxyMBean
private static void forwardPaxosCommit(Commit commit, ReplicaPlan.ForPaxosWrite replicaPlan)
{
InetAddressAndPort localEndpoint = FBUtilities.getBroadcastAddressAndPort();
// Get live replicas (excluding local node) to find a coordinator
EndpointsForToken liveReplicas = replicaPlan.live().filter(replica -> !replica.endpoint().equals(localEndpoint));
if (liveReplicas.isEmpty())
{
logger.warn("No live replicas available to forward Paxos commit for tracked keyspace");
logger.warn("No live replicas available to forward Paxos commit for {}.{} partition {}",
commit.metadata().keyspace, commit.metadata().name, commit.partitionKey());
return;
}
// Sort by proximity and select the closest as coordinator
EndpointsForToken sortedReplicas = DatabaseDescriptor.getNodeProximity().sortedByProximity(localEndpoint, liveReplicas);
InetAddressAndPort replicaCoordinator = sortedReplicas.get(0).endpoint();
Tracing.trace("Forwarding Paxos commit to replica coordinator {}", replicaCoordinator);
// Use respondAfterSend=true so coordinator responds after sending commits (not waiting for application)
// respondAfterSend=true so coordinator responds after sending commits (not waiting for application)
PaxosCommitForwardRequest forwardRequest = new PaxosCommitForwardRequest(commit, replicaPlan.consistencyLevel(), true);
Message<PaxosCommitForwardRequest> message = Message.out(PAXOS_COMMIT_FORWARD_REQ, forwardRequest);
// Wait for coordinator to confirm commits were sent before returning
Promise<NoPayload> promise = new AsyncPromise<>();
RequestCallback<NoPayload> callback = new RequestCallback<NoPayload>()
RequestCallbackWithFailure<NoPayload> callback = new RequestCallbackWithFailure<NoPayload>()
{
@Override
public void onResponse(Message<NoPayload> response)
@ -921,21 +929,27 @@ public class StorageProxy implements StorageProxyMBean
@Override
public void onFailure(InetAddressAndPort from, RequestFailure reason)
{
promise.setFailure(new RuntimeException("Failed to forward Paxos commit to " + from + ": " + reason));
if (reason.reason == RequestFailureReason.COORDINATOR_BEHIND)
promise.setFailure(new CoordinatorBehindException("Failed to forward Paxos commit to " + from + ": " + reason, reason.failure));
else
promise.setFailure(new RuntimeException("Failed to forward Paxos commit to " + from + ": " + reason, reason.failure));
}
};
MessagingService.instance().sendWithCallback(message, replicaCoordinator, callback);
try
{
// Wait for coordinator to confirm commits were sent
promise.get(DatabaseDescriptor.getWriteRpcTimeout(MILLISECONDS), MILLISECONDS);
}
catch (TimeoutException e)
{
logger.warn("Timeout waiting for forwarded Paxos commit response from {}", replicaCoordinator);
}
catch (InterruptedException e)
{
Thread.currentThread().interrupt();
throw new UncheckedInterruptedException(e);
}
catch (Exception e)
{
logger.warn("Error waiting for forwarded Paxos commit response from {}", replicaCoordinator, e);
@ -1026,35 +1040,56 @@ public class StorageProxy implements StorageProxyMBean
private static void commitPaxos(Commit proposal, ConsistencyLevel consistencyLevel, boolean allowHints, Dispatcher.RequestTime requestTime) throws WriteTimeoutException
{
checkArgument(!proposal.isEmpty());
// Check if this is a tracked keyspace
String keyspaceName = proposal.metadata().keyspace;
Keyspace keyspace = Keyspace.openIfExists(keyspaceName);
if (keyspace == null)
throw new KeyspaceNotDefinedException("Keyspace " + keyspaceName + " does not exist");
KeyspaceMetadata ksMetadata = keyspace.getMetadata();
if (ksMetadata.params.replicationType.isTracked())
long deadline = requestTime.computeDeadline(DatabaseDescriptor.getWriteRpcTimeout(NANOSECONDS));
while (nanoTime() < deadline)
{
// For tracked keyspaces, check if we need to forward or execute directly
Token tk = proposal.partitionKey().getToken();
ReplicaPlan.ForWrite replicaPlan = ReplicaPlans.forWrite(keyspace, consistencyLevel, tk, ReplicaPlans.writeAll);
if (isTrackedKeyspaceRequiringPaxosCommitForwarding(ksMetadata, proposal, replicaPlan.liveAndDown()))
try
{
// Forward to a replica coordinator
forwardPaxosCommit(proposal, consistencyLevel, replicaPlan);
Token tk = proposal.partitionKey().getToken();
ClusterMetadata cm = ClusterMetadata.current();
boolean shouldBeTracked = MigrationRouter.shouldUseTrackedForWrites(cm, keyspaceName, proposal.metadata().id, tk);
// Reconcile the mutation's ID with the current migration state.
Commit reconciled = proposal;
if (!shouldBeTracked && !proposal.mutation.id().isNone())
{
logger.warn("Stripping mutation ID {} from V1 paxos commit for {}.{} partition {} - keyspace migrated to untracked",
proposal.mutation.id(), keyspaceName, proposal.metadata().name, proposal.partitionKey());
Tracing.trace("Stripping mutation ID {} from V1 paxos commit for {}.{} partition {} - keyspace migrated to untracked",
proposal.mutation.id(), keyspaceName, proposal.metadata().name, proposal.partitionKey());
reconciled = proposal.withMutationId(MutationId.none());
}
if (shouldBeTracked)
{
ReplicaPlan.ForWrite replicaPlan = ReplicaPlans.forWrite(keyspace, consistencyLevel, tk, ReplicaPlans.writeAll);
if (requiresPaxosCommitForwarding(replicaPlan.liveAndDown()))
forwardPaxosCommit(reconciled, consistencyLevel, replicaPlan);
else
commitPaxosTracked(keyspace, reconciled, consistencyLevel, requestTime);
}
else
{
commitPaxosUntracked(keyspace, reconciled, consistencyLevel, allowHints, requestTime);
}
return;
}
else
catch (CoordinatorBehindException e)
{
// Execute directly using tracked logic
commitPaxosTracked(keyspace, proposal, consistencyLevel, requestTime);
casWriteMetrics.retryCoordinatorBehind.mark();
logger.warn("Retrying V1 Paxos commit after COORDINATOR_BEHIND for {}.{} partition {}",
keyspaceName, proposal.metadata().name, proposal.partitionKey());
Tracing.trace("Retrying V1 Paxos commit after COORDINATOR_BEHIND for {}.{} partition {}",
keyspaceName, proposal.metadata().name, proposal.partitionKey());
}
}
else
{
// For untracked keyspaces, use existing logic
commitPaxosUntracked(keyspace, proposal, consistencyLevel, allowHints, requestTime);
}
throw new WriteTimeoutException(WriteType.CAS, consistencyLevel, 0, consistencyLevel.blockFor(keyspace.getReplicationStrategy()));
}
public static void commitPaxosTracked(Keyspace keyspace, Commit proposal, ConsistencyLevel consistencyLevel, Dispatcher.RequestTime requestTime) throws WriteTimeoutException
@ -1152,7 +1187,7 @@ public class StorageProxy implements StorageProxyMBean
responseHandler.get();
}
private static void commitPaxosUntracked(Keyspace keyspace, Commit proposal, ConsistencyLevel consistencyLevel, boolean allowHints, Dispatcher.RequestTime requestTime) throws WriteTimeoutException
public static void commitPaxosUntracked(Keyspace keyspace, Commit proposal, ConsistencyLevel consistencyLevel, boolean allowHints, Dispatcher.RequestTime requestTime) throws WriteTimeoutException
{
boolean shouldBlock = consistencyLevel != ConsistencyLevel.ANY;
PartitionUpdate update = proposal.update;
@ -1245,48 +1280,35 @@ public class StorageProxy implements StorageProxyMBean
}
/**
* Checks if this commit needs to be forwarded to a replica coordinator for tracked keyspace support.
* Returns true if the local node is not a participant and the operation must be forwarded to a replica coordinator.
*/
private static boolean isTrackedKeyspaceRequiringPaxosCommitForwarding(KeyspaceMetadata ksMetadata, Commit proposal, EndpointsForToken participants)
private static boolean requiresPaxosCommitForwarding(EndpointsForToken participants)
{
if (!ksMetadata.params.replicationType.isTracked())
return false;
// Check if current coordinator is not a replica
InetAddressAndPort localEndpoint = FBUtilities.getBroadcastAddressAndPort();
boolean isLocalReplica = participants.endpoints().contains(localEndpoint);
return !isLocalReplica;
return !participants.endpoints().contains(localEndpoint);
}
/**
* Forwards a Paxos V1 commit operation to a replica coordinator for tracked keyspaces.
* Uses the replica plan to select the best live, non-local replica based on proximity.
*/
private static void forwardPaxosCommit(Commit proposal, ConsistencyLevel consistencyLevel, ReplicaPlan.ForWrite replicaPlan) throws WriteTimeoutException
private static void forwardPaxosCommit(Commit proposal,
ConsistencyLevel consistencyLevel,
ReplicaPlan.ForWrite replicaPlan)
throws WriteTimeoutException, CoordinatorBehindException
{
InetAddressAndPort localEndpoint = FBUtilities.getBroadcastAddressAndPort();
// Get live replicas and filter out local node
EndpointsForToken liveReplicas = replicaPlan.live().filter(replica -> !replica.endpoint().equals(localEndpoint));
if (liveReplicas.isEmpty())
{
// No live replica available, throw exception
throw new WriteTimeoutException(WriteType.CAS, consistencyLevel, 0, consistencyLevel.blockFor(replicaPlan.replicationStrategy()));
}
// Sort by proximity and select the best coordinator
EndpointsForToken sortedReplicas = DatabaseDescriptor.getNodeProximity().sortedByProximity(localEndpoint, liveReplicas);
InetAddressAndPort replicaCoordinator = sortedReplicas.get(0).endpoint();
// Create forward request with participant list
PaxosCommitForwardRequest forwardRequest = new PaxosCommitForwardRequest(proposal, consistencyLevel);
Message<PaxosCommitForwardRequest> message = Message.out(PAXOS_COMMIT_FORWARD_REQ, forwardRequest);
// Use AsyncPromise for proper callback handling
Promise<NoPayload> promise = new AsyncPromise<>();
RequestCallback<NoPayload> callback = new RequestCallback<NoPayload>()
RequestCallbackWithFailure<NoPayload> callback = new RequestCallbackWithFailure<NoPayload>()
{
@Override
public void onResponse(Message<NoPayload> response)
@ -1297,27 +1319,36 @@ public class StorageProxy implements StorageProxyMBean
@Override
public void onFailure(InetAddressAndPort from, RequestFailure reason)
{
promise.setFailure(new WriteTimeoutException(WriteType.CAS, consistencyLevel, 0, consistencyLevel.blockFor(replicaPlan.replicationStrategy())));
if (reason.reason == RequestFailureReason.COORDINATOR_BEHIND)
promise.setFailure(new CoordinatorBehindException("Forwarded Paxos commit rejected: handler says coordinator is behind", reason.failure));
else
promise.setFailure(new WriteTimeoutException(WriteType.CAS, consistencyLevel, 0, consistencyLevel.blockFor(replicaPlan.replicationStrategy()), reason.failure));
}
};
try
{
MessagingService.instance().sendWithCallback(message, replicaCoordinator, callback);
// Wait for response with timeout
promise.get(DatabaseDescriptor.getWriteRpcTimeout(java.util.concurrent.TimeUnit.MILLISECONDS), java.util.concurrent.TimeUnit.MILLISECONDS);
promise.get(DatabaseDescriptor.getWriteRpcTimeout(MILLISECONDS), MILLISECONDS);
}
catch (TimeoutException e)
{
throw new WriteTimeoutException(WriteType.CAS, consistencyLevel, 0, consistencyLevel.blockFor(replicaPlan.replicationStrategy()));
}
catch (InterruptedException e)
{
Thread.currentThread().interrupt();
throw new UncheckedInterruptedException(e);
}
catch (Exception e)
{
if (e instanceof WriteTimeoutException)
throw (WriteTimeoutException) e;
throw new WriteTimeoutException(WriteType.CAS, consistencyLevel, 0, consistencyLevel.blockFor(replicaPlan.replicationStrategy()));
Throwable cause = e.getCause();
if (cause instanceof WriteTimeoutException)
throw (WriteTimeoutException) cause;
if (cause instanceof CoordinatorBehindException)
throw (CoordinatorBehindException) cause;
throw new WriteTimeoutException(WriteType.CAS, consistencyLevel, 0, consistencyLevel.blockFor(replicaPlan.replicationStrategy()), e);
}
}
@ -4349,13 +4380,13 @@ public class StorageProxy implements StorageProxyMBean
boolean alreadyForwarded)
throws UnavailableException, RequestFailureException, RequestTimeoutException
{
// Get keyspace metadata to check if it's tracked
Keyspace keyspace = Keyspace.openIfExists(keyspaceName);
if (keyspace == null)
throw new KeyspaceNotDefinedException("Keyspace " + keyspaceName + " does not exist");
KeyspaceMetadata ksMetadata = keyspace.getMetadata();
if (!ksMetadata.params.replicationType.isTracked())
ClusterMetadata cm = ClusterMetadata.current();
TableMetadata tableMetadata = cm.schema.getTableMetadata(keyspaceName, cfName);
if (tableMetadata == null || !MigrationRouter.shouldUseTrackedForWrites(cm, keyspaceName, tableMetadata.id, key.getToken()))
return null; // Not tracked, no forwarding needed
// Property to disable top-level forwarding for testing
@ -4364,7 +4395,7 @@ public class StorageProxy implements StorageProxyMBean
// Check if current coordinator is not a replica
Token tk = key.getToken();
EndpointsForToken allReplicas = ReplicaLayout.forTokenWriteLiveAndDown(ClusterMetadata.current(), keyspace, tk)
EndpointsForToken allReplicas = ReplicaLayout.forTokenWriteLiveAndDown(cm, keyspace, tk)
.all();
EndpointsForToken liveReplicas = allReplicas.filter(FailureDetector.isReplicaAlive);
@ -4445,13 +4476,12 @@ public class StorageProxy implements StorageProxyMBean
SinglePartitionReadCommand firstCommand = group.queries.get(0);
String keyspaceName = firstCommand.metadata().keyspace;
// Get keyspace metadata to check if it's tracked
Keyspace keyspace = Keyspace.openIfExists(keyspaceName);
if (keyspace == null)
throw new KeyspaceNotDefinedException("Keyspace " + keyspaceName + " does not exist");
KeyspaceMetadata ksMetadata = keyspace.getMetadata();
if (!ksMetadata.params.replicationType.isTracked())
ClusterMetadata cm = ClusterMetadata.current();
if (!MigrationRouter.shouldUseTracked(cm, firstCommand))
return null; // Not tracked, no forwarding needed
// Property to disable top-level forwarding for testing
@ -4460,7 +4490,7 @@ public class StorageProxy implements StorageProxyMBean
// Check if current coordinator is not a replica
Token tk = firstCommand.partitionKey().getToken();
EndpointsForToken allReplicas = ReplicaLayout.forTokenWriteLiveAndDown(ClusterMetadata.current(), keyspace, tk)
EndpointsForToken allReplicas = ReplicaLayout.forTokenWriteLiveAndDown(cm, keyspace, tk)
.all();
EndpointsForToken liveReplicas = allReplicas.filter(FailureDetector.isReplicaAlive);

10
src/java/org/apache/cassandra/service/paxos/CasForwardHandler.java
View File

@ -40,7 +40,7 @@ import org.apache.cassandra.transport.Dispatcher;
/**
* Handler for forwarded CAS (Compare-And-Set) operations.
* Executes the CAS operation on behalf of the original coordinator,
* ensuring that MutationId generation happens on a replica coordinator for tracked keyspaces.
* ensuring that MutationId generation happens on a replica coordinator.
*
* TODO (expected): more comprehensive testing
*/
@ -60,7 +60,6 @@ public class CasForwardHandler implements IVerbHandler<CasForwardRequest>
ClientWarn.instance.captureWarnings();
try
{
// Validate keyspace exists and is tracked
KeyspaceMetadata ksMetadata = Schema.instance.getKeyspaceMetadata(request.keyspaceName);
if (ksMetadata == null)
{
@ -69,13 +68,6 @@ public class CasForwardHandler implements IVerbHandler<CasForwardRequest>
return;
}
if (!ksMetadata.params.replicationType.isTracked())
{
MessagingService.instance().respondWithFailure(RequestFailureReason.INCOMPATIBLE_SCHEMA, message);
logger.error("Asked to perform forwarded CAS operation, but keyspace {} is not tracked", request.keyspaceName);
return;
}
// Execute the forwarded CAS operation
logger.debug("Executing CAS operation for table {}.{} with key {}",
request.keyspaceName, request.cfName, request.key);

16
src/java/org/apache/cassandra/service/paxos/Commit.java
View File

@ -180,6 +180,12 @@ public class Commit
this.localDeletionTime = localDeletionTime;
}
public AcceptedWithTTL(Ballot ballot, Mutation mutation, long localDeletionTime)
{
super(ballot, mutation);
this.localDeletionTime = localDeletionTime;
}
boolean isExpired(long nowInSec)
{
return nowInSec >= localDeletionTime;
@ -194,7 +200,7 @@ public class Commit
@Override
public AcceptedWithTTL withMutationId(MutationId mutationId)
{
return new AcceptedWithTTL(ballot, makeMutation(mutationId).getOnlyUpdate(), localDeletionTime);
return new AcceptedWithTTL(ballot, makeMutation(mutationId), localDeletionTime);
}
}
@ -385,6 +391,12 @@ public class Commit
return new Commit(ballot, update);
}
public static Commit newProposal(Ballot ballot, Mutation mutation)
{
PartitionUpdate update = withTimestamp(mutation.getOnlyUpdate(), ballot.unixMicros());
return new Commit(ballot, new Mutation(mutation.id(), update, mutation.potentialTxnConflicts()));
}
public boolean isAfter(Commit other)
{
return other == null || ballot.uuidTimestamp() > other.ballot.uuidTimestamp();
@ -651,7 +663,7 @@ public class Commit
if (version >= MessagingService.VERSION_61)
{
// New format: deserialize Mutation
Mutation mutation = org.apache.cassandra.db.Mutation.serializer.deserialize(in, version);
Mutation mutation = Mutation.serializer.deserialize(in, version);
return mutationConstructor.apply(ballot, mutation);
}
else

14
src/java/org/apache/cassandra/service/paxos/ConsensusReadForwardHandler.java
View File

@ -40,8 +40,7 @@ import org.apache.cassandra.transport.Dispatcher;
/**
* Handler for forwarded consensus read operations.
* Executes the consensus read operation on behalf of the original coordinator,
* ensuring proper coordination for tracked keyspaces on a replica coordinator.
* Executes the consensus read operation on behalf of the original coordinator.
*
* TODO (expected): more comprehensive testing
*/
@ -55,13 +54,13 @@ public class ConsensusReadForwardHandler implements IVerbHandler<ConsensusReadFo
{
ConsensusReadForwardRequest request = message.payload;
SinglePartitionReadCommand command = request.command;
Tracing.trace("Executing forwarded consensus read operation for {}", command.partitionKey());
// Start capturing client warnings for the forwarded operation
ClientWarn.instance.captureWarnings();
try
{
// Validate keyspace exists and is tracked
String keyspaceName = command.metadata().keyspace;
KeyspaceMetadata ksMetadata = Schema.instance.getKeyspaceMetadata(keyspaceName);
if (ksMetadata == null)
@ -71,13 +70,6 @@ public class ConsensusReadForwardHandler implements IVerbHandler<ConsensusReadFo
return;
}
if (!ksMetadata.params.replicationType.isTracked())
{
MessagingService.instance().respondWithFailure(RequestFailureReason.INCOMPATIBLE_SCHEMA, message);
logger.error("Asked to perform forwarded consensus read operation, but keyspace {} is not tracked", keyspaceName);
return;
}
// Create a Group from the single command for reading
SinglePartitionReadCommand.Group group = SinglePartitionReadCommand.Group.one(command);
@ -85,7 +77,7 @@ public class ConsensusReadForwardHandler implements IVerbHandler<ConsensusReadFo
// 1. Check forwarding (returns null since we're on a replica)
// 2. Execute the consensus read with the appropriate protocol
logger.debug("Executing consensus read operation for table {}.{} with key {}",
keyspaceName, command.metadata().name, command.partitionKey());
command.metadata().keyspace, command.metadata().name, command.partitionKey());
Dispatcher.RequestTime requestTime = Dispatcher.RequestTime.forImmediateExecution();
PartitionIterator result = StorageProxy.readWithConsensusForwarded(group, request.consistencyLevel, requestTime);

34
src/java/org/apache/cassandra/service/paxos/Paxos.java
View File

@ -20,6 +20,7 @@ package org.apache.cassandra.service.paxos;
import java.io.IOException;
import java.util.Collection;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
@ -94,6 +95,7 @@ import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.service.ClientState;
import org.apache.cassandra.service.FailureRecordingCallback.AsMap;
import org.apache.cassandra.service.consensus.migration.ConsensusRequestRouter;
import org.apache.cassandra.service.paxos.Commit.Agreed;
import org.apache.cassandra.service.paxos.Commit.Proposal;
import org.apache.cassandra.service.paxos.PaxosPrepare.FoundIncompleteAccepted;
import org.apache.cassandra.service.paxos.PaxosPrepare.FoundIncompleteCommitted;
@ -740,6 +742,8 @@ public class Paxos
try (PaxosOperationLock lock = PaxosState.lock(partitionKey, metadata, proposeDeadline, consistencyForConsensus, true))
{
Paxos.Async<PaxosCommit.Status> commit = null;
Agreed agreed = null;
Participants commitParticipants = null;
done: while (true)
{
// read the current values and check they validate the conditions
@ -839,7 +843,11 @@ public class Paxos
// 1) reached a majority, in which case it was agreed, had no effect and we can do nothing; or
// 2) did not reach a majority, was not agreed, and was not user visible as a result so we can ignore it
if (!proposal.isEmpty())
commit = commit(proposal.agreed(), participants, consistencyForConsensus, consistencyForCommit, true);
{
agreed = proposal.agreed();
commitParticipants = participants;
commit = commit(agreed, participants, consistencyForConsensus, consistencyForCommit, true);
}
break done;
}
@ -872,8 +880,26 @@ public class Paxos
if (commit != null)
{
PaxosCommit.Status result = commit.awaitUntil(commitDeadline);
if (!result.isSuccess())
throw result.maybeFailure().markAndThrowAsTimeoutOrFailure(true, consistencyForCommit, failedAttemptsDueToContention);
while (!result.isSuccess())
{
Paxos.MaybeFailure failure = result.maybeFailure();
long coordinatorBehindCount = Collections.frequency(failure.failures.values(),
RequestFailureReason.COORDINATOR_BEHIND);
if (coordinatorBehindCount == 0
|| failure.successes + coordinatorBehindCount < failure.required
|| nanoTime() >= commitDeadline)
{
throw failure.markAndThrowAsTimeoutOrFailure(true, consistencyForCommit, failedAttemptsDueToContention);
}
casWriteMetrics.retryCoordinatorBehind.mark();
Tracing.trace("Retrying V2 Paxos commit after COORDINATOR_BEHIND for {}.{} partition {}, {} behind replicas out of {} required",
metadata.keyspace, metadata.name, partitionKey, coordinatorBehindCount, failure.required);
logger.warn("Retrying V2 Paxos commit after COORDINATOR_BEHIND for {}.{} partition {}, {} behind replicas out of {} required",
metadata.keyspace, metadata.name, partitionKey, coordinatorBehindCount, failure.required);
commit = commit(agreed, commitParticipants, consistencyForConsensus, consistencyForCommit, true);
result = commit.awaitUntil(commitDeadline);
}
}
Tracing.trace("CAS successful");
return casResult((RowIterator)null);
@ -1103,7 +1129,7 @@ public class Paxos
// is equal to the latest commit (even if the ballots aren't) we're done and can abort earlier,
// and in fact it's possible for a CAS to sometimes determine if side effects occurred by reading
// the underlying data and not witnessing the timestamp of its ballot (or any newer for the relevant data).
Proposal repropose = new Proposal(inProgress.ballot, inProgress.accepted.update);
Proposal repropose = new Proposal(inProgress.ballot, inProgress.accepted.mutation);
PaxosPropose.Status proposeResult = propose(repropose, inProgress.participants, false).awaitUntil(deadline);
switch (proposeResult.outcome)
{

61
src/java/org/apache/cassandra/service/paxos/Paxos2CommitForwardHandler.java
View File

@ -29,8 +29,8 @@ import org.apache.cassandra.net.IVerbHandler;
import org.apache.cassandra.net.Message;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.net.NoPayload;
import org.apache.cassandra.schema.KeyspaceMetadata;
import org.apache.cassandra.schema.Schema;
import org.apache.cassandra.service.replication.migration.MigrationRouter;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.ClusterMetadataService;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.utils.concurrent.ConditionAsConsumer;
@ -39,13 +39,11 @@ import static org.apache.cassandra.utils.concurrent.ConditionAsConsumer.newCondi
/**
* Handler for forwarded Paxos V2 commit requests.
* Executes the commit operation on behalf of the original coordinator,
* ensuring that MutationId generation happens on a replica coordinator.
*
* The PaxosCommit constructor handles mutation ID generation, so this handler
* simply delegates to PaxosCommit.commit() with the original commit.
* Delegates to PaxosCommit.commit() which handles mutation ID generation
* in its constructor.
*
* TODO (expected): more comprehensive testing
* TODO: should loop on CoordinatorBehindException rather than propagating failure to the forwarding coordinator
*/
public class Paxos2CommitForwardHandler implements IVerbHandler<Paxos2CommitForwardRequest>
{
@ -55,27 +53,38 @@ public class Paxos2CommitForwardHandler implements IVerbHandler<Paxos2CommitForw
@Override
public void doVerb(Message<Paxos2CommitForwardRequest> message)
{
// Ensure we have up-to-date cluster metadata before executing the forwarded commit
ClusterMetadataService.instance().fetchLogFromPeerOrCMS(message.from(), message.header.epoch);
Paxos2CommitForwardRequest request = message.payload;
Commit.Agreed commit = request.commit;
Tracing.trace("Executing forwarded Paxos V2 commit for {}", request.commit.partitionKey());
Tracing.trace("Executing forwarded Paxos V2 commit for {}.{} partition {}",
commit.metadata().keyspace, commit.metadata().name, commit.partitionKey());
try
{
String ksName = request.commit.metadata().keyspace;
KeyspaceMetadata ksMetadata = Schema.instance.getKeyspaceMetadata(ksName);
if (ksMetadata == null)
String ksName = commit.metadata().keyspace;
ClusterMetadata metadata = ClusterMetadata.current();
boolean shouldBeTracked = MigrationRouter.shouldUseTrackedForWrites(metadata,
ksName,
commit.metadata().id,
commit.partitionKey().getToken());
if (!shouldBeTracked && message.epoch().isAfter(metadata.epoch))
{
MessagingService.instance().respondWithFailure(RequestFailureReason.INCOMPATIBLE_SCHEMA, message);
logger.error("Failed to forward paxos commit for non-existent keyspace {}", ksName);
return;
metadata = ClusterMetadataService.instance().fetchLogFromPeerOrCMS(metadata, message.from(), message.epoch());
// shouldBeTracked isn't used after this, but is kept up to date just in case
shouldBeTracked = MigrationRouter.shouldUseTrackedForWrites(metadata,
ksName,
commit.metadata().id,
commit.partitionKey().getToken());
}
if (!ksMetadata.params.replicationType.isTracked())
if (metadata.schema.getKeyspaces().getNullable(ksName) == null)
{
MessagingService.instance().respondWithFailure(RequestFailureReason.INCOMPATIBLE_SCHEMA, message);
logger.error("Asked to perform forwarded paxos commit, but keyspace {} is not tracked", ksName);
logger.error("Failed to forward paxos commit for non-existent keyspace {}.{} partition {}",
ksName, commit.metadata().name, commit.partitionKey());
Tracing.trace("Failed to forward paxos commit for non-existent keyspace {}.{} partition {}",
ksName, commit.metadata().name, commit.partitionKey());
return;
}
@ -112,8 +121,21 @@ public class Paxos2CommitForwardHandler implements IVerbHandler<Paxos2CommitForw
}
else
{
MessagingService.instance().respondWithFailure(RequestFailureReason.UNKNOWN, message);
RequestFailureReason reason = RequestFailureReason.UNKNOWN;
if (status != null && status.maybeFailure() != null)
{
for (RequestFailureReason r : status.maybeFailure().failures.values())
{
if (r == RequestFailureReason.COORDINATOR_BEHIND)
{
reason = RequestFailureReason.COORDINATOR_BEHIND;
break;
}
}
}
MessagingService.instance().respondWithFailure(reason, message);
logger.error("Forwarded Paxos V2 commit failed with status: {}", status);
Tracing.trace("Forwarded Paxos V2 commit failed with status: {}", status);
}
}
catch (InterruptedException e)
@ -121,6 +143,7 @@ public class Paxos2CommitForwardHandler implements IVerbHandler<Paxos2CommitForw
Thread.currentThread().interrupt();
MessagingService.instance().respondWithFailure(RequestFailure.forException(e), message);
logger.error("Forwarded Paxos V2 commit interrupted", e);
Tracing.trace("Forwarded Paxos V2 commit interrupted");
}
}
catch (Exception e)

80
src/java/org/apache/cassandra/service/paxos/PaxosCommit.java
View File

@ -18,6 +18,7 @@
package org.apache.cassandra.service.paxos;
import java.util.Collections;
import java.util.concurrent.atomic.AtomicLongFieldUpdater;
import java.util.function.Consumer;
@ -43,11 +44,12 @@ import org.apache.cassandra.net.IVerbHandler;
import org.apache.cassandra.net.Message;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.net.NoPayload;
import org.apache.cassandra.net.RequestCallback;
import org.apache.cassandra.net.RequestCallbackWithFailure;
import org.apache.cassandra.net.Verb;
import org.apache.cassandra.replication.MutationId;
import org.apache.cassandra.replication.MutationTrackingService;
import org.apache.cassandra.service.paxos.Paxos.Participants;
import org.apache.cassandra.service.replication.migration.MigrationRouter;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.utils.FBUtilities;
@ -112,6 +114,7 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
final EndpointsForToken replicas;
final int required;
final OnDone onDone;
final boolean tracked;
@Nullable
final IntHashSet remoteReplicas;
@ -128,8 +131,7 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
public PaxosCommit(Agreed commit, boolean allowHints, ConsistencyLevel consistencyForConsensus, ConsistencyLevel consistencyForCommit, EndpointsForToken replicas, int required, OnDone onDone)
{
// Check if this is a tracked keyspace
boolean isTracked = commit.metadata().replicationType().isTracked();
boolean isTracked = MigrationRouter.shouldUseTrackedForWrites(commit.metadata().keyspace, commit.metadata().id, commit.partitionKey().getToken());
Agreed commitToUse = commit;
IntHashSet remoteReplicas = null;
@ -162,7 +164,18 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
remoteReplicas.add(metadata.directory.peerId(replica.endpoint()).id());
}
}
else if (!commit.mutation.id().isNone())
{
// Keyspace is now untracked but the commit has an ID (from system.paxos or a previous
// round when tracking was active). Strip it to avoid misrouting on replicas.
logger.warn("Stripping mutation ID {} from PaxosCommit for {}.{} partition {} - keyspace migrated to untracked",
commit.mutation.id(), commit.metadata().keyspace, commit.metadata().name, commit.partitionKey());
Tracing.trace("Stripping mutation ID {} from PaxosCommit for {}.{} partition {} - keyspace migrated to untracked",
commit.mutation.id(), commit.metadata().keyspace, commit.metadata().name, commit.partitionKey());
commitToUse = commit.withMutationId(MutationId.none());
}
this.tracked = isTracked;
this.commit = commitToUse;
this.allowHints = allowHints;
this.consistencyForConsensus = consistencyForConsensus;
@ -181,10 +194,8 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
*/
static Paxos.Async<Status> commit(Agreed commit, EndpointsForToken all, EndpointsForToken allLive, EndpointsForToken allDown, int required, boolean isUrgent, ConsistencyLevel consistencyForConsensus, ConsistencyLevel consistencyForCommit, /** @deprecated See CASSANDRA-17164 */ @Deprecated(since = "4.1") boolean allowHints)
{
// Check if this is a tracked keyspace requiring forwarding to a replica coordinator
if (isTrackedKeyspaceRequiringForwarding(commit, all))
{
// For async version, create a wrapper that handles forwarding
Status[] statusHolder = new Status[1];
ConditionAsConsumer<Status> condition = newConditionAsConsumer();
Consumer<Status> statusCapture = status -> {
@ -246,7 +257,6 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
*/
static <T extends Consumer<Status>> T commit(Agreed commit, EndpointsForToken all, EndpointsForToken allLive, EndpointsForToken allDown, int required, boolean isUrgent, ConsistencyLevel consistencyForConsensus, ConsistencyLevel consistencyForCommit, /** @deprecated See CASSANDRA-17164 */ @Deprecated(since = "4.1") boolean allowHints, T onDone)
{
// Check if this is a tracked keyspace requiring forwarding to a replica coordinator
if (isTrackedKeyspaceRequiringForwarding(commit, all))
{
forwardPaxos2Commit(commit, all, allLive, allDown, required, isUrgent, consistencyForConsensus, consistencyForCommit, onDone);
@ -328,13 +338,11 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
}
}
// Now send to remote replicas (and record local execution for non-tracked keyspaces)
boolean executeOnSelf = false;
for (int i = 0, mi = allLive.size(); i < mi ; ++i)
{
InetAddressAndPort endpoint = allLive.endpoint(i);
// Skip self if we already executed synchronously for tracked keyspace.
// Use direct comparison instead of shouldExecuteOnSelf to avoid dependence on USE_SELF_EXECUTION.
// Skip self if already executed synchronously (avoid shouldExecuteOnSelf to not depend on USE_SELF_EXECUTION)
if (localExecutedSynchronously && endpoint.equals(localEndpoint))
continue;
executeOnSelf |= isSelfOrSend(commitMessage, mutationMessage, endpoint);
@ -343,17 +351,14 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
for (int i = 0, mi = allDown.size(); i < mi ; ++i)
{
InetAddressAndPort endpoint = allDown.endpoint(i);
// Skip self if we already executed synchronously for tracked keyspace.
// We can't "retry" to self via network anyway, and we've already written to the journal.
// Skip self — already handled above
if (localExecutedSynchronously && endpoint.equals(localEndpoint))
continue;
onFailure(endpoint, RequestFailure.NODE_DOWN);
}
// Tracked if remoteReplicas != null, register write request with tracking service for tracked keyspaces
if (remoteReplicas != null)
if (isTrackedKeyspace && !remoteReplicas.isEmpty())
{
checkState(!remoteReplicas.isEmpty());
MutationTrackingService.instance().sentWriteRequest(commit.makeMutation(), remoteReplicas);
}
@ -390,7 +395,7 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
private boolean isTracked()
{
return !commit.mutation.id().equals(MutationId.none());
return tracked;
}
/**
@ -399,35 +404,21 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
@Override
public void onFailure(InetAddressAndPort from, RequestFailure reason)
{
super.onFailure(from, reason); // Populates failureResponses for failureReasonsAsMap()
if (logger.isTraceEnabled())
logger.trace("{} {} from {}", commit, reason, from);
// Track failed response for tracked keyspaces
if (isTracked())
MutationTrackingService.instance().retryFailedWrite(commit.mutation.id(), from, reason);
response(false, from);
Replica replica = replicas.lookup(from);
if (allowHints && shouldHint(replica))
if (allowHints && shouldHint(replica) && !isTracked())
submitHint(commit.makeMutation(), replica, null);
}
/**
* Record a success response
*/
public void onResponse(Message<NoPayload> response)
{
logger.trace("{} Success from {}", commit, response.from());
// Track successful response for tracked keyspaces
// (Local mutations are witnessed from Keyspace.applyInternalTracked)
if (isTracked())
MutationTrackingService.instance().receivedWriteResponse(commit.mutation.id(), response.from());
response(true, response.from());
}
/**
* Execute locally and record response
*/
@ -458,8 +449,9 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
@Override
public void onResponse(NoPayload response, InetAddressAndPort from)
{
// Track successful response for tracked keyspaces
if (isTracked())
logger.trace("{} {} from {}", commit, response != null ? "Success" : "Failure", from);
if (isTracked() && !from.equals(FBUtilities.getBroadcastAddressAndPort()))
{
if (response != null)
MutationTrackingService.instance().receivedWriteResponse(commit.mutation.id(), from);
@ -519,6 +511,15 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
@Override
public void doVerb(Message<Agreed> message)
{
Agreed agreed = message.payload;
ClusterMetadata metadata = ClusterMetadata.current();
boolean coordinatorSaysTracked = !agreed.mutation.id().isNone();
MigrationRouter.checkPaxosCommitMigration(metadata, message, message.from(),
agreed.metadata().keyspace, agreed.metadata().id,
agreed.partitionKey().getToken(),
coordinatorSaysTracked);
NoPayload response = execute(message.payload);
// NOTE: for correctness, this must be our last action, so that we cannot throw an error and send both a response and a failure response
if (response == null)
@ -543,7 +544,7 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
*/
private static boolean isTrackedKeyspaceRequiringForwarding(Agreed commit, EndpointsForToken all)
{
if (!commit.metadata().replicationType().isTracked())
if (!MigrationRouter.shouldUseTrackedForWrites(commit.metadata().keyspace, commit.metadata().id, commit.partitionKey().getToken()))
return false;
// Check if current coordinator is not a replica
@ -592,8 +593,7 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
required, isUrgent);
Message<Paxos2CommitForwardRequest> message = Message.out(Verb.PAXOS2_COMMIT_FORWARD_REQ, forwardRequest);
// Create callback to handle forwarding response
RequestCallback<NoPayload> callback = new RequestCallback<NoPayload>()
RequestCallbackWithFailure<NoPayload> callback = new RequestCallbackWithFailure<NoPayload>()
{
@Override
public void onResponse(Message<NoPayload> response)
@ -607,9 +607,9 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
{
logger.debug("Forwarded Paxos V2 commit to {} failed: {}", from, failure);
Tracing.trace("Forwarded Paxos V2 commit to {} failed: {}", from, failure);
// Populate the failure map with the actual failure reason; contacted=1, required=1 for forwarded request
// contacted=1, required=1 because this is a single forwarded request
onDone.accept(new Status(new Paxos.MaybeFailure(true, 1, 1, 0,
java.util.Collections.singletonMap(from, failure.reason))));
Collections.singletonMap(from, failure.reason))));
}
};
@ -622,7 +622,7 @@ public class PaxosCommit<OnDone extends Consumer<? super PaxosCommit.Status>> ex
logger.debug("Failed to send forwarded Paxos V2 commit to {}: {}", replicaCoordinator, e.getMessage());
Tracing.trace("Failed to send forwarded Paxos V2 commit: {}", e.getMessage());
onDone.accept(new Status(new Paxos.MaybeFailure(true, 1, 1, 0,
java.util.Collections.singletonMap(replicaCoordinator, UNKNOWN))));
Collections.singletonMap(replicaCoordinator, UNKNOWN))));
}
}

37
src/java/org/apache/cassandra/service/paxos/PaxosCommitAndPrepare.java
View File

@ -20,6 +20,9 @@ package org.apache.cassandra.service.paxos;
import java.io.IOException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.db.ConsistencyLevel;
import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.db.EmbeddableSinglePartitionReadCommand;
@ -29,6 +32,7 @@ import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.net.IVerbHandler;
import org.apache.cassandra.net.Message;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.replication.MutationId;
import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.service.consensus.migration.ConsensusKeyMigrationState.KeyMigrationState;
import org.apache.cassandra.service.paxos.Commit.Agreed;
@ -36,8 +40,8 @@ import org.apache.cassandra.service.paxos.PaxosPrepare.Rejected;
import org.apache.cassandra.service.paxos.PaxosPrepare.Response;
import org.apache.cassandra.service.reads.tracked.TrackedRead;
import org.apache.cassandra.service.reads.tracked.TrackedRead.Id;
import org.apache.cassandra.service.replication.migration.MigrationRouter;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.ClusterMetadataService;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.transport.Dispatcher.RequestTime;
import org.apache.cassandra.utils.concurrent.Future;
@ -52,6 +56,8 @@ import static org.apache.cassandra.service.paxos.PaxosPrepare.start;
public class PaxosCommitAndPrepare
{
private static final Logger logger = LoggerFactory.getLogger(PaxosCommitAndPrepare.class);
public static final RequestSerializer requestSerializer = new RequestSerializer();
public static final RequestHandler requestHandler = new RequestHandler();
@ -68,7 +74,22 @@ public class PaxosCommitAndPrepare
*
* All these things are tractable to do better, but for now doing something simple and correct.
*/
if (readCommand.metadata().replicationType().isTracked())
boolean shouldBeTracked = MigrationRouter.shouldUseTrackedForWrites(commit.metadata().keyspace,
commit.metadata().id,
commit.partitionKey().getToken());
// Reconcile the mutation's ID with the current migration state.
// The commit may have been saved to system.paxos under a different replication type.
if (!shouldBeTracked && !commit.mutation.id().isNone())
{
logger.warn("Stripping mutation ID {} from PaxosCommitAndPrepare for {}.{} partition {} - keyspace migrated to untracked",
commit.mutation.id(), commit.metadata().keyspace, commit.metadata().name, commit.partitionKey());
Tracing.trace("Stripping mutation ID {} from PaxosCommitAndPrepare for {}.{} partition {} - keyspace migrated to untracked",
commit.mutation.id(), commit.metadata().keyspace, commit.metadata().name, commit.partitionKey());
commit = commit.withMutationId(MutationId.none());
}
if (shouldBeTracked)
{
/*
* Consistency for consensus is tricky to pick here. The goal of sending this commit is to unblock the prepare
@ -169,7 +190,17 @@ public class PaxosCommitAndPrepare
@Override
public void doVerb(Message<Request> message)
{
ClusterMetadataService.instance().fetchLogFromPeerOrCMS(ClusterMetadata.current(), message.from(), message.epoch());
ClusterMetadata metadata = ClusterMetadata.current();
Agreed commit = message.payload.commit;
boolean coordinatorSaysTracked = !commit.mutation.id().isNone();
metadata = MigrationRouter.checkPaxosCommitMigration(metadata, message, message.from(),
commit.metadata().keyspace, commit.metadata().id,
commit.partitionKey().getToken(),
coordinatorSaysTracked);
if (message.payload.read != null)
MigrationRouter.checkPaxosPrepareReadMigration(metadata, message, message.from(), message.payload.read);
Future<Response> response = execute(message.payload, new RequestTime(message.createdAtNanos()));
if (response == null)

60
src/java/org/apache/cassandra/service/paxos/PaxosCommitForwardHandler.java
View File

@ -28,17 +28,21 @@ import org.apache.cassandra.net.IVerbHandler;
import org.apache.cassandra.net.Message;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.net.NoPayload;
import org.apache.cassandra.replication.MutationId;
import org.apache.cassandra.service.StorageProxy;
import org.apache.cassandra.service.replication.migration.MigrationRouter;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.ClusterMetadataService;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.transport.Dispatcher;
/**
* Handler for forwarded Paxos V1 commit requests.
* Executes the commit operation on behalf of the original coordinator,
* ensuring that MutationId generation happens on a replica coordinator.
* Routes the commit to the tracked or untracked path based on the current
* migration state of the keyspace for the affected partition.
*
* TODO (expected): more comprehensive testing
* TODO: should loop on CoordinatorBehindException rather than propagating failure to the forwarding coordinator
*/
public class PaxosCommitForwardHandler implements IVerbHandler<PaxosCommitForwardRequest>
{
@ -48,35 +52,59 @@ public class PaxosCommitForwardHandler implements IVerbHandler<PaxosCommitForwar
@Override
public void doVerb(Message<PaxosCommitForwardRequest> message)
{
// PaxosV1 when doing commit picks whatever the current replicas are to send the commits to
// so make sure we at least match what they would have picked
ClusterMetadataService.instance().fetchLogFromPeerOrCMS(message.from(), message.header.epoch);
PaxosCommitForwardRequest request = message.payload;
Commit proposal = request.proposal;
Tracing.trace("Executing forwarded Paxos commit for {}", request.proposal.partitionKey());
Tracing.trace("Executing forwarded Paxos commit for {}", proposal.partitionKey());
try
{
String ksName = request.proposal.metadata().keyspace;
String ksName = proposal.metadata().keyspace;
Keyspace keyspace = Keyspace.openIfExists(ksName);
if (keyspace == null)
{
MessagingService.instance().respondWithFailure(RequestFailureReason.INCOMPATIBLE_SCHEMA, message);
logger.error("Failed to forward paxos commit for non-existent keyspace {}", ksName);
logger.error("Failed to forward paxos commit for non-existent keyspace {}.{} partition {}",
ksName, proposal.metadata().name, proposal.partitionKey());
return;
}
if (!keyspace.getMetadata().params.replicationType.isTracked())
ClusterMetadata metadata = ClusterMetadata.current();
boolean shouldBeTracked = MigrationRouter.shouldUseTrackedForWrites(metadata,
ksName,
proposal.metadata().id,
proposal.partitionKey().getToken());
if (!shouldBeTracked && message.epoch().isAfter(metadata.epoch))
{
MessagingService.instance().respondWithFailure(RequestFailureReason.INCOMPATIBLE_SCHEMA, message);
logger.error("Asked to perform forwarded paxos commit, but keyspace {} is not tracked", ksName);
return;
metadata = ClusterMetadataService.instance().fetchLogFromPeerOrCMS(metadata, message.from(), message.epoch());
shouldBeTracked = MigrationRouter.shouldUseTrackedForWrites(metadata,
ksName,
proposal.metadata().id,
proposal.partitionKey().getToken());
}
// Call commitPaxosTracked which handles mutation ID generation, sending to all replicas,
// and tracking. The respondAfterSend flag determines if we wait for application.
StorageProxy.commitPaxosTracked(keyspace, request.proposal, request.consistencyLevel,
Dispatcher.RequestTime.forImmediateExecution(), request.respondAfterSend);
if (shouldBeTracked)
{
StorageProxy.commitPaxosTracked(keyspace, proposal, request.consistencyLevel,
Dispatcher.RequestTime.forImmediateExecution(), request.respondAfterSend);
}
else
{
// respondAfterSend is not propagated here — commitPaxosUntracked always blocks.
// During migration races this adds latency but doesn't affect correctness.
Commit reconciled = proposal;
if (!proposal.mutation.id().isNone())
{
logger.warn("Stripping mutation ID {} from forwarded PaxosCommit for {}.{} partition {} - keyspace is untracked at handler",
proposal.mutation.id(), ksName, proposal.metadata().name, proposal.partitionKey());
Tracing.trace("Stripping mutation ID {} from forwarded PaxosCommit for {}.{} partition {} - keyspace is untracked at handler",
proposal.mutation.id(), ksName, proposal.metadata().name, proposal.partitionKey());
reconciled = proposal.withMutationId(MutationId.none());
}
StorageProxy.commitPaxosUntracked(keyspace, reconciled, request.consistencyLevel,
true, Dispatcher.RequestTime.forImmediateExecution());
}
MessagingService.instance().respond(NoPayload.noPayload, message);
}

15
src/java/org/apache/cassandra/service/paxos/PaxosPrepare.java
View File

@ -71,6 +71,7 @@ import org.apache.cassandra.service.reads.tracked.TrackedRead;
import org.apache.cassandra.service.reads.tracked.TrackedRead.DataRequest;
import org.apache.cassandra.service.reads.tracked.TrackedRead.Id;
import org.apache.cassandra.service.reads.tracked.TrackedRead.SummaryRequest;
import org.apache.cassandra.service.replication.migration.MigrationRouter;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.ClusterMetadataService;
import org.apache.cassandra.tcm.Epoch;
@ -346,6 +347,7 @@ public class PaxosPrepare extends PaxosRequestCallback<PaxosPrepare.Response> im
private final List<Message<IReadResponse>> readResponses;
private boolean haveReadResponseWithLatest;
private boolean haveTrackedDataResponseIfNeeded;
private boolean isTrackedRead;
private boolean haveQuorumOfPermissions; // permissions => SUCCESS or READ_SUCCESS
private @Nonnull List<InetAddressAndPort> withLatest; // promised and have latest commit
private @Nullable List<InetAddressAndPort> needLatest; // promised without having witnessed latest commit, nor yet been refreshed by us
@ -430,7 +432,9 @@ public class PaxosPrepare extends PaxosRequestCallback<PaxosPrepare.Response> im
*/
static <R extends AbstractRequest<R>> void start(PaxosPrepare prepare, Participants participants, Message<R> send, BiFunction<R, RequestTime, Future<Response>> selfHandler)
{
if (send.payload.read != null && send.payload.read.metadata().replicationType().isTracked())
boolean tracked = send.payload.read != null && MigrationRouter.shouldUseTracked(send.payload.read);
prepare.isTrackedRead = tracked;
if (tracked)
startTracked(prepare, participants, send, selfHandler);
else
startUntracked(prepare, participants, send, selfHandler);
@ -554,7 +558,7 @@ public class PaxosPrepare extends PaxosRequestCallback<PaxosPrepare.Response> im
private boolean isTracked()
{
return request.read != null && request.read.metadata().replicationType().isTracked();
return isTrackedRead;
}
private boolean isDone()
@ -1226,7 +1230,12 @@ public class PaxosPrepare extends PaxosRequestCallback<PaxosPrepare.Response> im
@Override
public void doVerb(Message<Request> message)
{
ClusterMetadataService.instance().fetchLogFromPeerOrCMS(ClusterMetadata.current(), message.from(), message.epoch());
ClusterMetadata metadata = ClusterMetadataService.instance().fetchLogFromPeerOrCMS(ClusterMetadata.current(),
message.from(),
message.epoch());
if (message.payload.read != null)
MigrationRouter.checkPaxosPrepareReadMigration(metadata, message, message.from(), message.payload.read);
try
{

281
src/java/org/apache/cassandra/service/paxos/PaxosPrepareRefresh.java
View File

@ -21,18 +21,20 @@ package org.apache.cassandra.service.paxos;
import java.io.IOException;
import java.util.List;
import com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.db.Mutation;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.exceptions.RequestFailure;
import org.apache.cassandra.exceptions.RequestFailureReason;
import org.apache.cassandra.exceptions.RetryOnDifferentSystemException;
import org.apache.cassandra.exceptions.WriteTimeoutException;
import org.apache.cassandra.gms.FailureDetector;
import org.apache.cassandra.io.IVersionedSerializer;
import org.apache.cassandra.io.util.DataInputPlus;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.locator.EndpointsForToken;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.locator.Replica;
import org.apache.cassandra.net.IVerbHandler;
@ -44,6 +46,7 @@ import org.apache.cassandra.replication.MutationId;
import org.apache.cassandra.replication.MutationTrackingService;
import org.apache.cassandra.service.paxos.Commit.Agreed;
import org.apache.cassandra.service.paxos.Commit.Committed;
import org.apache.cassandra.service.replication.migration.MigrationRouter;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.ClusterMetadataService;
import org.apache.cassandra.tracing.Tracing;
@ -78,10 +81,11 @@ public class PaxosPrepareRefresh implements RequestCallbackWithFailure<PaxosPrep
void onRefreshSuccess(Ballot isSupersededBy, InetAddressAndPort from);
}
private Message<Request> send;
private volatile Message<Request> send;
private final Callbacks callbacks;
private final Paxos.Participants participants;
private final boolean isUrgent;
private boolean selfCallbackDelivered;
public PaxosPrepareRefresh(Ballot prepared, Paxos.Participants participants, Committed latestCommitted, Callbacks callbacks)
{
@ -97,145 +101,146 @@ public class PaxosPrepareRefresh implements RequestCallbackWithFailure<PaxosPrep
*/
void refresh(List<InetAddressAndPort> refresh)
{
// Check if forwarding is needed for tracked keyspaces
selfCallbackDelivered = false;
Committed commit = send.payload.missingCommit;
boolean tracked = MigrationRouter.shouldUseTrackedForWrites(commit.metadata().keyspace,
commit.metadata().id,
commit.partitionKey().getToken());
if (commit.metadata().replicationType().isTracked() && commit.mutation.id().isNone())
if (tracked && commit.mutation.id().isNone())
{
// Check if we can generate mutation ID locally (are we a replica?)
Replica localReplica = participants.all.byEndpoint().get(getBroadcastAddressAndPort());
if (localReplica != null)
if (localReplica == null)
{
// We ARE a replica - generate mutation ID and update the commit
String keyspaceName = commit.metadata().keyspace;
MutationId mutationId = MutationTrackingService.instance().nextMutationId(keyspaceName, commit.partitionKey().getToken());
Mutation mutationWithId = commit.makeMutation(mutationId);
Committed commitWithId = new Commit.Committed(commit.ballot, mutationWithId);
// Update the message payload with the new commit
this.send = Message.out(PAXOS2_PREPARE_REFRESH_REQ,
new Request(send.payload.promised, commitWithId),
isUrgent);
// For tracked keyspaces, we MUST ALWAYS write to the local journal since we generated the mutation ID.
// This is required for retry purposes: if a remote target fails, the ActiveLogReconciler will try
// to look up the mutation in the local journal. The node that generated the mutation ID is the "owner"
// and must have the mutation available for retries.
//
// We do this BEFORE the main refresh loop to ensure the mutation is in the journal before any
// failure callback can trigger reconciliation.
Response localResponse = null;
try
{
localResponse = RequestHandler.execute(this.send.payload, getBroadcastAddressAndPort());
if (localResponse == null)
logger.warn("Local execution failed for tracked mutation {}", mutationId);
}
catch (Exception e)
{
logger.warn("Exception writing tracked mutation {} locally", mutationId, e);
}
// If self is in the refresh list, report the local execution result to callbacks
// We need to do this because the main loop will skip self for tracked keyspaces
for (int i = 0, size = refresh.size(); i < size; ++i)
{
if (shouldExecuteOnSelf(refresh.get(i)))
{
if (localResponse != null)
callbacks.onRefreshSuccess(localResponse.isSupersededBy, getBroadcastAddressAndPort());
else
callbacks.onRefreshFailure(getBroadcastAddressAndPort(), RequestFailure.UNKNOWN);
break;
}
}
}
else
{
// We're NOT a replica - forward to a replica
forwardRefresh(refresh);
return;
}
if (!generateMutationIdAndPersistLocally(commit, refresh))
return;
}
else if (!tracked && !commit.mutation.id().isNone())
{
logger.warn("Stripping mutation ID {} from PaxosPrepareRefresh for {}.{} partition {} - keyspace migrated to untracked",
commit.mutation.id(), commit.metadata().keyspace, commit.metadata().name, commit.partitionKey());
Tracing.trace("Stripping mutation ID {} from PaxosPrepareRefresh for {}.{} partition {} - keyspace migrated to untracked",
commit.mutation.id(), commit.metadata().keyspace, commit.metadata().name, commit.partitionKey());
updateSendMessage(commit.withMutationId(MutationId.none()));
}
// For tracked keyspaces where we generated the ID above, we already wrote locally.
// For tracked keyspaces where the ID was already present, we still need to ensure local execution.
boolean isTracked = !send.payload.missingCommit.mutation.id().isNone();
dispatchRefresh(refresh, tracked);
}
// If we just generated the ID above, we already wrote locally - check by examining the original commit
boolean alreadyWroteLocally = commit.metadata().replicationType().isTracked()
&& commit.mutation.id().isNone()
&& participants.all.byEndpoint().get(getBroadcastAddressAndPort()) != null;
boolean localExecutedSync = alreadyWroteLocally;
/**
* Generates a mutation ID as the local replica, updates the send message, executes locally
* to persist to the journal (for ID ownership), and reports self's callback if self is a target.
*
* @return true if local write succeeded, false if it failed (all target callbacks reported as failure)
*/
private boolean generateMutationIdAndPersistLocally(Committed commit, List<InetAddressAndPort> refresh)
{
String keyspaceName = commit.metadata().keyspace;
MutationId mutationId = MutationTrackingService.instance().nextMutationId(keyspaceName, commit.partitionKey().getToken());
updateSendMessage(commit.withMutationId(mutationId));
selfCallbackDelivered = true;
// For tracked keyspaces where we DIDN'T generate the ID (it was already present), we still need to
// execute locally BEFORE sending to remotes if self is in the refresh list.
if (isTracked && !alreadyWroteLocally)
Response localResponse = null;
try
{
for (int i = 0, size = refresh.size(); i < size; ++i)
localResponse = RequestHandler.execute(this.send.payload, getBroadcastAddressAndPort());
if (localResponse == null)
logger.warn("Local execution failed for tracked mutation {}", mutationId);
}
catch (Exception e)
{
logger.warn("Exception writing tracked mutation {} locally", mutationId, e);
}
if (localResponse == null)
{
for (InetAddressAndPort target : refresh)
callbacks.onRefreshFailure(target, RequestFailure.UNKNOWN);
return false;
}
for (int i = 0, size = refresh.size(); i < size; ++i)
{
if (shouldExecuteOnSelf(refresh.get(i)))
{
if (shouldExecuteOnSelf(refresh.get(i)))
{
executeOnSelf(); // SYNCHRONOUS - journal write completes here
localExecutedSync = true;
break;
}
callbacks.onRefreshSuccess(localResponse.isSupersededBy, getBroadcastAddressAndPort());
break;
}
}
return true;
}
// Now send to remote nodes (and record local execution for non-tracked keyspaces)
boolean executeOnSelf = false;
for (int i = 0, size = refresh.size(); i < size ; ++i)
private void updateSendMessage(Committed commit)
{
this.send = Message.out(PAXOS2_PREPARE_REFRESH_REQ, new Request(send.payload.promised, commit), isUrgent);
}
/**
* Dispatches refresh to all targets. For tracked keyspaces, self is executed synchronously
* before remotes (unless already handled during ID generation). For untracked keyspaces,
* self is scheduled for async execution after remotes.
*/
private void dispatchRefresh(List<InetAddressAndPort> refresh, boolean tracked)
{
if (tracked && !selfCallbackDelivered)
executeSelfSynchronously(refresh);
boolean selfInList = false;
for (int i = 0, size = refresh.size(); i < size; ++i)
{
InetAddressAndPort destination = refresh.get(i);
if (shouldExecuteOnSelf(destination))
{
selfInList = true;
continue;
}
if (logger.isTraceEnabled())
logger.trace("Refresh {} and Confirm {} to {}", send.payload.missingCommit, Ballot.toString(send.payload.promised, "Promise"), destination);
if (Tracing.isTracing())
Tracing.trace("Refresh {} and Confirm {} to {}", send.payload.missingCommit.ballot, send.payload.promised, destination);
if (shouldExecuteOnSelf(destination))
{
// For tracked keyspaces, skip self - already executed synchronously above
if (!localExecutedSync)
executeOnSelf = true;
}
else
{
MessagingService.instance().sendWithCallback(send, destination, this);
}
MessagingService.instance().sendWithCallback(send, destination, this);
}
// Async local execution only for non-tracked keyspaces
if (executeOnSelf)
if (!tracked && selfInList)
PAXOS2_PREPARE_REFRESH_REQ.stage.execute(this::executeOnSelf);
}
private void executeSelfSynchronously(List<InetAddressAndPort> refresh)
{
for (int i = 0, size = refresh.size(); i < size; ++i)
{
if (shouldExecuteOnSelf(refresh.get(i)))
{
executeOnSelf();
break;
}
}
}
/**
* Forward the refresh operation to a replica coordinator.
* The replica will generate the mutation ID and send to all refresh nodes.
* The replica will generate (or find) the mutation ID and send the refresh to all target nodes.
*/
private void forwardRefresh(List<InetAddressAndPort> refreshTargets)
{
// Find a live replica to forward to (that's not us)
InetAddressAndPort targetReplica = null;
for (Replica replica : participants.all)
{
if (!replica.endpoint().equals(getBroadcastAddressAndPort()) &&
FailureDetector.instance.isAlive(replica.endpoint()))
{
targetReplica = replica.endpoint();
break;
}
}
InetAddressAndPort localEndpoint = getBroadcastAddressAndPort();
EndpointsForToken liveExcludingSelf = participants.allLive.filter(replica -> !replica.endpoint().equals(localEndpoint));
InetAddressAndPort targetReplica = liveExcludingSelf.isEmpty()
? null
: DatabaseDescriptor.getNodeProximity().sortedByProximity(localEndpoint, liveExcludingSelf).get(0).endpoint();
if (targetReplica == null)
{
logger.error("No live replica available to forward PaxosPrepareRefresh for {}",
logger.error("No live replica available to forward PaxosPrepareRefresh for {}.{} partition {}",
send.payload.missingCommit.metadata().keyspace, send.payload.missingCommit.metadata().name,
send.payload.missingCommit.partitionKey());
// Report failure for all refresh targets
for (InetAddressAndPort target : refreshTargets)
callbacks.onRefreshFailure(target, RequestFailure.UNKNOWN);
return;
@ -244,54 +249,68 @@ public class PaxosPrepareRefresh implements RequestCallbackWithFailure<PaxosPrep
logger.debug("Forwarding PaxosPrepareRefresh to replica {} for mutation ID generation", targetReplica);
Tracing.trace("Forwarding PaxosPrepareRefresh to replica {}", targetReplica);
// Create forward request with refresh targets
PrepareRefreshForwardRequest forwardRequest = new PrepareRefreshForwardRequest(
send.payload.promised,
send.payload.missingCommit,
refreshTargets,
isUrgent
);
ImmutableList<InetAddressAndPort> immutableRefreshTargets = ImmutableList.copyOf(refreshTargets);
PrepareRefreshForwardRequest forwardRequest = new PrepareRefreshForwardRequest(send.payload.promised,
send.payload.missingCommit,
immutableRefreshTargets,
isUrgent);
Message<PrepareRefreshForwardRequest> message = Message.out(
Verb.PAXOS_PREPARE_REFRESH_FORWARD_REQ, forwardRequest, isUrgent);
Message<PrepareRefreshForwardRequest> message = Message.out(Verb.PAXOS_PREPARE_REFRESH_FORWARD_REQ,
forwardRequest, isUrgent);
// Send and handle response
MessagingService.instance().sendWithCallback(message, targetReplica,
new ForwardCallback(refreshTargets));
new ForwardCallback(immutableRefreshTargets));
}
/**
* Callback for forwarded refresh operations.
* Translates forward response to individual refresh callbacks.
* Receives multiple onResponse() calls (non-final + final) as the forward handler
* streams per-target results back incrementally.
* Caches the mutation ID from the first response so subsequent refresh() calls
* can dispatch directly without forwarding.
*/
private class ForwardCallback implements RequestCallbackWithFailure<PrepareRefreshForwardResponse>
{
private final List<InetAddressAndPort> refreshTargets;
private final boolean[] reported;
ForwardCallback(List<InetAddressAndPort> refreshTargets)
{
this.refreshTargets = refreshTargets;
this.reported = new boolean[refreshTargets.size()];
}
@Override
public void onResponse(Message<PrepareRefreshForwardResponse> message)
public synchronized void onResponse(Message<PrepareRefreshForwardResponse> message)
{
PrepareRefreshForwardResponse response = message.payload;
// Report results for each target
for (int i = 0; i < refreshTargets.size(); i++)
Message<Request> currentSend = send;
if (!response.mutationId.isNone() && currentSend.payload.missingCommit.mutation.id().isNone())
updateSendMessage(currentSend.payload.missingCommit.withMutationId(response.mutationId));
if (response.targetIndex != null && !reported[response.targetIndex])
{
InetAddressAndPort target = refreshTargets.get(i);
Ballot supersededBy = response.supersededBy.get(i);
callbacks.onRefreshSuccess(supersededBy, target);
reported[response.targetIndex] = true;
InetAddressAndPort target = refreshTargets.get(response.targetIndex);
if (PrepareRefreshForwardHandler.FAILED_SENTINEL.equals(response.supersededBy))
callbacks.onRefreshFailure(target, RequestFailure.UNKNOWN);
else
callbacks.onRefreshSuccess(response.supersededBy, target);
}
}
@Override
public void onFailure(InetAddressAndPort from, RequestFailure reason)
public synchronized void onFailure(InetAddressAndPort from, RequestFailure reason)
{
// Report failure for all targets
for (InetAddressAndPort target : refreshTargets)
callbacks.onRefreshFailure(target, reason);
for (int i = 0; i < refreshTargets.size(); i++)
{
if (!reported[i])
{
reported[i] = true;
callbacks.onRefreshFailure(refreshTargets.get(i), reason);
}
}
}
}
@ -315,7 +334,10 @@ public class PaxosPrepareRefresh implements RequestCallbackWithFailure<PaxosPrep
{
response = RequestHandler.execute(send.payload, getBroadcastAddressAndPort());
if (response == null)
{
onFailure(getBroadcastAddressAndPort(), RequestFailure.UNKNOWN);
return;
}
}
catch (RetryOnDifferentSystemException e)
{
@ -365,7 +387,16 @@ public class PaxosPrepareRefresh implements RequestCallbackWithFailure<PaxosPrep
@Override
public void doVerb(Message<Request> message)
{
ClusterMetadataService.instance().fetchLogFromPeerOrCMS(ClusterMetadata.current(), message.from(), message.epoch());
ClusterMetadata metadata = ClusterMetadataService.instance().fetchLogFromPeerOrCMS(ClusterMetadata.current(),
message.from(),
message.epoch());
Committed commit = message.payload.missingCommit;
MigrationRouter.checkPaxosCommitMigration(metadata, message, message.from(),
commit.metadata().keyspace, commit.metadata().id,
commit.partitionKey().getToken(),
!commit.mutation.id().isNone());
try
{
Response response = execute(message.payload, message.from());

2
src/java/org/apache/cassandra/service/paxos/PaxosRepair.java
View File

@ -343,7 +343,7 @@ public class PaxosRepair extends AbstractPaxosRepair
// with a newer ballot)
FoundIncompleteAccepted incomplete = input.incompleteAccepted();
Proposal propose = new Proposal(incomplete.ballot, incomplete.accepted.update);
Proposal propose = new Proposal(incomplete.ballot, incomplete.accepted.mutation);
logger.trace("PaxosRepair of {} found incomplete {}", partitionKey(), incomplete.accepted);
return PaxosPropose.propose(propose, participants, false,
new ProposingRepair(propose)); // we don't know if we're done, so we must restart

27
src/java/org/apache/cassandra/service/paxos/PaxosState.java
View File

@ -59,6 +59,7 @@ import org.apache.cassandra.service.consensus.migration.ConsensusKeyMigrationSta
import org.apache.cassandra.service.paxos.uncommitted.PaxosBallotTracker;
import org.apache.cassandra.service.paxos.uncommitted.PaxosStateTracker;
import org.apache.cassandra.service.paxos.uncommitted.PaxosUncommittedTracker;
import org.apache.cassandra.service.replication.migration.MigrationRouter;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.utils.Nemesis;
@ -729,18 +730,28 @@ public class PaxosState implements PaxosOperationLock
Tracing.trace("Not committing proposal {} as ballot timestamp predates last truncation time", commit);
// Still acknowledge mutation ID for tracked keyspaces even though we're discarding
// This ensures the tracking service knows this replica "handled" the mutation.
// We call both startWriting and finishWriting - startWriting registers the mutation
// data (for reconciliation) and finishWriting marks it as witnessed.
// This is needed because there are cases where mutation IDs might be created for the same Paxos
// commit.
// the data. This ensures the tracking service considers this mutation handled by
// this replica, preventing false positives during reconciliation.
Mutation mutation = commit.makeMutation();
if (!mutation.id().isNone())
{
KeyspaceMetadata ksm = Schema.instance.getKeyspaceMetadata(mutation.getKeyspaceName());
if (ksm != null && ksm.params.replicationType.isTracked()
&& MutationTrackingService.instance().startWriting(mutation))
MutationTrackingService.instance().finishWriting(mutation);
if (ksm != null
&& MigrationRouter.shouldUseTrackedForWrites(ksm.name, mutation.getOnlyUpdate().metadata().id, mutation.key().getToken()))
{
if (MutationTrackingService.instance().startWriting(mutation))
MutationTrackingService.instance().finishWriting(mutation);
}
else if (ksm == null)
{
logger.warn("Paxos commit has mutation ID {} but keyspace {} not found in schema - skipping mutation tracking",
mutation.id(), mutation.getKeyspaceName());
}
else
{
logger.warn("Paxos commit has mutation ID {} but keyspace {}.{} partition {} is not tracked - skipping mutation tracking",
mutation.id(), mutation.getKeyspaceName(), mutation.getOnlyUpdate().metadata().name, mutation.key());
}
}
}

130
src/java/org/apache/cassandra/service/paxos/PrepareRefreshForwardHandler.java
View File

@ -18,10 +18,9 @@
package org.apache.cassandra.service.paxos;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.TimeUnit;
import javax.annotation.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -33,109 +32,97 @@ import org.apache.cassandra.exceptions.RequestFailureReason;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.net.IVerbHandler;
import org.apache.cassandra.net.Message;
import org.apache.cassandra.net.MessageFlag;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.net.RequestCallbackWithFailure;
import org.apache.cassandra.replication.MutationId;
import org.apache.cassandra.replication.MutationTrackingService;
import org.apache.cassandra.schema.KeyspaceMetadata;
import org.apache.cassandra.schema.Schema;
import org.apache.cassandra.service.paxos.Commit.Committed;
import org.apache.cassandra.tcm.ClusterMetadataService;
import org.apache.cassandra.service.replication.migration.MigrationRouter;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tracing.Tracing;
import org.apache.cassandra.utils.Clock;
import org.apache.cassandra.utils.FBUtilities;
import org.apache.cassandra.utils.concurrent.CountDownLatch;
import static org.apache.cassandra.net.Verb.PAXOS2_PREPARE_REFRESH_REQ;
import static org.apache.cassandra.service.paxos.PaxosRequestCallback.shouldExecuteOnSelf;
/**
* Handler for forwarded PaxosPrepareRefresh requests.
* Generates a mutation ID and sends the refresh to all target nodes.
*
* This handler is invoked when a non-replica coordinator forwards the refresh
* to a full replica that can generate the mutation ID.
*
* TODO (expected): more comprehensive testing
* Generates a mutation ID and sends the refresh to all target nodes,
* streaming per-target results back to the coordinator incrementally
* using the NOT_FINAL message flag pattern.
*/
public class PrepareRefreshForwardHandler implements IVerbHandler<PrepareRefreshForwardRequest>
{
public static final PrepareRefreshForwardHandler instance = new PrepareRefreshForwardHandler();
private static final Logger logger = LoggerFactory.getLogger(PrepareRefreshForwardHandler.class);
static final Ballot FAILED_SENTINEL = Ballot.none();
@Override
public void doVerb(Message<PrepareRefreshForwardRequest> message)
{
ClusterMetadataService.instance().fetchLogFromPeerOrCMS(message.from(), message.header.epoch);
ClusterMetadata metadata = ClusterMetadata.current();
PrepareRefreshForwardRequest request = message.payload;
Tracing.trace("Executing forwarded PaxosPrepareRefresh for {}", request.commit.partitionKey());
try
{
String ksName = request.commit.metadata().keyspace;
KeyspaceMetadata ksMetadata = Schema.instance.getKeyspaceMetadata(ksName);
if (ksMetadata == null)
{
MessagingService.instance().respondWithFailure(RequestFailureReason.INCOMPATIBLE_SCHEMA, message);
logger.error("Failed to forward paxos prepare refresh for non-existent keyspace {}", ksName);
return;
}
if (!ksMetadata.params.replicationType.isTracked())
{
MessagingService.instance().respondWithFailure(RequestFailureReason.INCOMPATIBLE_SCHEMA, message);
logger.error("Asked to perform forwarded prepare refresh, but keyspace {} is not tracked", ksName);
return;
}
Token token = request.commit.partitionKey().getToken();
if (metadata.schema.getKeyspaces().getNullable(ksName) == null)
{
MessagingService.instance().respondWithFailure(RequestFailureReason.INCOMPATIBLE_SCHEMA, message);
logger.error("Failed to forward prepare refresh for non-existent keyspace {}.{} partition {}",
ksName, request.commit.metadata().name, request.commit.partitionKey());
return;
}
MigrationRouter.checkPaxosCommitMigration(metadata, message, message.from(),
ksName, request.commit.metadata().id, token, true);
MutationId mutationId = MutationTrackingService.instance().nextMutationId(ksName, token);
Mutation mutationWithId = request.commit.makeMutation(mutationId);
Committed commitWithId = new Commit.Committed(request.commit.ballot, mutationWithId);
// Now send the refresh to all targets and collect responses
List<InetAddressAndPort> targets = request.refreshTargets;
List<Ballot> supersededBy = Collections.synchronizedList(new ArrayList<>(Collections.nCopies(targets.size(), null)));
CountDownLatch latch = CountDownLatch.newCountDownLatch(targets.size());
Message<PaxosPrepareRefresh.Request> refreshMsg = Message.out(
PAXOS2_PREPARE_REFRESH_REQ,
new PaxosPrepareRefresh.Request(request.promised, commitWithId),
request.isUrgent
);
// For tracked keyspaces, we MUST ALWAYS write to the local journal since we generated the mutation ID.
// This is required for retry purposes: if a remote target fails, the ActiveLogReconciler will try
// to look up the mutation in the local journal. The node that generated the mutation ID is the "owner"
// and must have the mutation available for retries.
//
// This is different from checking if self is in targets - even if we're not in targets,
// we're still the ID generator and need the mutation locally.
PaxosPrepareRefresh.Response localResponse = null;
try
{
PaxosPrepareRefresh.RequestHandler.execute(
new PaxosPrepareRefresh.Request(request.promised, commitWithId), FBUtilities.getBroadcastAddressAndPort());
// Note: we don't use the response since this node may not be in targets
localResponse = PaxosPrepareRefresh.RequestHandler.execute(new PaxosPrepareRefresh.Request(request.promised, commitWithId),
FBUtilities.getBroadcastAddressAndPort());
}
catch (Exception e)
{
// Log but continue - we still need to send to targets
logger.warn("Failed to execute local commit for tracked keyspace mutation {}", mutationId, e);
}
// Now send to remote targets
if (localResponse == null)
{
MessagingService.instance().respondWithFailure(RequestFailureReason.UNKNOWN, message);
logger.error("Aborting forwarded PaxosPrepareRefresh: local journal write failed for mutation {}", mutationId);
return;
}
List<InetAddressAndPort> targets = request.refreshTargets;
int[] remaining = new int[] { targets.size() };
Message<PaxosPrepareRefresh.Request> refreshMsg = Message.out(PAXOS2_PREPARE_REFRESH_REQ,
new PaxosPrepareRefresh.Request(request.promised, commitWithId),
request.isUrgent);
for (int i = 0; i < targets.size(); i++)
{
final int targetIndex = i;
InetAddressAndPort target = targets.get(i);
// Check if self is in targets for response tracking (separate from the local write above)
// We need to decrement the latch for the local target since we already executed above
if (shouldExecuteOnSelf(target))
{
latch.decrement();
respond(message, mutationId, targetIndex, localResponse.isSupersededBy, remaining);
continue;
}
@ -144,36 +131,18 @@ public class PrepareRefreshForwardHandler implements IVerbHandler<PrepareRefresh
@Override
public void onResponse(Message<PaxosPrepareRefresh.Response> response)
{
supersededBy.set(targetIndex, response.payload.isSupersededBy);
latch.decrement();
respond(message, mutationId, targetIndex, response.payload.isSupersededBy, remaining);
}
@Override
public void onFailure(InetAddressAndPort from, RequestFailure reason)
{
// Leave null to indicate we didn't get a definitive answer
latch.decrement();
respond(message, mutationId, targetIndex, FAILED_SENTINEL, remaining);
}
};
MessagingService.instance().sendWithCallback(refreshMsg, target, callback);
}
// Wait for all responses with timeout
long timeoutNanos = message.expiresAtNanos() - Clock.Global.nanoTime();
boolean completed = latch.await(Math.max(0, timeoutNanos), TimeUnit.NANOSECONDS);
if (!completed)
logger.warn("Forwarded PaxosPrepareRefresh timed out waiting for responses");
// Send aggregated response back to original coordinator
MessagingService.instance().respond(new PrepareRefreshForwardResponse(supersededBy), message);
}
catch (InterruptedException e)
{
Thread.currentThread().interrupt();
MessagingService.instance().respondWithFailure(RequestFailure.forException(e), message);
logger.error("Forwarded PaxosPrepareRefresh interrupted", e);
}
catch (Exception e)
{
@ -181,4 +150,19 @@ public class PrepareRefreshForwardHandler implements IVerbHandler<PrepareRefresh
logger.error("Failed to execute forwarded PaxosPrepareRefresh for {}", request.commit, e);
}
}
private void respond(Message<?> request, MutationId mutationId, int targetIndex,
@Nullable Ballot supersededBy, int[] remaining)
{
PrepareRefreshForwardResponse payload = new PrepareRefreshForwardResponse(mutationId, targetIndex, supersededBy);
Message<PrepareRefreshForwardResponse> response = request.responseWith(payload);
//noinspection SynchronizationOnLocalVariableOrMethodParameter
synchronized (remaining)
{
boolean isFinal = --remaining[0] == 0;
if (!isFinal)
response = response.withFlag(MessageFlag.NOT_FINAL);
MessagingService.instance().send(response, request.respondTo());
}
}
}

59
src/java/org/apache/cassandra/service/paxos/PrepareRefreshForwardResponse.java
View File

@ -19,30 +19,44 @@
package org.apache.cassandra.service.paxos;
import java.io.IOException;
import java.util.List;
import javax.annotation.Nullable;
import org.apache.cassandra.db.TypeSizes;
import org.apache.cassandra.io.IVersionedSerializer;
import org.apache.cassandra.io.util.DataInputPlus;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.utils.CollectionSerializers;
import org.apache.cassandra.replication.MutationId;
import org.apache.cassandra.utils.NullableSerializer;
/**
* Response from a forwarded PaxosPrepareRefresh operation.
* Contains the superseding ballot for each refresh target (null if confirmed).
* Each response carries a single target result (targetIndex + supersededBy),
* or just the mutation ID if targetIndex is null.
* Multiple non-final responses are sent incrementally as targets respond,
* followed by a final response for the last target.
*/
public class PrepareRefreshForwardResponse
{
public static final Serializer serializer = new Serializer();
/**
* List of superseding ballots, one per refresh target.
* Null entry means the promise was confirmed for that target.
*/
public final List<Ballot> supersededBy;
public final MutationId mutationId;
@Nullable
public final Integer targetIndex;
@Nullable
public final Ballot supersededBy;
public PrepareRefreshForwardResponse(List<Ballot> supersededBy)
public PrepareRefreshForwardResponse(MutationId mutationId)
{
this.mutationId = mutationId;
this.targetIndex = null;
this.supersededBy = null;
}
public PrepareRefreshForwardResponse(MutationId mutationId, int targetIndex, @Nullable Ballot supersededBy)
{
this.mutationId = mutationId;
this.targetIndex = targetIndex;
this.supersededBy = supersededBy;
}
@ -53,20 +67,39 @@ public class PrepareRefreshForwardResponse
@Override
public void serialize(PrepareRefreshForwardResponse response, DataOutputPlus out, int version) throws IOException
{
CollectionSerializers.serializeList(response.supersededBy, out, version, NULLABLE_BALLOT_SERIALIZER);
MutationId.serializer.serialize(response.mutationId, out);
boolean hasTarget = response.targetIndex != null;
out.writeBoolean(hasTarget);
if (hasTarget)
{
out.writeUnsignedVInt32(response.targetIndex);
NULLABLE_BALLOT_SERIALIZER.serialize(response.supersededBy, out, version);
}
}
@Override
public PrepareRefreshForwardResponse deserialize(DataInputPlus in, int version) throws IOException
{
List<Ballot> supersededBy = CollectionSerializers.deserializeList(in, version, NULLABLE_BALLOT_SERIALIZER);
return new PrepareRefreshForwardResponse(supersededBy);
MutationId mutationId = MutationId.serializer.deserialize(in);
boolean hasTarget = in.readBoolean();
if (!hasTarget)
return new PrepareRefreshForwardResponse(mutationId);
int targetIndex = in.readUnsignedVInt32();
Ballot supersededBy = NULLABLE_BALLOT_SERIALIZER.deserialize(in, version);
return new PrepareRefreshForwardResponse(mutationId, targetIndex, supersededBy);
}
@Override
public long serializedSize(PrepareRefreshForwardResponse response, int version)
{
return CollectionSerializers.serializedListSize(response.supersededBy, version, NULLABLE_BALLOT_SERIALIZER);
long size = MutationId.serializer.serializedSize(response.mutationId);
size += 1; // hasTarget boolean
if (response.targetIndex != null)
{
size += TypeSizes.sizeofUnsignedVInt(response.targetIndex);
size += NULLABLE_BALLOT_SERIALIZER.serializedSize(response.supersededBy, version);
}
return size;
}
}
}

3
src/java/org/apache/cassandra/service/reads/tracked/ReadReconciliations.java
View File

@ -35,6 +35,7 @@ import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.net.Message;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.net.Verb;
import org.apache.cassandra.replication.ActiveLogReconciler;
import org.apache.cassandra.replication.CoordinatorLogId;
import org.apache.cassandra.replication.ExpiredStatePurger;
import org.apache.cassandra.replication.IncomingMutations;
@ -397,7 +398,7 @@ public class ReadReconciliations implements ExpiredStatePurger.Expireable
if (!toPull.isEmpty())
{
PullMutationsRequest pull = new PullMutationsRequest(Offsets.Immutable.copy(toPull));
PullMutationsRequest pull = new PullMutationsRequest(Offsets.Immutable.copy(toPull), ActiveLogReconciler.Priority.HIGH);
logger.debug("Pulling {} from {}", pull, pullFrom);
MessagingService.instance().send(Message.out(Verb.MT_PULL_MUTATIONS_REQ, pull), pullFrom);
}

169
src/java/org/apache/cassandra/service/replication/migration/MigrationRouter.java
View File

@ -25,19 +25,28 @@ import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.cassandra.db.EmbeddableSinglePartitionReadCommand;
import org.apache.cassandra.db.IMutation;
import org.apache.cassandra.db.PartitionPosition;
import org.apache.cassandra.db.PartitionRangeReadCommand;
import org.apache.cassandra.db.SinglePartitionReadCommand;
import org.apache.cassandra.db.virtual.VirtualMutation;
import org.apache.cassandra.dht.AbstractBounds;
import org.apache.cassandra.dht.NormalizedRanges;
import org.apache.cassandra.dht.Range;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.exceptions.CoordinatorBehindException;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.metrics.TCMMetrics;
import org.apache.cassandra.net.Message;
import org.apache.cassandra.schema.KeyspaceMetadata;
import org.apache.cassandra.schema.SchemaConstants;
import org.apache.cassandra.schema.TableId;
import org.apache.cassandra.schema.TableMetadata;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.ClusterMetadataService;
import org.apache.cassandra.utils.Pair;
import static com.google.common.base.Preconditions.checkState;
@ -49,23 +58,14 @@ import static com.google.common.base.Preconditions.checkState;
*/
public class MigrationRouter
{
public static boolean shouldUseTracked(SinglePartitionReadCommand command)
private static final Logger logger = LoggerFactory.getLogger(MigrationRouter.class);
public static boolean shouldUseTracked(EmbeddableSinglePartitionReadCommand command)
{
// System keyspaces never use tracked replication
if (SchemaConstants.isSystemKeyspace(command.metadata().keyspace))
return false;
ClusterMetadata metadata = ClusterMetadata.current();
KeyspaceMigrationInfo migrationInfo = metadata.mutationTrackingMigrationState.getKeyspaceInfo(command.metadata().keyspace);
if (migrationInfo == null)
return command.metadata().replicationType().isTracked();
Token token = command.partitionKey().getToken();
boolean isTracked = command.metadata().replicationType().isTracked();
return migrationInfo.shouldUseTrackedForReads(isTracked, command.metadata().id(), token);
return shouldUseTracked(ClusterMetadata.current(), command);
}
/**
@ -215,13 +215,18 @@ public class MigrationRouter
PartitionRangeReadCommand command)
{
// System keyspaces never use tracked replication
if (SchemaConstants.isSystemKeyspace(command.metadata().keyspace))
String keyspace = command.metadata().keyspace;
if (SchemaConstants.isSystemKeyspace(keyspace) || command.metadata().kind != TableMetadata.Kind.REGULAR)
return ImmutableList.of(new RangeReadWithReplication(command, false));
KeyspaceMetadata ksm = metadata.schema.maybeGetKeyspaceMetadata(keyspace).orElse(null);
if (ksm == null)
return ImmutableList.of(new RangeReadWithReplication(command, false));
KeyspaceMigrationInfo migrationInfo = metadata.mutationTrackingMigrationState
.getKeyspaceInfo(command.metadata().keyspace);
.getKeyspaceInfo(keyspace);
boolean isTracked = command.metadata().replicationType().isTracked();
boolean isTracked = ksm.params.replicationType.isTracked();
// During migration, reads use untracked replication except for ranges that have
// completed migration to tracked. Therefore, we only need to split ranges when
@ -254,16 +259,26 @@ public class MigrationRouter
if (SchemaConstants.isSystemKeyspace(keyspace))
return false;
KeyspaceMigrationInfo migrationInfo = metadata.mutationTrackingMigrationState
.getKeyspaceInfo(keyspace);
KeyspaceMigrationInfo migrationInfo = metadata.mutationTrackingMigrationState.getKeyspaceInfo(keyspace);
if (migrationInfo == null)
return metadata.schema.getKeyspaceMetadata(keyspace).params.replicationType.isTracked();
{
KeyspaceMetadata ksm = metadata.schema.maybeGetKeyspaceMetadata(keyspace).orElse(null);
return ksm != null && ksm.params.replicationType.isTracked();
}
boolean isTracked = metadata.schema.getKeyspaceMetadata(keyspace).params.replicationType.isTracked();
KeyspaceMetadata ksm = metadata.schema.maybeGetKeyspaceMetadata(keyspace).orElse(null);
if (ksm == null)
return false;
boolean isTracked = ksm.params.replicationType.isTracked();
return migrationInfo.shouldUseTrackedForWrites(isTracked, tableId, token);
}
public static boolean shouldUseTrackedForWrites(String keyspace, TableId tableId, Token token)
{
return shouldUseTrackedForWrites(ClusterMetadata.current(), keyspace, tableId, token);
}
public static class RoutedMutations
{
public final List<? extends IMutation> trackedMutations;
@ -392,4 +407,116 @@ public class MigrationRouter
{
validateMutationReplication(mutation, MutationRouting.UNTRACKED);
}
/**
* Validate that coordinator and handler agree on tracked/untracked routing for a Paxos commit.
* Fetches log from coordinator only on mismatch when coordinator epoch is ahead.
*
* @return updated ClusterMetadata after any fetch
*/
public static ClusterMetadata checkPaxosCommitMigration(ClusterMetadata metadata,
Message<?> message,
InetAddressAndPort respondTo,
String keyspace,
TableId tableId,
Token token,
boolean coordinatorSaysTracked)
{
boolean handlerSaysTracked = shouldUseTrackedForWrites(metadata, keyspace, tableId, token);
if (coordinatorSaysTracked == handlerSaysTracked)
return metadata;
if (message.epoch().isAfter(metadata.epoch))
{
metadata = ClusterMetadataService.instance().fetchLogFromPeerOrCMS(metadata, respondTo, message.epoch());
handlerSaysTracked = shouldUseTrackedForWrites(metadata, keyspace, tableId, token);
if (coordinatorSaysTracked == handlerSaysTracked)
return metadata;
}
if (message.epoch().isBefore(metadata.epoch))
{
TCMMetrics.instance.coordinatorBehindReplication.mark();
logger.warn("COORDINATOR_BEHIND: Paxos commit migration mismatch for keyspace {} table {} token {}, coordinator {} at epoch {} is behind our epoch {}",
keyspace, tableId, token, respondTo, message.epoch(), metadata.epoch);
throw new CoordinatorBehindException(String.format("Paxos commit migration mismatch for keyspace: %s token %s, coordinator: %s is behind, our epoch = %s, their epoch = %s",
keyspace, token, respondTo, metadata.epoch, message.epoch()));
}
else
{
logger.error("Inconsistent Paxos commit routing at same epoch {} for keyspace {} table {} token {} - coordinator says tracked={}, handler says tracked={}",
metadata.epoch, keyspace, tableId, token, coordinatorSaysTracked, handlerSaysTracked);
throw new IllegalStateException(String.format("Inconsistent Paxos commit routing at epoch = %s. Keyspace: %s token: %s",
metadata.epoch, keyspace, token));
}
}
/**
* Validate that a Paxos prepare read type matches the handler's migration state.
* Uses the conditional-fetch pattern: check first, fetch only on mismatch when coordinator is ahead.
*
* Both directions are validated:
* - Tracked read → untracked handler: CRITICAL — tracked reads lose monotonicity when writes are untracked.
* - Untracked read → tracked handler: less bad (untracked quorum reads are self-consistent) but still
* indicates epoch disagreement that should be resolved.
*
* @return updated ClusterMetadata after any fetch
*/
public static ClusterMetadata checkPaxosPrepareReadMigration(ClusterMetadata metadata,
Message<?> message,
InetAddressAndPort respondTo,
EmbeddableSinglePartitionReadCommand read)
{
boolean coordinatorSaysTracked = read.isTracked();
boolean handlerSaysTracked = shouldUseTracked(metadata, read);
if (coordinatorSaysTracked == handlerSaysTracked)
return metadata;
if (message.epoch().isAfter(metadata.epoch))
{
metadata = ClusterMetadataService.instance().fetchLogFromPeerOrCMS(metadata, respondTo, message.epoch());
handlerSaysTracked = shouldUseTracked(metadata, read);
if (coordinatorSaysTracked == handlerSaysTracked)
return metadata;
}
if (message.epoch().isBefore(metadata.epoch))
{
TCMMetrics.instance.coordinatorBehindReplication.mark();
logger.warn("COORDINATOR_BEHIND: Paxos prepare read migration mismatch for {}.{} partition {}, coordinator {} at epoch {} is behind our epoch {}",
read.metadata().keyspace, read.metadata().name, read.partitionKey(), respondTo, message.epoch(), metadata.epoch);
throw new CoordinatorBehindException(String.format("Paxos prepare read migration mismatch for keyspace: %s token %s, coordinator: %s is behind, our epoch = %s, their epoch = %s",
read.metadata().keyspace, read.partitionKey().getToken(), respondTo, metadata.epoch, message.epoch()));
}
else
{
logger.error("Inconsistent Paxos prepare read routing at same epoch {} for {}.{} partition {} - coordinator says tracked={}, handler says tracked={}",
metadata.epoch, read.metadata().keyspace, read.metadata().name, read.partitionKey(), coordinatorSaysTracked, handlerSaysTracked);
throw new IllegalStateException(String.format("Inconsistent Paxos prepare read routing at epoch = %s. Keyspace: %s token: %s",
metadata.epoch, read.metadata().keyspace, read.partitionKey().getToken()));
}
}
public static boolean shouldUseTracked(ClusterMetadata metadata, EmbeddableSinglePartitionReadCommand command)
{
if (command.metadata().kind != TableMetadata.Kind.REGULAR)
return false;
String keyspace = command.metadata().keyspace;
if (SchemaConstants.isSystemKeyspace(keyspace))
return false;
KeyspaceMetadata ksm = metadata.schema.maybeGetKeyspaceMetadata(keyspace).orElse(null);
if (ksm == null)
return false;
KeyspaceMigrationInfo migrationInfo = metadata.mutationTrackingMigrationState.getKeyspaceInfo(keyspace);
boolean isTracked = ksm.params.replicationType.isTracked();
if (migrationInfo == null)
return isTracked;
Token token = command.partitionKey().getToken();
return migrationInfo.shouldUseTrackedForReads(isTracked, command.metadata().id(), token);
}
}

43
test/distributed/org/apache/cassandra/distributed/shared/ClusterUtils.java
View File

@ -1773,4 +1773,47 @@ public class ClusterUtils
return null;
});
}
/**
* Wait for all nodes in the cluster to reach at least the highest TCM epoch currently observed.
* Useful after tests that block TCM propagation to specific nodes.
*/
public static void awaitTCMCatchUp(ICluster<?> cluster)
{
long maxEpoch = 0;
boolean foundRunning = false;
for (int i = 1; i <= cluster.size(); i++)
{
IInvokableInstance inst = (IInvokableInstance) cluster.get(i);
if (inst.isShutdown())
continue;
foundRunning = true;
maxEpoch = Math.max(maxEpoch, inst.callOnInstance(() -> ClusterMetadata.current().epoch.getEpoch()));
}
if (!foundRunning)
throw new AssertionError("No running nodes to await TCM catch-up on");
long target = maxEpoch;
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(30);
List<String> lagging = new ArrayList<>();
for (int i = 1; i <= cluster.size(); i++)
{
IInvokableInstance inst = (IInvokableInstance) cluster.get(i);
if (inst.isShutdown())
continue;
int node = i;
while (System.nanoTime() < deadline)
{
long nodeEpoch = inst.callOnInstance(() -> ClusterMetadata.current().epoch.getEpoch());
if (nodeEpoch >= target)
break;
sleepUninterruptibly(100, TimeUnit.MILLISECONDS);
}
long finalEpoch = inst.callOnInstance(() -> ClusterMetadata.current().epoch.getEpoch());
if (finalEpoch < target)
lagging.add(String.format("node%d(epoch=%d)", node, finalEpoch));
}
if (!lagging.isEmpty())
throw new AssertionError(String.format("Timed out waiting for nodes to reach epoch %d; lagging: %s", target, lagging));
}
}

83
test/distributed/org/apache/cassandra/distributed/test/tracking/AssertingLatch.java Normal file
View File

@ -0,0 +1,83 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.distributed.test.tracking;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
/**
* A CountDownLatch wrapper that throws on timeout. Interruption during shutdown is benign.
*/
public class AssertingLatch
{
private static final long DEFAULT_TIMEOUT = 30;
private static final TimeUnit DEFAULT_UNIT = TimeUnit.SECONDS;
private final CountDownLatch latch;
private final String description;
private final long timeout;
private final TimeUnit unit;
public AssertingLatch(String description)
{
this(1, DEFAULT_TIMEOUT, DEFAULT_UNIT, description);
}
public AssertingLatch(int count, String description)
{
this(count, DEFAULT_TIMEOUT, DEFAULT_UNIT, description);
}
public AssertingLatch(int count, long timeout, TimeUnit unit, String description)
{
this.latch = new CountDownLatch(count);
this.description = description;
this.timeout = timeout;
this.unit = unit;
}
public void await()
{
try
{
if (!latch.await(timeout, unit))
throw new AssertionError("Timed out after " + timeout + " " + unit + " waiting for: " + description);
}
catch (InterruptedException e)
{
Thread.currentThread().interrupt();
}
}
public void countDown()
{
latch.countDown();
}
public void release()
{
while (latch.getCount() > 0)
latch.countDown();
}
public long getCount()
{
return latch.getCount();
}
}

685
test/distributed/org/apache/cassandra/distributed/test/tracking/PaxosMigrationTestUtils.java Normal file
View File

@ -0,0 +1,685 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.distributed.test.tracking;
import java.lang.reflect.Field;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.cassandra.Util;
import org.apache.cassandra.db.EmbeddableSinglePartitionReadCommand;
import org.apache.cassandra.db.Keyspace;
import org.apache.cassandra.db.Mutation;
import org.apache.cassandra.db.marshal.Int32Type;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.distributed.Cluster;
import org.apache.cassandra.distributed.api.ConsistencyLevel;
import org.apache.cassandra.distributed.api.Feature;
import org.apache.cassandra.distributed.api.IMessage;
import org.apache.cassandra.distributed.api.IMessageFilters;
import org.apache.cassandra.distributed.impl.Instance;
import org.apache.cassandra.exceptions.CasWriteTimeoutException;
import org.apache.cassandra.exceptions.CasWriteUnknownResultException;
import org.apache.cassandra.exceptions.RequestFailureReason;
import org.apache.cassandra.exceptions.WriteFailureException;
import org.apache.cassandra.exceptions.WriteTimeoutException;
import org.apache.cassandra.hints.HintsService;
import org.apache.cassandra.net.Message;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.net.Verb;
import org.apache.cassandra.replication.MutationId;
import org.apache.cassandra.replication.MutationTrackingService;
import org.apache.cassandra.schema.ReplicationType;
import org.apache.cassandra.service.paxos.Commit;
import org.apache.cassandra.tcm.ClusterMetadata;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
/**
* Utility methods for Paxos mutation tracking migration tests.
*
* Reduces boilerplate around deserializing messages, extracting MutationIds,
* generating synthetic failure responses, stranding nodes at old TCM epochs, and
* composing common cluster-level filter-and-count spy patterns.
*/
public class PaxosMigrationTestUtils
{
private static final AtomicInteger KEYSPACE_COUNTER = new AtomicInteger();
private PaxosMigrationTestUtils()
{
}
// Must be called inside callsOnInstance() or runOnInstance() on the receiving node
public static boolean messageHasMutationId(IMessage msg)
{
Message<?> deserialized = Instance.deserializeMessage(msg);
return payloadHasMutationId(deserialized.payload);
}
// Must be called inside callsOnInstance() on the receiving node.
// Returns true if the prepare message carries a tracked read, false if untracked, null if no read field.
public static Boolean messageReadIsTracked(IMessage msg)
{
Message<?> deserialized = Instance.deserializeMessage(msg);
Object payload = deserialized.payload;
try
{
Field readField = null;
Class<?> cls = payload.getClass();
while (cls != null && readField == null)
{
try { readField = cls.getDeclaredField("read"); }
catch (NoSuchFieldException e) { cls = cls.getSuperclass(); }
}
if (readField == null)
return null;
readField.setAccessible(true);
Object read = readField.get(payload);
if (read == null)
return null;
return ((EmbeddableSinglePartitionReadCommand) read).isTracked();
}
catch (IllegalAccessException e)
{
throw new RuntimeException(e);
}
}
// Must be called inside runOnInstance() on the receiving node
public static void respondWithTimeout(IMessage msg)
{
Message<?> deserialized = Instance.deserializeMessage(msg);
MessagingService.instance().respondWithFailure(RequestFailureReason.TIMEOUT, deserialized);
}
private static MutationId extractMutationIdFromPayload(Object payload)
{
if (payload instanceof Commit)
return ((Commit) payload).mutation.id();
if (payload instanceof Mutation)
return ((Mutation) payload).id();
try
{
for (Field f : payload.getClass().getDeclaredFields())
{
if (f.getName().equals("missingCommit") || f.getName().equals("mutation") || f.getName().equals("commit"))
{
f.setAccessible(true);
Object val = f.get(payload);
if (val instanceof Commit)
return ((Commit) val).mutation.id();
if (val instanceof Mutation)
return ((Mutation) val).id();
}
}
}
catch (IllegalAccessException e)
{
throw new RuntimeException(e);
}
return MutationId.none();
}
private static boolean payloadHasMutationId(Object payload)
{
return !extractMutationIdFromPayload(payload).isNone();
}
public static void awaitReplicationType(Cluster cluster, String keyspace, ReplicationType expected, int... nodes)
{
boolean expectTracked = expected == ReplicationType.tracked;
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(30);
for (int node : nodes)
{
while (System.nanoTime() < deadline)
{
boolean isTracked = cluster.get(node).callOnInstance(() ->
ClusterMetadata.current().schema.getKeyspaceMetadata(keyspace).params.replicationType.isTracked());
if (isTracked == expectTracked)
break;
Thread.yield();
}
boolean isTracked = cluster.get(node).callOnInstance(() ->
ClusterMetadata.current().schema.getKeyspaceMetadata(keyspace).params.replicationType.isTracked());
if (isTracked != expectTracked)
throw new AssertionError("Node " + node + " did not see replicationType " +
expected + " for " + keyspace + " within 30s");
}
}
// --- Cluster builder helpers ---
/**
* Build a Cluster.Builder preconfigured for Paxos migration tests: NETWORK + GOSSIP, given
* paxos_variant, long (10s) write/CAS contention timeouts. The caller is responsible for
* calling .start() and wrapping in init() (which requires TestBaseImpl access), and for
* invoking {@link #pauseHintsAndReconciler(Cluster)} after the cluster is live.
*/
public static Cluster.Builder buildPaxosCluster(int nodes, String paxosVariant)
{
return Cluster.build(nodes)
.withConfig(cfg -> cfg.with(Feature.NETWORK)
.with(Feature.GOSSIP)
.set("paxos_variant", paxosVariant)
.set("write_request_timeout", "10000ms")
.set("cas_contention_timeout", "10000ms"));
}
/**
* Pause hint delivery and the regular-priority mutation tracking reconciler on every node in
* the cluster. Intended to be called once in @BeforeClass after the cluster is started so that
* hints and reconciliation traffic don't bleed into tests.
*/
public static void pauseHintsAndReconciler(Cluster cluster)
{
cluster.forEach(instance -> instance.runOnInstance(() -> HintsService.instance.pauseDispatch()));
cluster.forEach(instance -> instance.runOnInstance(() -> MutationTrackingService.instance().pauseActiveReconcilerRegularPriority()));
}
// --- Keyspace / schema helpers ---
/**
* Creates a fresh keyspace with unique name (prefix_N) using SimpleStrategy at the requested
* RF and replication_type, plus a standard table {@code tbl (k int PRIMARY KEY, v int, v2 int)}
* covering every field used across the paxos-migration test suite.
*
* @return the generated keyspace name.
*/
public static String createKeyspace(Cluster cluster, String prefix, String replicationType, int rf)
{
String ks = prefix + '_' + KEYSPACE_COUNTER.incrementAndGet();
cluster.schemaChange("CREATE KEYSPACE " + ks + " WITH replication = " +
"{'class': 'SimpleStrategy', 'replication_factor': " + rf + "} " +
"AND replication_type='" + replicationType + "'");
cluster.schemaChange("CREATE TABLE " + ks + ".tbl (k int PRIMARY KEY, v int, v2 int)");
return ks;
}
/** Convenience overload defaulting to RF=3. */
public static String createKeyspace(Cluster cluster, String prefix, String replicationType)
{
return createKeyspace(cluster, prefix, replicationType, 3);
}
public static void alterReplicationType(Cluster cluster, String keyspace, ReplicationType replicationType)
{
cluster.schemaChange("ALTER KEYSPACE " + keyspace + " WITH replication = " +
"{'class': 'SimpleStrategy', 'replication_factor': 3} " +
"AND replication_type='" + replicationType + "'");
}
public static void alterReplicationTypeFrom(Cluster cluster,
int coordinator,
String keyspace,
ReplicationType replicationType,
ConsistencyLevel cl)
{
cluster.coordinator(coordinator).execute("ALTER KEYSPACE " + keyspace + " WITH replication = " +
"{'class': 'SimpleStrategy', 'replication_factor': 3} " +
"AND replication_type='" + replicationType + "'",
cl);
}
// --- Assertion helpers ---
public static void assertReplicasHaveValue(Cluster cluster, String keyspace, int key, Object value, int... nodes)
{
for (int node : nodes)
{
Util.spinAssertEquals(value,
() -> {
Object[][] r = cluster.get(node).executeInternal("SELECT v FROM " + keyspace + ".tbl WHERE k = " + key);
return r.length == 1 ? r[0][0] : null;
});
}
}
public static void assertReplicaHasNoRow(Cluster cluster, String keyspace, int key, int node)
{
Object[][] nodeResult = cluster.get(node).executeInternal("SELECT v FROM " + keyspace + ".tbl WHERE k = " + key);
assertEquals("Node " + node + " should have no row for k=" + key, 0, nodeResult.length);
}
public static void assertAllNodesSee(Cluster cluster, String keyspace, ReplicationType expected)
{
for (int i = 1; i <= cluster.size(); i++)
assertNodeSees(cluster, i, keyspace, expected);
}
public static void assertNodeSees(Cluster cluster, int node, String keyspace, ReplicationType expected)
{
boolean expectTracked = expected == ReplicationType.tracked;
boolean isTracked = cluster.get(node).callOnInstance(() ->
ClusterMetadata.current().schema.getKeyspaceMetadata(keyspace).params.replicationType.isTracked());
assertEquals("Node " + node + " should see replicationType " + expected + " for " + keyspace,
expectTracked, isTracked);
}
/**
* Fail-fast assertion: if the partitioner or token allocation changes such that {@code key}
* no longer maps to the expected replica set, every test that depends on that placement is
* silently invalid. Validate explicitly up-front so a placement regression fails loudly
* rather than via vacuous message counts.
*/
public static void assertReplicasAreExactly(Cluster cluster, String keyspace, int key, int[] expectedReplicas)
{
String[] actualReplicas = cluster.get(1).applyOnInstance((String ks, Integer k) -> {
Keyspace keyspaceObj = Keyspace.open(ks);
Token token = keyspaceObj.getColumnFamilyStore("tbl").getPartitioner()
.getToken(Int32Type.instance.decompose(k));
return keyspaceObj.getReplicationStrategy()
.calculateNaturalReplicas(token, ClusterMetadata.current())
.endpoints().stream()
.map(ep -> ep.getAddress().getHostAddress() + ":" + ep.getPort())
.toArray(String[]::new);
}, keyspace, key);
String[] expected = new String[expectedReplicas.length];
for (int i = 0; i < expectedReplicas.length; i++)
expected[i] = cluster.get(expectedReplicas[i]).broadcastAddress().getAddress().getHostAddress()
+ ":" + cluster.get(expectedReplicas[i]).broadcastAddress().getPort();
Set<String> actualSet = new HashSet<>(Arrays.asList(actualReplicas));
Set<String> expectedSet = new HashSet<>(Arrays.asList(expected));
assertEquals("KEY=" + key + " placement assumption violated. Actual replicas: " + actualSet
+ " Expected: " + expectedSet + " — tests that depend on this placement are invalid.",
expectedSet, actualSet);
}
/**
* Recognize a CAS-related write exception by simple-name string comparison. Uses string
* comparison because the exception may have been deserialized in a different classloader
* (in-JVM dtest nodes use isolated classloaders) and {@code instanceof} would return false.
*/
public static void assertCasException(Exception e)
{
String actual = e.getClass().getName();
boolean recognized = actual.equals(CasWriteTimeoutException.class.getName())
|| actual.equals(CasWriteUnknownResultException.class.getName())
|| actual.equals(WriteTimeoutException.class.getName())
|| actual.equals(WriteFailureException.class.getName());
assertTrue("Expected a CAS-related write exception but got " + actual + ": " + e.getMessage(),
recognized);
}
public static void assertCasApplied(Object[][] result)
{
assertNotNull("CAS should return a result", result);
assertEquals(1, result.length);
assertTrue("CAS should be applied", (boolean) result[0][0]);
}
public static void assertCasNotApplied(Object[][] result)
{
assertNotNull("CAS should return a result", result);
assertEquals(1, result.length);
assertFalse("CAS should NOT be applied", (boolean) result[0][0]);
}
// --- Async CAS helpers ---
/**
* Execute a CAS on the given coordinator at SERIAL/QUORUM asynchronously. The returned
* future completes with the result or completes exceptionally.
*/
public static CompletableFuture<Object[][]> casAsync(Cluster cluster, int coordinator, String cql)
{
return CompletableFuture.supplyAsync(() ->
cluster.coordinator(coordinator).execute(cql, ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM));
}
/**
* Execute a CAS expected to fail. The returned future completes with the raised Throwable
* (or null if the CAS unexpectedly succeeded).
*/
public static CompletableFuture<Throwable> casAsyncExpectingFailure(Cluster cluster, int coordinator, String cql)
{
return CompletableFuture.supplyAsync(() -> {
try
{
cluster.coordinator(coordinator).execute(cql, ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
return null;
}
catch (Throwable t)
{
return t;
}
});
}
// --- EpochPin: hold TCM traffic so a node stays at its old epoch ---
/**
* Install filters that pin a node at its current TCM epoch by holding inbound
* TCM_REPLICATION / TCM_NOTIFY_REQ messages targeted at the node, and outbound
* TCM_FETCH_CMS_LOG_REQ / TCM_FETCH_PEER_LOG_REQ messages originating from it, until
* {@link EpochPin#close()} releases the hold latch. Messages are held rather than
* dropped so they eventually flow and the node catches up before subsequent tests — an
* indefinite fetchLogFromPeerOrCMS stall would bleed into later tests.
*/
public static EpochPin epochPin(Cluster cluster, int node)
{
return new EpochPin(cluster, node);
}
/**
* Keep {@code nodes} from learning new TCM epochs proactively by dropping inbound
* TCM_REPLICATION and TCM_NOTIFY_REQ, while leaving on-demand catch-up
* (TCM_FETCH_PEER_LOG / TCM_FETCH_CMS_LOG, and their responses) untouched.
*
* This is the "replica behind" counterpart to {@link EpochPin}. EpochPin freezes a node
* completely (it also holds the outbound fetch requests), which would deadlock a handler
* that must call fetchLogFromPeerOrCMS to catch up. Dropping only the proactive push lets a
* behind replica fetch-and-catch-up the moment it receives a message carrying a higher epoch,
* which is exactly the MigrationRouter "coordinator ahead" branch under test.
*
* The returned filter is cleared by the usual {@code cluster.filters().reset()} in @After.
*/
public static IMessageFilters.Filter blockProactiveTcm(Cluster cluster, int... nodes)
{
return cluster.filters()
.inbound()
.verbs(Verb.TCM_REPLICATION.id, Verb.TCM_NOTIFY_REQ.id)
.to(nodes)
.drop();
}
public static final class EpochPin implements AutoCloseable
{
private final AssertingLatch releaseTcm;
private EpochPin(Cluster cluster, int node)
{
this.releaseTcm = new AssertingLatch("EpochPin release for node " + node);
cluster.filters()
.inbound()
.verbs(Verb.TCM_REPLICATION.id, Verb.TCM_NOTIFY_REQ.id)
.to(node)
.messagesMatching((from, to, msg) -> {
releaseTcm.await();
return false;
}).drop();
cluster.filters()
.inbound()
.verbs(Verb.TCM_FETCH_CMS_LOG_REQ.id, Verb.TCM_FETCH_PEER_LOG_REQ.id)
.from(node)
.messagesMatching((from, to, msg) -> {
releaseTcm.await();
return false;
}).drop();
}
@Override
public void close()
{
// Release the hold so held messages flow through the filter. We deliberately do NOT
// call .off() here; the filter remains installed so any late messages still flow
// rather than being silently dropped. Callers typically reset filters via cluster.filters().reset()
// in @After.
releaseTcm.countDown();
}
}
// --- MessageSpy: fluent wrapper over cluster filter + counter + latch boilerplate ---
/**
* Create a {@link Builder} that configures a spy (or dropper) over one or more message verbs
* on the given cluster. Returns a {@link MessageSpy} once {@link Builder#start()} is called.
*/
public static Builder on(Cluster cluster, Verb... verbs)
{
return new Builder(cluster, verbs);
}
/**
* Fluent builder for {@link MessageSpy}. Configures source/destination nodes,
* mutation-id checking, expected message count, hold/release behavior, and whether messages
* should be dropped or allowed through.
*/
public static final class Builder
{
private final Cluster cluster;
private final int[] verbIds;
private int[] fromNodes;
private int[] toNodes;
private boolean checkMutationId = false;
private boolean checkReadTracked = false;
private int expect = 0;
private boolean holdAll = false;
private int holdFirst = 0;
private boolean drop = false;
private Builder(Cluster cluster, Verb... verbs)
{
this.cluster = cluster;
this.verbIds = new int[verbs.length];
for (int i = 0; i < verbs.length; i++)
this.verbIds[i] = verbs[i].id;
}
public Builder from(int... nodes)
{
this.fromNodes = nodes;
return this;
}
public Builder to(int... nodes)
{
this.toNodes = nodes;
return this;
}
/** Deserialize each matching message on its target node and count those carrying a mutation id. */
public Builder checkMutationId()
{
this.checkMutationId = true;
return this;
}
/** Deserialize each matching message on its target node and count those carrying tracked vs untracked reads. */
public Builder checkReadTracked()
{
this.checkReadTracked = true;
return this;
}
/** Arm an internal {@link AssertingLatch} so {@link MessageSpy#await()} waits for N matching messages. */
public Builder expect(int count)
{
this.expect = count;
return this;
}
/**
* Hold every matching message on an internal latch until {@link MessageSpy#release()} is
* called. Messages are still counted and (if configured) checked for a mutation id before
* blocking.
*/
public Builder holdAll()
{
this.holdAll = true;
return this;
}
/** Hold only the first {@code n} matching messages; the rest flow through immediately. */
public Builder holdFirst(int n)
{
this.holdFirst = n;
return this;
}
/** Drop matching messages (don't deliver). Default is spy-only — messages flow through. */
public Builder drop()
{
this.drop = true;
return this;
}
public MessageSpy start()
{
return new MessageSpy(this);
}
}
/**
* A running message spy — counts matching messages, optionally holds them, optionally checks
* for mutation ids, and fires an expectation latch once the target count is reached.
*
* Closing via {@link #close()} turns the underlying filter off and releases any held messages.
*/
public static final class MessageSpy implements AutoCloseable
{
private final Cluster cluster;
private final AtomicInteger total = new AtomicInteger();
private final AtomicInteger withMutationId = new AtomicInteger();
private final AtomicInteger withTrackedRead = new AtomicInteger();
private final AtomicInteger withUntrackedRead = new AtomicInteger();
private final AtomicInteger held = new AtomicInteger();
private final AssertingLatch deliveryLatch;
private final AssertingLatch firstArrivalLatch;
private final AssertingLatch holdLatch;
private final IMessageFilters.Filter filter;
private final boolean drop;
private final int holdFirst;
private final boolean holdAll;
private MessageSpy(Builder b)
{
this.cluster = b.cluster;
this.drop = b.drop;
this.holdFirst = b.holdFirst;
this.holdAll = b.holdAll;
this.deliveryLatch = b.expect > 0 ? new AssertingLatch(b.expect, "MessageSpy delivery (expect=" + b.expect + ')')
: null;
this.firstArrivalLatch = new AssertingLatch("MessageSpy first arrival");
this.holdLatch = (b.holdAll || b.holdFirst > 0) ? new AssertingLatch("MessageSpy hold release") : null;
IMessageFilters.Builder fb = cluster.filters().inbound().verbs(b.verbIds);
if (b.fromNodes != null)
fb = fb.from(b.fromNodes);
if (b.toNodes != null)
fb = fb.to(b.toNodes);
final boolean checkId = b.checkMutationId;
final boolean checkRead = b.checkReadTracked;
this.filter = fb.messagesMatching((from, to, msg) -> {
total.incrementAndGet();
firstArrivalLatch.countDown();
if (checkId)
{
boolean hasId = cluster.get(to).callsOnInstance(() -> messageHasMutationId(msg)).call();
if (hasId)
withMutationId.incrementAndGet();
}
if (checkRead)
{
Boolean tracked = cluster.get(to).callsOnInstance(() -> messageReadIsTracked(msg)).call();
if (tracked != null)
{
if (tracked) withTrackedRead.incrementAndGet();
else withUntrackedRead.incrementAndGet();
}
}
// Hold before signaling deliveryLatch so the held-message-count stays accurate
// while the hold is still active.
boolean shouldHold = holdAll || (holdFirst > 0 && total.get() <= holdFirst);
if (shouldHold)
{
held.incrementAndGet();
holdLatch.await();
}
if (deliveryLatch != null)
deliveryLatch.countDown();
return drop;
}).drop();
}
public int total()
{
return total.get();
}
public int withMutationId()
{
return withMutationId.get();
}
public int withTrackedRead()
{
return withTrackedRead.get();
}
public int withUntrackedRead()
{
return withUntrackedRead.get();
}
public int held()
{
return held.get();
}
/** Block until the {@code expect(N)} count is reached. Throws if no expectation was set. */
public void await()
{
if (deliveryLatch == null)
throw new IllegalStateException("await() requires expect(N) to have been set at builder time");
deliveryLatch.await();
}
/** Block until at least one matching message has arrived. */
public void awaitFirstArrival()
{
firstArrivalLatch.await();
}
/** Release messages blocked by {@code holdAll()} / {@code holdFirst(N)}. */
public void release()
{
if (holdLatch == null)
throw new IllegalStateException("release() requires holdAll() or holdFirst(N) to have been set at builder time");
holdLatch.countDown();
}
@Override
public void close()
{
// Release any held messages so filter threads don't leak if the test fails early.
if (holdLatch != null)
holdLatch.countDown();
filter.off();
}
}
}

354
test/distributed/org/apache/cassandra/distributed/test/tracking/PaxosMutationTrackingForwardingV1Test.java Normal file
View File

@ -0,0 +1,354 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.distributed.test.tracking;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.apache.cassandra.config.CassandraRelevantProperties;
import org.apache.cassandra.distributed.Cluster;
import org.apache.cassandra.distributed.api.ConsistencyLevel;
import org.apache.cassandra.distributed.shared.ClusterUtils;
import org.apache.cassandra.distributed.test.TestBaseImpl;
import org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.MessageSpy;
import org.apache.cassandra.metrics.ClientRequestsMetricsHolder;
import org.apache.cassandra.net.Verb;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.alterReplicationType;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertCasApplied;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertReplicasAreExactly;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertReplicasHaveValue;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.buildPaxosCluster;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.casAsync;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.createKeyspace;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.messageHasMutationId;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.on;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.pauseHintsAndReconciler;
import static org.apache.cassandra.schema.ReplicationType.tracked;
import static org.apache.cassandra.schema.ReplicationType.untracked;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
/**
* Tests for V1 Paxos handler-level forwarding paths with CAS-level forwarding disabled.
*
* Tests the V1 commitPaxos forwarding path (StorageProxy.forwardPaxosCommit) which is invoked when
* the coordinator is not a replica for a tracked keyspace.
*
* KEY=5 maps to replicas on nodes 1,2,3 (node 4 excluded) with Murmur3Partitioner RF=3.
*/
public class PaxosMutationTrackingForwardingV1Test extends TestBaseImpl
{
private static Cluster cluster;
private static final int KEY = 5;
@BeforeClass
public static void setup() throws Throwable
{
CassandraRelevantProperties.DISABLE_CONSENSUS_REQUEST_FORWARDING.setBoolean(true);
cluster = init(buildPaxosCluster(4, "v1").start());
pauseHintsAndReconciler(cluster);
}
@AfterClass
public static void teardown()
{
if (cluster != null)
cluster.close();
CassandraRelevantProperties.DISABLE_CONSENSUS_REQUEST_FORWARDING.reset();
}
@After
public void resetFilters()
{
cluster.filters().reset();
ClusterUtils.awaitTCMCatchUp(cluster);
}
private String newKeyspace(String replicationType)
{
String ks = createKeyspace(cluster, "pmt_fwd_v1", replicationType);
assertReplicasAreExactly(cluster, ks, KEY, new int[]{ 1, 2, 3 });
return ks;
}
/*
* V1 commitPaxos forwarding from non-replica coordinator
*
* Covers: StorageProxy.forwardPaxosCommit() via commitPaxos()
* forwardPaxosCommit(reconciled, consistencyLevel, replicaPlan);
*
* With CAS-level forwarding disabled, a CAS from node 4 (non-replica for KEY=5)
* executes locally through V1 doPaxos -> commitPaxos. Inside commitPaxos,
* shouldBeTracked=true and requiresPaxosCommitForwarding returns
* true (node 4 not in replicas) -> forwardPaxosCommit fires, forwarding the commit to a replica
* via PaxosCommitForwardHandler.
*/
@Test
public void testV1CommitForwardingFromNonReplica()
{
String ks = newKeyspace("tracked");
try (MessageSpy spy = on(cluster, Verb.PAXOS_COMMIT_FORWARD_REQ)
.expect(1)
.start();
MessageSpy commitSpy = on(cluster, Verb.PAXOS_COMMIT_REQ)
.to(1, 2, 3)
.checkMutationId()
.expect(2)
.start())
{
Object[][] result = cluster.coordinator(4)
.execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 42) IF NOT EXISTS",
ConsistencyLevel.SERIAL,
ConsistencyLevel.QUORUM);
assertCasApplied(result);
spy.await();
commitSpy.await();
assertEquals("PAXOS_COMMIT_FORWARD_REQ should have been sent exactly once",
1, spy.total());
// The forward target re-coordinates on the tracked keyspace via commitPaxosTracked:
// it commits locally and sends PAXOS_COMMIT_REQ to the two other replicas, each
// carrying the freshly assigned mutation ID. Verify that on the wire, not just the
// forward count.
assertEquals("Forward target should send 2 sub-commits to the other replicas",
2, commitSpy.total());
assertEquals("Every forwarded sub-commit on a tracked keyspace must carry a mutation ID",
2, commitSpy.withMutationId());
}
assertReplicasHaveValue(cluster, ks, KEY, 42, 1, 2, 3);
}
/*
* V1 commit forwarding with migration to untracked during forward
*
* Covers: PaxosCommitForwardHandler.doVerb() untracked fallback path
*
* CAS from node 4 (non-replica) forwards the commit to a replica via
* PAXOS_COMMIT_FORWARD_REQ. The forward is delayed at the replica. During the delay,
* the keyspace migrates to untracked. When the handler processes the forward, it finds
* the keyspace is untracked and commits via the untracked path. No mutation ID needs to
* be stripped because the proposal arrives with MutationId.none() — the ID is only
* assigned inside commitPaxosTracked, which forwardPaxosCommit bypasses.
*/
@Test
public void testV1CommitForwardingFallbackToUntracked() throws Throwable
{
String ks = newKeyspace("tracked");
// Delay PAXOS_COMMIT_FORWARD_REQ at the receiving replica until ALTER completes.
// After migration to untracked, the forward handler commits via the untracked path.
// The proposal never had a mutation ID (it is only assigned inside commitPaxosTracked),
// so we verify no PAXOS_COMMIT_REQ carries one.
try (MessageSpy hold = on(cluster, Verb.PAXOS_COMMIT_FORWARD_REQ)
.holdAll()
.start();
MessageSpy commitSpy = on(cluster, Verb.PAXOS_COMMIT_REQ)
.to(1, 2, 3)
.checkMutationId()
.start())
{
CompletableFuture<Object[][]> casResult = casAsync(cluster, 4,
"INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 99) IF NOT EXISTS");
hold.awaitFirstArrival();
alterReplicationType(cluster, ks, untracked);
hold.release();
Object[][] result = casResult.get(60, TimeUnit.SECONDS);
assertCasApplied(result);
assertReplicasHaveValue(cluster, ks, KEY, 99, 1, 2, 3);
assertEquals("No PAXOS_COMMIT_REQ should carry a mutation ID after fallback to untracked",
0, commitSpy.withMutationId());
}
}
/*
* V1 commit forwarding with COORDINATOR_BEHIND retry after migration
*
* Covers: StorageProxy.java forwardPaxosCommit callback + commitPaxos retry loop
*
* The forward handler (node 1) still thinks tracked, calls commitPaxosTracked, sends
* PAXOS_COMMIT_REQ to nodes 2,3 which have already migrated to untracked. They reject
* with COORDINATOR_BEHIND. The forward handler catches the CoordinatorBehindException
* and sends COORDINATOR_BEHIND back to the original coordinator (node 4). The coordinator's
* retry loop should catch this and retry with fresh metadata (untracked path).
*
* Without the fix: the callback wraps COORDINATOR_BEHIND as WriteTimeoutException, which
* propagates out of the retry loop and the CAS fails with CasWriteTimeout.
*/
@Test
public void testV1CommitForwardingRetryAfterCoordinatorBehind()
{
String ks = newKeyspace("tracked");
@SuppressWarnings("Convert2MethodRef")
long retryBefore = cluster.get(4).callOnInstance(() -> ClientRequestsMetricsHolder.casWriteMetrics.retryCoordinatorBehind.getCount());
// Pre-insert so the CAS condition is met on first try
cluster.coordinator(1).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 1)",
ConsistencyLevel.ALL);
// Strategy: Hold PAXOS_COMMIT_REQ at nodes 2 and 3 (the non-forwarding replicas).
// While held, ALTER the keyspace to untracked. When released, nodes 2,3 see the
// commit request from the stale forward handler (node 1) and reject with
// COORDINATOR_BEHIND. This bubbles back to node 4's retry loop.
AssertingLatch commitArrived = new AssertingLatch("PAXOS_COMMIT_REQ at nodes 2,3");
AssertingLatch alterDone = new AssertingLatch("V1 commit forwarding retry - alter to untracked");
// Count ALL inbound PAXOS_COMMIT_REQ at nodes 2,3 regardless of source. After the retry,
// node 4 (now on untracked) calls commitPaxosUntracked directly and sends to replicas
// {1,2,3}, yielding 2 more messages at {2,3}. Total = initial 2 + retry 2 = 4.
// If the retry didn't happen (regression), total would be 2 and the CAS would fail.
AtomicInteger commitsAtReplicas = new AtomicInteger();
// Hold PAXOS_COMMIT_REQ at nodes 2 and 3 until ALTER completes.
// Also check retry messages (from node 4) for mutation ID absence. One filter handles
// both roles because the from-node distinguishes initial (node 1) from retry (node 4)
// and two overlapping MessageSpy instances on the same verb+destination would not
// compose cleanly — the cluster filter system applies ALL matching filters.
AtomicInteger retryCommitsWithId = new AtomicInteger();
cluster.filters()
.verbs(Verb.PAXOS_COMMIT_REQ.id)
.to(2, 3)
.messagesMatching((from, to, msg) -> {
commitsAtReplicas.incrementAndGet();
// Only hold commits from node 1 (the forward handler acting as coordinator)
if (from == 1)
{
commitArrived.countDown();
alterDone.await();
}
else if (from == 4)
{
boolean hasId = cluster.get(to).callsOnInstance(() -> messageHasMutationId(msg)).call();
if (hasId)
retryCommitsWithId.incrementAndGet();
}
return false;
}).drop();
// Start the CAS from node 4 (non-replica) async
CompletableFuture<Object[][]> casResult = casAsync(cluster, 4,
"UPDATE " + ks + ".tbl SET v = 100 WHERE k = " + KEY + " IF v = 1");
// Wait for the PAXOS_COMMIT_REQ to arrive (proves forward path was taken), then ALTER
// to untracked while the commit is held. All nodes see this immediately. Release the
// held commits in finally so the held filter thread never strands if a step throws.
try
{
commitArrived.await();
alterReplicationType(cluster, ks, untracked);
}
finally
{
// Release the held commits — nodes 2,3 now see migration mismatch -> COORDINATOR_BEHIND
alterDone.countDown();
}
// With the fix: node 4's retry loop catches CoordinatorBehindException, retries
// with fresh metadata (untracked), succeeds.
// Without the fix: WriteTimeoutException escapes the loop -> test fails.
try
{
Object[][] result = casResult.get(30, TimeUnit.SECONDS);
assertCasApplied(result);
}
catch (Exception e)
{
throw new AssertionError("CAS should succeed via COORDINATOR_BEHIND retry but got: " + e.getMessage(), e);
}
// Verify the write took effect on each replica individually. The retry path uses
// commitPaxosUntracked at QUORUM, so all 3 replicas (1,2,3) should have the data
// (node 1 gets it via the tracked local write in the initial forward, nodes 2,3 via retry).
assertReplicasHaveValue(cluster, ks, KEY, 100, 1, 2, 3);
// Authenticate that the retry actually happened by counting inbound PAXOS_COMMIT_REQ at
// the two non-forwarding replicas. Initial attempt (from node 1, held then released) = 2,
// retry (from node 4 directly on untracked path) = 2. Total = 4. A result of 2 would mean
// the CAS succeeded without retry (bug).
assertEquals("Expected initial (2) + retry (2) = 4 PAXOS_COMMIT_REQ at nodes 2,3",
4, commitsAtReplicas.get());
// Retry messages from node 4 (untracked path) must not carry mutation IDs
assertEquals("Retry PAXOS_COMMIT_REQ from node 4 should NOT carry mutation IDs (untracked path)",
0, retryCommitsWithId.get());
// Verify the retryCoordinatorBehind metric was incremented on the coordinator (node 4).
// StorageProxy.commitPaxos marks this when CoordinatorBehindException is caught and retried.
@SuppressWarnings("Convert2MethodRef")
long retryAfter = cluster.get(4).callOnInstance(() -> ClientRequestsMetricsHolder.casWriteMetrics.retryCoordinatorBehind.getCount());
assertTrue("casWriteMetrics.retryCoordinatorBehind should have been incremented on coordinator node 4",
retryAfter > retryBefore);
}
/*
* V1 commit forwarding reached via an untracked -> tracked migration.
* The keyspace is created untracked -- where a non-replica coordinator commits directly -- then migrated to
* tracked. Once writes are tracked, a non-replica (node 4) must forward the commit to a replica to obtain a
* mutation id (inverse of testV1CommitForwardingFallbackToUntracked).
*/
@Test
public void testV1CommitForwardingDuringMigrationToTracked()
{
String ks = newKeyspace("untracked");
alterReplicationType(cluster, ks, tracked);
ClusterUtils.awaitTCMCatchUp(cluster);
try (MessageSpy spy = on(cluster, Verb.PAXOS_COMMIT_FORWARD_REQ)
.expect(1)
.start();
MessageSpy commitSpy = on(cluster, Verb.PAXOS_COMMIT_REQ)
.to(1, 2, 3)
.checkMutationId()
.expect(2)
.start())
{
Object[][] result = cluster.coordinator(4)
.execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 42) IF NOT EXISTS",
ConsistencyLevel.SERIAL,
ConsistencyLevel.QUORUM);
assertCasApplied(result);
spy.await();
commitSpy.await();
assertEquals("Commit must be forwarded once after migration to tracked",
1, spy.total());
assertEquals("Forward target should send 2 sub-commits to the other replicas",
2, commitSpy.total());
assertEquals("Every forwarded sub-commit on the migrated-to-tracked keyspace must carry a mutation ID",
2, commitSpy.withMutationId());
}
assertReplicasHaveValue(cluster, ks, KEY, 42, 1, 2, 3);
}
}

611
test/distributed/org/apache/cassandra/distributed/test/tracking/PaxosMutationTrackingForwardingV2Test.java Normal file
View File

@ -0,0 +1,611 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.distributed.test.tracking;
import java.lang.reflect.Field;
import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.apache.cassandra.config.CassandraRelevantProperties;
import org.apache.cassandra.distributed.Cluster;
import org.apache.cassandra.distributed.api.ConsistencyLevel;
import org.apache.cassandra.distributed.impl.Instance;
import org.apache.cassandra.distributed.shared.ClusterUtils;
import org.apache.cassandra.distributed.test.TestBaseImpl;
import org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.MessageSpy;
import org.apache.cassandra.metrics.ClientRequestsMetricsHolder;
import org.apache.cassandra.net.Message;
import org.apache.cassandra.net.Verb;
import org.apache.cassandra.service.paxos.Commit;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.alterReplicationType;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertCasApplied;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertCasNotApplied;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertReplicaHasNoRow;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertReplicasAreExactly;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertReplicasHaveValue;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.buildPaxosCluster;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.casAsync;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.createKeyspace;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.messageHasMutationId;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.on;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.pauseHintsAndReconciler;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.respondWithTimeout;
import static org.apache.cassandra.schema.ReplicationType.tracked;
import static org.apache.cassandra.schema.ReplicationType.untracked;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
public class PaxosMutationTrackingForwardingV2Test extends TestBaseImpl
{
private static Cluster cluster;
private static final int KEY = 5;
@BeforeClass
public static void setup() throws Throwable
{
CassandraRelevantProperties.DISABLE_CONSENSUS_REQUEST_FORWARDING.setBoolean(true);
cluster = init(buildPaxosCluster(4, "v2").start());
pauseHintsAndReconciler(cluster);
}
@AfterClass
public static void teardown()
{
if (cluster != null)
cluster.close();
CassandraRelevantProperties.DISABLE_CONSENSUS_REQUEST_FORWARDING.reset();
}
@After
public void resetFilters()
{
cluster.filters().reset();
ClusterUtils.awaitTCMCatchUp(cluster);
}
private String newKeyspace(String replicationType)
{
String ks = createKeyspace(cluster, "pmt_fwd", replicationType);
ClusterUtils.awaitTCMCatchUp(cluster);
assertReplicasAreExactly(cluster, ks, KEY, new int[]{ 1, 2, 3 });
return ks;
}
@Test
public void testPrepareRefreshForwardHandler()
{
String ks = newKeyspace("untracked");
// Partial commit on node 1 only (commits to replicas 2 and 3 blocked) so they remain stale
// when the keyspace flips to tracked. The subsequent CAS from node 4 will force a
// prepare-refresh that exercises PrepareRefreshForwardHandler.
cluster.filters().verbs(Verb.PAXOS_COMMIT_REQ.id).from(1).to(2, 3, 4).drop();
try
{
cluster.coordinator(1).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 42) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
fail("CAS should have thrown because commits to nodes 2,3,4 were blocked");
}
catch (Exception e)
{
// First CAS is expected to throw: the commit phase was intentionally blocked to 2,3,4.
}
cluster.filters().reset();
alterReplicationType(cluster, ks, tracked);
ClusterUtils.awaitTCMCatchUp(cluster);
// Count PAXOS_PREPARE_REFRESH_FORWARD_REQ arrivals at any replica.
MessageSpy forwardSpy = on(cluster, Verb.PAXOS_PREPARE_REFRESH_FORWARD_REQ)
.start();
// Custom spy: extract the mutation ID string from each PAXOS2_PREPARE_REFRESH_REQ payload so
// we can assert that the forward handler produced a non-empty mutation ID. MessageSpy can
// only count by ID presence — it cannot capture the string representation — so this filter
// stays manual.
AtomicInteger refreshWithMutationId = new AtomicInteger();
Set<String> observedMutationIds = ConcurrentHashMap.newKeySet();
cluster.filters()
.verbs(Verb.PAXOS2_PREPARE_REFRESH_REQ.id)
.messagesMatching((from, to, msg) -> {
String mutId = cluster.get(to).callsOnInstance(() -> {
try
{
Message<?> deserialized = Instance.deserializeMessage(msg);
Object payload = deserialized.payload;
Field f = payload.getClass().getDeclaredField("missingCommit");
f.setAccessible(true);
Commit commit = (Commit) f.get(payload);
if (!commit.mutation.id().isNone())
return commit.mutation.id().toString();
return null;
}
catch (Exception e)
{
throw new RuntimeException(e);
}
}).call();
if (mutId != null)
{
refreshWithMutationId.incrementAndGet();
observedMutationIds.add(mutId);
}
return false;
}).drop();
// Drop prepare responses from node 3 so quorum can only be formed by node 1 + node 2.
// Node 1 has the latest committed ballot; when its response arrives (in any order relative
// to node 2), the coordinator will see withLatest()=1 < quorum=2 with
// haveReadResponseWithLatest=true, triggering refreshStaleParticipants rather than
// FOUND_INCOMPLETE_COMMITTED.
cluster.filters()
.verbs(Verb.PAXOS2_PREPARE_RSP.id)
.from(3)
.to(4)
.drop();
// The CAS may timeout during the commit phase due to migration state, but the handler's
// effect is observable via the intercepted PAXOS2_PREPARE_REFRESH_REQ messages.
boolean casThrew = false;
Object[][] casResult = null;
try
{
try
{
casResult = cluster.coordinator(4).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 99) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
}
catch (Exception e)
{
casThrew = true;
}
// Node 2 is the sole stale participant (it responded with an older commit). Node 3 never
// responded to prepare (its PAXOS2_PREPARE_RSP was dropped), so it is NOT in needLatest
// and is not targeted for refresh. The forward sends refresh only to node 2.
int forwards = forwardSpy.total();
assertTrue("At least 1 forward should have been sent, got " + forwards,
forwards >= 1);
// The stale participant (node 2) should receive a refresh with a mutation ID. This may
// come via a network PAXOS2_PREPARE_REFRESH_REQ (counted here) or via local execution when
// the forward handler is itself the target (shouldExecuteOnSelf).
int refreshes = refreshWithMutationId.get();
assertTrue("Expected 1 refresh with mutation ID, got " + refreshes,
refreshes >= 1);
assertFalse("Observed mutation IDs should not be empty", observedMutationIds.isEmpty());
// Two deterministic possibilities for the CAS outcome:
// - CAS succeeded: IF NOT EXISTS saw v=42 → not applied
// - CAS threw: migration state disrupted commit phase
// In either case v=99 must not be committed: the IF NOT EXISTS precondition fails because
// the first CAS's v=42 is present on node 1's system.paxos.
if (!casThrew)
assertCasNotApplied(casResult);
// Node 1 always has the data (original committer). Node 2 received the refresh (it was
// the sole stale participant targeted by refreshStaleParticipants). Node 3 never responded
// to prepare (its PAXOS2_PREPARE_RSP was dropped) so it was never in needLatest and was
// not targeted for refresh.
// Assert while filters are still active — the PAXOS2_PREPARE_RSP block on node 3 keeps
// it from receiving any refresh.
assertReplicasHaveValue(cluster, ks, KEY, 42, 1, 2);
assertReplicaHasNoRow(cluster, ks, KEY, 3);
}
finally
{
cluster.filters().reset();
forwardSpy.close();
}
}
/**
* FAILED_SENTINEL path when quorum is NOT achievable: the sole refresh target (node 2) fails
* (message intercepted with immediate failure), causing a FAILED_SENTINEL response. The CAS
* must fail (not hang) because quorum cannot be met (only node 1 has the latest commit;
* node 3 never responded to prepare so is not targeted for refresh).
*
* Verifies sentinel response on the wire, CAS fails with appropriate error, no data on stale
* nodes, no deadlock.
*/
@Test
public void testPrepareRefreshForwardHandlerTargetFailure()
{
String ks = newKeyspace("untracked");
cluster.filters().verbs(Verb.PAXOS_COMMIT_REQ.id).from(1).to(2, 3, 4).drop();
try
{
cluster.coordinator(1).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 42) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
fail("CAS should have thrown because commits to nodes 2,3,4 were blocked");
}
catch (Exception e)
{
// Expected — commit phase intentionally blocked.
}
cluster.filters().reset();
alterReplicationType(cluster, ks, tracked);
ClusterUtils.awaitTCMCatchUp(cluster);
// Intercept PAXOS2_PREPARE_REFRESH_REQ at nodes 2 and 3, immediately respond with failure
// (simulating timeout), and drop the original. In practice only node 2 will be targeted
// (node 3's prepare response is dropped below so it won't be in needLatest). This triggers
// FAILED_SENTINEL instantly without waiting for the real write_request_timeout.
// respondWithTimeout actively produces a response, so this filter stays manual.
cluster.filters()
.verbs(Verb.PAXOS2_PREPARE_REFRESH_REQ.id)
.to(2, 3)
.messagesMatching((from, to, msg) -> {
cluster.get(to).runOnInstance(() -> respondWithTimeout(msg));
return true;
}).drop();
// Block reconciler pushes so we get clean signal that the refresh was rejected
// without the high-priority reconciler replicating data from node 1's journal.
cluster.filters().verbs(Verb.MT_PUSH_MUTATION_REQ.id).drop();
// Drop prepare responses from node 3 so quorum can only be formed by node 1 + node 2.
// Node 1 has the latest committed ballot, triggering refreshStaleParticipants.
// Node 2's refresh receives an immediate failure (FAILED_SENTINEL), quorum unreachable, CAS must throw.
try (MessageSpy forwardSpy = on(cluster, Verb.PAXOS_PREPARE_REFRESH_FORWARD_REQ)
.start())
{
cluster.filters()
.verbs(Verb.PAXOS2_PREPARE_RSP.id)
.from(3)
.to(4)
.drop();
boolean casThrew = false;
try
{
cluster.coordinator(4).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 99) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
}
catch (Exception e)
{
casThrew = true;
}
assertTrue("CAS should have thrown due to quorum unreachable from FAILED_SENTINEL responses", casThrew);
int forwards = forwardSpy.total();
assertTrue("At least 1 forward should have been sent, got " + forwards, forwards >= 1);
// Node 1 always has the data (original committer); neither node 2 nor node 3 received
// the refresh (it was intercepted with a failure response) and reconciler pushes are blocked.
assertReplicasHaveValue(cluster, ks, KEY, 42, 1);
assertReplicaHasNoRow(cluster, ks, KEY, 2);
assertReplicaHasNoRow(cluster, ks, KEY, 3);
}
}
/**
* When node 3's prepare response is dropped, only node 2 becomes a refresh target (it
* responded with an older commit). The single refresh to node 2 succeeds, achieving quorum
* (node 1 + node 2 = 2). Verifies that quorum is reached with a single successful refresh
* when one electorate member is unresponsive during prepare.
*/
@Test
public void testPrepareRefreshForwardHandlerPartialFailureAchievesQuorum()
{
String ks = newKeyspace("untracked");
cluster.filters().verbs(Verb.PAXOS_COMMIT_REQ.id).from(1).to(2, 3, 4).drop();
try
{
cluster.coordinator(1).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 42) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
fail("CAS should have thrown because commits to nodes 2,3,4 were blocked");
}
catch (Exception e)
{
// Expected — commit phase intentionally blocked.
}
cluster.filters().reset();
alterReplicationType(cluster, ks, tracked);
ClusterUtils.awaitTCMCatchUp(cluster);
// Defensive filter: intercept PAXOS2_PREPARE_REFRESH_REQ at node 3 in case it somehow
// ends up in needLatest. In practice, node 3's prepare response is dropped below so it
// will not be targeted for refresh. Node 2's refresh proceeds unblocked, giving quorum
// (node 1 + node 2 = 2 >= quorum).
cluster.filters()
.verbs(Verb.PAXOS2_PREPARE_REFRESH_REQ.id)
.to(3)
.messagesMatching((from, to, msg) -> {
cluster.get(to).runOnInstance(() -> respondWithTimeout(msg));
return true;
}).drop();
// Drop prepare responses from node 3 so quorum can only be formed by node 1 + node 2.
// Node 1 has the latest committed ballot, triggering refreshStaleParticipants.
// Only node 2 is targeted for refresh (node 3 never responded to prepare). The single
// refresh succeeds: quorum = 2 with node 1 (has data) + node 2 (refresh success).
try (MessageSpy forwardSpy = on(cluster, Verb.PAXOS_PREPARE_REFRESH_FORWARD_REQ)
.start())
{
cluster.filters()
.verbs(Verb.PAXOS2_PREPARE_RSP.id)
.from(3)
.to(4)
.drop();
Object[][] casResult = cluster.coordinator(4).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 99) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
// CAS succeeded -- IF NOT EXISTS saw v=42 already present.
assertCasNotApplied(casResult);
int forwards = forwardSpy.total();
assertTrue("At least 1 forward should have been sent, got " + forwards, forwards >= 1);
// Node 1 always has the data (original committer). Node 2 received the refresh
// successfully. Node 3 has no data (it was never targeted for refresh because it did
// not respond to prepare).
// Assert while filters are still active — the PAXOS2_PREPARE_REFRESH_REQ block on
// node 3 keeps it clean.
assertReplicasHaveValue(cluster, ks, KEY, 42, 1, 2);
assertReplicaHasNoRow(cluster, ks, KEY, 3);
}
}
// V2 Commit Forwarding Tests (Paxos2CommitForwardHandler)
@Test
public void testV2CommitForwardingFromNonReplica()
{
String ks = newKeyspace("tracked");
MessageSpy forwardSpy = on(cluster, Verb.PAXOS2_COMMIT_FORWARD_REQ)
.expect(1)
.start();
// The forward target re-coordinates on the tracked keyspace: it commits locally and sends
// PAXOS_COMMIT_REQ to the two other replicas, each carrying the freshly assigned mutation ID.
MessageSpy commitSpy = on(cluster, Verb.PAXOS_COMMIT_REQ)
.to(1, 2, 3)
.checkMutationId()
.expect(2)
.start();
Object[][] result = cluster.coordinator(4).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 42) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
assertCasApplied(result);
forwardSpy.await();
commitSpy.await();
assertEquals("PAXOS2_COMMIT_FORWARD_REQ should have been sent exactly once", 1, forwardSpy.total());
assertEquals("Forward target should send 2 sub-commits to the other replicas", 2, commitSpy.total());
assertEquals("Every forwarded sub-commit on a tracked keyspace must carry a mutation ID",
2, commitSpy.withMutationId());
assertReplicasHaveValue(cluster, ks, KEY, 42, 1, 2, 3);
forwardSpy.close();
commitSpy.close();
}
@Test
public void testV2CommitForwardingFallbackToUntracked() throws Throwable
{
String ks = newKeyspace("tracked");
// Hold PAXOS2_COMMIT_FORWARD_REQ so we can ALTER the keyspace to untracked mid-flight;
// the forward handler should then commit on the untracked path (no mutation IDs on
// PAXOS_COMMIT_REQ).
MessageSpy hold = on(cluster, Verb.PAXOS2_COMMIT_FORWARD_REQ)
.holdAll()
.start();
// The forward handler on the target replica commits locally and sends PAXOS_COMMIT_REQ to
// the other 2 replicas. Quorum = 2, so the 2nd remote commit may still be in-flight when
// the CAS returns — expect(2) waits for both.
MessageSpy commitSpy = on(cluster, Verb.PAXOS_COMMIT_REQ)
.to(1, 2, 3)
.checkMutationId()
.expect(2)
.start();
CompletableFuture<Object[][]> casFuture = casAsync(cluster, 4,
"INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 99) IF NOT EXISTS");
try
{
hold.awaitFirstArrival();
alterReplicationType(cluster, ks, untracked);
}
finally
{
hold.release();
}
Object[][] result = casFuture.get(60, TimeUnit.SECONDS);
assertCasApplied(result);
commitSpy.await();
cluster.filters().reset();
assertReplicasHaveValue(cluster, ks, KEY, 99, 1, 2, 3);
// After fallback to untracked no commit should carry a mutation ID.
assertEquals("No PAXOS_COMMIT_REQ should carry a mutation ID after fallback to untracked",
0, commitSpy.withMutationId());
hold.close();
commitSpy.close();
}
@Test
public void testV2CommitForwardingRetryAfterCoordinatorBehind()
{
String ks = newKeyspace("tracked");
@SuppressWarnings("Convert2MethodRef")
long retryBefore = cluster.get(4).callOnInstance(() -> ClientRequestsMetricsHolder.casWriteMetrics.retryCoordinatorBehind.getCount());
// Pre-insert so the CAS condition (IF v = 1) is met on first try.
cluster.coordinator(1).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 1)",
ConsistencyLevel.ALL);
AtomicInteger forwardTarget = new AtomicInteger(-1);
AssertingLatch commitArrived = new AssertingLatch("PAXOS_COMMIT_REQ from forward target");
AssertingLatch alterDone = new AssertingLatch("ALTER to untracked to complete");
AtomicInteger commitsAtReplicas = new AtomicInteger();
AtomicInteger retryCommitsWithId = new AtomicInteger();
// Learn which replica the snitch chose for the forward — any of {1,2,3} is possible.
cluster.filters()
.verbs(Verb.PAXOS2_COMMIT_FORWARD_REQ.id)
.messagesMatching((from, to, msg) -> {
forwardTarget.set(to);
return false;
}).drop();
// Hold PAXOS_COMMIT_REQ from the forward target until ALTER completes. Count ALL inbound
// PAXOS_COMMIT_REQ at replicas to verify the retry fires, and check retry messages (from
// node 4) for mutation ID absence. Conditional behaviour inside one filter — stays manual.
cluster.filters()
.verbs(Verb.PAXOS_COMMIT_REQ.id)
.to(1, 2, 3)
.messagesMatching((from, to, msg) -> {
commitsAtReplicas.incrementAndGet();
if (from == forwardTarget.get())
{
commitArrived.countDown();
alterDone.await();
}
else if (from == 4)
{
boolean hasId = cluster.get(to).callsOnInstance(() -> messageHasMutationId(msg)).call();
if (hasId)
retryCommitsWithId.incrementAndGet();
}
return false;
}).drop();
CompletableFuture<Object[][]> casFuture = casAsync(cluster, 4,
"UPDATE " + ks + ".tbl SET v = 100 WHERE k = " + KEY + " IF v = 1");
try
{
commitArrived.await();
alterReplicationType(cluster, ks, untracked);
}
finally
{
alterDone.release();
commitArrived.release();
}
// With the fix: node 4's retry loop catches CoordinatorBehindException and retries with
// fresh metadata (untracked), succeeding. Without the fix, WriteTimeoutException escapes
// the loop and the test fails.
try
{
Object[][] result = casFuture.get(30, TimeUnit.SECONDS);
assertCasApplied(result);
}
catch (Exception e)
{
throw new AssertionError("CAS should succeed via COORDINATOR_BEHIND retry but got: " + e.getMessage(), e);
}
assertReplicasHaveValue(cluster, ks, KEY, 100, 1, 2, 3);
// Retry verification: forward target sends to 2 other replicas (initial attempt), node 4
// retries directly on the untracked path sending to all 3 replicas. Total = 5.
assertTrue("Forward target should have been identified", forwardTarget.get() > 0);
assertEquals("Expected initial (2) + retry (3) = 5 PAXOS_COMMIT_REQ at replicas",
5, commitsAtReplicas.get());
// Retry messages from node 4 (untracked path) must not carry mutation IDs.
assertEquals("Retry PAXOS_COMMIT_REQ from node 4 should NOT carry mutation IDs (untracked path)",
0, retryCommitsWithId.get());
// Verify the retryCoordinatorBehind metric was incremented on the coordinator (node 4).
// Paxos.cas() marks this when COORDINATOR_BEHIND responses are detected and a retry fires.
@SuppressWarnings("Convert2MethodRef")
long retryAfter = cluster.get(4).callOnInstance(() -> ClientRequestsMetricsHolder.casWriteMetrics.retryCoordinatorBehind.getCount());
assertTrue("casWriteMetrics.retryCoordinatorBehind should have been incremented on coordinator node 4",
retryAfter > retryBefore);
}
/*
* Commit forwarding reached via an untracked -> tracked migration. The keyspace is created untracked -- where
* a non-replica coordinator commits directly -- then migrated to tracked. Once writes are tracked, a non-replica
* (node 4) must forward the commit to a replica to obtain a mutation id, exercising the migration-state-aware
* forwarding decision (the inverse of testV2CommitForwardingFallbackToUntracked).
*/
@Test
public void testV2CommitForwardingDuringMigrationToTracked()
{
String ks = newKeyspace("untracked");
alterReplicationType(cluster, ks, tracked);
ClusterUtils.awaitTCMCatchUp(cluster);
MessageSpy forwardSpy = on(cluster, Verb.PAXOS2_COMMIT_FORWARD_REQ)
.expect(1)
.start();
// The forward target re-coordinates on the now-tracked keyspace: it commits locally and
// sends PAXOS_COMMIT_REQ to the two other replicas, each carrying a mutation ID.
MessageSpy commitSpy = on(cluster, Verb.PAXOS_COMMIT_REQ)
.to(1, 2, 3)
.checkMutationId()
.expect(2)
.start();
Object[][] result = cluster.coordinator(4).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (" + KEY + ", 42) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
assertCasApplied(result);
forwardSpy.await();
commitSpy.await();
assertEquals("Commit must be forwarded once after migration to tracked", 1, forwardSpy.total());
assertEquals("Forward target should send 2 sub-commits to the other replicas", 2, commitSpy.total());
assertEquals("Every forwarded sub-commit on the migrated-to-tracked keyspace must carry a mutation ID",
2, commitSpy.withMutationId());
assertReplicasHaveValue(cluster, ks, KEY, 42, 1, 2, 3);
forwardSpy.close();
commitSpy.close();
}
}

407
test/distributed/org/apache/cassandra/distributed/test/tracking/PaxosMutationTrackingMigrationV1Test.java Normal file
View File

@ -0,0 +1,407 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.cassandra.distributed.test.tracking;
import java.util.concurrent.CompletableFuture;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.apache.cassandra.distributed.Cluster;
import org.apache.cassandra.distributed.api.ConsistencyLevel;
import org.apache.cassandra.distributed.shared.ClusterUtils;
import org.apache.cassandra.distributed.test.TestBaseImpl;
import org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.EpochPin;
import org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.MessageSpy;
import org.apache.cassandra.net.Verb;
import org.apache.cassandra.replication.MutationTrackingService;
import org.apache.cassandra.service.replication.migration.MutationTrackingMigrationState;
import org.apache.cassandra.tcm.ClusterMetadata;
import static java.util.concurrent.TimeUnit.SECONDS;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.alterReplicationType;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.alterReplicationTypeFrom;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertAllNodesSee;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertCasApplied;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertCasException;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertCasNotApplied;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertReplicaHasNoRow;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.assertReplicasHaveValue;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.awaitReplicationType;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.buildPaxosCluster;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.casAsync;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.createKeyspace;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.epochPin;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.on;
import static org.apache.cassandra.distributed.test.tracking.PaxosMigrationTestUtils.pauseHintsAndReconciler;
import static org.apache.cassandra.schema.ReplicationType.tracked;
import static org.apache.cassandra.schema.ReplicationType.untracked;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
/**
* Tests that Paxos V1 CAS operations work correctly during mutation tracking migration.
*
* Each test documents the exact message verb being verified and asserts that the
* expected code path was taken (not just that the CAS "succeeds").
*
* Uses a shared 3-node cluster with paxos_variant=v1.
*/
public class PaxosMutationTrackingMigrationV1Test extends TestBaseImpl
{
private static Cluster cluster;
@BeforeClass
public static void setup() throws Throwable
{
cluster = init(buildPaxosCluster(3, "v1").start());
pauseHintsAndReconciler(cluster);
}
@AfterClass
public static void teardown()
{
if (cluster != null)
cluster.close();
}
@After
public void resetFilters()
{
cluster.filters().reset();
ClusterUtils.awaitTCMCatchUp(cluster);
}
/*
* Message: PAXOS_COMMIT_REQ
* Path: V1 -> StorageProxy.commitPaxosTracked
* Verifies: MigrationRouter.shouldUseTrackedForWrites() returns true during migration
*/
@Test
public void testCasDuringMigrationToTracked()
{
String ks = createKeyspace(cluster, "pmt_v1", "untracked");
cluster.coordinator(1).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (1, 0)",
ConsistencyLevel.QUORUM);
alterReplicationType(cluster, ks, tracked);
// Precondition: migration is active and schema reads tracked on every node.
for (int i = 1; i <= cluster.size(); i++)
{
final String keyspace = ks;
int nodeId = i;
cluster.get(i).runOnInstance(() -> {
ClusterMetadata cm = ClusterMetadata.current();
MutationTrackingMigrationState state = cm.mutationTrackingMigrationState;
assertTrue("Node " + nodeId + ": keyspace should be migrating",
state.isMigrating(keyspace));
assertTrue("Node " + nodeId + ": schema should show tracked",
cm.schema.getKeyspaceMetadata(keyspace).params.replicationType.isTracked());
});
}
// Count PAXOS_COMMIT_REQ messages that carry a mutation ID. Counting raw messages is not
// enough -- 2 untracked commits on RF=3 would also produce count==2. The tracked path
// differs from the untracked path by carrying a non-none mutation ID on the Commit payload.
try (MessageSpy spy = on(cluster, Verb.PAXOS_COMMIT_REQ)
.checkMutationId()
.expect(2)
.start())
{
Object[][] result = cluster.coordinator(1).execute("UPDATE " + ks + ".tbl SET v = 1 WHERE k = 1 IF v = 0",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
spy.await();
assertCasApplied(result);
assertEquals("Tracked CAS should send exactly 2 PAXOS_COMMIT_REQ carrying a mutation ID (to 2 remote replicas)",
2, spy.withMutationId());
}
for (int i = 1; i <= cluster.size(); i++)
{
int nodeId = i;
cluster.get(i).runOnInstance(() ->
assertTrue("Node " + nodeId + ": MutationTrackingService should be enabled",
MutationTrackingService.isEnabled()));
}
assertReplicasHaveValue(cluster, ks, 1, 1, 1, 2, 3);
}
/*
* Scenario: first CAS commits locally but remote commits are dropped. Migrate to untracked.
* Second CAS discovers the stale ballot and recommits with the ID stripped.
*
* The coordinator strips the ID BEFORE sending, so the handler never sees a mismatch.
* Zero mutation IDs on outbound commits proves stripping worked (no COORDINATOR_BEHIND rejection).
*
* V1 message: PAXOS_COMMIT_REQ via StorageProxy.sendCommit()
*/
@Test
public void testStaleIdStrippedOnRecommit()
{
String ks = createKeyspace(cluster, "pmt_v1", "tracked");
// Drop remote commits to leave an uncommitted ballot with a mutation ID on node 1.
// The tracked commit path writes locally synchronously, so node 1 has the commit.
cluster.filters().verbs(Verb.PAXOS_COMMIT_REQ.id).drop();
boolean threw = false;
try
{
cluster.coordinator(1).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (1, 1) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
}
catch (Exception e)
{
threw = true;
assertCasException(e);
}
assertTrue("First CAS should have thrown (commits were blocked)", threw);
cluster.filters().reset();
// Migrate to untracked (instant). The stale ballot in system.paxos has a mutation ID.
alterReplicationType(cluster, ks, untracked);
assertAllNodesSee(cluster, ks, untracked);
// Spy on PAXOS_COMMIT_REQ messages to remote replicas. The V1 repair path's
// beginAndRepairPaxos loop may run 1+ iterations depending on timing (when a replica
// acknowledges the previous repair before the next prepare). Counting total messages
// is non-deterministic, but the INVARIANT is: every message after migration-to-untracked
// must have mutation_id stripped (.id().isNone() == true). If stripping failed, the
// handler would reject with CoordinatorBehindException -> CAS times out.
try (MessageSpy spy = on(cluster, Verb.PAXOS_COMMIT_REQ)
.to(2, 3)
.checkMutationId()
.start())
{
// Second CAS: INSERT IF NOT EXISTS. The first CAS's local commit left v=1 on node 1.
// After beginAndRepairPaxos repairs, all replicas have v=1. The condition IF NOT EXISTS
// therefore evaluates FALSE -> CAS NOT applied, result[0][0] == false.
// If stripping failed, CAS would time out (no exception/result here).
Object[][] result = cluster.coordinator(1).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (1, 1) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
assertCasNotApplied(result);
assertReplicasHaveValue(cluster, ks, 1, 1, 1, 2, 3);
// Core invariant: every repair commit must have its mutation_id stripped after migration
// to untracked. Otherwise handlers reject with CoordinatorBehindException and the CAS
// would time out (it didn't -- see CAS success above).
assertEquals("No PAXOS_COMMIT_REQ should carry a mutation ID after migration to untracked",
0, spy.withMutationId());
}
}
/*
* Scenario: tracked -> untracked migration completes, then a fresh CAS runs.
* No stale ballots, no epoch mismatches. The commit goes through the untracked path
* and all replicas accept on the first try.
*
* Message: PAXOS_COMMIT_REQ
* V1 path: StorageProxy.commitPaxosUntracked()
*
* Exact inbound count == 2 proves no COORDINATOR_BEHIND retry.
*/
@Test
public void testCommitAfterMigrationToUntracked()
{
String ks = createKeyspace(cluster, "pmt_v1", "tracked");
alterReplicationType(cluster, ks, untracked);
assertAllNodesSee(cluster, ks, untracked);
try (MessageSpy spy = on(cluster, Verb.PAXOS_COMMIT_REQ)
.to(2, 3)
.checkMutationId()
.expect(2)
.start())
{
// Fresh CAS on untracked keyspace -- clean commit, no stale data, no race
Object[][] result = cluster.coordinator(1).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (1, 42) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
spy.await();
assertCasApplied(result);
// Exactly 2 PAXOS_COMMIT_REQ (one per remote replica, no retry)
assertEquals("PAXOS_COMMIT_REQ should be sent to exactly 2 remote replicas (no retry)",
2, spy.total());
assertEquals("PAXOS_COMMIT_REQ should NOT carry mutation IDs on untracked keyspace",
0, spy.withMutationId());
}
assertReplicasHaveValue(cluster, ks, 1, 42, 1, 2, 3);
}
/*
* Scenario: CAS on tracked keyspace. Inbound filter delays PAXOS_COMMIT_REQ delivery
* at nodes 2, 3 while we ALTER to untracked. When released, the handlers see the stale
* epoch and reject with COORDINATOR_BEHIND. The coordinator retries with fresh routing.
*
* Message: PAXOS_COMMIT_REQ
* V1 retry: StorageProxy.commitPaxos() catches CoordinatorBehindException (thrown by
* AbstractWriteResponseHandler.get()), re-evaluates routing, resends.
*
* Inbound count == 4 proves the retry occurred (initial batch 2 + retry batch 2).
* Of those 4, only the initial 2 should carry mutation IDs (tracked era); the retry
* batch (untracked era) should have IDs stripped.
*/
@Test
public void testCommitCoordinatorBehindRetry() throws Throwable
{
String ks = createKeyspace(cluster, "pmt_v1", "tracked");
// holdFirst(2) blocks the initial batch at the destination until release() is called;
// subsequent messages (the retry batch) pass through immediately. The initial batch
// arrives with old epoch (tracked), but by the time the handler processes it, the
// node is at new epoch (untracked) -> COORDINATOR_BEHIND -> coordinator retries.
try (MessageSpy hold = on(cluster, Verb.PAXOS_COMMIT_REQ)
.from(1)
.to(2, 3)
.holdFirst(2)
.checkMutationId()
.expect(4)
.start())
{
CompletableFuture<Object[][]> casResult = casAsync(cluster, 1,
"INSERT INTO " + ks + ".tbl (k, v) VALUES (1, 42) IF NOT EXISTS");
hold.awaitFirstArrival();
try
{
// ALTER to untracked while commits are delayed at the destination
alterReplicationType(cluster, ks, untracked);
assertAllNodesSee(cluster, ks, untracked);
}
finally
{
hold.release();
}
Object[][] result = casResult.get(60, SECONDS);
hold.await();
assertCasApplied(result);
// Initial batch: 2 inbound (to nodes 2, 3) -> both COORDINATOR_BEHIND -> retry.
// Retry batch: 2 more inbound (to nodes 2, 3 again).
// Total = 4 proves the COORDINATOR_BEHIND retry occurred.
assertEquals("Expected initial (2) + retry (2) = 4 PAXOS_COMMIT_REQ",
4, hold.total());
// Only the initial batch (tracked era) carries mutation IDs. The retry batch
// (untracked era) must have IDs stripped. Total-with-id == 2 proves this:
// if retry messages also had IDs, the count would be 4.
assertEquals("Only the initial stale batch should carry mutation IDs (retry batch should not)",
2, hold.withMutationId());
}
assertReplicasHaveValue(cluster, ks, 1, 42, 1, 2, 3);
assertAllNodesSee(cluster, ks, untracked);
}
/*
* Covers: StorageProxy.commitPaxos() -- WriteTimeoutException thrown when the V1
* commitPaxosTracked response handler times out because maybeFetchLogs blocks before
* delivering the COORDINATOR_BEHIND failure callback.
*
* Node 2 as coordinator is blocked from receiving TCM updates, so it stays at the
* old epoch (tracked). After ALTER to untracked, nodes 1,3 are at the new epoch.
* Every commit from node 2 triggers COORDINATOR_BEHIND on nodes 1,3. Node 2 can't
* catch up, so the first commit attempt's inner timeout fires, failing the CAS without retrying.
*/
@Test
public void testCommitRetryLoopTimeout()
{
String ks = createKeyspace(cluster, "pmt_v1", "tracked");
// EpochPin strands node 2 at its current (tracked) epoch by holding TCM traffic in/out.
// "Hold" rather than "drop": messages block on a latch with a safety-net timeout so they
// eventually flow after the test, preventing an indefinite fetchLogFromPeerOrCMS stall
// on node 2 that would bleed into later tests.
try (EpochPin ignored = epochPin(cluster, 2))
{
// ALTER to untracked via node 1 at CL.ONE so we don't wait for schema agreement
// on TCM-blocked node 2.
alterReplicationTypeFrom(cluster, 1, ks, untracked, ConsistencyLevel.ONE);
// Wait for nodes 1,3 to see untracked. Node 2 should still see tracked.
awaitReplicationType(cluster, ks, untracked, 1, 3);
final String keyspace = ks;
assertTrue("Node 2 should still see tracked (TCM blocked)",
cluster.get(2).callOnInstance(() ->
ClusterMetadata.current().schema.getKeyspaceMetadata(keyspace)
.params.replicationType.isTracked()));
// Spy on PAXOS_COMMIT_REQ from node 2 to verify commit attempts were made.
try (MessageSpy spy = on(cluster, Verb.PAXOS_COMMIT_REQ)
.from(2)
.checkMutationId()
.start())
{
// CAS from node 2. Node 2 thinks tracked -> commitPaxosTracked sends PAXOS_COMMIT_REQ
// to nodes 1,3. They detect epoch mismatch -> COORDINATOR_BEHIND. CBE propagates back;
// maybeFetchLogs on node 2 tries to fetch but the fetches are held by the EpochPin.
// The CAS ultimately fails via the inner response-handler timeout at
// write_request_timeout, before the held fetch is released.
boolean threw = false;
try
{
cluster.coordinator(2).execute("INSERT INTO " + ks + ".tbl (k, v) VALUES (1, 42) IF NOT EXISTS",
ConsistencyLevel.SERIAL, ConsistencyLevel.QUORUM);
}
catch (Exception e)
{
threw = true;
assertTrue("Expected CAS timeout but got: " + e.getMessage(),
e.getMessage().contains("CAS operation timed out"));
}
// Exactly one iteration fires: inner timeout expires before the held TCM fetch
// releases and allows cb.onFailure to let commitPaxos catch the CBE and retry.
assertEquals("Exactly 1 iteration runs (held fetch outlasts write_request_timeout); 2 commits = 1 iteration x 2 replicas",
2, spy.total());
// All commits from stranded node 2 carry mutation IDs (tracked path),
// proving COORDINATOR_BEHIND is the rejection reason (untracked handlers reject tracked commits)
assertEquals("All commit attempts from stranded coordinator should carry mutation IDs (tracked path)",
spy.total(), spy.withMutationId());
assertTrue("CAS should have failed", threw);
// Node 2 is the coordinator AND a replica; its local commit (executeOnSelf) fires
// before the quorum wait, so v=42 is locally durable. Nodes 1,3 rejected with
// COORDINATOR_BEHIND — they never applied the commit.
assertReplicasHaveValue(cluster, ks, 1, 42, 2);
assertReplicaHasNoRow(cluster, ks, 1, 1);
assertReplicaHasNoRow(cluster, ks, 1, 3);
}
}
}
}

1472
test/distributed/org/apache/cassandra/distributed/test/tracking/PaxosMutationTrackingMigrationV2Test.java Normal file
View File

File diff suppressed because it is too large Load Diff

241
test/unit/org/apache/cassandra/service/replication/migration/MigrationRouterTest.java
View File

@ -37,10 +37,13 @@ import org.apache.cassandra.config.CassandraRelevantProperties;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.DataRange;
import org.apache.cassandra.db.DecoratedKey;
import org.apache.cassandra.db.EmbeddableSinglePartitionReadCommand;
import org.apache.cassandra.db.Mutation;
import org.apache.cassandra.db.PartitionPosition;
import org.apache.cassandra.db.PartitionRangeReadCommand;
import org.apache.cassandra.db.ReadCommand;
import org.apache.cassandra.db.ReadKind;
import org.apache.cassandra.db.SinglePartitionReadCommand;
import org.apache.cassandra.db.Slices;
import org.apache.cassandra.db.filter.ClusteringIndexSliceFilter;
import org.apache.cassandra.db.filter.ColumnFilter;
@ -56,6 +59,11 @@ import org.apache.cassandra.dht.Murmur3Partitioner;
import org.apache.cassandra.dht.NormalizedRanges;
import org.apache.cassandra.dht.Range;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.exceptions.CoordinatorBehindException;
import org.apache.cassandra.locator.InetAddressAndPort;
import org.apache.cassandra.net.Message;
import org.apache.cassandra.net.NoPayload;
import org.apache.cassandra.net.Verb;
import org.apache.cassandra.replication.MutationId;
import org.apache.cassandra.schema.DistributedSchema;
import org.apache.cassandra.schema.KeyspaceMetadata;
@ -68,6 +76,7 @@ import org.apache.cassandra.schema.Tables;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.cassandra.tcm.Epoch;
import org.apache.cassandra.utils.Clock;
import org.apache.cassandra.utils.FBUtilities;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
@ -590,10 +599,9 @@ public class MigrationRouterTest
assertTrue(MigrationRouter.shouldUseTrackedForWrites(metadata, TEST_KEYSPACE, testTable.id, tokenOutsidePending));
}
/**
* Test mutation routing with multiple tables - some tracked, some untracked.
* This verifies that routeMutations correctly filters mutations to separate tracked/untracked tables.
* Test mutation routing with multiple tables during migration to tracked.
* All tables in the mutation are routed as tracked (both migrating and already-completed).
*/
@Test
public void testMultiTableMutationRouting_ToTracked()
@ -608,7 +616,7 @@ public class MigrationRouterTest
ClusterMetadata.Transformer transformer = metadata.transformer();
// table1 migrating to untracked, table2 complete
// table1 still migrating (pending), table2 migration complete
MutationTrackingMigrationState migrationState = metadata.mutationTrackingMigrationState.withKeyspaceMigrating(ksm.name, Collections.singleton(table1.id), transformer.epoch());
metadata = transformer.with(migrationState).build().metadata;
@ -628,4 +636,229 @@ public class MigrationRouterTest
assertEquals(0, routed.untrackedMutations.size());
}
// Null keyspace metadata guard tests
@Test
public void testShouldUseTrackedForWritesWithNullKeyspace()
{
// ClusterMetadata without the test keyspace — simulates concurrent keyspace drop
ClusterMetadata metadata = ClusterMetadata.current();
// Ensure the non-existent keyspace is not in the metadata
assertFalse("Keyspace should not exist in metadata",
metadata.schema.getKeyspaces().containsKeyspace("nonexistent_ks"));
// Should return false, not NPE
assertFalse(MigrationRouter.shouldUseTrackedForWrites(metadata, "nonexistent_ks",
TableId.generate(), createToken(100)));
}
@Test
public void testShouldUseTrackedForWritesWithNullKeyspaceDuringMigration()
{
// Create metadata WITH migration info but WITHOUT the keyspace itself
ClusterMetadata metadata = ClusterMetadata.current();
// Add migration info for a keyspace that doesn't exist in the schema
KeyspaceMigrationInfo migrationInfo = new KeyspaceMigrationInfo("dropped_ks",
Collections.emptyMap(),
Epoch.create(1));
MutationTrackingMigrationState migrationState = new MutationTrackingMigrationState(Epoch.create(1),
ImmutableMap.of("dropped_ks", migrationInfo));
metadata = withMigrationInfo(metadata, migrationState);
// Should return false, not NPE
assertFalse(MigrationRouter.shouldUseTrackedForWrites(metadata, "dropped_ks",
TableId.generate(), createToken(100)));
}
@Test
public void testSplitRangeReadForMigrationWithNullKeyspace()
{
ClusterMetadata metadata = ClusterMetadata.current();
// Create a range read command for a non-existent keyspace using a table metadata
// that references a keyspace not in the cluster metadata
TableMetadata nonexistentTable = TableMetadata.builder("nonexistent_ks", "test_table")
.addPartitionKeyColumn("pk", UTF8Type.instance)
.addRegularColumn("value", UTF8Type.instance)
.partitioner(partitioner)
.build();
PartitionRangeReadCommand cmd = createRangeCommand(nonexistentTable, createToken(0), createToken(1000));
List<MigrationRouter.RangeReadWithReplication> result =
MigrationRouter.splitRangeRead(metadata, cmd);
// Should return single untracked entry, not NPE
assertEquals(1, result.size());
assertFalse("Should be untracked for non-existent keyspace", result.get(0).useTracked);
}
@Test
public void testShouldUseTrackedWithNullKeyspace()
{
TableMetadata nonexistentTable = TableMetadata.builder("nonexistent_ks", "test_table")
.addPartitionKeyColumn("pk", UTF8Type.instance)
.addRegularColumn("value", UTF8Type.instance)
.partitioner(partitioner)
.build();
DecoratedKey key = partitioner.decorateKey(UTF8Type.instance.decompose("test_key"));
SinglePartitionReadCommand cmd = SinglePartitionReadCommand.fullPartitionRead(nonexistentTable, 0, key);
assertFalse(MigrationRouter.shouldUseTracked(cmd));
}
/**
* Minimal test-only stub for EmbeddableSinglePartitionReadCommand that lets us control
* the kind() (and therefore isTracked()) without constructing a full TrackedRead.DataRequest
* or TrackedRead.SummaryRequest.
*/
private static EmbeddableSinglePartitionReadCommand readStub(TableMetadata table, DecoratedKey key, ReadKind kind)
{
return new EmbeddableSinglePartitionReadCommand()
{
@Override
public ReadKind kind()
{
return kind;
}
@Override
public TableMetadata metadata()
{
return table;
}
@Override
public DecoratedKey partitionKey()
{
return key;
}
};
}
@Test
public void testCheckPaxosCommitMigration_Agreement()
{
ClusterMetadata metadata = createMetadata(true, Collections.emptyList());
TableMetadata table = metadata.schema.getKeyspaceMetadata(TEST_KEYSPACE).getTableOrViewNullable(TEST_TABLE);
Token token = createToken(0L);
Message<NoPayload> msg = Message.builder(Verb._TEST_1, NoPayload.noPayload).withEpoch(metadata.epoch).build();
InetAddressAndPort respondTo = FBUtilities.getBroadcastAddressAndPort();
// Tracked keyspace, no migration → handler says tracked. Coordinator says tracked → agreement.
ClusterMetadata result = MigrationRouter.checkPaxosCommitMigration(metadata, msg, respondTo,
TEST_KEYSPACE, table.id, token, true);
Assert.assertSame("Metadata should be returned unchanged on agreement", metadata, result);
}
@Test
public void testCheckPaxosCommitMigration_CoordinatorBehind()
{
ClusterMetadata metadata = createMetadata(true, Collections.emptyList());
TableMetadata table = metadata.schema.getKeyspaceMetadata(TEST_KEYSPACE).getTableOrViewNullable(TEST_TABLE);
Token token = createToken(0L);
// Message epoch is strictly before metadata.epoch.
Message<NoPayload> msg = Message.builder(Verb._TEST_1, NoPayload.noPayload).withEpoch(Epoch.EMPTY).build();
InetAddressAndPort respondTo = FBUtilities.getBroadcastAddressAndPort();
try
{
// Coordinator says untracked (stale), handler says tracked → disagreement with older epoch → CBE.
MigrationRouter.checkPaxosCommitMigration(metadata, msg, respondTo, TEST_KEYSPACE, table.id, token, false);
Assert.fail("Expected CoordinatorBehindException");
}
catch (CoordinatorBehindException e)
{
Assert.assertTrue("Exception message should mention coordinator is behind: " + e.getMessage(),
e.getMessage().contains("coordinator") && e.getMessage().contains("behind"));
}
}
@Test
public void testCheckPaxosCommitMigration_SameEpochDisagreement()
{
ClusterMetadata metadata = createMetadata(true, Collections.emptyList());
TableMetadata table = metadata.schema.getKeyspaceMetadata(TEST_KEYSPACE).getTableOrViewNullable(TEST_TABLE);
Token token = createToken(0L);
Message<NoPayload> msg = Message.builder(Verb._TEST_1, NoPayload.noPayload).withEpoch(metadata.epoch).build();
InetAddressAndPort respondTo = FBUtilities.getBroadcastAddressAndPort();
try
{
// Same epoch, but coordinator says untracked while handler says tracked → inconsistent routing → ISE.
MigrationRouter.checkPaxosCommitMigration(metadata, msg, respondTo, TEST_KEYSPACE, table.id, token, false);
Assert.fail("Expected IllegalStateException");
}
catch (IllegalStateException e)
{
Assert.assertTrue("Exception message should mention inconsistent routing: " + e.getMessage(),
e.getMessage().contains("Inconsistent"));
}
}
@Test
public void testCheckPaxosPrepareReadMigration_Agreement()
{
ClusterMetadata metadata = createMetadata(true, Collections.emptyList());
TableMetadata table = metadata.schema.getKeyspaceMetadata(TEST_KEYSPACE).getTableOrViewNullable(TEST_TABLE);
DecoratedKey key = partitioner.decorateKey(UTF8Type.instance.decompose("test_key"));
Message<NoPayload> msg = Message.builder(Verb._TEST_1, NoPayload.noPayload).withEpoch(metadata.epoch).build();
InetAddressAndPort respondTo = FBUtilities.getBroadcastAddressAndPort();
// Tracked keyspace, no migration → handler says tracked. Coordinator says tracked (TRACKED_DATA) → agreement.
EmbeddableSinglePartitionReadCommand trackedRead = readStub(table, key, ReadKind.TRACKED_DATA);
ClusterMetadata result = MigrationRouter.checkPaxosPrepareReadMigration(metadata, msg, respondTo, trackedRead);
Assert.assertSame("Metadata should be returned unchanged on agreement", metadata, result);
}
@Test
public void testCheckPaxosPrepareReadMigration_CoordinatorBehind()
{
ClusterMetadata metadata = createMetadata(true, Collections.emptyList());
TableMetadata table = metadata.schema.getKeyspaceMetadata(TEST_KEYSPACE).getTableOrViewNullable(TEST_TABLE);
DecoratedKey key = partitioner.decorateKey(UTF8Type.instance.decompose("test_key"));
// Message epoch is strictly before metadata.epoch.
Message<NoPayload> msg = Message.builder(Verb._TEST_1, NoPayload.noPayload).withEpoch(Epoch.EMPTY).build();
InetAddressAndPort respondTo = FBUtilities.getBroadcastAddressAndPort();
try
{
// Coordinator says untracked (stale), handler says tracked → disagreement with older epoch → CBE.
EmbeddableSinglePartitionReadCommand untrackedRead = readStub(table, key, ReadKind.UNTRACKED);
MigrationRouter.checkPaxosPrepareReadMigration(metadata, msg, respondTo, untrackedRead);
Assert.fail("Expected CoordinatorBehindException");
}
catch (CoordinatorBehindException e)
{
Assert.assertTrue("Exception message should mention coordinator is behind: " + e.getMessage(),
e.getMessage().contains("coordinator") && e.getMessage().contains("behind"));
}
}
@Test
public void testCheckPaxosPrepareReadMigration_SameEpochDisagreement()
{
ClusterMetadata metadata = createMetadata(true, Collections.emptyList());
TableMetadata table = metadata.schema.getKeyspaceMetadata(TEST_KEYSPACE).getTableOrViewNullable(TEST_TABLE);
DecoratedKey key = partitioner.decorateKey(UTF8Type.instance.decompose("test_key"));
Message<NoPayload> msg = Message.builder(Verb._TEST_1, NoPayload.noPayload).withEpoch(metadata.epoch).build();
InetAddressAndPort respondTo = FBUtilities.getBroadcastAddressAndPort();
try
{
// Same epoch, but coordinator says untracked (UNTRACKED) while handler says tracked → ISE.
EmbeddableSinglePartitionReadCommand untrackedRead = readStub(table, key, ReadKind.UNTRACKED);
MigrationRouter.checkPaxosPrepareReadMigration(metadata, msg, respondTo, untrackedRead);
Assert.fail("Expected IllegalStateException");
}
catch (IllegalStateException e)
{
Assert.assertTrue("Exception message should mention inconsistent routing: " + e.getMessage(),
e.getMessage().contains("Inconsistent"));
}
}
}