Merge 9893f3b2fd into 10557d7ffe

.gitmodules

@@ -1,4 +1,4 @@
 [submodule "modules/accord"]
 	path = modules/accord
-	url = https://github.com/apache/cassandra-accord.git
+	url = https://github.com/belliottsmith/cassandra-accord.git
-	branch = trunk
+	branch = rerebootstrap

UNFINISHED.md

@@ -0,0 +1,28 @@
+### D1. Partial log unavailability (the piece you're deferring)
+
+ Current state, so it's recoverable later:
+ - Cleanup.isUnavailable (Cleanup.java:229-233) — any(LOG_UNAVAILABLE) or all(LOG_INCOMPLETE) && saveStatus < Stable, with your TODO (required): how does this interact with lost ownership?.
+ - SafeCommandStore.registerTransitiveDeps (~:565) — per-range subtraction via redundantBefore.removeLogUnavailableOrIncomplete(txnId, tmp), with TODO (required): if we only part-filter we're still going to have problems, as we won't be able to
+   update the transaction.
+ - RedundantBefore.Bounds.depBound — folds LOG_INCOMPLETE/LOG_UNAVAILABLE into the dep bound when UNREADY is ahead of the applied bound; this is what keeps deps sound after a restart, when the in-memory refuses map is gone.
+ - MapReduceCommandStores.Refuse{NONE,DEPS,ALL} + per-message abort() — the in-memory, coarse (whole-scope) refusal, only alive between unsafeRefuseRequests and unsafeAcceptRequests.
+
+ These four must agree on one predicate. Concretely the questions to settle:
+ 1. Is any ever right for LOG_UNAVAILABLE? The dangerous arm isn't FULL (throwing is merely unavailability) but non-FULL, where logUnavailable(...) returns ERASE — a compaction decision that discards the record for ranges whose log is intact.
+    If nothing else changes, splitting just that arm (throw on any, erase only on all) is strictly safer and is a one-line intermediate.
+ 2. Lost ownership / retired ranges break all. Ranges that are WAS_OWNED or locally retired never carry LOG_INCOMPLETE, so all(...) is false and we don't refuse even though the owned remainder is incomplete. Your "treat pre-log-point ranges as
+    locally-retired-like" idea is exactly the missing normalisation, and note the patch already does the dual of it in SafeCommandStore.refuses(participants) (removeLocallyRetired(participants) before folding). A shared helper — "the
+    participants we must have a complete log for" = participants − retired − wasOwned − logIncomplete/Unavailable — would let Cleanup, refuses() and registerTransitive share one definition instead of three.
+ 3. Recovery cannot answer partially. BeginRecovery refuses on refuses.max != NONE, which only covers the in-memory window, not the post-restart window. There is no wire representation for "valid only for a sub-range" — but ReadOk.unavailable
+    already is exactly that for reads, so a RecoverNack{unavailable} (and the analogous field on the deps replies) is the smallest precedent-following addition; it would also let Commit/Apply stop depending on the coarse Refuse.MinMax.
+ 4. Future distinction you mentioned: if you do split further, the axis that matters to consumers looks like "records absent below the bound" (incomplete) vs "records possibly wrong below the bound" (corrupted) vs "records absent but decisions
+    recoverable from peers". Worth writing that down in BootstrapReason/RedundantStatus.Property javadoc before adding a third status, since the persisted bit space is nearly full (see N9).
+
+ ### D2. Empty / short DurabilityResults
+
+ Now benign-ish but still wrong: DurabilityService.submit's canUseDurableBefore && isAlreadyMet shortcut calls reportSuccess() while success == DurabilityResults.EMPTY, so markBootstrapping marks nothing and the attempt completes with missing
+ == valid → onFailedBootstrap → retry. Since each retry re-derives a fresh maxConflict and can be satisfied by durableBefore again, an idle range can retry indefinitely, with the unhelpful "Unknown failure" from complete()'s fail runnable.
+ Options: (a) populate DurabilityResults.of(ranges, request.min, <nodes>) on the shortcut path; (b) make the shortcut ineligible when the requester needs per-range bounds/readable (a flag on the request); (c) fail prepareToBootstrap explicitly
+ if byTxnId() doesn't cover the requested ranges, so the diagnostic names the cause. Also sync(...)/close(...) still return success(null) for empty ranges — an empty DurabilityResults would be safer for every caller that dereferences the value.
+ Worth adding an Invariants.require(success.byTxnId() covers commitRanges) at the markBootstrapping site so this class of shortfall is loud rather than a retry loop.
+

doc/modules/cassandra/pages/architecture/accord-architecture.adoc

@@ -87,7 +87,7 @@ finally {
 }
 ....
 
-In other words, `CommandStore` collects the `PreLoadContext`, state
+In other words, `CommandStore` collects the `ExecutionContext`, state
 required to be in memory for command execution (possible dependencies,
 such as `TxnId`s, and `Key`s of commands, but also `CommandsForKeys`
 that will be needed during execution). Once the context is collected and
@@ -103,11 +103,11 @@ to ensure transactional integrity, changes to commands are tracked and
 are recorded into `Journal` for crash-recovery. `ProgressLog` and
 `CommandsForKey` are up
 
-On Cassandra side, concurrent execution is controlled by `AccordTask`,
+On Cassandra side, concurrent execution is controlled by `SafeTask`,
 which contains cache loading logic and persistence callbacks. Since
 Accord may potentially hold a large number of command states in memory,
 their states may be _shrunk_ to their binary representation to save some
-memory, or they can get fully evicted. This also means that `AccordTask`
+memory, or they can get fully evicted. This also means that `SafeTask`
 will have to reload relevant dependencies from preload context before
 command execution can begin.
 

modules/accord

@@ -1 +1 @@
-Subproject commit fff32de2e915772fbc70d16b1c32346313877838
+Subproject commit f4c8ae02f9393c5eb14720b5074bd45135ee5b00

src/java/org/apache/cassandra/concurrent/CassandraThread.java

@@ -18,14 +18,24 @@
 
 package org.apache.cassandra.concurrent;
 
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+
 import org.apache.cassandra.metrics.ThreadLocalMetrics;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
+import org.apache.cassandra.service.accord.execution.Task;
+import org.apache.cassandra.service.accord.execution.TaskRunner;
 
 import io.netty.util.concurrent.FastThreadLocalThread;
 
-public class CassandraThread extends FastThreadLocalThread
+public class CassandraThread extends FastThreadLocalThread implements TaskRunner
 {
     private ThreadLocalMetrics threadLocalMetrics;
     private ExecutorLocals executorLocals;
+    private AccordExecutor accordActiveExecutor;
+    private AccordExecutor accordLockedExecutor;
+    private int accordLockedExecutorDepth;
+    private volatile Task accordActiveTask;
+    private static final AtomicReferenceFieldUpdater<CassandraThread, Task> accordActiveTaskUpdater = AtomicReferenceFieldUpdater.newUpdater(CassandraThread.class, Task.class, "accordActiveTask");
 
     private final ImmediateTaskHolder immediateTaskHolder;
 
@@ -89,6 +99,55 @@ public class CassandraThread extends FastThreadLocalThread
         return current != null ? current : ExecutorLocals.none();
     }
 
+    public final AccordExecutor accordActiveExecutor()
+    {
+        return accordActiveExecutor;
+    }
+
+    public final void setAccordActiveExecutor(AccordExecutor newExecutor)
+    {
+        accordActiveExecutor = newExecutor;
+    }
+
+    @Override
+    public final AccordExecutor accordLockedExecutor()
+    {
+        return accordLockedExecutor;
+    }
+
+    @Override
+    public final boolean tryEnterAccordLockedExecutor(AccordExecutor newLockedExecutor)
+    {
+        if (accordLockedExecutor == null) accordLockedExecutor = newLockedExecutor;
+        else if (accordLockedExecutor != newLockedExecutor) return false;
+        ++accordLockedExecutorDepth;
+        return true;
+    }
+
+    @Override
+    public final void exitAccordLockedExecutor()
+    {
+        if (--accordLockedExecutorDepth == 0)
+            accordLockedExecutor = null;
+    }
+
+    public final Task accordActiveTask()
+    {
+        return accordActiveTask;
+    }
+
+    // to be called only by the thread itself, so can (eventually) avoid any memory barriers
+    public final Task accordActiveSelfTask()
+    {
+        // TODO (expected): with newer JDK use accordActiveTaskUpdater.getPlain
+        return accordActiveTask;
+    }
+
+    public final void setAccordActiveTask(Task newActiveTask)
+    {
+        accordActiveTaskUpdater.lazySet(this, newActiveTask);
+    }
+
     // final to avoid skipping of the cleanup logic in child classes
     final public void run()
     {

src/java/org/apache/cassandra/concurrent/Stage.java

@@ -47,7 +47,7 @@ public enum Stage
     MUTATION           (true,  "MutationStage",         "request",  DatabaseDescriptor::getConcurrentWriters,        DatabaseDescriptor::setConcurrentWriters,        Stage::multiThreadedLowSignalStage),
     COUNTER_MUTATION   (true,  "CounterMutationStage",  "request",  DatabaseDescriptor::getConcurrentCounterWriters, DatabaseDescriptor::setConcurrentCounterWriters, Stage::multiThreadedLowSignalStage),
     VIEW_MUTATION      (true,  "ViewMutationStage",     "request",  DatabaseDescriptor::getConcurrentViewWriters,    DatabaseDescriptor::setConcurrentViewWriters,    Stage::multiThreadedLowSignalStage),
-    ACCORD_MIGRATION   (false, "AccordMigrationStage", "request",   DatabaseDescriptor::getAccordConcurrentOps,      DatabaseDescriptor::setConcurrentAccordOps,      Stage::multiThreadedLowSignalStage),
+    ACCORD_MIGRATION   (false, "AccordMigrationStage",  "request",  DatabaseDescriptor::getAccordConcurrentMigrationOps, DatabaseDescriptor::setConcurrentAccordMigrationOps,             Stage::multiThreadedLowSignalStage),
     GOSSIP             (true,  "GossipStage",           "internal", () -> 1,                                         null,                                            Stage::singleThreadedStage),
     REQUEST_RESPONSE   (false, "RequestResponseStage",  "request",  FBUtilities::getAvailableProcessors,             null,                                            Stage::multiThreadedLowSignalStage),
     ANTI_ENTROPY       (false, "AntiEntropyStage",      "internal", () -> 1,                                         null,                                            Stage::singleThreadedStage),

src/java/org/apache/cassandra/config/AccordConfig.java

@@ -27,6 +27,7 @@ import accord.api.ProtocolModifiers.CoordinatorBacklogExecution;
 import accord.api.ProtocolModifiers.FastExecution;
 import accord.api.ProtocolModifiers.ReplicaExecution;
 import accord.api.ProtocolModifiers.SendStableMessages;
+import accord.api.ProtocolModifiers.UniqueTimestampOnConflict;
 import accord.primitives.TxnId;
 import accord.utils.Invariants;
 
@@ -34,8 +35,6 @@ import org.apache.cassandra.journal.Params;
 import org.apache.cassandra.service.accord.serializers.Version;
 import org.apache.cassandra.service.consensus.TransactionalMode;
 
-import static org.apache.cassandra.config.AccordConfig.CatchupMode.NORMAL;
-import static org.apache.cassandra.config.AccordConfig.QueuePriorityModel.HLC_FIFO;
 import static org.apache.cassandra.config.AccordConfig.QueueShardModel.THREAD_POOL_PER_SHARD;
 import static org.apache.cassandra.config.AccordConfig.QueueSubmissionModel.SYNC;
 import static org.apache.cassandra.config.AccordConfig.RangeIndexMode.in_memory;
@@ -134,25 +133,51 @@ public class AccordConfig
         FIFO,
 
         /**
-         * If the work has an associated TxnId, prioritise by its HLC (and FIFO otherwise)
+         * If the work has an associated TxnId of Ballot, prioritise by the newest HLC (and FIFO otherwise)
          */
         HLC_FIFO,
 
         /**
-         * Prioritise Apply, Stable, Commit, Accept, and PreAccept messages in that order.
+         * If the work has an associated TxnId, prioritise by its HLC (and FIFO otherwise)
-         * Within a given message type, prioritise by HLC.
-         * Other messages will be mixed with PreAccept messages, but using the next counter rather than the HLC of the TxnId.
-         * Note: this can have some performance edge cases for contended keys, as we may process Stable messages for later commands before
-         *       we process earlier Accept/Commit, which may delay execution
          */
-        PHASE_HLC_FIFO,
+        ORIG_HLC_FIFO
+    }
+
+    public enum QueueBalancingModel
+    {
+        /**
+         * Always pick the task by priority
+         */
+        PRIORITY_ONLY,
 
         /**
-         * Prioritise Apply, Stable, Commit, Accept, and PreAccept messages from the original coordinator only, in that order.
+         * Pick by phase first, so the highest phase with work ALWAYS runs.
-         * Within a given message type, prioritise by HLC.
+         * Within a phase, pick by priority.
-         * Other messages will be mixed with PreAccept messages, but using the next counter rather than the HLC of the TxnId.
          */
-        ORIG_PHASE_HLC_FIFO
+        PHASE_ONLY,
+
+        /**
+         * Pick by phase first, selecting the phase that has processed the least work recently relative to arrivals.
+         * Within a phase, pick by priority.
+         */
+        PHASE_FAIR,
+
+        /**
+         * While phases are within a threshold of imbalance, pick tasks by priority.
+         * Once the threshold is crossed, over-processed phases have a small penalty applied
+         * and work is prioritised by phase with the phase with the least recent work processed picked first,
+         * until the imbalance is resolved.
+         *
+         * Within a phase, pick by priority.
+         */
+        BLENDED_PRIORITY_PHASE_FAIR,
+    }
+
+    public enum UniqueTimestampReservations
+    {
+        NONE,
+        SMALL_SHARED,
+        HISTOGRAM
     }
 
     public QueueShardModel queue_shard_model = THREAD_POOL_PER_SHARD;
@@ -168,7 +193,38 @@ public class AccordConfig
      */
     public volatile OptionaldPositiveInt queue_thread_count = OptionaldPositiveInt.UNDEFINED;
 
-    public QueuePriorityModel queue_priority_model = HLC_FIFO;
+    public QueuePriorityModel queue_priority_model;
+    public QueueBalancingModel queue_balancing_model;
+
+    public Integer queue_flow_imbalance_onset = null;
+    public Integer queue_flow_imbalance_width_shift = null;
+
+    public String queue_active_limits;
+
+    public Boolean queue_nonsync_enabled;
+
+    /**
+     * Size at which we will begin processing a task that is ASYNC, INCR OR INCR_ATOMIC.
+     * Note that a size of zero will effectively give implicit priority to INCR_ATOMIC tasks, as they may immediately
+     * take a FIFO queue slot (which is processed preferentially).
+     */
+    public Integer queue_nonsync_min_batch_size;
+
+    /**
+     * If there are more than min_batch_size keys ready for an ASYNC, INCR or INCR_ATOMIC task,
+     * process up to this many keys at once.
+     */
+    public Integer queue_nonsync_max_batch_size;
+
+    /**
+     * An ASYNC, INCR or INCR_ATOMIC task that is ready to run but waiting for batch_size work will proceed
+     * once this number of tasks are blocked behind it, regardless of batch_size.
+     */
+    public Integer queue_nonsync_blocked_limit;
+    /**
+     * The number of threads that may be used to execute distributed requests for migration tasks
+     */
+    public volatile OptionaldPositiveInt migration_concurrency = OptionaldPositiveInt.UNDEFINED;
 
     /**
      * If set, the signal loop does not match park/unpark pairs, but instead consumers perform timed-park spin waits
@@ -181,15 +237,6 @@ public class AccordConfig
      */
     public DurationSpec.LongMicrosecondsBound queue_stop_check_interval;
 
-    /**
-     * If set, the signal loop reduces the number of threads it is using when the time spent parked exceeds real-time
-     * by this interval.
-     */
-    public DurationSpec.LongMicrosecondsBound queue_signal_stop_check_interval_credit;
-
-    // yield to other executor threads after executing this many tasks in a row, if there are waiting threads and tasks
-    public int queue_yield_interval = 100;
-
     /**
      * If the HLC is older than this, queue by FIFO instead
      */
@@ -204,9 +251,6 @@ public class AccordConfig
      */
     public volatile OptionaldPositiveInt command_store_shard_count = OptionaldPositiveInt.UNDEFINED;
 
-    public volatile OptionaldPositiveInt max_queued_loads = OptionaldPositiveInt.UNDEFINED;
-    public volatile OptionaldPositiveInt max_queued_range_loads = OptionaldPositiveInt.UNDEFINED;
-
     public volatile OptionaldPositiveInt progress_log_concurrency = OptionaldPositiveInt.UNDEFINED;
     public DurationSpec.IntMillisecondsBound progress_log_query_fallback_timeout = new DurationSpec.IntMillisecondsBound("1m");
 
@@ -225,6 +269,7 @@ public class AccordConfig
     public String expire_epoch_wait = "10s";
     // we don't want to wait ages for durability as it blocks other durability progress; even this might be too long, as we can always retry
     public String expire_durability = "10s*attempts <= 30s";
+    public String slow_durability = "10s";
     public String slow_syncpoint_preaccept = "10s";
     public String slow_txn_preaccept = "30ms <= p50*2 <= 1000ms";
     public String slow_read = "30ms <= p50*2 <= 1000ms";
@@ -261,12 +306,12 @@ public class AccordConfig
      */
     public volatile TransactionalRangeMigration range_migration = TransactionalRangeMigration.auto;
 
-    public enum CatchupMode
+    public enum CatchupFallbackMode
     {
-        DISABLED,
+        IGNORE,
-        NORMAL,
+        EXIT,
-        FALLBACK_TO_HARD,
+        REBOOTSTRAP,
-        HARD
+        REBOOTSTRAP_AND_CATCHUP
     }
 
     /**
@@ -296,6 +341,9 @@ public class AccordConfig
     public Boolean send_minimal;
     // note: simulator incompatible (for now)
     public Boolean precise_micros;
+    public UniqueTimestampReservations unique_timestamp_reservations;
+    public UniqueTimestampOnConflict unique_timestamp_on_conflict;
+    public DurationSpec.IntMillisecondsBound unique_timestamp_reservation_range;
 
     public boolean ephemeral_reads = true;
     public boolean state_cache_listener_jfr_enabled = false;
@@ -311,16 +359,19 @@ public class AccordConfig
     public int commands_for_key_prune_interval = 64;
     public DurationSpec.IntSecondsBound max_conflicts_prune_delta = new DurationSpec.IntSecondsBound(1);
 
+    public int catchup_on_start_max_attempts = 5;
+    public boolean catchup_on_start = true;
     public DurationSpec.IntSecondsBound catchup_on_start_success_latency = new DurationSpec.IntSecondsBound(60);
     public DurationSpec.IntSecondsBound catchup_on_start_fail_latency = new DurationSpec.IntSecondsBound(900);
-    public int catchup_on_start_max_attempts = 5;
+    // TODO (required): default this to EXIT or REBOOTSTRAP
-    // TODO (required): roll this back to catchup_on_start_exit_on_failure: true
+    public CatchupFallbackMode catchup_on_start_on_timeout = CatchupFallbackMode.IGNORE;
-    public boolean catchup_on_start_exit_on_failure = false;
+    public CatchupFallbackMode catchup_on_start_on_error = CatchupFallbackMode.IGNORE;
-    public CatchupMode catchup_on_start = NORMAL;
+    public CatchupFallbackMode catchup_on_start_on_rebootstrap_fallback = CatchupFallbackMode.IGNORE;
+    public DurationSpec.IntSecondsBound shutdown_grace_period = new DurationSpec.IntSecondsBound(15 * 60);
+
     public boolean execute_waiting_on_start = true;
     public DurationSpec.IntSecondsBound execute_waiting_on_start_timeout = new DurationSpec.IntSecondsBound(0);
     public boolean execute_waiting_on_start_fail_on_timeout = false;
-    public DurationSpec.IntSecondsBound shutdown_grace_period = new DurationSpec.IntSecondsBound(15 * 60);
 
     public enum RangeIndexMode { in_memory, journal_sai }
     public RangeIndexMode range_index_mode = in_memory;
@@ -358,7 +409,17 @@ public class AccordConfig
              * Replay journal entries for commands that are not durable to the data or command stores.
              * THIS MODE IS NOT YET SAFE TO RUN
              */
-            NON_DURABLE
+            NON_DURABLE,
+
+            /**
+             * Don't replay, simply rebootstrap, marking our log as incomplete.
+             */
+            REBOOTSTRAP_INCOMPLETE,
+
+            /**
+             * Don't replay, simply rebootstrap, marking our log as corrupted/unavailable.
+             */
+            REBOOTSTRAP_RESET
         }
 
         public enum ReplaySavePoint

src/java/org/apache/cassandra/config/DatabaseDescriptor.java

@@ -2967,6 +2967,20 @@ public class DatabaseDescriptor
         conf.accord.queue_thread_count = new OptionaldPositiveInt(concurrent_operations);
     }
 
+    public static int getAccordConcurrentMigrationOps()
+    {
+        return conf.accord.migration_concurrency.or(2 * FBUtilities.getAvailableProcessors());
+    }
+
+    public static void setConcurrentAccordMigrationOps(int concurrent_operations)
+    {
+        if (concurrent_operations < 0)
+        {
+            throw new IllegalArgumentException("Concurrent accord operations must be non-negative");
+        }
+        conf.accord.migration_concurrency = new OptionaldPositiveInt(concurrent_operations);
+    }
+
     public static int getFlushWriters()
     {
         return conf.memtable_flush_writers;

src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java

@@ -62,7 +62,8 @@ import accord.impl.CommandChange;
 import accord.impl.progresslog.DefaultProgressLog;
 import accord.impl.progresslog.DefaultProgressLog.ModeFlag;
 import accord.impl.progresslog.TxnStateKind;
-import accord.local.CatchupHard;
+import accord.local.BootstrapReason;
+import accord.local.Catchup;
 import accord.local.Cleanup;
 import accord.local.Command;
 import accord.local.CommandStore;
@@ -71,9 +72,9 @@ import accord.local.CommandStores.LatentStoreSelector;
 import accord.local.Commands;
 import accord.local.Commands.NotifyWaitingOnPlus;
 import accord.local.DurableBefore;
+import accord.local.ExecutionContext;
 import accord.local.MaxConflicts;
 import accord.local.Node;
-import accord.local.PreLoadContext;
 import accord.local.SafeCommand;
 import accord.local.SafeCommandStore;
 import accord.local.StoreParticipants;
@@ -132,11 +133,8 @@ import org.apache.cassandra.schema.Indexes;
 import org.apache.cassandra.schema.Schema;
 import org.apache.cassandra.schema.TableId;
 import org.apache.cassandra.schema.TableMetadata;
-import org.apache.cassandra.service.accord.AccordCache;
-import org.apache.cassandra.service.accord.AccordCacheEntry;
 import org.apache.cassandra.service.accord.AccordCommandStore;
 import org.apache.cassandra.service.accord.AccordCommandStores;
-import org.apache.cassandra.service.accord.AccordExecutor;
 import org.apache.cassandra.service.accord.AccordKeyspace;
 import org.apache.cassandra.service.accord.AccordOperations;
 import org.apache.cassandra.service.accord.AccordService;
@@ -154,6 +152,10 @@ import org.apache.cassandra.service.accord.debug.DebugTxnDepsAll;
 import org.apache.cassandra.service.accord.debug.DebugTxnDepsOrdered;
 import org.apache.cassandra.service.accord.debug.DebugTxnGraph;
 import org.apache.cassandra.service.accord.debug.TxnKindsAndDomains;
+import org.apache.cassandra.service.accord.execution.AccordCache;
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
+import org.apache.cassandra.service.accord.execution.TaskInfo;
 import org.apache.cassandra.service.accord.journal.AccordJournal;
 import org.apache.cassandra.service.consensus.migration.ConsensusMigrationState;
 import org.apache.cassandra.service.consensus.migration.TableMigrationState;
@@ -165,6 +167,8 @@ import org.apache.cassandra.utils.concurrent.Future;
 import static accord.coordinate.Infer.InvalidIf.NotKnownToBeInvalid;
 import static accord.impl.CommandChange.Field.ACCEPTED;
 import static accord.impl.CommandChange.Field.PROMISED;
+import static accord.local.BootstrapReason.LOG_CORRUPTED;
+import static accord.local.BootstrapReason.LOG_INCOMPLETE;
 import static accord.local.RedundantStatus.Property.GC_BEFORE;
 import static accord.local.RedundantStatus.Property.LOCALLY_APPLIED;
 import static accord.local.RedundantStatus.Property.LOCALLY_DURABLE_TO_COMMAND_STORE;
@@ -182,6 +186,7 @@ import static java.util.concurrent.TimeUnit.NANOSECONDS;
 import static org.apache.cassandra.cql3.statements.RequestValidations.invalidRequest;
 import static org.apache.cassandra.db.virtual.AbstractLazyVirtualTable.OnTimeout.BEST_EFFORT;
 import static org.apache.cassandra.db.virtual.AbstractLazyVirtualTable.OnTimeout.FAIL;
+import static org.apache.cassandra.db.virtual.AccordDebugKeyspace.CommandStoreOpsTable.CommandStoreOp.REBOOTSTRAP_CORRUPTED;
 import static org.apache.cassandra.db.virtual.VirtualTable.Sorted.ASC;
 import static org.apache.cassandra.db.virtual.VirtualTable.Sorted.SORTED;
 import static org.apache.cassandra.db.virtual.VirtualTable.Sorted.UNSORTED;
@@ -331,22 +336,22 @@ public class AccordDebugKeyspace extends VirtualKeyspace
                 int executorId = executor.executorId();
                 collector.partition(executorId).collect(rows -> {
                     int uniquePos = 0;
-                    AccordExecutor.TaskInfo prev = null;
+                    TaskInfo prev = null;
-                    for (AccordExecutor.TaskInfo info : executor.taskSnapshot())
+                    for (TaskInfo info : executor.taskSnapshot())
                     {
                         if (prev != null && info.status() == prev.status() && info.position() == prev.position()) ++uniquePos;
                         else uniquePos = 0;
                         prev = info;
-                        PreLoadContext preLoadContext = info.preLoadContext();
+                        ExecutionContext executionContext = info.preLoadContext();
                         rows.add(info.status().name(), info.position(), uniquePos)
                                  .lazyCollect(columns -> {
                                      columns.add("description", info.describe())
                                             .add("command_store_id", info.commandStoreId())
-                                            .add("txn_id", preLoadContext, PreLoadContext::primaryTxnId, TO_STRING)
+                                            .add("txn_id", executionContext, ExecutionContext::primaryTxnId, TO_STRING)
-                                            .add("txn_id_additional", preLoadContext, PreLoadContext::additionalTxnId, TO_STRING)
+                                            .add("txn_id_additional", executionContext, ExecutionContext::additionalTxnId, TO_STRING)
-                                            .add("keys", preLoadContext, PreLoadContext::keys, TO_STRING)
+                                            .add("keys", executionContext, ExecutionContext::keys, TO_STRING)
-                                            .add("keys_loading", preLoadContext, PreLoadContext::loadKeys, TO_STRING)
+                                            .add("keys_loading", executionContext, ExecutionContext::loadKeys, TO_STRING)
-                                            .add("keys_loading_for", preLoadContext, PreLoadContext::loadKeysFor, TO_STRING);
+                                            .add("keys_loading_for", executionContext, ExecutionContext::loadKeysFor, TO_STRING);
                         });
                     }
                 });
@@ -740,7 +745,7 @@ public class AccordDebugKeyspace extends VirtualKeyspace
             collector.partition(commandStore.id())
                      .collect(rows -> {
                          // TODO (desired): support maybe execute immediately with safeStore
-                         Future<?> future = toFuture(commandStore.chain((PreLoadContext.Empty) metadata::toString, safeStore -> { addRows(safeStore, rows); }));
+                         Future<?> future = toFuture(commandStore.chain((ExecutionContext.Empty) metadata::toString, safeStore -> { addRows(safeStore, rows); }));
                          if (!future.awaitUntilThrowUncheckedOnInterrupt(collector.deadlineNanos()))
                              throw new InternalTimeoutException();
                      });
@@ -1650,7 +1655,7 @@ public class AccordDebugKeyspace extends VirtualKeyspace
         {
             try (AccordCommandStore.ExclusiveCaches caches = commandStore.lockCaches())
             {
-                AccordCacheEntry<TxnId, Command> entry = caches.commands().getUnsafe(txnId);
+                AccordCacheEntry<TxnId, Command, ?> entry = caches.commands().getUnsafe(txnId);
                 return entry == null ? null : entry.getExclusive();
             }
         }
@@ -1794,7 +1799,7 @@ public class AccordDebugKeyspace extends VirtualKeyspace
                         SafeCommand safeCommand = safeStore.unsafeGet(txnId);
                         Command command = safeCommand.current();
                         if (command.saveStatus() == SaveStatus.Applying)
-                            return Commands.applyChain(safeStore, command);
+                            return Commands.applyChain(safeStore, command.asExecuted());
                         Commands.maybeExecute(safeStore, safeCommand, command, true, true, NotifyWaitingOnPlus.adapter(ignore -> {}, true, true));
                         return AsyncChains.success(null);
                     });
@@ -1875,7 +1880,7 @@ public class AccordDebugKeyspace extends VirtualKeyspace
             Node node = AccordService.unsafeInstance().node();
             if (Route.isFullRoute(route))
             {
-                PrepareRecovery.recover(node, node.someSequentialExecutor(), txnId, NotKnownToBeInvalid, (FullRoute<?>) route, null, LatentStoreSelector.standard(), (success, fail) -> {
+                PrepareRecovery.recover(node, node.someExclusiveExecutor(), txnId, NotKnownToBeInvalid, (FullRoute<?>) route, null, LatentStoreSelector.standard(), (success, fail) -> {
                     if (fail != null) result.setFailure(fail);
                     else result.setSuccess(null);
                 });
@@ -1895,7 +1900,7 @@ public class AccordDebugKeyspace extends VirtualKeyspace
             AccordService.getBlocking(accord.node()
                                             .commandStores()
                                             .forId(commandStoreId)
-                                            .chain(PreLoadContext.contextFor(txnId, TXN_OPS), apply)
+                                            .chain(ExecutionContext.unsequenced(txnId, TXN_OPS), apply)
                                             .flatMap(i -> i));
         }
 
@@ -1987,8 +1992,10 @@ public class AccordDebugKeyspace extends VirtualKeyspace
             UNSET_PROGRESS_LOG_MODE("Unset the specified progress log mode."),
             TRY_EXECUTE_LISTENING("Try to execute all of the transactions (and their dependencies) that have registered listeners on other transactions."),
             REPLAY("Run journal replay for all transactions"),
-            REBOOTSTRAP("Rebootstrap the command store. This invalidates the local journal, synchronises its data via data repair and rejoins the distributed state machine."),
+            // TODO (expected): specify ranges
-            HARD_CATCHUP("Hard catchup the command store. This invalidates the local journal for any ranges not up to date with some quorum, synchronises its data via data repair and rejoins the distributed state machine."),
+            REBOOTSTRAP_CORRUPTED("Rebootstrap the command store. This invalidates the local journal, synchronises its data via data repair and rejoins the distributed state machine."),
+            REBOOTSTRAP_INCOMPLETE("Rebootstrap the command store. This invalidates the local journal, synchronises its data via data repair and rejoins the distributed state machine."),
+            REBOOTSTRAP_IF_BEHIND("Hard catchup the command store. This invalidates the local journal for any ranges not up to date with some quorum, synchronises its data via data repair and rejoins the distributed state machine."),
             ;
 
             final String description;
@@ -2081,13 +2088,15 @@ public class AccordDebugKeyspace extends VirtualKeyspace
                     };
                     break;
                 }
-                case REBOOTSTRAP:
+                case REBOOTSTRAP_CORRUPTED:
-                    allFunction = () -> node.commandStores().rebootstrap(node);
+                case REBOOTSTRAP_INCOMPLETE:
-                    function = commandStore -> commandStore.rebootstrap(node);
+                    BootstrapReason reason = op == REBOOTSTRAP_CORRUPTED ? LOG_CORRUPTED : LOG_INCOMPLETE;
+                    allFunction = () -> node.commandStores().rebootstrap(node, reason);
+                    function = commandStore -> commandStore.rebootstrap(node, reason);
                     break;
-                case HARD_CATCHUP:
+                case REBOOTSTRAP_IF_BEHIND:
-                    allFunction = () -> CatchupHard.catchup(node, Arrays.asList(node.commandStores().all())).beginAsResult();
+                    allFunction = () -> Catchup.rebootstrapIfBehind(node, Arrays.asList(node.commandStores().all())).beginAsResult();
-                    function = commandStore -> CatchupHard.catchup(node, Collections.singletonList(commandStore)).beginAsResult();
+                    function = commandStore -> Catchup.rebootstrapIfBehind(node, Collections.singletonList(commandStore)).beginAsResult();
                     break;
             }
 
@@ -2483,7 +2492,6 @@ public class AccordDebugKeyspace extends VirtualKeyspace
         {
             throw new InvalidRequestException("Unknown bucket_mode '" + value + '\'');
         }
-
     }
 
     private static int checkNonNegative(Object value, String field, int ifNull)

src/java/org/apache/cassandra/db/virtual/QueriesTable.java

@@ -25,8 +25,8 @@ import org.apache.cassandra.db.marshal.LongType;
 import org.apache.cassandra.db.marshal.UTF8Type;
 import org.apache.cassandra.dht.LocalPartitioner;
 import org.apache.cassandra.schema.TableMetadata;
-import org.apache.cassandra.service.accord.AccordExecutor;
 import org.apache.cassandra.service.accord.AccordService;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
 
 import static java.lang.Long.max;
 import static java.util.concurrent.TimeUnit.NANOSECONDS;

src/java/org/apache/cassandra/exceptions/ExceptionSerializer.java

@@ -44,6 +44,9 @@ import static org.apache.cassandra.db.TypeSizes.sizeofUnsignedVInt;
  */
 public class ExceptionSerializer
 {
+    // allow plenty of room for UTF8 encoding
+    private static final int MAX_MESSAGE_CHAR_LENGTH = Short.MAX_VALUE / 4;
+
     public static class RemoteException extends RuntimeException
     {
         public final String originalClass;
@@ -72,10 +75,16 @@ public class ExceptionSerializer
 
     static String getMessageWithOriginatingHost(Throwable t, boolean isFirstException)
     {
-        if (isFirstException)
+        String message;
-            return "Remote exception from host " + FBUtilities.getBroadcastAddressAndPort().toString() + (t.getLocalizedMessage() != null ? " - " + t.getLocalizedMessage() : "");
+        if (isFirstException) message = "Remote exception from host " + FBUtilities.getBroadcastAddressAndPort().toString() + (t.getLocalizedMessage() != null ? " - " + t.getLocalizedMessage() : "");
-        else
+        else message = t.getLocalizedMessage();
-            return t.getLocalizedMessage();
+
+        if (message == null)
+            return message;
+
+        if (message.length() > MAX_MESSAGE_CHAR_LENGTH)
+            message = message.substring(0, MAX_MESSAGE_CHAR_LENGTH);
+        return message;
     }
 
     private static final IVersionedSerializer<StackTraceElement> stackTraceElementSerializer = new IVersionedSerializer<StackTraceElement>()

src/java/org/apache/cassandra/gms/FailureDetector.java

@@ -377,6 +377,10 @@ public class FailureDetector implements IFailureDetector, FailureDetectorMBean
             logger.debug("Still not marking nodes down due to local pause");
             return;
         }
+
+        if (!isAlive(ep))
+            return; // don't convict nodes that are already down - this helps Accord on startup which doesn't report itself alive in Gossip until ready to serve traffic
+
         double phi = hbWnd.phi(now);
         logger.trace("PHI for {} : {}", ep, phi);
 

src/java/org/apache/cassandra/io/util/PathUtils.java

@@ -662,7 +662,6 @@ public final class PathUtils
         DeleteOnExit.clearOnExitThreads();
     }
 
-
     private static final class DeleteOnExit implements Runnable
     {
         private boolean isRegistered;

src/java/org/apache/cassandra/journal/Compactor.java

@@ -17,7 +17,6 @@
  */
 package org.apache.cassandra.journal;
 
-import java.io.IOException;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.List;
@@ -29,6 +28,7 @@ import org.slf4j.LoggerFactory;
 
 import org.apache.cassandra.concurrent.ScheduledExecutorPlus;
 import org.apache.cassandra.concurrent.Shutdownable;
+import org.apache.cassandra.journal.SegmentCompactor.Stopped;
 import org.apache.cassandra.utils.concurrent.WaitQueue;
 
 import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
@@ -100,9 +100,9 @@ public final class Compactor<K, V> implements Runnable, Shutdownable
 
             compacted.signalAll();
         }
-        catch (IOException e)
+        catch (Stopped e)
         {
-            throw new RuntimeException("Could not compact segments: " + toCompact);
+            logger.info("Cancelling compaction of {}", toCompact);
         }
     }
 
@@ -118,6 +118,7 @@ public final class Compactor<K, V> implements Runnable, Shutdownable
         logger.debug("Shutting down " + executor);
         if (scheduled != null)
             scheduled.cancel(false);
+        segmentCompactor.stop();
         executor.shutdown();
     }
 

src/java/org/apache/cassandra/journal/SegmentCompactor.java

@@ -17,18 +17,20 @@
  */
 package org.apache.cassandra.journal;
 
-import java.io.IOException;
 import java.util.Collection;
 
 public interface SegmentCompactor<K, V>
 {
     SegmentCompactor<?, ?> NOOP = (SegmentCompactor<Object, Object>) (segments) -> segments;
 
+    class Stopped extends RuntimeException {}
+
     static <K, V> SegmentCompactor<K, V> noop()
     {
         //noinspection unchecked
         return (SegmentCompactor<K, V>) NOOP;
     }
 
-    Collection<StaticSegment<K, V>> compact(Collection<StaticSegment<K, V>> segments) throws IOException;
+    Collection<StaticSegment<K, V>> compact(Collection<StaticSegment<K, V>> segments) throws Stopped;
+    default void stop() {}
 }

src/java/org/apache/cassandra/metrics/AccordCacheMetrics.java

@@ -24,8 +24,8 @@ import java.util.function.ToLongFunction;
 
 import com.codahale.metrics.Gauge;
 
-import org.apache.cassandra.service.accord.AccordExecutor;
 import org.apache.cassandra.service.accord.IAccordService;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
 
 import static org.apache.cassandra.metrics.AccordMetricUtils.fromAccordService;
 import static org.apache.cassandra.metrics.CassandraMetricsRegistry.Metrics;
@@ -46,7 +46,7 @@ public class AccordCacheMetrics
         public AccordCacheGlobalMetrics()
         {
             DefaultNameFactory factory = new DefaultNameFactory(ACCORD_CACHE);
-            this.usedBytes = Metrics.gauge(factory.createMetricName("UsedBytes"), fromAccordService(sumExecutors(executor -> executor.cacheUnsafe().weightedSize()), 0L));
+            this.usedBytes = Metrics.gauge(factory.createMetricName("UsedBytes"), fromAccordService(sumExecutors(AccordExecutor::weightedSize), 0L));
             this.unreferencedBytes = Metrics.gauge(factory.createMetricName("UnreferencedBytes"), fromAccordService(sumExecutors(executor -> executor.cacheUnsafe().unreferencedBytes()), 0L));
         }
 

src/java/org/apache/cassandra/metrics/AccordExecutorMetrics.java

@@ -23,10 +23,10 @@ import java.util.concurrent.TimeUnit;
 import com.codahale.metrics.Gauge;
 
 import org.apache.cassandra.metrics.ShardedDecayingHistograms.ShardedDecayingHistogram;
-import org.apache.cassandra.service.accord.AccordExecutor;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
 
 import static org.apache.cassandra.metrics.CassandraMetricsRegistry.Metrics;
-import static org.apache.cassandra.service.accord.AccordExecutor.HISTOGRAMS;
+import static org.apache.cassandra.service.accord.execution.AccordExecutor.HISTOGRAMS;
 
 public class AccordExecutorMetrics
 {

src/java/org/apache/cassandra/metrics/AccordReplicaMetrics.java

@@ -33,11 +33,11 @@ import org.apache.cassandra.metrics.LogLinearDecayingHistograms.LogLinearDecayin
 import org.apache.cassandra.metrics.ShardedDecayingHistograms.DecayingHistogramsShard;
 import org.apache.cassandra.metrics.ShardedDecayingHistograms.ShardedDecayingHistogram;
 import org.apache.cassandra.service.accord.AccordCommandStore;
-import org.apache.cassandra.service.accord.AccordSafeCommandStore;
+import org.apache.cassandra.service.accord.execution.SaferCommandStore;
 import org.apache.cassandra.tracing.Tracing;
 
 import static org.apache.cassandra.metrics.CassandraMetricsRegistry.Metrics;
-import static org.apache.cassandra.service.accord.AccordExecutor.HISTOGRAMS;
+import static org.apache.cassandra.service.accord.execution.AccordExecutor.HISTOGRAMS;
 import static org.apache.cassandra.utils.Clock.Global.currentTimeMillis;
 
 public class AccordReplicaMetrics
@@ -168,7 +168,7 @@ public class AccordReplicaMetrics
 
         private static LogLinearDecayingHistograms.Buffer buffer(SafeCommandStore safeStore)
         {
-            return ((AccordSafeCommandStore) safeStore).histogramBuffer();
+            return ((SaferCommandStore) safeStore).histogramBuffer();
         }
 
         @Override

src/java/org/apache/cassandra/metrics/LogLinearDecayingHistograms.java

@@ -59,6 +59,9 @@ public class LogLinearDecayingHistograms
 
         private void add(long histogramIndex, long value)
         {
+            if (!Invariants.expect(value >= 0, "Negative value reported: %s", value))
+                return;
+
             Invariants.require(histogramIndex <= HISTOGRAM_INDEX_MASK);
             Invariants.require(value >= 0);
             if (value <= LARGE_VALUE) value <<= HISTOGRAM_INDEX_BITS;
@@ -73,6 +76,11 @@ public class LogLinearDecayingHistograms
             buffer[bufferCount++] = value;
         }
 
+        public void clear()
+        {
+            bufferCount = 0;
+        }
+
         public void flush(long at)
         {
             for (int i = 0 ; i < bufferCount ; ++i)
@@ -112,6 +120,9 @@ public class LogLinearDecayingHistograms
 
         public void increment(long value, long at)
         {
+            if (!Invariants.expect(value >= 0, "Negative value reported: %s", value))
+                return;
+
             int index = index(value);
             if (bufferCount >= buffer.length)
             {
@@ -122,6 +133,7 @@ public class LogLinearDecayingHistograms
             updateDecay(at);
             long v = Double.doubleToRawLongBits(increment) & VALUE_MASK;
             v |= histogramIndex | ((long)index << HISTOGRAM_INDEX_BITS);
+            Invariants.require(Double.longBitsToDouble(v & VALUE_MASK) > 0);
             buffer[bufferCount++] = v;
         }
 

src/java/org/apache/cassandra/metrics/LogLinearHistogram.java

@@ -54,6 +54,9 @@ public class LogLinearHistogram
 
     public void increment(long value)
     {
+        if (!Invariants.expect(value >= 0, "Negative value reported: %s", value))
+            return;
+
         int index = index(value);
         buckets(index)[index]++;
         ++totalCount;
@@ -61,6 +64,9 @@ public class LogLinearHistogram
 
     public void decrement(long value)
     {
+        if (!Invariants.expect(value >= 0, "Negative value reported: %s", value))
+            return;
+
         int index = index(value);
         buckets(index)[index]--;
         --totalCount;
@@ -68,6 +74,9 @@ public class LogLinearHistogram
 
     public void replace(long decrement, long increment)
     {
+        if (!Invariants.expect(decrement >= 0 && increment >= 0, "Negative value(s?) reported: %s and %s", decrement,  increment))
+            return;
+
         int decrementIndex = index(decrement);
         int incrementIndex = index(increment);
         long[] buckets = buckets(Math.max(decrementIndex, incrementIndex));

src/java/org/apache/cassandra/service/accord/AccordCacheEntry.java

@@ -1,764 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.cassandra.service.accord;
-
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.List;
-import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
-import java.util.function.BiConsumer;
-
-import javax.annotation.Nullable;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.primitives.Ints;
-
-import accord.utils.ArrayBuffers.BufferList;
-import accord.utils.IntrusiveLinkedList;
-import accord.utils.IntrusiveLinkedListNode;
-import accord.utils.Invariants;
-import accord.utils.async.Cancellable;
-
-import org.apache.cassandra.service.accord.AccordCache.Adapter;
-import org.apache.cassandra.service.accord.AccordCache.Adapter.Shrink;
-import org.apache.cassandra.utils.ObjectSizes;
-
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.EVICTED;
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.FAILED_TO_LOAD;
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.FAILED_TO_SAVE;
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.LOADED;
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.LOADING;
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.MODIFIED;
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.SAVING;
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.WAITING_TO_LOAD;
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.WAITING_TO_SAVE;
-
-/**
- * Global (per CommandStore) state of a cached entity (Command or CommandsForKey).
- */
-public class AccordCacheEntry<K, V> extends IntrusiveLinkedListNode
-{
-    public enum Status
-    {
-        UNINITIALIZED,
-
-        UNUSED1, // spacing to permit easier bit masks
-
-        WAITING_TO_LOAD(UNINITIALIZED),
-        LOADING(WAITING_TO_LOAD),
-
-        /**
-         * Consumers should never see this state
-         */
-        FAILED_TO_LOAD(LOADING),
-
-        UNUSED2, // spacing to permit easier bit masks
-        UNUSED3, // spacing to permit easier bit masks
-        UNUSED4, // spacing to permit easier bit masks
-
-        LOADED(true, false, UNINITIALIZED, LOADING),
-        MODIFIED(true, false, LOADED),
-
-        UNUSED5, // spacing to permit easier bit masks
-        UNUSED6, // spacing to permit easier bit masks
-
-        WAITING_TO_SAVE(true, true, MODIFIED),
-        SAVING(true, true, MODIFIED, WAITING_TO_SAVE),
-
-        /**
-         * Attempted to save but failed. Shouldn't normally happen unless we have a bug in serialization,
-         * or commit log has been stopped.
-         */
-        FAILED_TO_SAVE(true, true, SAVING),
-
-        UNUSED7, // spacing to permit easier bit masks
-
-        EVICTED(WAITING_TO_LOAD, LOADING, LOADED, FAILED_TO_LOAD),
-        ;
-
-        static final Status[] VALUES = values();
-        static
-        {
-            MODIFIED.permittedFrom |= 1 << MODIFIED.ordinal();
-            MODIFIED.permittedFrom |= 1 << SAVING.ordinal();
-            MODIFIED.permittedFrom |= 1 << FAILED_TO_SAVE.ordinal();
-            LOADED.permittedFrom |= 1 << SAVING.ordinal();
-            LOADED.permittedFrom |= 1 << MODIFIED.ordinal();
-            for (Status status : VALUES)
-            {
-                if (status.name().startsWith("UNUSED")) continue;
-                Invariants.require((status.ordinal() & IS_LOADED) != 0 == status.loaded);
-                Invariants.require(((status.ordinal() & IS_LOADED) != 0 && (status.ordinal() & IS_NESTED) != 0) == status.nested);
-                Invariants.require(((status.ordinal() & IS_LOADING_OR_WAITING_MASK) == IS_LOADING_OR_WAITING) == (status == LOADING || status == WAITING_TO_LOAD));
-                Invariants.require(((status.ordinal() & IS_SAVING_OR_WAITING_MASK) == IS_SAVING_OR_WAITING) == (status == SAVING || status == WAITING_TO_SAVE));
-            }
-        }
-
-        final boolean loaded;
-        final boolean nested;
-        int permittedFrom;
-
-        Status(Status ... statuses)
-        {
-            this(false, false, statuses);
-        }
-
-        Status(boolean loaded, boolean nested, Status ... statuses)
-        {
-            this.loaded = loaded;
-            this.nested = nested;
-            for (Status status : statuses)
-                permittedFrom |= 1 << status.ordinal();
-        }
-    }
-
-    static final int STATUS_MASK = 0x0000001F;
-    static final int SHRUNK = 0x00000040;
-    static final int NO_EVICT = 0x00000020;
-    static final int IS_NOT_EVICTED = 0xF;
-    static final int IS_LOADED = 0x8;
-    static final int IS_NESTED = 0x4;
-    static final int IS_LOADING_OR_WAITING_MASK = 0x6;
-    static final int IS_LOADING_OR_WAITING = 0x2;
-    static final int IS_SAVING_OR_WAITING_MASK = 0xE;
-    static final int IS_SAVING_OR_WAITING = 0xC;
-    static final long EMPTY_SIZE = ObjectSizes.measure(new AccordCacheEntry<>(null, null));
-
-    private final K key;
-    final AccordCache.Type<K, V, ?>.Instance owner;
-
-    private Object state;
-    private int status;
-    int sizeOnHeap;
-    private volatile int references;
-    private static final AtomicIntegerFieldUpdater<AccordCacheEntry> referencesUpdater = AtomicIntegerFieldUpdater.newUpdater(AccordCacheEntry.class, "references");
-
-    AccordCacheEntry(K key, AccordCache.Type<K, V, ?>.Instance owner)
-    {
-        this.key = key;
-        this.owner = owner;
-    }
-
-    void unlink()
-    {
-        remove();
-    }
-
-    boolean isUnqueued()
-    {
-        return isFree();
-    }
-
-    public K key()
-    {
-        return key;
-    }
-
-    public int references()
-    {
-        return references;
-    }
-
-    public int increment()
-    {
-        return referencesUpdater.incrementAndGet(this);
-    }
-
-    public int decrement()
-    {
-        return referencesUpdater.decrementAndGet(this);
-    }
-
-    boolean isLoaded()
-    {
-        return (status & IS_LOADED) != 0;
-    }
-
-    boolean isModified()
-    {
-        return (status & IS_NOT_EVICTED) >= MODIFIED.ordinal();
-    }
-
-    boolean isNested()
-    {
-        Invariants.require(isLoaded());
-        return (status & IS_NESTED) != 0;
-    }
-
-    boolean isShrunk()
-    {
-        return (status & SHRUNK) != 0;
-    }
-
-    public boolean is(Status status)
-    {
-        return (this.status & STATUS_MASK) == status.ordinal();
-    }
-
-    boolean isLoadingOrWaiting()
-    {
-        return (status & IS_LOADING_OR_WAITING_MASK) == IS_LOADING_OR_WAITING;
-    }
-
-    boolean isSavingOrWaiting()
-    {
-        return (status & IS_SAVING_OR_WAITING_MASK) == IS_SAVING_OR_WAITING;
-    }
-
-    public boolean isComplete()
-    {
-        return !is(LOADING) && !is(SAVING);
-    }
-
-    int noEvictGeneration()
-    {
-        Invariants.require(isNoEvict());
-        return (status >>> 8) & 0xffff;
-    }
-
-    int noEvictMaxAge()
-    {
-        Invariants.require(isNoEvict());
-        return status >>> 24;
-    }
-
-    boolean isNoEvict()
-    {
-        return (status & NO_EVICT) != 0;
-    }
-
-    int sizeOnHeap()
-    {
-        return sizeOnHeap;
-    }
-
-    void updateSize(AccordCache.Type<K, V, ?> parent)
-    {
-        // TODO (expected): we aren't weighing the keys
-        int newSizeOnHeap = Ints.saturatedCast(EMPTY_SIZE + estimateOnHeapSize(parent.adapter()));
-        parent.updateSize(newSizeOnHeap, newSizeOnHeap - sizeOnHeap, references == 0, true);
-        sizeOnHeap = newSizeOnHeap;
-    }
-
-    void initSize(AccordCache.Type<K, V, ?> parent)
-    {
-        // TODO (expected): we aren't weighing the keys
-        sizeOnHeap = Ints.saturatedCast(EMPTY_SIZE);
-        parent.updateSize(sizeOnHeap, sizeOnHeap, false, false);
-        parent.objectSize.increment(EMPTY_SIZE);
-    }
-
-    @Override
-    public String toString()
-    {
-        return "Node{" + status() +
-               ", key=" + key() +
-               ", references=" + references +
-               "}@" + Integer.toHexString(System.identityHashCode(this));
-    }
-
-    public Status status()
-    {
-        return Status.VALUES[(status & STATUS_MASK)];
-    }
-
-    private void setStatus(Status newStatus)
-    {
-        Invariants.require((newStatus.permittedFrom & (1 << (status & STATUS_MASK))) != 0, "%s not permitted from %s", newStatus, status());
-        setStatusUnsafe(newStatus);
-    }
-
-    private void setStatusUnsafe(Status newStatus)
-    {
-        status &= ~STATUS_MASK;
-        status |= newStatus.ordinal();
-    }
-
-    public void initialize(V value)
-    {
-        Invariants.require(state == null);
-        setStatus(LOADED);
-        state = value;
-    }
-
-    public void readyToLoad()
-    {
-        Invariants.require(state == null);
-        setStatus(WAITING_TO_LOAD);
-        state = new WaitingToLoad();
-    }
-
-    public void markNoEvict(int generation, int maxAge)
-    {
-        Invariants.require((maxAge & ~0xff) == 0);
-        Invariants.require((generation & ~0xffff) == 0);
-        status |= NO_EVICT;
-        status |= generation << 8;
-        status |= maxAge << 24;
-    }
-
-    public LoadingOrWaiting loadingOrWaiting()
-    {
-        return (LoadingOrWaiting)state;
-    }
-
-    void notifyListeners(BiConsumer<AccordCache.Listener<K, V>, AccordCacheEntry<K, V>> notify)
-    {
-        owner.notifyListeners(notify, this);
-    }
-
-    public interface LoadExecutor<P1, P2>
-    {
-        <K, V> Cancellable load(P1 p1, P2 p2, AccordCacheEntry<K, V> entry);
-    }
-
-    // functions as both an identity object, and a register of listeners
-    public static class UniqueSave
-    {
-        @Nullable List<Runnable> onSuccess;
-        void onSuccess(Runnable onSuccess)
-        {
-            if (this.onSuccess == null)
-                this.onSuccess = new ArrayList<>();
-            this.onSuccess.add(onSuccess);
-        }
-
-        static void notify(List<Runnable> onSuccess)
-        {
-            if (onSuccess != null)
-            {
-                onSuccess.forEach(run -> {
-                    try { run.run(); }
-                    catch (Throwable t)
-                    {
-                        Thread thread = Thread.currentThread();
-                        thread.getUncaughtExceptionHandler().uncaughtException(thread, t);
-                    }
-                });
-            }
-        }
-    }
-
-    public interface SaveExecutor
-    {
-        Cancellable save(AccordCacheEntry<?, ?> saving, UniqueSave identity, Runnable save);
-    }
-
-    public <P1, P2> Loading load(LoadExecutor<P1, P2> loadExecutor, P1 p1, P2 p2)
-    {
-        Invariants.require(is(WAITING_TO_LOAD), "%s", this);
-
-        WaitingToLoad cur = (WaitingToLoad)state;
-        Loading loading = cur.load(loadExecutor.load(p1, p2, this));
-        setStatus(LOADING);
-        state = loading;
-        return loading;
-    }
-
-    public Loading testLoad()
-    {
-        Invariants.require(is(WAITING_TO_LOAD));
-        Loading loading = ((WaitingToLoad)state).load(() -> {});
-        setStatus(LOADING);
-        state = loading;
-        return loading;
-    }
-
-    public Loading loading()
-    {
-        Invariants.require(is(LOADING), "%s", this);
-        return (Loading) state;
-    }
-
-    // must own the cache's lock when invoked. this is true of most methods in the class,
-    // but this one is less obvious so named as to draw attention
-    public V getExclusive()
-    {
-        Invariants.require(owner == null || owner.commandStore == null || owner.commandStore.executor().isOwningThread());
-        Invariants.require(isLoaded(), "%s", this);
-        if (isShrunk())
-        {
-            AccordCache.Type<K, V, ?> parent = owner.parent();
-            inflate(owner.commandStore, key, parent.adapter());
-            updateSize(parent);
-        }
-
-        return (V)unwrap();
-    }
-
-    public Object getOrShrunkExclusive()
-    {
-        Invariants.require(owner == null || owner.commandStore == null || owner.commandStore.executor().isOwningThread());
-        Invariants.require(isLoaded(), "%s", this);
-        return unwrap();
-    }
-
-    public V tryGetExclusive()
-    {
-        Invariants.require(owner == null || owner.commandStore == null || owner.commandStore.executor().isOwningThread());
-        if (!isLoaded() || isShrunk())
-            return null;
-        return (V)unwrap();
-    }
-
-    private Object unwrap()
-    {
-        return isNested() ? ((Nested)state).state : state;
-    }
-
-    // must own the cache's lock when invoked
-    void setExclusive(V value)
-    {
-        if (value == state)
-            return;
-
-        Saving cancel = is(SAVING) ? ((Saving)state) : null;
-        if (is(WAITING_TO_SAVE))
-        {
-            ((WaitingToSave<K, V>) state).state = value;
-            if (owner.parent().adapter().canSave(value, null))
-                save();
-        }
-        else
-        {
-            setStatus(MODIFIED);
-            state = value;
-        }
-        updateSize(owner.parent());
-        // TODO (expected): do we want to cancel in-progress saving?
-        if (cancel != null && cancel.identity.onSuccess == null)
-            cancel.saving.cancel();
-    }
-
-    public void loaded(V value)
-    {
-        setStatus(LOADED);
-        state = value;
-        updateSize(owner.parent());
-    }
-
-    public void testLoaded(V value)
-    {
-        setStatus(LOADED);
-        state = value;
-    }
-
-    public void failedToLoad()
-    {
-        setStatus(FAILED_TO_LOAD);
-        state = null;
-    }
-
-    Shrink tryShrink()
-    {
-        if (!isLoaded())
-            return Shrink.EVICT;
-
-        AccordCache.Type<K, V, ?> parent = owner.parent();
-        Adapter<K, V, ?> adapter = parent.adapter();
-        if (isShrunk() || state == null)
-            return Shrink.EVICT;
-
-        V cur = (V)unwrap();
-        Shrink shrink = adapter.decideFullShrink(key, cur);
-        if (shrink == Shrink.PERFORM_WITHOUT_LOCK)
-            return Shrink.PERFORM_WITHOUT_LOCK;
-
-        Object upd = adapter.fullShrink(key, cur);
-        if (upd == null || upd == cur)
-            return Shrink.EVICT;
-        applyShrink(parent, cur, upd);
-        return Shrink.DONE;
-    }
-
-    V tryGetFull()
-    {
-        return isShrunk() ? null : (V)unwrap();
-    }
-
-    Object tryGetShrunk()
-    {
-        return isShrunk() ? unwrap() : null;
-    }
-
-    boolean isNull()
-    {
-        return state == null;
-    }
-
-    boolean saveWhenReady()
-    {
-        V full = isShrunk() ? null : (V)state;
-        Object shrunk = isShrunk() ? state : null;
-        if (owner.parent().adapter().canSave(full, shrunk))
-            return save();
-
-        setStatus(WAITING_TO_SAVE);
-        UniqueSave identity = new UniqueSave();
-        state = new WaitingToSave<>(identity, state);
-        return true;
-    }
-
-    /**
-     * Submits a save runnable to the specified executor. When the runnable
-     * has completed, the state save will have either completed or failed.
-     */
-    @VisibleForTesting
-    boolean save()
-    {
-        WaitingToSave<K, V> waitingToSave = is(WAITING_TO_SAVE) ? (WaitingToSave<K, V>)state : null;
-        Object state = waitingToSave == null ? this.state : waitingToSave.state;
-        V full = isShrunk() ? null : (V)state;
-        Object shrunk = isShrunk() ? state : null;
-        Runnable save = owner.parent().adapter().save(owner.commandStore, key, full, shrunk);
-
-        UniqueSave identity = waitingToSave == null ? new UniqueSave() : waitingToSave.identity;
-        if (null == save) // null mutation -> null Runnable -> no change on disk
-        {
-            setStatus(LOADED);
-            if (waitingToSave != null)
-                this.state = state;
-            UniqueSave.notify(identity.onSuccess);
-            return false;
-        }
-        else
-        {
-            setStatus(SAVING);
-            Cancellable saving = owner.parent().parent().saveExecutor.save(this, identity, save);
-            this.state = new Saving(saving, identity, state);
-            return true;
-        }
-    }
-
-    boolean saved(Object identity, Throwable fail)
-    {
-        if (identity instanceof UniqueSave)
-            UniqueSave.notify(((UniqueSave) identity).onSuccess);
-
-        if (!is(SAVING))
-            return false;
-
-        Saving saving = (Saving) state;
-        if (saving.identity != identity)
-            return false;
-
-        if (fail != null)
-        {
-            setStatus(FAILED_TO_SAVE);
-            state = new FailedToSave(fail, ((Saving)state).state);
-            return false;
-        }
-        else
-        {
-            setStatus(LOADED);
-            state = saving.state;
-            return true;
-        }
-    }
-
-    protected void saved()
-    {
-        Invariants.require(is(MODIFIED));
-        setStatus(LOADED);
-    }
-
-    public SavingOrWaitingToSave savingOrWaitingToSave()
-    {
-        return (SavingOrWaitingToSave) state;
-    }
-
-    public AccordCacheEntry<K, V> evicted()
-    {
-        if (isNoEvict())
-            setStatusUnsafe(EVICTED);
-        else setStatus(EVICTED);
-        state = null;
-        return this;
-    }
-
-    public Throwable failure()
-    {
-        return ((FailedToSave)state).cause;
-    }
-
-    void tryApplyShrink(Object cur, Object upd, IntrusiveLinkedList<AccordCacheEntry<?,?>> queue)
-    {
-        if (references() > 0 || !isUnqueued())
-            return;
-
-        if (isLoaded() && unwrap() == cur && upd != cur && upd != null)
-            applyShrink(owner.parent(), cur, upd);
-        queue.addLast(this);
-    }
-
-    private void applyShrink(AccordCache.Type<K, V, ?> parent, Object cur, Object upd)
-    {
-        if (isNested()) ((Nested)this.state).state = upd;
-        else this.state = upd;
-        status |= SHRUNK;
-        updateSize(parent);
-    }
-
-    private void inflate(AccordCommandStore commandStore, K key, Adapter<K, V, ?> adapter)
-    {
-        Invariants.require(isShrunk());
-        if (isNested())
-        {
-            Nested nested = (Nested) state;
-            nested.state = adapter.inflate(commandStore, key, nested.state);
-        }
-        else
-        {
-            state = adapter.inflate(commandStore, key, state);
-        }
-        status &= ~SHRUNK;
-    }
-
-    private long estimateOnHeapSize(Adapter<K, V, ?> adapter)
-    {
-        Object current = unwrap();
-        if (current == null) return 0;
-        else if (isShrunk()) return adapter.estimateShrunkHeapSize(current);
-        return adapter.estimateHeapSize((V)current);
-    }
-
-    public static abstract class LoadingOrWaiting
-    {
-        Collection<AccordTask<?>> waiters;
-
-        public LoadingOrWaiting()
-        {
-        }
-
-        public LoadingOrWaiting(Collection<AccordTask<?>> waiters)
-        {
-            this.waiters = waiters;
-        }
-
-        public Collection<AccordTask<?>> waiters()
-        {
-            return waiters != null ? waiters : Collections.emptyList();
-        }
-
-        public BufferList<AccordTask<?>> copyWaiters()
-        {
-            BufferList<AccordTask<?>> list = new BufferList<>();
-            if (waiters != null)
-                list.addAll(waiters);
-            return list;
-        }
-
-        public void add(AccordTask<?> waiter)
-        {
-            if (waiters == null)
-                waiters = new ArrayList<>();
-            waiters.add(waiter);
-        }
-
-        public void remove(AccordTask<?> waiter)
-        {
-            if (waiters != null)
-            {
-                waiters.remove(waiter);
-                if (waiters.isEmpty())
-                    waiters = null;
-            }
-        }
-    }
-
-    static class WaitingToLoad extends LoadingOrWaiting
-    {
-        public Loading load(Cancellable loading)
-        {
-            Invariants.paranoid(waiters == null || !waiters.isEmpty());
-            Loading result = new Loading(waiters, loading);
-            waiters = Collections.emptyList();
-            return result;
-        }
-    }
-
-    static class Loading extends LoadingOrWaiting
-    {
-        public final Cancellable loading;
-
-        public Loading(Collection<AccordTask<?>> waiters, Cancellable loading)
-        {
-            super(waiters);
-            this.loading = loading;
-        }
-    }
-
-    static class Nested
-    {
-        Object state;
-    }
-
-    static class SavingOrWaitingToSave extends Nested
-    {
-        final UniqueSave identity;
-
-        SavingOrWaitingToSave(UniqueSave identity, Object state)
-        {
-            this.identity = identity;
-            this.state = state;
-        }
-    }
-
-    static class Saving extends SavingOrWaitingToSave
-    {
-        final Cancellable saving;
-
-        Saving(Cancellable saving, UniqueSave identity, Object state)
-        {
-            super(identity, state);
-            this.saving = saving;
-        }
-    }
-
-    static class WaitingToSave<K, V> extends SavingOrWaitingToSave
-    {
-        WaitingToSave(UniqueSave identity, Object state)
-        {
-            super(identity, state);
-        }
-    }
-
-    static class FailedToSave extends Nested
-    {
-        final Throwable cause;
-
-        FailedToSave(Throwable cause, Object state)
-        {
-            this.cause = cause;
-            this.state = state;
-        }
-
-        public Throwable failure()
-        {
-            return cause;
-        }
-    }
-
-    public static <K, V> AccordCacheEntry<K, V> createReadyToLoad(K key, AccordCache.Type<K, V, ?>.Instance owner)
-    {
-        AccordCacheEntry<K, V> node = new AccordCacheEntry<>(key, owner);
-        node.readyToLoad();
-        return node;
-    }
-}

src/java/org/apache/cassandra/service/accord/AccordCommandStore.java

@@ -27,7 +27,6 @@ import java.util.Objects;
 import java.util.concurrent.Callable;
 import java.util.concurrent.atomic.AtomicLong;
 import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
-import java.util.concurrent.locks.Lock;
 import java.util.function.Consumer;
 import java.util.function.Function;
 import java.util.function.IntFunction;
@@ -41,7 +40,6 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import accord.api.Agent;
-import accord.api.AsyncExecutor;
 import accord.api.DataStore;
 import accord.api.Journal;
 import accord.api.LocalListeners;
@@ -57,14 +55,13 @@ import accord.impl.progresslog.TxnState;
 import accord.local.Command;
 import accord.local.CommandStore;
 import accord.local.CommandStores.RangesForEpoch;
-import accord.local.CommandSummaries;
+import accord.local.ExecutionContext;
+import accord.local.ExecutionContext.Empty;
 import accord.local.MaxConflicts;
 import accord.local.MaxDecidedRX;
 import accord.local.MinimalCommand;
 import accord.local.MinimalCommand.MinimalWithDeps;
 import accord.local.NodeCommandStoreService;
-import accord.local.PreLoadContext;
-import accord.local.PreLoadContext.Empty;
 import accord.local.RedundantBefore;
 import accord.local.RedundantBefore.Bounds;
 import accord.local.RedundantStatus.Property;
@@ -74,7 +71,6 @@ import accord.local.cfk.CommandsForKey;
 import accord.primitives.PartialTxn;
 import accord.primitives.Range;
 import accord.primitives.Ranges;
-import accord.primitives.RoutableKey;
 import accord.primitives.Route;
 import accord.primitives.SaveStatus;
 import accord.primitives.Status;
@@ -100,9 +96,17 @@ import org.apache.cassandra.schema.TableId;
 import org.apache.cassandra.schema.TableMetadata;
 import org.apache.cassandra.schema.TableMetadataRef;
 import org.apache.cassandra.service.accord.AccordDurableOnFlush.ReportDurable;
-import org.apache.cassandra.service.accord.AccordKeyspace.CommandsForKeyAccessor;
 import org.apache.cassandra.service.accord.IAccordService.AccordCompactionInfo;
-import org.apache.cassandra.service.accord.api.TokenKey;
+import org.apache.cassandra.service.accord.execution.AccordCache;
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
+import org.apache.cassandra.service.accord.execution.ExclusiveExecutor;
+import org.apache.cassandra.service.accord.execution.SafeTask;
+import org.apache.cassandra.service.accord.execution.SaferCommand;
+import org.apache.cassandra.service.accord.execution.SaferCommandStore;
+import org.apache.cassandra.service.accord.execution.SaferCommandsForKey;
+import org.apache.cassandra.service.accord.execution.TaskRunner;
+import org.apache.cassandra.service.accord.execution.Unterminatable;
 import org.apache.cassandra.service.accord.journal.AccordJournal;
 import org.apache.cassandra.service.accord.journal.JournalRangeIndex;
 import org.apache.cassandra.service.accord.txn.TxnRead;
@@ -139,10 +143,10 @@ public class AccordCommandStore extends CommandStore
     public static class Caches
     {
         private final AccordCache global;
-        private final AccordCache.Type<TxnId, Command, AccordSafeCommand>.Instance commands;
+        private final AccordCache.Type<TxnId, Command, SaferCommand>.Instance commands;
-        private final AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>.Instance commandsForKeys;
+        private final AccordCache.Type<RoutingKey, CommandsForKey, SaferCommandsForKey>.Instance commandsForKeys;
 
-        Caches(AccordCache global, AccordCache.Type<TxnId, Command, AccordSafeCommand>.Instance commandCache, AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>.Instance commandsForKeyCache)
+        Caches(AccordCache global, AccordCache.Type<TxnId, Command, SaferCommand>.Instance commandCache, AccordCache.Type<RoutingKey, CommandsForKey, SaferCommandsForKey>.Instance commandsForKeyCache)
         {
             this.global = global;
             this.commands = commandCache;
@@ -154,44 +158,47 @@ public class AccordCommandStore extends CommandStore
             return global;
         }
 
-        public final AccordCache.Type<TxnId, Command, AccordSafeCommand>.Instance commands()
+        public final AccordCache.Type<TxnId, Command, SaferCommand>.Instance commands()
         {
             return commands;
         }
 
-        public final AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>.Instance commandsForKeys()
+        public final AccordCache.Type<RoutingKey, CommandsForKey, SaferCommandsForKey>.Instance commandsForKeys()
         {
             return commandsForKeys;
         }
     }
 
-    public static final class ExclusiveCaches extends Caches implements CommandStoreCaches<AccordSafeCommand, AccordSafeCommandsForKey>
+    public static final class ExclusiveCaches extends Caches implements CommandStoreCaches<SaferCommand, SaferCommandsForKey>
     {
-        private final Lock lock;
+        private final AccordExecutor owner;
 
-        public ExclusiveCaches(Lock lock, AccordCache global, AccordCache.Type<TxnId, Command, AccordSafeCommand>.Instance commands, AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>.Instance commandsForKeys)
+        public ExclusiveCaches(AccordExecutor owner, AccordCache global, AccordCache.Type<TxnId, Command, SaferCommand>.Instance commands, AccordCache.Type<RoutingKey, CommandsForKey, SaferCommandsForKey>.Instance commandsForKeys)
         {
             super(global, commands, commandsForKeys);
-            this.lock = lock;
+            this.owner = owner;
         }
 
         @Override
-        public AccordSafeCommand acquireIfLoaded(TxnId txnId)
+        public SaferCommand acquireIfLoaded(TxnId txnId)
         {
-            return commands().acquireIfLoaded(txnId);
+            // note: we must return false if the entry is locked to enforce ordering.
+            // note importantly that this is also coupled to the safety of synchronously releasing ExclusiveExecutor.owner,
+            // rather than waiting until the (potentially asynchronous) cleanup of the task completes
+            return commands().acquireIfLoadedAndPermitted(txnId);
         }
 
         @Override
-        public AccordSafeCommandsForKey acquireIfLoaded(RoutingKey key)
+        public SaferCommandsForKey acquireIfLoaded(RoutingKey key)
         {
-            return commandsForKeys().acquireIfLoaded(key);
+            return commandsForKeys().acquireIfLoadedAndPermitted(key);
         }
 
         @Override
         public void close()
         {
-            global().tryShrinkOrEvict(lock);
+            global().tryShrinkOrEvict(owner.unsafeLock());
-            lock.unlock();
+            owner.unlock(TaskRunner.get());
         }
     }
 
@@ -211,12 +218,10 @@ public class AccordCommandStore extends CommandStore
         = AtomicReferenceFieldUpdater.newUpdater(AccordCommandStore.class, Termination.class, "terminated");
     static final AtomicLong nextSafeRedundantBeforeTicket = new AtomicLong();
 
-    private static final AtomicLong lastSystemTimestampMicros = new AtomicLong();
-
     public final String loggingId;
     public final Journal journal;
     private final AccordExecutor sharedExecutor;
-    final AccordExecutor.SequentialExecutor exclusiveExecutor;
+    private final ExclusiveExecutor exclusiveExecutor;
     private final ExclusiveCaches caches;
     private final RangeIndex rangeIndex;
     private final TableId tableId;
@@ -225,8 +230,8 @@ public class AccordCommandStore extends CommandStore
     volatile SafeRedundantBefore safeRedundantBefore;
     volatile Termination terminated;
 
-    private AccordSafeCommandStore current;
+    private SaferCommandStore current;
-    LogLinearDecayingHistograms.Buffer metricsBuffer;
+    public LogLinearDecayingHistograms.Buffer metricsBuffer;
 
     public AccordCommandStore(int id,
                               NodeCommandStoreService node,
@@ -249,21 +254,23 @@ public class AccordCommandStore extends CommandStore
         maybeLoadRedundantBefore(journal.loadRedundantBefore(id()));
         maybeLoadBootstrapBeganAt(journal.loadBootstrapBeganAt(id()));
         maybeLoadSafeToRead(journal.loadSafeToRead(id()));
-
+        maybeLoadRangesForEpoch(journal.loadRangesForEpoch(id()));
-        tableId = (TableId)rangesForEpoch.all().stream().map(r -> r.start().prefix()).reduce((a, b) -> {
+        RangesForEpoch ranges = this.rangesForEpoch;
+        Invariants.require(ranges != null && !ranges.all().isEmpty(), "CommandStore %d created with no ranges", id);
+        tableId = (TableId)ranges.all().stream().map(r -> r.start().prefix()).reduce((a, b) -> {
             Invariants.require(a.equals(b), "CommandStore created with multiple distinct TableId (%s and %s)", a, b);
             return a;
         }).orElseThrow(() -> Invariants.illegalState("CommandStore %d created with no ranges", id));
 
-        final AccordCache.Type<TxnId, Command, AccordSafeCommand>.Instance commands;
+        final AccordCache.Type<TxnId, Command, SaferCommand>.Instance commands;
-        final AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>.Instance commandsForKey;
+        final AccordCache.Type<RoutingKey, CommandsForKey, SaferCommandsForKey>.Instance commandsForKey;
         try (AccordExecutor.ExclusiveGlobalCaches exclusive = sharedExecutor.lockCaches())
         {
             commands = exclusive.commands.newInstance(this);
             commandsForKey = exclusive.commandsForKey.newInstance(this);
-            this.caches = new ExclusiveCaches(sharedExecutor.unsafeLock(), exclusive.global, commands, commandsForKey);
+            this.caches = new ExclusiveCaches(sharedExecutor, exclusive.global, commands, commandsForKey);
         }
-        this.exclusiveExecutor = sharedExecutor.executor(id);
+        this.exclusiveExecutor = sharedExecutor.newExclusiveExecutor(id);
 
         {
             AccordConfig.RangeIndexMode mode = getAccord().range_index_mode;
@@ -296,12 +303,6 @@ public class AccordCommandStore extends CommandStore
         return exclusiveExecutor.inExecutor();
     }
 
-    void tryPreSetup(AccordTask<?> task)
-    {
-        if (inStore() && current != null)
-            task.presetup(current.task);
-    }
-
     public final TableId tableId()
     {
         return tableId;
@@ -315,7 +316,7 @@ public class AccordCommandStore extends CommandStore
     // TODO (desired): we use this for executing callbacks with mutual exclusivity,
     //  but we don't need to block the actual CommandStore - could quite easily
     //  inflate a separate queue dynamically in AccordExecutor
-    public AsyncExecutor taskExecutor()
+    public ExclusiveExecutor exclusiveExecutor()
     {
         return exclusiveExecutor;
     }
@@ -323,13 +324,13 @@ public class AccordCommandStore extends CommandStore
     public ExclusiveCaches lockCaches()
     {
         //noinspection LockAcquiredButNotSafelyReleased
-        caches.lock.lock();
+        caches.owner.lock(TaskRunner.get());
         return caches;
     }
 
     public ExclusiveCaches tryLockCaches()
     {
-        if (caches.lock.tryLock())
+        if (caches.owner.tryLock(TaskRunner.get()))
             return caches;
         return null;
     }
@@ -357,128 +358,70 @@ public class AccordCommandStore extends CommandStore
         journal.saveCommand(id, new CommandUpdate(before, after), onFlush);
     }
 
-    boolean validateCommand(TxnId txnId, Command evicting)
-    {
-        if (!Invariants.isParanoid())
-            return true;
-
-        Command reloaded = loadCommand(txnId);
-        return Objects.equals(evicting, reloaded);
-    }
-
     @VisibleForTesting
     public void sanityCheckCommand(RedundantBefore redundantBefore, Command command)
     {
         ((AccordJournal) journal).sanityCheck(id, redundantBefore, command);
     }
 
-    CommandsForKey loadCommandsForKey(RoutableKey key)
-    {
-        CommandsForKey cfk = CommandsForKeyAccessor.load(id, (TokenKey) key);
-        if (cfk == null)
-            return null;
-        RedundantBefore.QuickBounds bounds = safeGetRedundantBefore().get(key);
-        if (!Invariants.expect(bounds != null, "No RedundantBefore information found when loading key %s", key))
-            return cfk;
-        return cfk.withCleanCfkBeforeAtLeast(bounds.cleanCfkBefore(), false);
-    }
-
-    boolean validateCommandsForKey(RoutableKey key, CommandsForKey evicting)
-    {
-        if (!Invariants.isParanoid())
-            return true;
-
-        CommandsForKey reloaded = CommandsForKeyAccessor.load(id, (TokenKey) key);
-        return Objects.equals(evicting, reloaded);
-    }
-
-    @Nullable
-    Runnable saveCommandsForKey(RoutingKey key, CommandsForKey after, Object serialized)
-    {
-        return CommandsForKeyAccessor.systemTableUpdater(id, (TokenKey) key, after, serialized, nextSystemTimestampMicros());
-    }
-
-    public long nextSystemTimestampMicros()
-    {
-        return lastSystemTimestampMicros.accumulateAndGet(node.now(), (a, b) -> Math.max(a + 1, b));
-    }
     @Override
-    public <T> AsyncChain<T> chain(PreLoadContext loadCtx, Function<? super SafeCommandStore, T> function)
+    public <T> AsyncChain<T> chain(ExecutionContext context, Function<? super SafeCommandStore, T> function)
     {
-        return AccordTask.create(this, loadCtx, function).chain();
+        return SafeTask.create(this, context, function).chain();
     }
 
     @Override
-    public AsyncChain<Void> chain(PreLoadContext preLoadContext, Consumer<? super SafeCommandStore> consumer)
+    public AsyncChain<Void> chain(ExecutionContext context, Consumer<? super SafeCommandStore> consumer)
     {
-        return AccordTask.create(this, preLoadContext, consumer).chain();
+        return SafeTask.create(this, context, consumer).chain();
-    }
-
-    @Override
-    public AsyncChain<Void> priorityChain(PreLoadContext preLoadContext, Consumer<? super SafeCommandStore> consumer)
-    {
-        return AccordTask.create(this, preLoadContext, consumer).priorityChain();
-    }
-
-    @Override
-    public <T> AsyncChain<T> priorityChain(PreLoadContext preLoadContext, Function<? super SafeCommandStore, T> function)
-    {
-        return AccordTask.create(this, preLoadContext, function).priorityChain();
     }
 
     @Override
     public <T> AsyncChain<T> chain(Callable<T> call)
     {
-        return taskExecutor().chain(call);
+        return exclusiveExecutor().chain(call);
     }
 
     @Override
     public void execute(Runnable run)
     {
-        taskExecutor().execute(run);
+        exclusiveExecutor().execute(run);
     }
 
     @Override
     public boolean tryExecuteImmediately(Runnable run)
     {
-        return taskExecutor().tryExecuteImmediately(run);
+        return exclusiveExecutor().tryExecuteImmediately(run);
     }
 
-    public AccordSafeCommandStore begin(AccordTask<?> operation, @Nullable CommandSummaries commandsForRanges)
+    public SaferCommandStore begin(SaferCommandStore safeStore)
     {
         require(current == null);
-        current = AccordSafeCommandStore.create(operation, commandsForRanges, this);
+        current = safeStore;
         return current;
     }
 
+    public void complete(SaferCommandStore store)
+    {
+        require(current == store);
+        current = null;
+    }
+
     public boolean hasSafeStore()
     {
         return current != null;
     }
 
-    DataStore dataStore()
+    public DataStore dataStore()
     {
         return dataStore;
     }
 
-    ProgressLog progressLog()
+    public ProgressLog progressLog()
     {
         return progressLog;
     }
 
-    public void complete(AccordSafeCommandStore store)
-    {
-        require(current == store);
-        current.postExecute();
-        current = null;
-    }
-
-    public void abort(AccordSafeCommandStore store)
-    {
-        Invariants.require(store == current);
-        current = null;
-    }
-
     @Override
     public void shutdown()
     {
@@ -662,7 +605,7 @@ public class AccordCommandStore extends CommandStore
                 public Ready() { super(1); }
                 @Override public void run() { decrement(); }
 
-                void maybeFlush(ExclusiveCaches caches, AccordCacheEntry<RoutingKey, CommandsForKey> e)
+                void maybeFlush(ExclusiveCaches caches, AccordCacheEntry<RoutingKey, CommandsForKey, ?> e)
                 {
                     if (e.isModified())
                     {
@@ -677,7 +620,7 @@ public class AccordCommandStore extends CommandStore
             {
                 if (ranges == null)
                 {
-                    for (AccordCacheEntry<RoutingKey, CommandsForKey> e : caches.commandsForKeys())
+                    for (AccordCacheEntry<RoutingKey, CommandsForKey, ?> e : caches.commandsForKeys())
                         ready.maybeFlush(caches, e);
                 }
                 else
@@ -729,7 +672,7 @@ public class AccordCommandStore extends CommandStore
             if (!maybeShouldReplay(txnId))
                 return AsyncChains.success(null);
 
-            return commandStore.chain(PreLoadContext.contextFor(txnId, "Replay"), safeStore -> {
+            return commandStore.chain(ExecutionContext.unsequenced(txnId, "Replay"), safeStore -> {
                 Replay replay = shouldReplay(txnId, safeStore.unsafeGet(txnId).current().participants());
                 if (replay == Replay.NONE)
                     return null;
@@ -776,7 +719,7 @@ public class AccordCommandStore extends CommandStore
 
     AsyncChain<Boolean> saveState(Descriptor descriptor)
     {
-        return chain((AccordExecutor.Unterminatable)() -> "Save State", safeStore -> {
+        return chain((Unterminatable)() -> "Save State", safeStore -> {
             File storeDir = storeSaveDir();
             {
                 File[] tmpDirs = listTmpSaveDirs(storeDir);
@@ -883,7 +826,7 @@ public class AccordCommandStore extends CommandStore
                 {
                     File rjbf = new File(savePoint, "reject_before");
                     if (rjbf.exists())
-                        mxc = mxc.with(readOne(rjbf, rejectBefore));
+                        mxc = mxc.update(readOne(rjbf, rejectBefore));
                 }
                 dll = readList(new File(savePoint, "listeners"), txnListener);
                 dpl = readList(new File(savePoint, "progress_log"), progressLogState);
@@ -914,7 +857,7 @@ public class AccordCommandStore extends CommandStore
     // TODO (expected): handle journal failures, and consider how we handle partial failures.
     //  Very likely we will not be able to safely or cleanly handle partial failures of this logic, but decide and document.
     // TODO (desired): consider merging with PersistentField? This version is cheaper to manage which may be preferable at the CommandStore level.
-    static class SafeRedundantBefore
+    public static class SafeRedundantBefore
     {
         final long ticket;
         final RedundantBefore redundantBefore;
@@ -929,6 +872,15 @@ public class AccordCommandStore extends CommandStore
         {
             return a.ticket >= b.ticket ? a : b;
         }
+
+        public static Runnable updater(AccordCommandStore commandStore, RedundantBefore newRedundantBefore)
+        {
+            long ticket = nextSafeRedundantBeforeTicket.incrementAndGet();
+            SafeRedundantBefore update = new SafeRedundantBefore(ticket, newRedundantBefore);
+            return () -> {
+                safeRedundantBeforeUpdater.accumulateAndGet(commandStore, update, SafeRedundantBefore::max);
+            };
+        }
     }
 
     private @Nullable TableMetadata tableMetadata()

src/java/org/apache/cassandra/service/accord/AccordCommandStores.java

@@ -31,13 +31,14 @@ import org.slf4j.LoggerFactory;
 
 import accord.api.Agent;
 import accord.api.DataStore;
+import accord.api.ExclusiveAsyncExecutor;
 import accord.api.Journal;
 import accord.api.LocalListeners;
 import accord.api.ProgressLog;
 import accord.local.CommandStores;
 import accord.local.NodeCommandStoreService;
-import accord.local.SequentialAsyncExecutor;
 import accord.local.ShardDistributor;
+import accord.utils.Invariants;
 import accord.utils.RandomSource;
 import accord.utils.Reduce;
 import accord.utils.async.AsyncChain;
@@ -54,14 +55,20 @@ import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.journal.Descriptor;
 import org.apache.cassandra.schema.TableId;
 import org.apache.cassandra.service.accord.AccordCommandStore.DurablyAppliedTo;
-import org.apache.cassandra.service.accord.AccordExecutor.AccordExecutorFactory;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
-import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.service.accord.execution.AccordExecutor.AccordExecutorFactory;
+import org.apache.cassandra.service.accord.execution.AccordExecutorAsyncSubmit;
+import org.apache.cassandra.service.accord.execution.AccordExecutorSemiSyncSubmit;
+import org.apache.cassandra.service.accord.execution.AccordExecutorSignalLoop;
+import org.apache.cassandra.service.accord.execution.AccordExecutorSimple;
+import org.apache.cassandra.service.accord.execution.AccordExecutorSyncSubmit;
 
 import static org.apache.cassandra.config.AccordConfig.QueueShardModel.THREAD_PER_SHARD;
+import static org.apache.cassandra.config.AccordConfig.QueueShardModel.THREAD_PER_SHARD_SYNC_QUEUE;
 import static org.apache.cassandra.config.DatabaseDescriptor.getAccord;
-import static org.apache.cassandra.service.accord.AccordExecutor.Mode.RUN_WITHOUT_LOCK;
+import static org.apache.cassandra.service.accord.execution.AccordExecutor.Mode.RUN_WITHOUT_LOCK;
-import static org.apache.cassandra.service.accord.AccordExecutor.Mode.RUN_WITH_LOCK;
+import static org.apache.cassandra.service.accord.execution.AccordExecutor.Mode.RUN_WITH_LOCK;
-import static org.apache.cassandra.service.accord.AccordExecutor.constant;
+import static org.apache.cassandra.service.accord.execution.AccordExecutor.constant;
 import static org.apache.cassandra.service.accord.journal.ReplayMarkers.saveDirectory;
 import static org.apache.cassandra.utils.Clock.Global.nanoTime;
 
@@ -73,7 +80,6 @@ public class AccordCommandStores extends CommandStores implements CacheSize, Shu
     private final int mask;
 
     private long cacheSize, workingSetSize;
-    private int maxQueuedLoads, maxQueuedRangeLoads;
     private boolean shrinkingOn;
 
     AccordCommandStores(NodeCommandStoreService node, Agent agent, DataStore store, RandomSource random,
@@ -87,12 +93,9 @@ public class AccordCommandStores extends CommandStores implements CacheSize, Shu
 
         cacheSize = DatabaseDescriptor.getAccordCacheSizeInMiB() << 20;
         workingSetSize = DatabaseDescriptor.getAccordWorkingSetSizeInMiB() << 20;
-        AccordConfig config = DatabaseDescriptor.getAccord();
-        maxQueuedLoads = maxQueuedLoads(config);
-        maxQueuedRangeLoads = maxQueuedRangeLoads(config);
         shrinkingOn = DatabaseDescriptor.getAccordCacheShrinkingOn();
         refreshCapacities();
-        ScheduledExecutors.scheduledFastTasks.scheduleWithFixedDelay(() -> {
+        ScheduledExecutors.scheduledTasks.scheduleWithFixedDelay(() -> {
             for (AccordExecutor executor : executors)
             {
                 executor.executeDirectlyWithLock(() -> {
@@ -122,19 +125,21 @@ public class AccordCommandStores extends CommandStores implements CacheSize, Shu
                     break;
                 case EXEC_ST:
                     factory = AccordExecutorSimple::new;
+                    Invariants.require(executors.length == 1, "Executors that hold their lock during execution cannot support multiple executor shards, as they cannot lock multiple executors at once");
                     maxThreads = 1;
                     break;
             }
 
+            QueueShardModel shardModel = config.queue_shard_model;
+            Invariants.require(shardModel != THREAD_PER_SHARD_SYNC_QUEUE || executors.length == 1, "Executors that hold their lock during execution cannot support multiple executor shards, as they cannot lock multiple executors at once");
             for (int id = 0; id < executors.length; id++)
             {
-                QueueShardModel shardModel = config.queue_shard_model;
                 String baseName = AccordExecutor.class.getSimpleName() + '[' + id;
                 switch (shardModel)
                 {
                     case THREAD_PER_SHARD:
                     case THREAD_PER_SHARD_SYNC_QUEUE:
-                        executors[id] = factory.get(id, shardModel == THREAD_PER_SHARD ? RUN_WITHOUT_LOCK : RUN_WITH_LOCK, 1, constant(baseName + ']'), agent);
+                        executors[id] = factory.get(id, shardModel == THREAD_PER_SHARD_SYNC_QUEUE ? RUN_WITH_LOCK : RUN_WITHOUT_LOCK, 1, constant(baseName + ']'), agent);
                         break;
                     case THREAD_POOL_PER_SHARD:
                         int threads = Math.min(maxThreads, Math.max(DatabaseDescriptor.getAccordConcurrentOps() / executors.length, 1));
@@ -148,10 +153,10 @@ public class AccordCommandStores extends CommandStores implements CacheSize, Shu
     }
 
     @Override
-    public SequentialAsyncExecutor someSequentialExecutor()
+    public ExclusiveAsyncExecutor someExclusiveExecutor()
     {
         int idx = ((int) Thread.currentThread().getId()) & mask;
-        return executors[idx].newSequentialExecutor();
+        return executors[idx].newExclusiveExecutor();
     }
 
     public synchronized void setCapacity(long bytes)
@@ -173,13 +178,6 @@ public class AccordCommandStores extends CommandStores implements CacheSize, Shu
         refreshCapacities();
     }
 
-    public synchronized void setMaxQueuedLoads(int total, int range)
-    {
-        maxQueuedLoads = total;
-        maxQueuedRangeLoads = range;
-        refreshCapacities();
-    }
-
     public synchronized void setShrinking(boolean on)
     {
         shrinkingOn = on;
@@ -213,14 +211,11 @@ public class AccordCommandStores extends CommandStores implements CacheSize, Shu
     {
         long capacityPerExecutor = cacheSize / executors.length;
         long workingSetPerExecutor = workingSetSize < 0 ? Long.MAX_VALUE : workingSetSize / executors.length;
-        int maxLoadsPerExecutor = Math.max(1, (maxQueuedLoads + executors.length - 1) / executors.length);
-        int maxRangeLoadsPerExecutor = Math.max(1, (maxQueuedRangeLoads + executors.length - 1) / executors.length);
         for (AccordExecutor executor : executors)
         {
             executor.executeDirectlyWithLock(() -> {
                 executor.setCapacity(capacityPerExecutor);
                 executor.setWorkingSetSize(workingSetPerExecutor);
-                executor.setMaxQueuedLoads(maxLoadsPerExecutor, maxRangeLoadsPerExecutor);
                 executor.cacheExclusive().setShrinkingOn(shrinkingOn);
             });
         }
@@ -341,21 +336,4 @@ public class AccordCommandStores extends CommandStores implements CacheSize, Shu
                 return Math.max(1, config.queue_shard_count.or(DatabaseDescriptor.getAvailableProcessors() / 8));
         }
     }
-
-    private static int threads(AccordConfig config)
-    {
-        return config.queue_thread_count.or(2 * FBUtilities.getAvailableProcessors());
-    }
-
-    public static int maxQueuedLoads(AccordConfig config)
-    {
-        return config.max_queued_loads.or(FBUtilities.getAvailableProcessors());
-    }
-
-    public static int maxQueuedRangeLoads(AccordConfig config)
-    {
-        return config.max_queued_range_loads.or(maxQueuedLoads(config) / 4);
-    }
-
-
 }

src/java/org/apache/cassandra/service/accord/AccordDataStore.java

@@ -20,24 +20,23 @@ package org.apache.cassandra.service.accord;
 
 import java.util.ArrayList;
 import java.util.List;
-import java.util.Map;
-import java.util.concurrent.TimeUnit;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import accord.api.DataStore;
 import accord.local.CommandStore;
+import accord.local.CommandStores.RestrictedStoreSelector;
+import accord.local.MapReduceConsumeCommandStores;
 import accord.local.Node;
 import accord.local.RedundantBefore;
 import accord.local.SafeCommandStore;
 import accord.primitives.Ranges;
-import accord.primitives.SyncPoint;
+import accord.primitives.Timestamp;
 import accord.primitives.TxnId;
 import accord.utils.Invariants;
-import accord.utils.Reduce;
+import accord.utils.SortedArrays.SortedArrayList;
 import accord.utils.UnhandledEnum;
-import accord.utils.async.AsyncResult;
 import accord.utils.async.AsyncResults;
 
 import org.apache.cassandra.concurrent.ScheduledExecutors;
@@ -56,10 +55,9 @@ import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.service.accord.AccordDurableOnFlush.ReportDurable;
 import org.apache.cassandra.streaming.PreviewKind;
 import org.apache.cassandra.tcm.ClusterMetadata;
+import org.apache.cassandra.utils.progress.ProgressEvent;
+import org.apache.cassandra.utils.progress.ProgressListener;
 
-import static accord.local.durability.DurabilityService.SyncLocal.NoLocal;
-import static accord.local.durability.DurabilityService.SyncRemote.Quorum;
-import static accord.primitives.Txn.Kind.ExclusiveSyncPoint;
 import static accord.utils.Invariants.illegalArgument;
 
 public class AccordDataStore implements DataStore
@@ -94,12 +92,12 @@ public class AccordDataStore implements DataStore
         AccordDurableOnFlush.notifyOnDurable(cfs, commandStore, ReportDurable.of(redundantBefore, flags));
     }
 
-    public FetchResult image(Node node, SafeCommandStore safeStore, Ranges ranges, SyncPoint syncPoint, FetchRanges callback)
+    public FetchResult image(Node node, SafeCommandStore safeStore, Ranges ranges, TxnId atLeast, SortedArrayList<Node.Id> readable, FetchRanges callback)
     {
         AccordFetchCoordinator coordinator;
         try
         {
-            coordinator = new AccordFetchCoordinator(node, ranges, syncPoint, callback, safeStore.commandStore());
+            coordinator = new AccordFetchCoordinator(node, ranges, atLeast, readable, callback, safeStore.commandStore());
         }
         catch (Throwable t)
         {
@@ -110,11 +108,10 @@ public class AccordDataStore implements DataStore
         return coordinator.result();
     }
 
-    public FetchResult sync(Node node, SafeCommandStore safeStore, Map<TxnId, Ranges> rangesById, FetchRanges callback)
+    @Override
+    public FetchResult sync(Node node, SafeCommandStore safeStore, Ranges ranges, TxnId atLeast, SortedArrayList<Node.Id> readable, FetchRanges callback)
     {
         TableId tableId = ((AccordCommandStore)safeStore.commandStore()).tableId();
-        Invariants.require(rangesById.values().stream().flatMap(Ranges::stream).allMatch(r -> tableId.equals(r.prefix())));
-        Ranges ranges = rangesById.values().stream().reduce(Ranges::with).get();
 
         ClusterMetadata cm = ClusterMetadata.current();
         TableMetadata tableMetadata = cm.schema.getTableMetadata(tableId);
@@ -137,27 +134,23 @@ public class AccordDataStore implements DataStore
         // TODO (expected): add some automatic slicing of ranges and retry/back-off logic; but for now,
         //  since this is done at the command store level, and this is already a slice of a node, this should be fine
         SyncResult syncResult = new SyncResult();
+        ProgressListener listener = new ProgressListener()
+        {
+            StartingRangeFetch starting = callback.starting(ranges);
+            { Invariants.require(starting != null); }
 
-        logger.info("Requesting quorum durability before initiating repair");
+            @Override
-        List<AsyncResult<Void>> syncs = new ArrayList<>();
+            public void progress(String tag, ProgressEvent event)
-        for (Map.Entry<TxnId, Ranges> e : rangesById.entrySet())
-            syncs.add(node.durability().sync("Sync Data Store", ExclusiveSyncPoint, e.getKey(), e.getValue(), NoLocal, Quorum, 1L, TimeUnit.HOURS));
-
-        AsyncResults.reduce(syncs, Reduce.toNull()).invoke((success, fail) -> {
-            if (fail != null)
             {
-                logger.error("{} failed to achieve quorum durability before repair for rebootstrap of {}", safeStore.commandStore(), ranges);
-                syncResult.tryFailure(fail);
-                return;
-            }
-
-            RepairCoordinator coord = StorageService.instance.newRepairCoordinator(tableMetadata.keyspace, options(tableMetadata, ranges));
-            coord.addProgressListener((tag, event) -> {
                 switch (event.getType())
                 {
                     default: throw new UnhandledEnum(event.getType());
+                    case SUCCESS:
+                        callback.fetched(ranges);
+                        syncResult.trySuccess(null);
+                        // fall-through to ensure started
                     case START:
-                        callback.starting(ranges);
+                        reportStarted();
                         break;
                     case PROGRESS:
                     case COMPLETE:
@@ -165,21 +158,40 @@ public class AccordDataStore implements DataStore
                         break;
                     case ABORT:
                     case ERROR:
-                        IllegalStateException ex = new IllegalStateException(String.format("Repair failed: %s", event));
+                        RuntimeException ex = new RuntimeException(String.format("Repair failed (%s): %s", event.getType(), event.getMessage()));
                         callback.fail(ranges, ex);
                         syncResult.tryFailure(ex);
                         break;
-                    case SUCCESS:
-                        callback.fetched(ranges);
-                        syncResult.trySuccess(null);
-                        break;
                 }
-            });
+            }
 
-            ScheduledExecutors.optionalTasks.submit(coord).addCallback((s, f) -> {
+            private void reportStarted()
-                if (f != null)
+            {
-                    syncResult.tryFailure(f);
+                StartingRangeFetch start = this.starting;
-            });
+                if (start == null)
+                    return;
+                this.starting = null;
+                node.commandStores().mapReduceConsume(new RestrictedStoreSelector(ranges, 0, Long.MAX_VALUE), new MapReduceConsumeCommandStores<Ranges, Timestamp>(ranges)
+                {
+                    @Override public Timestamp reduce(Timestamp o1, Timestamp o2) { return Timestamp.max(o1, o2); }
+                    @Override public void accept(Timestamp result, Throwable failure)
+                    {
+                        if (failure != null) syncResult.tryFailure(failure);
+                        else start.started(result);
+                    }
+                    @Override public TxnId primaryTxnId() { return null; }
+                    @Override public String reason() { return "Compute MaxConflict to report for fetch"; }
+                    @Override protected Timestamp applyInternal(SafeCommandStore safeStore) { return safeStore.commandStore().maxConflict(TxnId.NONE, ranges); }
+                });
+            }
+        };
+
+        RepairCoordinator coord = StorageService.instance.newRepairCoordinator(tableMetadata.keyspace, options(tableMetadata, ranges));
+        coord.addProgressListener(listener);
+
+        ScheduledExecutors.optionalTasks.submit(coord).addCallback((s, f) -> {
+            if (f != null)
+                syncResult.tryFailure(f);
         });
 
         return syncResult;

src/java/org/apache/cassandra/service/accord/AccordDurableOnFlush.java

@@ -34,8 +34,9 @@ import org.apache.cassandra.db.lifecycle.View;
 import org.apache.cassandra.db.memtable.Memtable;
 import org.apache.cassandra.schema.Schema;
 import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.service.accord.execution.Unstoppable;
 
-class AccordDurableOnFlush implements BiConsumer<Long, TableMetadata>
+public class AccordDurableOnFlush implements BiConsumer<Long, TableMetadata>
 {
     private static final Logger logger = LoggerFactory.getLogger(AccordDurableOnFlush.class);
 
@@ -98,6 +99,11 @@ class AccordDurableOnFlush implements BiConsumer<Long, TableMetadata>
         {
             return redundantBefore.toString();
         }
+
+        public static void reportMaybeTerminate(AccordCommandStore commandStore, int flags)
+        {
+            commandStore.maybeTerminated(ReportDurable.isCommandStoreFlush(flags), ReportDurable.isDataStoreFlush(flags));
+        }
     }
 
     private Int2ObjectHashMap<ReportDurable> commandStores = new Int2ObjectHashMap<>();
@@ -185,7 +191,7 @@ class AccordDurableOnFlush implements BiConsumer<Long, TableMetadata>
     static void notifyNow(CommandStore commandStore, ReportDurable report)
     {
         logger.debug("{} reporting flush with {}", commandStore, report);
-        commandStore.execute((AccordExecutor.Unstoppable) () -> "Report Durable", safeStore -> {
+        commandStore.execute((Unstoppable) () -> "Report Durable", safeStore -> {
             safeStore.reportDurable(report.redundantBefore, report.flags);
         }, commandStore.agent());
     }

src/java/org/apache/cassandra/service/accord/AccordExecutorSignalLoop.java

@@ -1,472 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.cassandra.service.accord;
-
-import java.util.concurrent.ConcurrentLinkedQueue;
-import java.util.concurrent.TimeUnit;
-import java.util.function.IntFunction;
-
-import javax.annotation.Nullable;
-
-import accord.api.Agent;
-import accord.utils.Invariants;
-import accord.utils.QuadFunction;
-import accord.utils.QuintConsumer;
-
-import org.apache.cassandra.service.accord.debug.DebugExecution.DebugTask;
-import org.apache.cassandra.utils.concurrent.SignalLock;
-
-import static accord.utils.Invariants.nonNull;
-import static org.apache.cassandra.service.accord.debug.DebugExecution.DEBUG_EXECUTION;
-
-public class AccordExecutorSignalLoop extends AccordExecutorAbstractLoop
-{
-    private static class ShutdownException extends RuntimeException {}
-
-    private final SignalLock lock;
-    private final AccordExecutorLoops loops;
-    private int readyToRunTarget = 1;
-    private final int readyToRunLimit;
-    // TODO (desired): intrusive queue using Task.next, but a little challenging because we reuse SequentialQueueTask so have ABA problem
-    private final ConcurrentLinkedQueue<Task> readyToRun = new ConcurrentLinkedQueue<>();
-
-    private Task pendingSequentialHead, pendingSequentialTail;
-    private Task pendingCleanupHead, pendingCleanupTail;
-    private Task pendingNewHead, pendingNewTail;
-    private int pendingCount;
-
-    private boolean shutdown;
-
-    public AccordExecutorSignalLoop(int executorId, Mode mode, int threads, long spinInterval, long stopCheckInterval, TimeUnit units, IntFunction<String> name, Agent agent)
-    {
-        this(new SignalLock(threads, spinInterval, stopCheckInterval, units), executorId, mode, threads, name, agent);
-    }
-
-    public AccordExecutorSignalLoop(SignalLock lock, int executorId, Mode mode, int threads, IntFunction<String> name, Agent agent)
-    {
-        super(lock, executorId, agent);
-        Invariants.require(threads < SignalLock.MAX_THREADS);
-        this.lock = lock;
-        this.loops = new AccordExecutorLoops(mode, threads, name, this::task);
-        this.readyToRunLimit = Math.min(threads * 4, SignalLock.MAX_SIGNAL_COUNT);
-    }
-
-    <P1s, P1a, P2, P3, P4> void submit(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Task> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4)
-    {
-        Task next = async.apply(p1a, p2, p3, p4);
-        Task prev = push(next);
-        if (prev == null)
-            lock.signalLockWork();
-    }
-
-    @Override
-    final void beforeUnlockExternal()
-    {
-    }
-
-    private Task pollReadyToRun()
-    {
-        return readyToRun.poll();
-    }
-
-    private void addReadyToRun(Task task)
-    {
-        readyToRun.add(task);
-    }
-
-    private boolean hasReadyToRun()
-    {
-        return !readyToRun.isEmpty();
-    }
-
-    private LoopTask task(int index, String id, Mode mode)
-    {
-        return new LoopTask(index, id);
-    }
-
-    class LoopTask extends AccordExecutorLoops.LoopTask
-    {
-        final int index;
-
-        LoopTask(int index, String id)
-        {
-            super(id);
-            this.index = index;
-        }
-
-        private Task awaitWork()
-        {
-            while (true)
-            {
-                if (lock.awaitAsyncOrLock(index))
-                {
-                    try
-                    {
-                        if (DEBUG_EXECUTION) debug.onEnterLock();
-                        fetchWorkExclusive();
-                    }
-                    catch (Throwable t)
-                    {
-                        unlock();
-                        throw t;
-                    }
-
-                    if (!unlockAndAcquire())
-                        continue;
-                }
-
-                if (shutdown)
-                    throw new ShutdownException();
-
-                return nonNull(pollReadyToRun());
-            }
-        }
-
-        private Task cleanupAndMaybeGetWork(@Nullable Task cleanup)
-        {
-            if (cleanup == null)
-                return null;
-
-            if (lock.tryAcquireAsyncWork())
-            {
-                if (shutdown)
-                    throw new ShutdownException();
-
-                return pushCleanupAndReturn(cleanup, nonNull(pollReadyToRun()));
-            }
-
-            if (!tryLock())
-                return pushCleanupAndReturn(cleanup, null);
-
-            try
-            {
-                completeTaskExclusive(cleanup);
-                clearRunning();
-                fetchWorkExclusive();
-            }
-            catch (Throwable t)
-            {
-                unlock();
-                throw t;
-            }
-
-            if (unlockAndAcquire())
-            {
-                if (shutdown)
-                    throw new ShutdownException();
-
-                return nonNull(pollReadyToRun());
-            }
-            return null;
-        }
-
-        private Task pushCleanupAndReturn(Task cleanup, Task result)
-        {
-            clearRunning();
-            cleanup.setReadyToCleanup();
-            if (push(cleanup) == null)
-                lock.signalLockWork();
-            return result;
-        }
-
-        final boolean tryLock()
-        {
-            return onTryLock(lock.tryLock(index));
-        }
-
-        final boolean unlockAndAcquire()
-        {
-            if (Invariants.isParanoid()) paranoidUnlockExclusive();
-            if (DEBUG_EXECUTION) debug.onExitLock();
-            return lock.unlockAndAcquireAsyncWork();
-        }
-
-        private boolean enqueueOnePending()
-        {
-            if (pendingCount == 0)
-                return false;
-
-            if (pendingSequentialHead != null)
-            {
-                pendingSequentialHead = enqueueOneCleanup(pendingSequentialHead);
-                if (pendingSequentialHead == null)
-                    pendingSequentialTail = null;
-            }
-            else if (pendingCleanupHead != null)
-            {
-                pendingCleanupHead = enqueueOneCleanup(pendingCleanupHead);
-                if (pendingCleanupHead == null)
-                    pendingCleanupTail = null;
-            }
-            else
-            {
-                pendingNewHead = enqueueOneSubmit(pendingNewHead);
-                if (pendingNewHead == null)
-                    pendingNewTail = null;
-            }
-            --pendingCount;
-            return true;
-        }
-
-        private void fetchWorkExclusive()
-        {
-            boolean hasReadyToRun = hasReadyToRun();
-            boolean hadWaitingToRun = hasAlreadyWaitingToRun();
-            {
-                int prevPendingUnqueued = pendingCount;
-                updateAndEnqueuePendingUntilHasWaitingToRun(prevPendingUnqueued / 2);
-            }
-
-            if (hadWaitingToRun && hasReadyToRun)
-            {
-                lock.addAndGetEnabledThreadCount(1);
-            }
-            else if (hadWaitingToRun)
-            {
-                readyToRunTarget = Math.min(readyToRunLimit, readyToRunTarget + (1+readyToRunTarget)/2);
-            }
-            else if (hasReadyToRun && readyToRunTarget > 1)
-            {
-                --readyToRunTarget;
-            }
-
-            boolean hasDrainedSignal = false;
-            while (true)
-            {
-                long state = lock.state();
-                int signals = SignalLock.asyncSignalCount(state);
-                int waiters = SignalLock.waitingEnabledThreadCount(state);
-                if (signals >= readyToRunTarget)
-                {
-                    if (enqueueOnePending() || (updatePendingUnqueued() && enqueueOnePending())) continue;
-                    else if (hasDrainedSignal)
-                        lock.signalLockWorkExclusive();
-                    return;
-                }
-                else if (waiters > 0 && signals > 1 && SignalLock.activeEnabledThreadCount(state) == 1)
-                {
-                    // ensure at least one other thread is running if there's enough work for it; it will spin up other threads if necessary
-                    lock.propagateAsyncWorkSignals(1);
-                }
-
-                Task task = pollAlreadyWaitingToRunExclusive();
-                if (task == null)
-                {
-                    if (updateAndEnqueuePendingUntilHasWaitingToRun(0))
-                        continue;
-
-                    lock.clearLockWork();
-                    hasDrainedSignal = true;
-                    if (!updateAndEnqueuePendingUntilHasWaitingToRun(0))
-                    {
-                        if (tasks == 0)
-                            notifyQuiescentExclusive();
-                        return;
-                    }
-                }
-                else
-                {
-                    try { task.preRunExclusive(); }
-                    catch (Throwable t)
-                    {
-                        try { task.fail(t); }
-                        catch (Throwable t2) { try { t.addSuppressed(t2); } catch (Throwable t3) {} }
-                        try { completeTaskExclusive(task); }
-                        catch (Throwable t2) { try { t.addSuppressed(t2); } catch (Throwable t3) {} }
-                        continue;
-                    }
-                    if (DEBUG_EXECUTION) DebugTask.get(task).onPreRun();
-
-                    addReadyToRun(task);
-                    boolean incremented = lock.incrementAsyncWork(false);
-                    Invariants.require(incremented);
-                }
-            }
-        }
-
-        private boolean updatePendingUnqueued()
-        {
-            if (!hasUnqueued())
-                return false;
-
-            int count = 0;
-            Task addSequentialHead = null, addSequentialTail = null;
-            Task addCleanupHead = null, addCleanupTail = null;
-            Task addNewHead = null, addNewTail = null;
-            {
-                Task cur = Task.reverse(acquireUnqueuedExclusive());
-                while (cur != null)
-                {
-                    Task next = cur.next;
-                    if (!cur.isReadyToCleanup())
-                    {
-                        if (addNewHead == null) addNewHead = addNewTail = setNextNull(cur);
-                        else addNewHead = reverseOne(addNewHead, cur);
-                    }
-                    else if (cur instanceof SequentialQueueTask)
-                    {
-                        if (addSequentialHead == null) addSequentialHead = addSequentialTail = setNextNull(cur);
-                        else addSequentialHead = reverseOne(addSequentialHead, cur);
-                    }
-                    else
-                    {
-                        if (addCleanupHead == null) addCleanupHead = addCleanupTail = setNextNull(cur);
-                        else addCleanupHead = reverseOne(addCleanupHead, cur);
-                    }
-                    ++count;
-                    cur = next;
-                }
-            }
-
-            pendingCount += count;
-            if (addSequentialHead != null)
-            {
-                if (pendingSequentialHead == null) pendingSequentialHead = addSequentialHead;
-                else pendingSequentialTail.next = addSequentialHead;
-                pendingSequentialTail = addSequentialTail;
-            }
-            if (addCleanupHead != null)
-            {
-                if (pendingCleanupHead == null) pendingCleanupHead = addCleanupHead;
-                else pendingCleanupTail.next = addCleanupHead;
-                pendingCleanupTail = addCleanupTail;
-            }
-            if (addNewHead != null)
-            {
-                if (pendingNewHead == null) pendingNewHead = addNewHead;
-                else pendingNewTail.next = addNewHead;
-                pendingNewTail = addNewTail;
-            }
-            return true;
-        }
-
-        private Task reverseOne(Task prev, Task cur)
-        {
-            cur.next = prev;
-            return cur;
-        }
-
-        private Task setNextNull(Task cur)
-        {
-            cur.next = null;
-            return cur;
-        }
-
-        private boolean updateAndEnqueuePendingUntilHasWaitingToRun(int processAtLeast)
-        {
-            updatePendingUnqueued();
-            int count = 0;
-            while (enqueueOnePending())
-            {
-                if (++count >= processAtLeast && hasAlreadyWaitingToRun())
-                    return true;
-            }
-            return hasAlreadyWaitingToRun();
-        }
-
-        @Override
-        public void run()
-        {
-            lock.register(index, Thread.currentThread());
-            Task task = null;
-            while (true)
-            {
-                try
-                {
-                    try { task = cleanupAndMaybeGetWork(task); }
-                    catch (Throwable t) { task = null; throw t; }
-                    if (task == null)
-                        task = awaitWork();
-
-                    try
-                    {
-                        setRunning(task);
-                        task.runInternal();
-                    }
-                    catch (Throwable t)
-                    {
-                        try { task.fail(t); }
-                        catch (Throwable t2)
-                        {
-                            try
-                            {
-                                t2.addSuppressed(t);
-                                agent.onException(t2);
-                            }
-                            catch (Throwable t3) { /* empty to ensure we definitely loop so we cleanup the task */ }
-                        }
-                    }
-                }
-                catch (ShutdownException ignore)
-                {
-                    break;
-                }
-                catch (Throwable t)
-                {
-                    agent.onException(t);
-                }
-            }
-        }
-    }
-
-    @Override
-    public void shutdown()
-    {
-        lock.lock();
-        try
-        {
-            shutdown = true;
-            lock.signalAllRegistered();
-        }
-        finally
-        {
-            lock.unlock();
-        }
-    }
-
-    @Override
-    AccordExecutorLoops loops()
-    {
-        return loops;
-    }
-
-    @Override
-    boolean isInLoop()
-    {
-        return loops.isInLoop();
-    }
-
-    @Override
-    boolean isOwningThread()
-    {
-        return lock.isOwner();
-    }
-
-    @Override
-    public boolean isTerminated()
-    {
-        return loops.isTerminated();
-    }
-
-    @Override
-    public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException
-    {
-        return loops.awaitTermination(timeout, unit);
-    }
-}

src/java/org/apache/cassandra/service/accord/AccordFetchCoordinator.java

@@ -38,7 +38,6 @@ import accord.impl.AbstractFetchCoordinator;
 import accord.local.CommandStore;
 import accord.local.Node;
 import accord.local.SafeCommandStore;
-import accord.primitives.PartialDeps;
 import accord.primitives.PartialTxn;
 import accord.primitives.Participants;
 import accord.primitives.Range;
@@ -46,12 +45,12 @@ import accord.primitives.Ranges;
 import accord.primitives.Routable;
 import accord.primitives.Seekable;
 import accord.primitives.Seekables;
-import accord.primitives.SyncPoint;
 import accord.primitives.Timestamp;
 import accord.primitives.Txn;
 import accord.primitives.TxnId;
 import accord.topology.TopologyException;
 import accord.utils.Invariants;
+import accord.utils.SortedArrays.SortedArrayList;
 import accord.utils.async.AsyncChain;
 import accord.utils.async.AsyncChains;
 
@@ -245,7 +244,7 @@ public class AccordFetchCoordinator extends AbstractFetchCoordinator implements
                     logger.info("Reporting failure of plan {} for bootstrap of {} from {}", planId, range, from, fail);
                     fail(from, Ranges.of(range), fail);
                 }
-            }, ((AccordCommandStore) commandStore()).taskExecutor());
+            }, ((AccordCommandStore) commandStore()).exclusiveExecutor());
         }
     }
 
@@ -395,9 +394,9 @@ public class AccordFetchCoordinator extends AbstractFetchCoordinator implements
 
     private final Map<TimeUUID, IncomingStream> streams = new HashMap<>();
 
-    public AccordFetchCoordinator(Node node, Ranges ranges, SyncPoint syncPoint, DataStore.FetchRanges fetchRanges, CommandStore commandStore) throws TopologyException
+    public AccordFetchCoordinator(Node node, Ranges ranges, TxnId atLeast, SortedArrayList<Node.Id> readable, DataStore.FetchRanges fetchRanges, CommandStore commandStore) throws TopologyException
     {
-        super(node, node.someSequentialExecutor(), ranges, syncPoint, fetchRanges, commandStore);
+        super(node, node.someExclusiveExecutor(), ranges, atLeast, readable, fetchRanges, commandStore);
     }
 
     @Override
@@ -431,13 +430,6 @@ public class AccordFetchCoordinator extends AbstractFetchCoordinator implements
         super.onDone(success, failure);
     }
 
-    @Override
-    protected PartialTxn rangeReadTxn(Ranges ranges)
-    {
-        StreamingRead read = new StreamingRead(FBUtilities.getBroadcastAddressAndPort(), ranges);
-        return new PartialTxn.InMemory(Txn.Kind.Read, ranges, read, noopQuery, null, TableMetadatasAndKeys.none(Routable.Domain.Range));
-    }
-
     @Override
     protected synchronized void onReadOk(Node.Id from, CommandStore commandStore, Data data, Ranges received)
     {
@@ -463,24 +455,22 @@ public class AccordFetchCoordinator extends AbstractFetchCoordinator implements
 
     public static class AccordFetchRequest extends FetchRequest
     {
-        public AccordFetchRequest(long sourceEpoch, TxnId syncId, Ranges ranges, PartialDeps partialDeps, PartialTxn partialTxn)
+        public AccordFetchRequest(long sourceEpoch, TxnId syncId, Ranges ranges, PartialTxn partialTxn)
         {
-            super(sourceEpoch, syncId, ranges, partialDeps, partialTxn);
+            super(sourceEpoch, syncId, ranges, partialTxn);
-        }
-
-        @Override
-        protected AsyncChain<Data> beginRead(SafeCommandStore safeStore, Timestamp executeAt, PartialTxn txn, Participants<?> execute)
-        {
-            AsyncChain<Data> result = super.beginRead(safeStore, executeAt, txn, execute);
-            // TODO (required): verify that streaming snapshots have all been created by now, so we won't stream any data that arrives after this
-            readStarted(safeStore);
-            return result;
         }
     }
 
     @Override
-    protected FetchRequest newFetchRequest(long sourceEpoch, TxnId syncId, Ranges ranges, PartialDeps partialDeps, PartialTxn partialTxn)
+    protected FetchRequest newFetchRequest(long sourceEpoch, TxnId syncId, Ranges ranges)
     {
-        return new AccordFetchRequest(sourceEpoch, syncId, ranges, partialDeps, partialTxn);
+        return new AccordFetchRequest(sourceEpoch, syncId, ranges, rangeReadTxn(ranges));
     }
+
+    private PartialTxn rangeReadTxn(Ranges ranges)
+    {
+        StreamingRead read = new StreamingRead(FBUtilities.getBroadcastAddressAndPort(), ranges);
+        return new PartialTxn.InMemory(Txn.Kind.Read, ranges, read, noopQuery, null, TableMetadatasAndKeys.none(Routable.Domain.Range));
+    }
+
 }

src/java/org/apache/cassandra/service/accord/AccordKeyspace.java

@@ -26,6 +26,7 @@ import java.util.Iterator;
 import java.util.List;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
 import java.util.function.Consumer;
 
 import com.google.common.annotations.VisibleForTesting;
@@ -105,6 +106,7 @@ import org.apache.cassandra.schema.Tables;
 import org.apache.cassandra.schema.Types;
 import org.apache.cassandra.schema.UserFunctions;
 import org.apache.cassandra.schema.Views;
+import org.apache.cassandra.service.accord.api.AccordAgent;
 import org.apache.cassandra.service.accord.api.TokenKey;
 import org.apache.cassandra.service.accord.serializers.CommandSerializers;
 import org.apache.cassandra.utils.AbstractIterator;
@@ -122,6 +124,7 @@ import static org.apache.cassandra.config.AccordConfig.RangeIndexMode.journal_sa
 import static org.apache.cassandra.db.partitions.PartitionUpdate.singleRowUpdate;
 import static org.apache.cassandra.db.rows.BTreeRow.singleCellRow;
 import static org.apache.cassandra.schema.SchemaConstants.ACCORD_KEYSPACE_NAME;
+import static org.apache.cassandra.utils.Clock.Global.currentTimeMillis;
 
 public class AccordKeyspace
 {
@@ -339,7 +342,7 @@ public class AccordKeyspace
         {
             ByteBuffer bytes;
             if (serialized instanceof ByteBuffer) bytes = (ByteBuffer) serialized;
-            else bytes = Serialize.toBytesWithoutKey(commandsForKey.maximalPrune()); // TODO (expected): we only need to strip pruned, not prune additional txns
+            else bytes = Serialize.toBytesWithoutKey(commandsForKey.maybePrune(0, AccordAgent.cfkHlcPruneDelta(DatabaseDescriptor.getAccord()))); // TODO (expected): we only need to strip pruned, not prune additional txns
             return makeUpdate(storeId, key, timestampMicros, bytes);
         }
 
@@ -353,9 +356,15 @@ public class AccordKeyspace
                                                  BufferCell.live(CFKAccessor.data, timestampMicros, bytes)));
         }
 
-        public static Runnable systemTableUpdater(int storeId, TokenKey key, CommandsForKey update, Object serialized, long timestampMicros)
+        private static final AtomicLong lastSystemTimestampMicros = new AtomicLong();
+        private static long nextSystemTimestampMicros()
         {
-            PartitionUpdate upd = makeUpdate(storeId, key, update, serialized, timestampMicros);
+            return lastSystemTimestampMicros.accumulateAndGet(currentTimeMillis(), (a, b) -> Math.max(a + 1, b));
+        }
+
+        public static Runnable systemTableUpdater(int storeId, TokenKey key, CommandsForKey update, Object serialized)
+        {
+            PartitionUpdate upd = makeUpdate(storeId, key, update, serialized, nextSystemTimestampMicros());
             return () -> {
                 ColumnFamilyStore cfs = AccordColumnFamilyStores.commandsForKey;
                 try (OpOrder.Group group = Keyspace.writeOrder.start())

src/java/org/apache/cassandra/service/accord/AccordMessageSink.java

@@ -254,6 +254,14 @@ public class AccordMessageSink implements MessageSink
                     slowAtNanos = nowNanos + slow.computeWait(attempt, NANOSECONDS);
                 break;
             }
+
+            case ACCORD_WAIT_UNTIL_APPLIED_REQ:
+            {
+                TimeoutStrategy slow = slowPreaccept(txnId);
+                if (slow != null)
+                    slowAtNanos = nowNanos + slow.computeWait(attempt, NANOSECONDS);
+                break;
+            }
         }
 
         Message<Request> message = Message.out(verb, request, expiresAtNanos);

src/java/org/apache/cassandra/service/accord/AccordResult.java

@@ -201,7 +201,9 @@ public class AccordResult<V> extends AsyncFuture<V> implements BiConsumer<V, Thr
         }
         else
         {
-            logger.error("Unexpected exception", fail);
+            AccordAgent.handleException(fail);
+            if (!AccordAgent.expectedException(fail))
+                logger.error("Unexpected exception", fail);
             JVMStabilityInspector.inspectThrowable(fail);
             report = bookkeeping.newFailed(txnId, keysOrRanges);
         }

src/java/org/apache/cassandra/service/accord/AccordSafeCommand.java

@@ -1,151 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.cassandra.service.accord;
-
-import java.util.Objects;
-
-import com.google.common.annotations.VisibleForTesting;
-
-import accord.api.Journal;
-import accord.local.Command;
-import accord.local.SafeCommand;
-import accord.primitives.TxnId;
-
-import org.apache.cassandra.utils.concurrent.Ref;
-
-public class AccordSafeCommand extends SafeCommand implements AccordSafeState<TxnId, Command>
-{
-    public static class DebugAccordSafeCommand extends AccordSafeCommand
-    {
-        final Ref<?> selfRef;
-        public DebugAccordSafeCommand(AccordCacheEntry<TxnId, Command> global)
-        {
-            super(global);
-            selfRef = new Ref<>(this, null);
-            selfRef.debug(global.key().toString());
-        }
-
-        @Override
-        public void markUnsafe()
-        {
-            super.markUnsafe();
-            selfRef.release();
-        }
-
-        public static void trace(AccordSafeCommand safeCommand, String message)
-        {
-            ((DebugAccordSafeCommand)safeCommand).selfRef.debug(message);
-        }
-    }
-
-    private boolean unsafe;
-    private final AccordCacheEntry<TxnId, Command> global;
-    private Command original;
-    private Command current;
-
-    public AccordSafeCommand(AccordCacheEntry<TxnId, Command> global)
-    {
-        super(global.key());
-        this.global = global;
-        this.original = null;
-        this.current = null;
-    }
-
-    @Override
-    public boolean equals(Object o)
-    {
-        if (this == o) return true;
-        if (o == null || getClass() != o.getClass()) return false;
-        AccordSafeCommand that = (AccordSafeCommand) o;
-        return Objects.equals(original, that.original) && Objects.equals(current, that.current);
-    }
-
-    @Override
-    public int hashCode()
-    {
-        throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public String toString()
-    {
-        return "AccordSafeCommand{" +
-               "invalidated=" + unsafe +
-               ", global=" + global +
-               ", original=" + original +
-               ", current=" + current +
-               '}';
-    }
-
-    @Override
-    public AccordCacheEntry<TxnId, Command> global()
-    {
-        checkNotInvalidated();
-        return global;
-    }
-
-    @Override
-    public Command current()
-    {
-        checkNotInvalidated();
-        return current;
-    }
-
-    @Override
-    @VisibleForTesting
-    public void set(Command command)
-    {
-        checkNotInvalidated();
-        this.current = command;
-    }
-
-    @Override
-    public Command original()
-    {
-        checkNotInvalidated();
-        return original;
-    }
-
-    public Journal.CommandUpdate update()
-    {
-        return new Journal.CommandUpdate(original, current);
-    }
-
-    @Override
-    public void preExecute()
-    {
-        checkNotInvalidated();
-        original = global.getExclusive();
-        current = original;
-        if (isUnset())
-            uninitialised();
-    }
-
-    @Override
-    public void markUnsafe()
-    {
-        unsafe = true;
-    }
-
-    @Override
-    public boolean isUnsafe()
-    {
-        return unsafe;
-    }
-}

src/java/org/apache/cassandra/service/accord/AccordSafeCommandsForKey.java

@@ -1,158 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.cassandra.service.accord;
-
-import java.util.Objects;
-
-import com.google.common.annotations.VisibleForTesting;
-
-import accord.api.RoutingKey;
-import accord.local.cfk.CommandsForKey;
-import accord.local.cfk.NotifySink;
-import accord.local.cfk.SafeCommandsForKey;
-
-public class AccordSafeCommandsForKey extends SafeCommandsForKey implements AccordSafeState<RoutingKey, CommandsForKey>
-{
-    public static class CommandsForKeyCacheEntry extends AccordCacheEntry<RoutingKey, CommandsForKey>
-    {
-        private NotifySink overrideSink;
-
-        CommandsForKeyCacheEntry(RoutingKey key, AccordCache.Type<RoutingKey, CommandsForKey, ?>.Instance owner)
-        {
-            super(key, owner);
-        }
-    }
-
-    private boolean invalidated;
-    private final AccordCacheEntry<RoutingKey, CommandsForKey> global;
-    private CommandsForKey original;
-    private CommandsForKey current;
-
-    public AccordSafeCommandsForKey(AccordCacheEntry<RoutingKey, CommandsForKey> global)
-    {
-        super(global.key());
-        this.global = global;
-        this.original = null;
-        this.current = null;
-//        if (overrideSink() == null)
-//            overrideSink(new RecordingNotifySink());
-    }
-
-    @Override
-    public boolean equals(Object o)
-    {
-        if (this == o) return true;
-        if (o == null || getClass() != o.getClass()) return false;
-        AccordSafeCommandsForKey that = (AccordSafeCommandsForKey) o;
-        return Objects.equals(original, that.original) && Objects.equals(current, that.current);
-    }
-
-    @Override
-    public int hashCode()
-    {
-        throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public String toString()
-    {
-        return "AccordSafeCommandsForKey{" +
-               "invalidated=" + invalidated +
-               ", global=" + global +
-               ", original=" + original +
-               ", current=" + current +
-               '}';
-    }
-
-    @Override
-    public boolean hasUpdate()
-    {
-        boolean hasUpdate = AccordSafeState.super.hasUpdate();
-
-        // cfk initialization is legal, but doesn't need to be propagated to the cache (and would
-        // cause an exception to be thrown if it were). Making an exception on the cache side could
-        // throw away applied cfk updates as well, so it's special cased here
-        if (hasUpdate && original == null && current != null && current.size() == 0)
-            return false;
-
-        return hasUpdate;
-    }
-
-    @Override
-    public AccordCacheEntry<RoutingKey, CommandsForKey> global()
-    {
-        checkNotInvalidated();
-        return global;
-    }
-
-    @Override
-    public CommandsForKey current()
-    {
-        checkNotInvalidated();
-        return current;
-    }
-
-    @Override
-    @VisibleForTesting
-    public void set(CommandsForKey cfk)
-    {
-        checkNotInvalidated();
-        this.current = cfk;
-    }
-
-    @Override
-    public void overrideSink(NotifySink overrideSink)
-    {
-        ((CommandsForKeyCacheEntry)global).overrideSink = overrideSink;
-    }
-
-    @Override
-    public NotifySink overrideSink()
-    {
-        return ((CommandsForKeyCacheEntry)global).overrideSink;
-    }
-
-    public CommandsForKey original()
-    {
-        checkNotInvalidated();
-        return original;
-    }
-
-    @Override
-    public void preExecute()
-    {
-        checkNotInvalidated();
-        original = global.getExclusive();
-        current = original;
-        if (isUnset())
-            initialize();
-    }
-
-    @Override
-    public void markUnsafe()
-    {
-        invalidated = true;
-    }
-
-    @Override
-    public boolean isUnsafe()
-    {
-        return invalidated;
-    }
-}

src/java/org/apache/cassandra/service/accord/AccordService.java

@@ -56,12 +56,16 @@ import accord.impl.RequestCallbacks;
 import accord.impl.SizeOfIntersectionSorter;
 import accord.impl.progresslog.DefaultProgressLog;
 import accord.impl.progresslog.DefaultProgressLogs;
+import accord.local.BootstrapReason;
 import accord.local.Catchup;
-import accord.local.CatchupHard;
+import accord.local.Catchup.Unsuccessful;
 import accord.local.CommandStores;
 import accord.local.Node;
 import accord.local.Node.Id;
 import accord.local.ShardDistributor.EvenSplit;
+import accord.local.TimeService;
+import accord.local.UniqueTimeService;
+import accord.local.UniqueTimeService.AtomicUniqueTime;
 import accord.local.UniqueTimeService.AtomicUniqueTimeWithStaleReservation;
 import accord.local.durability.DurabilityService;
 import accord.local.durability.ShardDurability;
@@ -92,8 +96,9 @@ import accord.utils.async.AsyncResults;
 
 import org.apache.cassandra.concurrent.Shutdownable;
 import org.apache.cassandra.config.AccordConfig;
-import org.apache.cassandra.config.AccordConfig.CatchupMode;
+import org.apache.cassandra.config.AccordConfig.CatchupFallbackMode;
 import org.apache.cassandra.config.AccordConfig.JournalConfig.ReplaySavePoint;
+import org.apache.cassandra.config.AccordConfig.UniqueTimestampReservations;
 import org.apache.cassandra.config.CassandraRelevantProperties;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.db.ColumnFamilyStore;
@@ -126,6 +131,7 @@ import org.apache.cassandra.service.accord.api.AccordTopologySorter;
 import org.apache.cassandra.service.accord.api.AccordViolationHandler;
 import org.apache.cassandra.service.accord.api.CompositeTopologySorter;
 import org.apache.cassandra.service.accord.api.TokenKey.KeyspaceSplitter;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
 import org.apache.cassandra.service.accord.interop.AccordInteropAdapter.AccordInteropFactory;
 import org.apache.cassandra.service.accord.journal.AccordJournal;
 import org.apache.cassandra.service.accord.journal.ReplayMarkers;
@@ -165,6 +171,8 @@ import static accord.api.Journal.TopologyUpdate;
 import static accord.api.ProtocolModifiers.FastExecution.MAY_BYPASS_SAFESTORE;
 import static accord.coordinate.Coordination.CoordinationKind.Client;
 import static accord.impl.progresslog.DefaultProgressLog.ModeFlag.CATCH_UP;
+import static accord.local.BootstrapReason.LOG_CORRUPTED;
+import static accord.local.BootstrapReason.LOG_INCOMPLETE;
 import static accord.local.durability.DurabilityService.SyncLocal.Self;
 import static accord.local.durability.DurabilityService.SyncRemote.All;
 import static accord.messages.SimpleReply.Ok;
@@ -172,15 +180,20 @@ import static accord.primitives.Txn.Kind.ExclusiveSyncPoint;
 import static accord.primitives.Txn.Kind.Write;
 import static accord.primitives.TxnId.MediumPath.NoMediumPath;
 import static accord.primitives.TxnId.MediumPath.TrackStable;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
 import static java.util.concurrent.TimeUnit.MINUTES;
 import static java.util.concurrent.TimeUnit.NANOSECONDS;
 import static java.util.concurrent.TimeUnit.SECONDS;
 import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
 import static org.apache.cassandra.concurrent.ExecutorFactory.SimulatorThreadTag.JOB;
 import static org.apache.cassandra.concurrent.ExecutorFactory.SystemThreadTag.DAEMON;
-import static org.apache.cassandra.config.AccordConfig.CatchupMode.FALLBACK_TO_HARD;
+import static org.apache.cassandra.config.AccordConfig.CatchupFallbackMode.EXIT;
-import static org.apache.cassandra.config.AccordConfig.CatchupMode.HARD;
+import static org.apache.cassandra.config.AccordConfig.CatchupFallbackMode.REBOOTSTRAP;
+import static org.apache.cassandra.config.AccordConfig.CatchupFallbackMode.REBOOTSTRAP_AND_CATCHUP;
+import static org.apache.cassandra.config.AccordConfig.JournalConfig.ReplayMode.REBOOTSTRAP_INCOMPLETE;
+import static org.apache.cassandra.config.AccordConfig.JournalConfig.ReplayMode.REBOOTSTRAP_RESET;
 import static org.apache.cassandra.config.AccordConfig.JournalConfig.ReplayMode.RESET;
+import static org.apache.cassandra.config.AccordConfig.UniqueTimestampReservations.HISTOGRAM;
 import static org.apache.cassandra.config.DatabaseDescriptor.getAccord;
 import static org.apache.cassandra.config.DatabaseDescriptor.getAccordGlobalDurabilityCycle;
 import static org.apache.cassandra.config.DatabaseDescriptor.getAccordShardDurabilityCycle;
@@ -384,6 +397,11 @@ public class AccordService implements IAccordService, Shutdownable
             AccordService as = new AccordService(tcmIdToAccord(tcmId));
             unsafeInstance = replyInstance = as;
             as.localStartup();
+
+            AccordReplicaMetrics.touch();
+            AccordSystemMetrics.touch();
+            AccordExecutorMetrics.touch();
+            AccordViolationHandler.setup();
         }
     }
 
@@ -397,11 +415,6 @@ public class AccordService implements IAccordService, Shutdownable
             return as;
 
         as.distributedStartupInternal();
-
-        AccordReplicaMetrics.touch();
-        AccordSystemMetrics.touch();
-        AccordExecutorMetrics.touch();
-        AccordViolationHandler.setup();
         return as;
     }
 
@@ -464,6 +477,7 @@ public class AccordService implements IAccordService, Shutdownable
         agent.setup(localId);
         AccordTimeService time = new AccordTimeService();
         this.scheduler = new AccordScheduler();
+        // TODO (expected): can we pass ImmediateExecutor rather than Scheduler?
         final RequestCallbacks callbacks = new RequestCallbacks(time, scheduler);
         this.dataStore = new AccordDataStore();
         this.journal = new AccordJournal(DatabaseDescriptor.getAccord().journal);
@@ -474,7 +488,7 @@ public class AccordService implements IAccordService, Shutdownable
         this.node = new Node(localId,
                              messageSink,
                              topologyService,
-                             time, new AtomicUniqueTimeWithStaleReservation(time),
+                             time, uniqueTimeService(time),
                              () -> dataStore,
                              new KeyspaceSplitter(new EvenSplit<>(getAccord().commandStoreShardCount(), getPartitioner().accordSplitter())),
                              agent,
@@ -524,6 +538,8 @@ public class AccordService implements IAccordService, Shutdownable
             ProtocolModifiers.Configure.setSendStableMessages(config.send_stable);
         if (config.send_minimal != null)
             ProtocolModifiers.Configure.setSendMinimal(config.send_minimal);
+        if (config.unique_timestamp_on_conflict != null)
+            ProtocolModifiers.Configure.setUniqueTimestampOnConflict(config.unique_timestamp_on_conflict);
     }
 
     @Override
@@ -532,7 +548,16 @@ public class AccordService implements IAccordService, Shutdownable
         if (state != State.INIT)
             return;
 
-        boolean rebootstrap = false;
+        BootstrapReason rebootstrap = null;
+        if (getAccord().journal.replay == REBOOTSTRAP_RESET)
+        {
+            rebootstrap = LOG_CORRUPTED;
+        }
+        else if (getAccord().journal.replay == REBOOTSTRAP_INCOMPLETE)
+        {
+            rebootstrap = LOG_INCOMPLETE;
+        }
+        else
         {
             long startMarker = ReplayMarkers.readStartMarker();
             long stopMarker = ReplayMarkers.readStopMarker();
@@ -551,7 +576,7 @@ public class AccordService implements IAccordService, Shutdownable
 
                     case REBOOTSTRAP:
                         logger.info("Stop marker is older than start marker ({}<{}). Rebootstrapping.", stopMarker, startMarker);
-                        rebootstrap = true;
+                        rebootstrap = LOG_INCOMPLETE;
                 }
             }
         }
@@ -592,7 +617,7 @@ public class AccordService implements IAccordService, Shutdownable
 
             // restore save points before starting the journal so we can validate consistency between journal and save point state (where possible)
             Long2LongHashMap minSegments;
-            if (rebootstrap) minSegments = null;
+            if (rebootstrap != null) minSegments = null;
             else
             {
                 switch (getAccord().journal.replaySavePoint)
@@ -614,11 +639,11 @@ public class AccordService implements IAccordService, Shutdownable
             }
 
             node.commandStores().forAllUnsafe(cs -> cs.unsafeProgressLog().start());
-            if (rebootstrap)
+            if (rebootstrap != null)
             {
                 // rebootstrap expects the durability service to process visibility sync points for command stores
                 node.durability().start();
-                getBlocking(node.commandStores().rebootstrap(node));
+                getBlocking(node.commandStores().rebootstrap(node, rebootstrap));
             }
             else
             {
@@ -704,82 +729,96 @@ public class AccordService implements IAccordService, Shutdownable
 
     void catchup()
     {
-        AccordConfig spec = DatabaseDescriptor.getAccord();
+        AccordConfig config = DatabaseDescriptor.getAccord();
-        if (spec.catchup_on_start == CatchupMode.DISABLED)
+        if (!config.catchup_on_start)
         {
             logger.info("Catchup disabled; continuing to startup");
             return;
         }
 
-        CatchupMode mode = spec.catchup_on_start;
         BootstrapState bootstrapState = SystemKeyspace.getBootstrapState();
         if (bootstrapState == COMPLETED)
         {
             node.commandStores().forAllUnsafe(commandStore -> ((DefaultProgressLog)commandStore.unsafeProgressLog()).setMode(CATCH_UP));
             try
             {
-                long maxLatencyNanos = spec.catchup_on_start_fail_latency.toNanoseconds();
+                CatchupFallbackMode onError = config.catchup_on_start_on_error;
+                CatchupFallbackMode onTimeout = config.catchup_on_start_on_timeout;
+
+                long maxLatencyNanos = config.catchup_on_start_fail_latency.toNanoseconds();
                 int attempts = 1;
                 while (true)
                 {
-                    logger.info("Catchup ({}) with quorum...", mode);
+                    logger.info("Catchup with quorum...");
                     long start = nanoTime();
                     long failAt = start + maxLatencyNanos;
-                    Future<Void> f;
+                    Future<Unsuccessful> f = toFuture(Catchup.catchup(node, failAt, NANOSECONDS));
-                    {
+                    f.awaitThrowUncheckedOnInterrupt();
-                        AsyncChain<Void> submit;
-                        if (mode == HARD)
-                        {
-                            if (!node.durability().isStarted())
-                                node.durability().start();
-                            submit = CatchupHard.catchup(node);
-                        }
-                        else submit = Catchup.catchup(node);
-                        f = toFuture(submit);
-                    }
-                    if (!f.awaitUntilThrowUncheckedOnInterrupt(failAt))
-                    {
-                        if (spec.catchup_on_start_exit_on_failure)
-                        {
-                            logger.error("Catchup exceeded maximum latency of {}ns; shutting down", maxLatencyNanos);
-                            throw new RuntimeException("Could not catchup with peers");
-                        }
-                        logger.error("Catchup exceeded maximum latency of {}ns; continuing to startup", maxLatencyNanos);
-                        break;
-                    }
-
                     Throwable failed = f.cause();
+                    Unsuccessful result = f.getNow();
+
                     if (failed != null)
                     {
-                        if (spec.catchup_on_start_exit_on_failure)
+                        switch (onError)
-                            throw new RuntimeException("Could not catchup with peers", failed);
+                        {
-
+                            default: throw new UnhandledEnum(onError);
-                        logger.error("Could not catchup with peers; continuing to startup");
+                            case EXIT:
-                        break;
+                                logger.error("Could not catchup with peers; exiting", failed);
+                                throw new RuntimeException("Could not catchup with peers", failed);
+                            case IGNORE:
+                                logger.error("Could not catchup with peers; continuing to startup", failed);
+                                return;
+                            case REBOOTSTRAP:
+                            case REBOOTSTRAP_AND_CATCHUP:
+                                logger.error("Could not catchup with peers; rebootstrapping", failed);
+                                getBlocking(node.commandStores().rebootstrap(node, LOG_INCOMPLETE));
+                                if (onError == REBOOTSTRAP)
+                                    return;
+                                ++attempts;
+                                onError = onTimeout = rebootstrapFallbackMode(config);
+                                continue;
+                        }
                     }
 
                     long end = nanoTime();
                     double seconds = NANOSECONDS.toMillis(end - start)/1000.0;
                     logger.info("Finished catchup with all quorums. {}s elapsed.", String.format("%.2f", seconds));
 
-                    if (seconds <= spec.catchup_on_start_success_latency.toSeconds())
+                    if (result == null)
-                        break;
-
-                    if (++attempts > spec.catchup_on_start_max_attempts)
                     {
-                        if (spec.catchup_on_start_exit_on_failure)
+                        if (seconds <= config.catchup_on_start_success_latency.toSeconds())
-                        {
+                            return;
-                            logger.error("Catchup was slow, aborting after {} attempts and shutting down", attempts);
-                            throw new RuntimeException("Could not catchup with peers");
-                        }
 
-                        logger.info("Catchup was slow; continuing to startup after {} attempts.", attempts - 1);
+                        if (attempts < config.catchup_on_start_max_attempts)
-                        break;
+                        {
+                            logger.info("Catchup was slow, so we may behind again; retrying");
+                            ++attempts;
+                            continue;
+                        }
                     }
 
-                    logger.info("Catchup was slow, so we may behind again; retrying");
+                    switch (onTimeout)
-                    if (mode == FALLBACK_TO_HARD)
+                    {
-                        mode = HARD;
+                        default: throw new UnhandledEnum(onTimeout);
+                        case EXIT:
+                            if (result == null) logger.error("Catchup was slow; aborting after {} attempts and shutting down", attempts);
+                            else logger.error("Catchup was incomplete for {}; aborting after {} attempts and shutting down", result.ranges, attempts);
+                            throw new RuntimeException("Could not catchup with peers");
+                        case IGNORE:
+                            if (result == null) logger.info("Catchup was slow, continuing to startup after {} attempts", attempts);
+                            else logger.info("Catchup was incomplete for {}, continuing to startup after {} attempts", result.ranges, attempts);
+                            return;
+                        case REBOOTSTRAP:
+                        case REBOOTSTRAP_AND_CATCHUP:
+                            if (result == null) logger.info("Catchup was slow, rebootstrapping after {} attempts", attempts);
+                            else logger.info("Catchup was incomplete for {}, rebootstrapping after {} attempts", result.ranges, attempts);
+                            getBlocking(node.commandStores().rebootstrap(node, result == null ? null : result.ranges, LOG_INCOMPLETE));
+                            if (onTimeout == REBOOTSTRAP)
+                                return;
+                            ++attempts;
+                            onError = onTimeout = rebootstrapFallbackMode(config);
+                            continue;
+                    }
                 }
             }
             finally
@@ -793,6 +832,18 @@ public class AccordService implements IAccordService, Shutdownable
         }
     }
 
+    private static CatchupFallbackMode rebootstrapFallbackMode(AccordConfig config)
+    {
+        CatchupFallbackMode mode = config.catchup_on_start_on_rebootstrap_fallback;
+        if (mode == REBOOTSTRAP_AND_CATCHUP)
+        {
+            CatchupFallbackMode newMode = EXIT;
+            logger.warn("Converting {} to {} after first failed rebootstrap, to avoid infinite loop", mode, newMode);
+            mode = newMode;
+        }
+        return mode;
+    }
+
     /**
      * Startup is broken up in two phases: local and distributed startup. During local startup, we replay up to
      *  the latest epoch known to the node prior to restart. After that, we replay journal itself, and only after
@@ -952,7 +1003,7 @@ public class AccordService implements IAccordService, Shutdownable
     }
 
     @Override
-    public AsyncResult<Void> sync(Object requestedBy, TxnId minBound, Ranges ranges, @Nullable Collection<Id> include, DurabilityService.SyncLocal syncLocal, DurabilityService.SyncRemote syncRemote, long timeout, TimeUnit timeoutUnits)
+    public AsyncResult<?> sync(Object requestedBy, TxnId minBound, Ranges ranges, @Nullable Collection<Id> include, DurabilityService.SyncLocal syncLocal, DurabilityService.SyncRemote syncRemote, long timeout, TimeUnit timeoutUnits)
     {
         return node.durability().sync(requestedBy, ExclusiveSyncPoint, minBound, ranges, include, syncLocal, syncRemote, timeout, timeoutUnits);
     }
@@ -964,7 +1015,7 @@ public class AccordService implements IAccordService, Shutdownable
             return syncInternal(minBound, keys, syncLocal, syncRemote);
 
         return KeyBarriers.find(node, minBound, keys.get(0).toUnseekable(), syncLocal, syncRemote).chain()
-                          .flatMap(found -> KeyBarriers.await(node, node.someSequentialExecutor(), found, syncLocal, syncRemote))
+                          .flatMap(found -> KeyBarriers.await(node, node.someExclusiveExecutor(), found, syncLocal, syncRemote))
                           .flatMap(success -> {
                               if (success)
                                   return null;
@@ -1436,4 +1487,22 @@ public class AccordService implements IAccordService, Shutdownable
     {
         return journal.configuration();
     }
+
+    private static UniqueTimeService uniqueTimeService(TimeService time)
+    {
+        AccordConfig config = getAccord();
+        UniqueTimestampReservations reservations = config.unique_timestamp_reservations;
+        if (reservations == null)
+            reservations = HISTOGRAM;
+
+        switch (reservations)
+        {
+            default: throw UnhandledEnum.unknown(reservations);
+            case NONE: return new AtomicUniqueTime(time);
+            case SMALL_SHARED: return new AtomicUniqueTimeWithStaleReservation(time);
+            case HISTOGRAM:
+                long millis = config.unique_timestamp_reservation_range == null ? 50 : config.unique_timestamp_reservation_range.toMilliseconds();
+                return new UniqueTimeService.AtomicUniqueAutoStaleTimes(time, millis, MILLISECONDS);
+        }
+    }
 }

src/java/org/apache/cassandra/service/accord/IAccordService.java

@@ -63,6 +63,7 @@ import org.apache.cassandra.net.Message;
 import org.apache.cassandra.schema.TableId;
 import org.apache.cassandra.service.accord.api.AccordScheduler;
 import org.apache.cassandra.service.accord.api.AccordTopologySorter;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
 import org.apache.cassandra.service.accord.topology.AccordEndpointMapper;
 import org.apache.cassandra.service.accord.topology.AccordSyncPropagator;
 import org.apache.cassandra.service.accord.topology.AccordSyncPropagator.Notification;
@@ -87,8 +88,8 @@ public interface IAccordService
     IVerbHandler<? extends Request> requestHandler();
     IVerbHandler<? extends Reply> responseHandler();
 
-    AsyncResult<Void> sync(Object requestedBy, @Nullable TxnId minBound, Ranges ranges, @Nullable Collection<Id> include, SyncLocal syncLocal, SyncRemote syncRemote, long timeout, TimeUnit timeoutUnits);
+    AsyncResult<?> sync(Object requestedBy, @Nullable TxnId minBound, Ranges ranges, @Nullable Collection<Id> include, SyncLocal syncLocal, SyncRemote syncRemote, long timeout, TimeUnit timeoutUnits);
-    AsyncChain<Void> sync(@Nullable TxnId minBound, Keys keys, SyncLocal syncLocal, SyncRemote syncRemote);
+    AsyncChain<?> sync(@Nullable TxnId minBound, Keys keys, SyncLocal syncLocal, SyncRemote syncRemote);
     AsyncChain<Timestamp> maxConflict(Ranges ranges);
 
     @Nonnull
@@ -401,13 +402,13 @@ public interface IAccordService
         }
 
         @Override
-        public AsyncResult<Void> sync(Object requestedBy, @Nullable TxnId onOrAfter, Ranges ranges, @Nullable Collection<Id> include, SyncLocal syncLocal, SyncRemote syncRemote, long timeout, TimeUnit timeoutUnits)
+        public AsyncResult<?> sync(Object requestedBy, @Nullable TxnId onOrAfter, Ranges ranges, @Nullable Collection<Id> include, SyncLocal syncLocal, SyncRemote syncRemote, long timeout, TimeUnit timeoutUnits)
         {
             return delegate.sync(requestedBy, onOrAfter, ranges, include, syncLocal, syncRemote, timeout, timeoutUnits);
         }
 
         @Override
-        public AsyncChain<Void> sync(@Nullable TxnId onOrAfter, Keys keys, SyncLocal syncLocal, SyncRemote syncRemote)
+        public AsyncChain<?> sync(@Nullable TxnId onOrAfter, Keys keys, SyncLocal syncLocal, SyncRemote syncRemote)
         {
             return delegate.sync(onOrAfter, keys, syncLocal, syncRemote);
         }

src/java/org/apache/cassandra/service/accord/InMemoryRangeIndex.java

@@ -44,6 +44,7 @@ import accord.primitives.Unseekables;
 import accord.utils.async.Cancellable;
 
 import org.apache.cassandra.io.util.File;
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry;
 import org.apache.cassandra.service.accord.serializers.CommandStoreSerializers;
 
 import static org.apache.cassandra.io.util.CompressedFrameDataInputPlus.readList;
@@ -88,7 +89,7 @@ public class InMemoryRangeIndex extends InMemoryRangeSummaryIndex implements Ran
             {
                 for (TxnId txnId : load)
                 {
-                    AccordCacheEntry<TxnId, Command> entry = caches.commands().getUnsafe(txnId);
+                    AccordCacheEntry<TxnId, Command, ?> entry = caches.commands().getUnsafe(txnId);
                     if (entry == null)
                     {
                         loadFromDisk.add(txnId);
@@ -115,7 +116,6 @@ public class InMemoryRangeIndex extends InMemoryRangeSummaryIndex implements Ran
 
         public void finish(Map<Timestamp, Summary> into)
         {
-            cleanupExclusive(null);
             owner.search(this, into::put, null);
         }
 

src/java/org/apache/cassandra/service/accord/RangeIndex.java

@@ -41,6 +41,7 @@ import accord.utils.Invariants;
 import org.apache.cassandra.exceptions.UnknownTableException;
 import org.apache.cassandra.io.util.DataInputBuffer;
 import org.apache.cassandra.io.util.File;
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry;
 import org.apache.cassandra.service.accord.journal.CommandChanges;
 import org.apache.cassandra.service.accord.serializers.Version;
 
@@ -59,10 +60,10 @@ public interface RangeIndex
 
         protected abstract AccordCommandStore commandStore();
 
-        protected abstract void loadExclusive(Map<Timestamp, CommandSummaries.Summary> into, AccordCommandStore.Caches caches);
+        public abstract void loadExclusive(Map<Timestamp, CommandSummaries.Summary> into, AccordCommandStore.Caches caches);
-        protected abstract void load(Map<Timestamp, CommandSummaries.Summary> into, BooleanSupplier abort);
+        public abstract void load(Map<Timestamp, CommandSummaries.Summary> into, BooleanSupplier abort);
-        protected abstract void finish(Map<Timestamp, CommandSummaries.Summary> into);
+        public abstract void finish(Map<Timestamp, CommandSummaries.Summary> into);
-        protected abstract void cleanupExclusive(AccordCommandStore.Caches caches);
+        public abstract void cleanupExclusive(AccordCommandStore.Caches caches);
 
         protected CommandSummaries.Summary loadFromDisk(TxnId txnId)
         {
@@ -82,7 +83,7 @@ public interface RangeIndex
             return null;
         }
 
-        public CommandSummaries.Summary ifRelevant(AccordCacheEntry<TxnId, Command> state)
+        public CommandSummaries.Summary ifRelevant(AccordCacheEntry<TxnId, Command, ?> state)
         {
             if (state.key().domain() != Routable.Domain.Range)
                 return null;

src/java/org/apache/cassandra/service/accord/api/AccordAgent.java

@@ -39,16 +39,16 @@ import accord.api.Result;
 import accord.api.RoutingKey;
 import accord.api.Tracing;
 import accord.coordinate.Coordination;
-import accord.coordinate.Exhausted;
+import accord.coordinate.CoordinationFailed;
-import accord.coordinate.Preempted;
-import accord.coordinate.Timeout;
 import accord.local.Command;
+import accord.local.CommandStore;
 import accord.local.LogUnavailableException;
 import accord.local.Node;
 import accord.local.SafeCommand;
 import accord.local.SafeCommandStore;
 import accord.local.TimeService;
 import accord.messages.MessageType;
+import accord.primitives.Ballot;
 import accord.primitives.Keys;
 import accord.primitives.Participants;
 import accord.primitives.Ranges;
@@ -84,6 +84,7 @@ import org.apache.cassandra.service.accord.txn.TxnResult;
 import org.apache.cassandra.utils.Clock;
 import org.apache.cassandra.utils.JVMStabilityInspector;
 import org.apache.cassandra.utils.NoSpamLogger;
+import org.apache.cassandra.utils.NoSpamLogger.NoDuplicateSpamLogStatement;
 
 import static accord.primitives.Routable.Domain.Key;
 import static accord.utils.SortedArrays.SortedArrayList.ofSorted;
@@ -107,12 +108,14 @@ import static org.apache.cassandra.service.accord.api.AccordWaitStrategies.retry
 import static org.apache.cassandra.service.accord.api.AccordWaitStrategies.slowRead;
 import static org.apache.cassandra.service.accord.api.AccordWaitStrategies.slowTxnPreaccept;
 import static org.apache.cassandra.service.accord.txn.TxnResult.Kind.txn_data;
+import static org.apache.cassandra.utils.NoSpamLogger.NoDuplicateSpamLogStatement.exceptionId;
 
 // TODO (expected): merge with AccordService
 public class AccordAgent implements Agent, OwnershipEventListener
 {
     private static final Logger logger = LoggerFactory.getLogger(AccordAgent.class);
     private static final NoSpamLogger noSpamLogger = NoSpamLogger.getLogger(logger, 1L, MINUTES);
+    private static final NoDuplicateSpamLogStatement noSpamException = new NoDuplicateSpamLogStatement(logger, "", 1L, MINUTES);
     private static final ReplicaEventListener replicaEventListener = new AccordReplicaMetrics.Listener();
 
     private static BiConsumer<TxnId, Throwable> onFailedBarrier;
@@ -155,6 +158,12 @@ public class AccordAgent implements Agent, OwnershipEventListener
         config = DatabaseDescriptor.getAccord();
     }
 
+    @Override
+    public void onSuccessfulBootstrap(CommandStore commandStore, int attempt, long epoch, Ranges ranges)
+    {
+        logger.info("{}: Completed bootstrap of {} on epoch {}", commandStore, ranges, epoch);
+    }
+
     @Override
     public void onFailedBootstrap(int attempts, String phase, Ranges ranges, Runnable retry, Runnable fail, Throwable failure)
     {
@@ -212,13 +221,19 @@ public class AccordAgent implements Agent, OwnershipEventListener
             return;
 
         AccordSystemMetrics.metrics.errors.inc();
-        if (t instanceof CancellationException || t instanceof TimeoutException || t instanceof Timeout || t instanceof Preempted || t instanceof Exhausted || t instanceof LogUnavailableException)
+        if (expectedException(t)) // TODO (required): leaky logger, permitting multiple messages per time period and reporting how many were dropped
-            // TODO (required): leaky logger, permitting multiple messages per time period and reporting how many were dropped
+            noSpamException.warn(exceptionId(t), t);
-            noSpamLogger.warn("", t);
         else
             JVMStabilityInspector.uncaughtException(Thread.currentThread(), t);
     }
 
+    public static boolean expectedException(Throwable t)
+    {
+        if (t instanceof CancellationException)
+            return t.getCause() == null;
+        return t instanceof TimeoutException || t instanceof LogUnavailableException || t instanceof CoordinationFailed;
+    }
+
     @Override
     public void onException(Throwable t)
     {
@@ -249,6 +264,11 @@ public class AccordAgent implements Agent, OwnershipEventListener
     // TODO (expected): we probably want additional configuration here so we can prune on shorter time horizons when we have a lot of transactions on a single key
     @Override
     public long cfkHlcPruneDelta()
+    {
+        return cfkHlcPruneDelta(config);
+    }
+
+    public static long cfkHlcPruneDelta(AccordConfig config)
     {
         return config.commands_for_key_prune_delta.to(MICROSECONDS);
     }
@@ -308,24 +328,12 @@ public class AccordAgent implements Agent, OwnershipEventListener
     @Override
     public long slowCoordinatorDelay(Node node, SafeCommandStore safeStore, TxnId txnId, TimeUnit units, int attempt)
     {
-        SafeCommand safeCommand = safeStore.unsafeGetNoCleanup(txnId);
+        Command command = command(safeStore, txnId, "slowCoordinatorDelay");
-        if (safeCommand == null)
+        Ballot promised = promised(command);
-        {
-            noSpamLogger.warn("{} invoked slowCoordinatorDelay for {} without having it in cache", safeStore.commandStore(), txnId, new RuntimeException());
-            return recover(txnId).computeWait(attempt, units);
-        }
-
-        Command command = safeCommand.current();
-        if (command == null)
-        {
-            noSpamLogger.warn("{} invoked slowCoordinatorDelay for {} without knowing the command", safeStore.commandStore(), txnId, new RuntimeException());
-            return recover(txnId).computeWait(attempt, units);
-        }
-
 
         // TODO (expected): make this a configurable calculation on normal request latencies (like ContentionStrategy)
         long nowMicros = MILLISECONDS.toMicros(Clock.Global.currentTimeMillis());
-        long mostRecentStart = mostRecentStart(command, nowMicros);
+        long mostRecentStart = mostRecentStart(txnId, promised, nowMicros);
         long waitMicros = recover(txnId).computeWait(attempt, MICROSECONDS);
         long startTime = mostRecentStart + waitMicros;
         if (startTime < nowMicros)
@@ -338,7 +346,7 @@ public class AccordAgent implements Agent, OwnershipEventListener
         }
 
         RoutingKey homeKey = command.route().homeKey();
-        Shard shard = node.topology().active().forEpochIfKnown(homeKey, command.txnId().epoch());
+        Shard shard = node.topology().active().forEpochIfKnown(homeKey, txnId.epoch());
 
         startTime = nonClashingStartTime(startTime, shard == null ? null : shard.nodes, node.id(), ONE_SECOND, random);
         long delayMicros = Math.max(1, startTime - nowMicros);
@@ -346,15 +354,15 @@ public class AccordAgent implements Agent, OwnershipEventListener
         return units.convert(delayMicros, MICROSECONDS);
     }
 
-    private static long mostRecentStart(Command command, long nowMicros)
+    private static long mostRecentStart(TxnId txnId, Ballot promised, long nowMicros)
     {
         // TODO (expected): make this a configurable calculation on normal request latencies (like ContentionStrategy)
-        long promisedHlc = command.promised().hlc();
+        long promisedHlc = promised.hlc();
         if (promisedHlc > nowMicros + ONE_MINUTE)
             promisedHlc = 0;
-        long result = Math.max(command.txnId().hlc(), promisedHlc);
+        long result = Math.max(txnId.hlc(), promisedHlc);
         if (result > nowMicros + ONE_SECOND)
-            noSpamLogger.warn("max({},{})>{}", command.txnId(), command.promised(), nowMicros);
+            noSpamLogger.warn("max({},{})>{}", txnId, promised, nowMicros);
         return result;
     }
 
@@ -388,25 +396,36 @@ public class AccordAgent implements Agent, OwnershipEventListener
         return newStartTime;
     }
 
-    @Override
+    private static Ballot promised(@Nullable Command command)
-    public long slowReplicaDelay(Node node, SafeCommandStore safeStore, TxnId txnId, int attempt, BlockedUntil blockedUntil, TimeUnit units)
+    {
+        return command == null ? Ballot.ZERO : command.promised();
+    }
+
+    private static Command command(SafeCommandStore safeStore, TxnId txnId, String source)
     {
         SafeCommand safeCommand = safeStore.unsafeGetNoCleanup(txnId);
         if (safeCommand == null)
         {
-            noSpamLogger.warn("{} invoked slowReplicaDelay for {} without having it in cache", safeStore.commandStore(), txnId, new RuntimeException());
+            noSpamLogger.warn("{} invoked {} for {} without having it in cache", safeStore.commandStore(), source, txnId, new RuntimeException());
-            return fetch(txnId).computeWait(attempt, units);
+            return null;
         }
 
         Command command = safeCommand.current();
         if (command == null)
         {
-            noSpamLogger.warn("{} invoked slowReplicaDelay for {} without knowing the command", safeStore.commandStore(), txnId, new RuntimeException());
+            noSpamLogger.warn("{} invoked {} for {} without knowing the command", safeStore.commandStore(), source, txnId, new RuntimeException());
-            return fetch(txnId).computeWait(attempt, units);
+            return null;
         }
 
+        return command;
+    }
+
+    @Override
+    public long slowReplicaDelay(Node node, SafeCommandStore safeStore, TxnId txnId, int attempt, BlockedUntil blockedUntil, TimeUnit units)
+    {
+        Ballot promised = promised(command(safeStore, txnId, "slowReplicaDelay"));
         long nowMicros = MILLISECONDS.toMicros(Clock.Global.currentTimeMillis());
-        long mostRecentStart = mostRecentStart(command, nowMicros);
+        long mostRecentStart = mostRecentStart(txnId, promised, nowMicros);
         long waitMicros = fetch(txnId).computeWait(attempt, units);
         long startTime = mostRecentStart + waitMicros;
         if (startTime < nowMicros)

src/java/org/apache/cassandra/service/accord/api/AccordWaitStrategies.java

@@ -38,7 +38,7 @@ import static org.apache.cassandra.service.TimeoutStrategy.LatencySourceFactory.
 
 public class AccordWaitStrategies
 {
-    static TimeoutStrategy slowTxnPreaccept, slowSyncPointPreaccept, slowRead;
+    static TimeoutStrategy slowTxnPreaccept, slowSyncPointPreaccept, slowRead, slowDurability;
     static TimeoutStrategy expireTxn, expireSyncPoint, expireDurability, expireEpochWait;
     static TimeoutStrategy fetchTxn, fetchSyncPoint;
     static RetryStrategy recoverTxn, recoverSyncPoint, retrySyncPoint, retryDurability, retryBootstrap, retryJoinBootstrap;
@@ -88,6 +88,7 @@ public class AccordWaitStrategies
         setSlowRead(config.slow_read);
         setSlowTxnPreaccept(config.slow_txn_preaccept);
         setSlowSyncPointPreaccept(config.slow_syncpoint_preaccept);
+        setSlowDurability(config.slow_durability);
         setExpireTxn(config.expire_txn);
         setExpireSyncPoint(config.expire_syncpoint);
         setExpireDurability(config.expire_durability);
@@ -119,6 +120,11 @@ public class AccordWaitStrategies
         slowSyncPointPreaccept = TimeoutStrategy.parse(spec, none());
     }
 
+    public static void setSlowDurability(String spec)
+    {
+        slowDurability = TimeoutStrategy.parse(spec, none());
+    }
+
     public static void setExpireTxn(String spec)
     {
         expireTxn = TimeoutStrategy.parse(spec, rw(accordReadMetrics, accordWriteMetrics));

src/java/org/apache/cassandra/service/accord/debug/DebugBlockedTxns.java

@@ -37,7 +37,7 @@ import accord.api.RoutingKey;
 import accord.local.Command;
 import accord.local.CommandStore;
 import accord.local.CommandStores;
-import accord.local.PreLoadContext;
+import accord.local.ExecutionContext;
 import accord.local.SafeCommandStore;
 import accord.local.cfk.SafeCommandsForKey;
 import accord.primitives.RoutingKeys;
@@ -53,8 +53,6 @@ import org.apache.cassandra.service.accord.IAccordService;
 import org.apache.cassandra.service.accord.api.TokenKey;
 import org.apache.cassandra.utils.concurrent.Future;
 
-import static accord.local.LoadKeys.SYNC;
-import static accord.local.LoadKeysFor.READ_WRITE;
 import static java.util.Collections.emptyList;
 
 public class DebugBlockedTxns
@@ -178,7 +176,7 @@ public class DebugBlockedTxns
 
     private AsyncChain<Txn> visitRootTxnAsync(CommandStore commandStore, TxnId txnId)
     {
-        return commandStore.chain(PreLoadContext.contextFor(txnId, "Populate txn_blocked_by"), safeStore -> {
+        return commandStore.chain(ExecutionContext.unsequenced(txnId, "Populate txn_blocked_by"), safeStore -> {
             Command command = safeStore.unsafeGetNoCleanup(txnId).current();
             if (command == null || command.saveStatus() == SaveStatus.Uninitialised)
                 return null;
@@ -188,7 +186,7 @@ public class DebugBlockedTxns
 
     private AsyncChain<Txn> visitTxnAsync(CommandStore commandStore, TxnId txnId, Timestamp rootExecuteAt, @Nullable TokenKey byKey, int depth, boolean recurse)
     {
-        return commandStore.chain(PreLoadContext.contextFor(txnId, "Populate txn_blocked_by"), safeStore -> {
+        return commandStore.chain(ExecutionContext.unsequenced(txnId, "Populate txn_blocked_by"), safeStore -> {
             Command command = safeStore.unsafeGetNoCleanup(txnId).current();
             if (command == null || command.saveStatus() == SaveStatus.Uninitialised)
                 return null;
@@ -231,7 +229,7 @@ public class DebugBlockedTxns
 
     private AsyncChain<Void> visitKeysAsync(CommandStore commandStore, TokenKey key, Timestamp rootExecuteAt, int depth)
     {
-        return commandStore.chain(PreLoadContext.contextFor(RoutingKeys.of(key.toUnseekable()), SYNC, READ_WRITE, "Populate txn_blocked_by"), safeStore -> {
+        return commandStore.chain(ExecutionContext.unsequencedReadWrite(RoutingKeys.of(key.toUnseekable()), "Populate txn_blocked_by"), safeStore -> {
             visitKeysSync(safeStore, key, rootExecuteAt, depth);
         });
     }

src/java/org/apache/cassandra/service/accord/debug/DebugExecution.java

@@ -29,7 +29,7 @@ import accord.local.Command;
 
 import org.apache.cassandra.config.CassandraRelevantProperties;
 import org.apache.cassandra.metrics.LogLinearHistogram;
-import org.apache.cassandra.service.accord.AccordExecutor.Task;
+import org.apache.cassandra.service.accord.execution.Task;
 import org.apache.cassandra.utils.Closeable;
 import org.apache.cassandra.utils.WithResources;
 
@@ -40,10 +40,10 @@ public class DebugExecution
 {
     private static final Logger logger = LoggerFactory.getLogger(DebugExecution.class);
     public static final boolean DEBUG_EXECUTION = CassandraRelevantProperties.ACCORD_DEBUG_EXECUTION.getBoolean(false);
-    private static final long REPORT_MIN_LATENCY_MICROS = 20_000;
+    private static final long REPORT_MIN_LATENCY_MICROS = 50_000;
     private static final long REPORT_CPU_RATIO = 2;
-    private static final long REPORT_MAX_LATENCY_MICROS = 50_000;
+    private static final long REPORT_MAX_LATENCY_MICROS = 100_000;
-    private static final long REPORT_CPU_MICROS = 10_000;
+    private static final long REPORT_CPU_MICROS = 50_000;
 
     // TODO (expected): use sharded histogram so we can report global stats
     public static class DebugExecutor
@@ -94,48 +94,21 @@ public class DebugExecution
             long lockedForCpuMicros = (unlockedAtCpu - lockedAtCpu)/1000;
             if (lockedForMicros >= REPORT_MAX_LATENCY_MICROS)
             {
-                report("Held lock for {}us (cpu:{}us)\n", lockedForMicros, lockedForCpuMicros);
+                report("Held lock for {}us (cpu:{}us)", lockedForMicros, lockedForCpuMicros);
             }
             else if (lockedForMicros >= REPORT_MIN_LATENCY_MICROS && (lockedForMicros / lockedForCpuMicros) >= REPORT_CPU_RATIO)
             {
-                report("Held lock for {}us with cpu time only {}us\n", lockedForMicros, lockedForCpuMicros);
+                report("Held lock for {}us with cpu time only {}us", lockedForMicros, lockedForCpuMicros);
             }
             locked.increment(lockedForMicros);
         }
     }
 
-    public static class DebugExecutorLoop
+    public static class DebugExclusiveExecutor
     {
-        final DebugExecutor owner;
+        public static DebugExclusiveExecutor maybeDebug(DebugExecutor owner, int commandStoreId)
-        long lockAt;
-
-        public DebugExecutorLoop(DebugExecutor owner)
         {
-            this.owner = owner;
+            return DEBUG_EXECUTION ? new DebugExclusiveExecutor(owner, commandStoreId) : null;
-        }
-
-        public void onLock()
-        {
-            lockAt = nanoTime();
-        }
-
-        public void onEnterLock()
-        {
-            owner.onEnterLock(lockAt);
-            lockAt = 0;
-        }
-
-        public void onExitLock()
-        {
-            owner.onExitLock();
-        }
-    }
-
-    public static class DebugSequentialExecutor
-    {
-        public static DebugSequentialExecutor maybeDebug(DebugExecutor owner, int commandStoreId)
-        {
-            return DEBUG_EXECUTION ? new DebugSequentialExecutor(owner, commandStoreId) : null;
         }
 
         final DebugExecutor owner;
@@ -144,7 +117,7 @@ public class DebugExecution
         long setTaskAt, waitingAt;
         Task prev;
 
-        public DebugSequentialExecutor(DebugExecutor owner, int commandStoreId)
+        public DebugExclusiveExecutor(DebugExecutor owner, int commandStoreId)
         {
             this.owner = owner;
             this.commandStoreId = commandStoreId;
@@ -159,15 +132,16 @@ public class DebugExecution
         public void onComplete(Task completed)
         {
             long readyAt = setTaskAt;
+            long runningAt = DebugTask.get(completed).runningAt;
             if (waitingAt > setTaskAt)
             {
                 readyAt = waitingAt;
-                long waitingMicros = (completed.runningAt - waitingAt)/1000;
+                long waitingMicros = (runningAt - waitingAt)/1000;
                 owner.sequentialExecutorWaitingToRunLatency.increment(waitingMicros);
                 if (waitingMicros > REPORT_MAX_LATENCY_MICROS)
                     report("{} spent {}us blocked by a direct execution on queue {}", completed, waitingMicros, commandStoreId);
             }
-            long atHeadMicros = (completed.runningAt - readyAt)/1000;
+            long atHeadMicros = (runningAt - readyAt)/1000;
             owner.sequentialExecutorSetHeadToRunLatency.increment(atHeadMicros);
             if (atHeadMicros > REPORT_MAX_LATENCY_MICROS)
             {
@@ -199,8 +173,8 @@ public class DebugExecution
         }
 
         public List<Command> sanityCheck; // for AccordTask only
-        long polledAt, preRunAt, runCompleteAt, completedAt;
+        long polledAt, preRunAt, runningAt, runCompleteAt, completeAt, completedAt;
-        long releasedRangeScannerAt, releasedCommandsAt, releasedCommandsForKeyAt;
+        long releasedRangeScannerAt, releasedStateAt;
         long runningAtCpu, runCompleteAtCpu;
         Thread thread;
 
@@ -218,6 +192,7 @@ public class DebugExecution
         {
             thread = Thread.currentThread();
             runningAtCpu = nowCpu();
+            runningAt = nanoTime();
         }
 
         public void onRunComplete()
@@ -231,32 +206,32 @@ public class DebugExecution
             releasedRangeScannerAt = nanoTime();
         }
 
-        public void onReleasedCommands()
+        public void onReleasedState()
         {
-            releasedCommandsAt = nanoTime();
+            releasedStateAt = nanoTime();
         }
 
-        public void onReleasedCommandsForKeys()
+        public void onComplete()
         {
-            releasedCommandsForKeyAt = nanoTime();
+            completeAt = nanoTime();
         }
 
         public void onCompleted(DebugExecutor owner)
         {
             completedAt = nanoTime();
-            if (task.runningAt > 0 && polledAt > 0)
+            if (runningAt > 0 && polledAt > 0)
             {
-                long pollToRunMicros = (task.runningAt - polledAt)/1000;
+                long pollToRunMicros = (runningAt - polledAt)/1000;
                 owner.pollToRun.increment(pollToRunMicros);
                 long runningMicros = -1;
                 if (runCompleteAt > 0)
                 {
-                    runningMicros = (runCompleteAt - task.runningAt) / 1000;
+                    runningMicros = (runCompleteAt - runningAt) / 1000;
                     owner.running.increment(runningMicros);
                 }
-                long runToCleanMicros = (task.cleanupAt - runCompleteAt)/1000;
+                long runToCleanMicros = (completeAt - runCompleteAt) / 1000;
                 owner.runToCleanup.increment(runToCleanMicros);
-                long cleanupMicros = (completedAt - task.cleanupAt)/1000;
+                long cleanupMicros = (completedAt - completeAt) / 1000;
                 owner.cleanup.increment(cleanupMicros);
                 long totalMicros = (completedAt - polledAt)/1000;
                 owner.taskTotal.increment(totalMicros);
@@ -266,7 +241,7 @@ public class DebugExecution
                     String reason = "";
                     if (totalMicros > REPORT_MAX_LATENCY_MICROS) reason += "LONG TIME ";
                     if (totalCpu > REPORT_CPU_MICROS) reason += "HIGH CPU ";
-                    if ((totalMicros > REPORT_MIN_LATENCY_MICROS && (totalMicros/totalCpu) >= REPORT_CPU_RATIO)) reason += "LOW RATIO ";
+                    if ((totalMicros > REPORT_MIN_LATENCY_MICROS && (totalCpu == 0 || (totalMicros/totalCpu) >= REPORT_CPU_RATIO))) reason += "LOW RATIO ";
                     report("{}{}: total {}us cpu:{}us ({}), pollToRun {}us, running {}us, runToClean {}us, cleanup {}us",
                            reason, task, totalMicros, totalCpu, thread, pollToRunMicros, runningMicros, runToCleanMicros, cleanupMicros);
                 }

src/java/org/apache/cassandra/service/accord/debug/DebugTxnGraph.java

@@ -40,7 +40,7 @@ import javax.annotation.Nullable;
 import accord.local.Command;
 import accord.local.CommandStore;
 import accord.local.CommandStores;
-import accord.local.PreLoadContext;
+import accord.local.ExecutionContext;
 import accord.local.SafeCommandStore;
 import accord.primitives.PartialDeps;
 import accord.primitives.Participants;
@@ -221,7 +221,7 @@ public abstract class DebugTxnGraph<T, P>
 
     private AsyncChain<TxnInfos<T>> submitRoot(CommandStore commandStore, TxnId txnId)
     {
-        return commandStore.chain(PreLoadContext.contextFor(txnId, "Populate txn_graph"), safeStore -> {
+        return commandStore.chain(ExecutionContext.unsequenced(txnId, "Populate txn_graph"), safeStore -> {
             Command command = safeStore.unsafeGetNoCleanup(txnId).current();
             if (command == null || command.saveStatus() == SaveStatus.Uninitialised)
                 return AsyncChains.<TxnInfos<T>>success(null);
@@ -232,7 +232,7 @@ public abstract class DebugTxnGraph<T, P>
     private AsyncChain<TxnInfos<T>> submitParent(CommandStore commandStore, TxnId txnId, P param, Map<TxnId, SaveInfo> infos, Set<TxnId> visitedParent, int depth)
     {
 
-        return commandStore.chain(PreLoadContext.contextFor(txnId, "Populate txn_graph"), safeStore -> {
+        return commandStore.chain(ExecutionContext.unsequenced(txnId, "Populate txn_graph"), safeStore -> {
             Command command = safeStore.unsafeGetNoCleanup(txnId).current();
             if (command == null || command.saveStatus() == SaveStatus.Uninitialised)
                 return AsyncChains.<TxnInfos<T>>success(null);
@@ -344,7 +344,7 @@ public abstract class DebugTxnGraph<T, P>
 
     private AsyncChain<Void> populateTxnAsync(CommandStore commandStore, TxnId txnId, Map<TxnId, SaveInfo> visited)
     {
-        return commandStore.chain(PreLoadContext.contextFor(txnId, "Populate txn_graph"), safeStore -> {
+        return commandStore.chain(ExecutionContext.unsequenced(txnId, "Populate txn_graph"), safeStore -> {
             Command command = safeStore.unsafeGetNoCleanup(txnId).current();
             visited.putIfAbsent(txnId, command == null || command.saveStatus() == SaveStatus.Uninitialised ? SaveInfo.NONE : new SaveInfo(command.saveStatus(), command.executeAtIfKnown()));
         });

src/java/org/apache/cassandra/service/accord/execution/AbstractLockLoop.java

@@ -16,47 +16,43 @@
  * limitations under the License.
  */
 
-package org.apache.cassandra.service.accord;
+package org.apache.cassandra.service.accord.execution;
 
 import java.util.concurrent.locks.Lock;
+import java.util.function.Consumer;
+import java.util.function.Function;
 
 import accord.api.Agent;
-import accord.utils.QuadFunction;
-import accord.utils.QuintConsumer;
 
-import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.service.accord.execution.Loops.LoopTask;
-import org.apache.cassandra.service.accord.AccordExecutorLoops.LoopTask;
-import org.apache.cassandra.service.accord.debug.DebugExecution.DebugExecutorLoop;
 
-import static org.apache.cassandra.service.accord.AccordExecutor.Mode.RUN_WITH_LOCK;
 import static org.apache.cassandra.service.accord.debug.DebugExecution.DEBUG_EXECUTION;
+import static org.apache.cassandra.service.accord.execution.AccordExecutor.Mode.RUN_WITH_LOCK;
 
-abstract class AccordExecutorAbstractLockLoop extends AccordExecutorAbstractLoop
+abstract class AbstractLockLoop extends AbstractLoop
 {
-    private static final int YIELD_INTERVAL = DatabaseDescriptor.getAccord().queue_yield_interval;
     int runningThreads;
     boolean shutdown;
 
-    AccordExecutorAbstractLockLoop(Lock lock, int executorId, Agent agent)
+    AbstractLockLoop(Lock lock, int executorId, Agent agent)
     {
         super(lock, executorId, agent);
     }
 
     abstract void notifyWork();
     abstract void notifyWorkExclusive();
-    void loopYieldExclusive() throws InterruptedException {}
     abstract void awaitExclusive() throws InterruptedException;
-    abstract <P1s, P1a, P2, P3, P4> void submitExternal(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Task> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4);
+    abstract <P1> void submitExternal(Consumer<P1> sync, Function<P1, Task> async, P1 p1);
 
-    <P1s, P1a, P2, P3, P4> void submit(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Task> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4)
+    final <P1> void submit(Consumer<P1> sync, Function<P1, Task> async, P1 p1)
     {
         // if we're a loop thread, we will poll the waitingToRun queue when we come around
         // NOTE: this assumes no synchronous blocking tasks are submitted to this executor
-        if (isInLoop() || isOwningThread()) push(async.apply(p1a, p2, p3, p4));
+        if (isInLoop() || isOwningThread()) push(async.apply(p1));
-        else submitExternal(sync, async, p1s, p1a, p2, p3, p4);
+        else submitExternal(sync, async, p1);
     }
 
-    <P1s, P2, P3, P4> void submitExternalExclusive(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, P1s p1s, P2 p2, P3 p3, P4 p4)
+    final <P1> void submitExternalExclusive(Consumer<P1> sync, Function<P1, Task> async, P1 p1)
     {
         try
         {
@@ -66,11 +62,11 @@ abstract class AccordExecutorAbstractLockLoop extends AccordExecutorAbstractLoop
             }
             catch (Throwable t)
             {
-                try { sync.accept(this, p1s, p2, p3, p4); }
+                try { sync.accept(p1); }
                 catch (Throwable t2) { t.addSuppressed(t2); }
                 throw t;
             }
-            sync.accept(this, p1s, p2, p3, p4);
+            sync.accept(p1);
         }
         finally
         {
@@ -84,6 +80,73 @@ abstract class AccordExecutorAbstractLockLoop extends AccordExecutorAbstractLoop
             notifyWorkExclusive();
     }
 
+    final boolean hasWaitingToRun()
+    {
+        updateWaitingToRunExclusive();
+        return hasAlreadyWaitingToRun();
+    }
+
+    final Task pollWaitingToRunExclusive()
+    {
+        updateWaitingToRunExclusive();
+        return pollAlreadyWaitingToRunExclusive();
+    }
+
+    final void updateWaitingToRunExclusive()
+    {
+        drainUnqueuedExclusive();
+        super.updateWaitingToRunExclusive();
+    }
+
+    final void drainUnqueuedExclusive()
+    {
+        Task cur = Task.reverse(acquireUnqueuedExclusive());
+        while (cur != null)
+            cur = enqueueOneExclusive(cur);
+    }
+
+    final void drainUnqueuedNewWorkExclusive()
+    {
+        Task cur = acquireUnqueuedExclusive();
+        Task prev = null, requeue = null, requeueLast = null;
+        while (cur != null)
+        {
+            Task next = cur.next;
+            if (cur.isNewWork())
+            {
+                cur.next = prev;
+                prev = cur;
+            }
+            else
+            {
+                if (requeue == null) requeue = cur;
+                else requeueLast.next = cur;
+                requeueLast = cur;
+            }
+            cur = next;
+        }
+
+        if (requeue != null)
+        {
+            while (true)
+            {
+                Task next = unqueued;
+                requeueLast.next = next;
+                if (unqueuedUpdater.compareAndSet(this, next, requeue))
+                    break;
+            }
+        }
+
+        cur = prev;
+        while (cur != null)
+        {
+            Task next = cur.next;
+            cur.submitExclusiveNoExcept();
+            cur.next = null;
+            cur = next;
+        }
+    }
+
     @Override
     final void beforeUnlockExternal()
     {
@@ -130,36 +193,22 @@ abstract class AccordExecutorAbstractLockLoop extends AccordExecutorAbstractLoop
             @Override
             public void run()
             {
-                Thread self = Thread.currentThread();
+                Thread thread = Thread.currentThread();
+                TaskRunner self = TaskRunner.get(thread);
+                self.setAccordActiveExecutor(AbstractLockLoop.this);
+                setWrapped(self);
+
                 Task task;
                 while (true)
                 {
-                    lock();
+                    lock(self);
                     try
                     {
                         enterLockLoop();
                         while (true)
                         {
                             task = pollWaitingToRunExclusive();
-
+                            if (task != null) prepareRunComplete(self, task);
-                            if (task != null)
-                            {
-                                setRunning(task);
-                                try
-                                {
-                                    task.preRunExclusive();
-                                    task.runInternal();
-                                }
-                                catch (Throwable t)
-                                {
-                                    task.fail(t);
-                                }
-                                finally
-                                {
-                                    completeTaskExclusive(task);
-                                    clearRunning();
-                                }
-                            }
                             else
                             {
                                 if (shutdown)
@@ -179,13 +228,12 @@ abstract class AccordExecutorAbstractLockLoop extends AccordExecutorAbstractLoop
                     catch (Throwable t)
                     {
                         exitLockLoop();
-
                         try { agent.onException(t); }
                         catch (Throwable t2) { }
                     }
                     finally
                     {
-                        unlock();
+                        unlock(self);
                     }
                 }
             }
@@ -196,42 +244,39 @@ abstract class AccordExecutorAbstractLockLoop extends AccordExecutorAbstractLoop
     {
         return new LoopTask(name)
         {
-            final DebugExecutorLoop debug = DEBUG_EXECUTION ? new DebugExecutorLoop(AccordExecutorAbstractLockLoop.this.debug) : null;
             @Override
             public void run()
             {
-                int count = 0;
+                Thread thread = Thread.currentThread();
+                TaskRunner self = TaskRunner.get(thread);
+                self.setAccordActiveExecutor(AbstractLockLoop.this);
+                setWrapped(self);
+
                 Task task = null;
                 while (true)
                 {
-                    if (DEBUG_EXECUTION) debug.onLock();
+                    lock(self);
-                    lock();
                     try
                     {
-                        if (DEBUG_EXECUTION) debug.onEnterLock();
                         enterLockLoop();
                         if (task != null)
                         {
                             Task tmp = task;
                             task = null;
-                            completeTaskExclusive(tmp);
+                            tmp.completeExclusiveNoExcept();
-                            clearRunning();
-                        }
-
-                        if (count >= YIELD_INTERVAL)
-                        {
-                            loopYieldExclusive();
-                            count = 0;
                         }
 
                         while (true)
                         {
                             task = pollWaitingToRunExclusive();
-
                             if (task != null)
                             {
-                                setRunning(task);
+                                if (!task.prepareExclusiveNoExcept())
-                                task.preRunExclusive();
+                                {
+                                    task = null;
+                                    continue;
+                                }
+
                                 if (DEBUG_EXECUTION) debug.onExitLock();
                                 exitLockLoop();
                                 break;
@@ -250,56 +295,22 @@ abstract class AccordExecutorAbstractLockLoop extends AccordExecutorAbstractLoop
                             awaitExclusive();
                             if (DEBUG_EXECUTION) debug.onEnterLock();
                             resumeLoop();
-                            count = 0;
                         }
                     }
                     catch (Throwable t)
                     {
-                        if (task != null)
+                        try { agent.onException(t); }
-                        {
+                        catch (Throwable t2) { }
-                            try { task.fail(t); }
-                            catch (Throwable t2) { t.addSuppressed(t2); }
-                            try { completeTaskExclusive(task); }
-                            catch (Throwable t2) { t.addSuppressed(t2); }
-                            try { agent.onException(t); }
-                            catch (Throwable t2) { /* nothing we can sensibly do after already reporting */ }
-                            task = null;
-                        }
-                        else
-                        {
-                            try { agent.onException(t); }
-                            catch (Throwable t2) { /* nothing we can sensibly do after already reporting */ }
-                        }
                         exitLockLoop();
                         continue;
                     }
                     finally
                     {
                         if (DEBUG_EXECUTION) debug.onExitLock();
-                        unlock();
+                        unlock(self);
                     }
 
-                    try
+                    task.runNoExcept(self);
-                    {
-                        ++count;
-                        task.runInternal();
-                    }
-                    catch (Throwable t)
-                    {
-                        try { task.fail(t); }
-                        catch (Throwable t2)
-                        {
-                            try
-                            {
-                                t2.addSuppressed(t);
-                                agent.onException(t2);
-                            }
-                            catch (Throwable t3)
-                            {
-                                // empty to ensure we definitely loop so we cleanup the task
-                            }
-                        }
-                    }
                 }
             }
         };

src/java/org/apache/cassandra/service/accord/execution/AbstractLoop.java

@@ -16,7 +16,7 @@
  * limitations under the License.
  */
 
-package org.apache.cassandra.service.accord;
+package org.apache.cassandra.service.accord.execution;
 
 import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
 import java.util.concurrent.locks.Lock;
@@ -27,28 +27,25 @@ import accord.utils.Invariants;
 
 import org.apache.cassandra.concurrent.DebuggableTask.DebuggableTaskRunner;
 
-abstract class AccordExecutorAbstractLoop extends AccordExecutor
+import static org.apache.cassandra.service.accord.execution.Task.State.REGISTERED;
-{
-    private volatile Task unqueued;
-    private static final AtomicReferenceFieldUpdater<AccordExecutorAbstractLoop, Task> unqueuedUpdater = AtomicReferenceFieldUpdater.newUpdater(AccordExecutorAbstractLoop.class, Task.class, "unqueued");
 
-    AccordExecutorAbstractLoop(Lock lock, int executorId, Agent agent)
+abstract class AbstractLoop extends AccordExecutor
+{
+    volatile Task unqueued;
+    static final AtomicReferenceFieldUpdater<AbstractLoop, Task> unqueuedUpdater = AtomicReferenceFieldUpdater.newUpdater(AbstractLoop.class, Task.class, "unqueued");
+
+    AbstractLoop(Lock lock, int executorId, Agent agent)
     {
         super(lock, executorId, agent);
     }
 
-    abstract AccordExecutorLoops loops();
+    abstract Loops loops();
 
     boolean hasUnqueued()
     {
         return unqueued != null;
     }
 
-    Task unqueued()
-    {
-        return unqueued;
-    }
-
     final Task push(Task submit)
     {
         Invariants.require(submit.next == null);
@@ -67,30 +64,18 @@ abstract class AccordExecutorAbstractLoop extends AccordExecutor
         if (hasUnqueued() || tasks > 0)
             return true;
 
-        lock();
+        TaskRunner self = TaskRunner.get();
+        lock(self);
         try
         {
             return hasUnqueued() || tasks > 0;
         }
         finally
         {
-            unlock();
+            unlock(self);
         }
     }
 
-    final void updateWaitingToRunExclusive()
-    {
-        drainUnqueuedExclusive();
-        super.updateWaitingToRunExclusive();
-    }
-
-    final void drainUnqueuedExclusive()
-    {
-        Task cur = Task.reverse(acquireUnqueuedExclusive());
-        while (cur != null)
-            cur = enqueueOneExclusive(cur);
-    }
-
     final Task acquireUnqueuedExclusive()
     {
         return unqueuedUpdater.getAndSet(this, null);
@@ -101,26 +86,16 @@ abstract class AccordExecutorAbstractLoop extends AccordExecutor
         Invariants.require(cur != null);
         Task next = cur.next;
         cur.next = null;
-        if (cur.isReadyToCleanup()) completeTaskExclusive(cur);
+        if (cur.compareTo(REGISTERED) < 0) cur.submitExclusiveNoExcept();
-        else cur.submitExclusive(this);
+        else cur.completeExclusiveNoExcept();
         return next;
     }
 
-    final Task enqueueOneCleanup(Task cur)
+    final Task destructiveNext(Task cur)
     {
         Invariants.require(cur != null);
         Task next = cur.next;
         cur.next = null;
-        completeTaskExclusive(cur);
-        return next;
-    }
-
-    final Task enqueueOneSubmit(Task cur)
-    {
-        Invariants.require(cur != null);
-        Task next = cur.next;
-        cur.next = null;
-        cur.submitExclusive(this);
         return next;
     }
 

src/java/org/apache/cassandra/service/accord/execution/AbstractSemiSyncSubmit.java

@@ -16,26 +16,26 @@
  * limitations under the License.
  */
 
-package org.apache.cassandra.service.accord;
+package org.apache.cassandra.service.accord.execution;
 
 import java.util.concurrent.locks.Lock;
+import java.util.function.Consumer;
+import java.util.function.Function;
 
 import accord.api.Agent;
-import accord.utils.QuadFunction;
-import accord.utils.QuintConsumer;
 
-abstract class AccordExecutorAbstractSemiSyncSubmit extends AccordExecutorAbstractLockLoop
+abstract class AbstractSemiSyncSubmit extends AbstractLockLoop
 {
-    AccordExecutorAbstractSemiSyncSubmit(Lock lock, int executorId, Agent agent)
+    AbstractSemiSyncSubmit(Lock lock, int executorId, Agent agent)
     {
         super(lock, executorId, agent);
     }
 
     abstract void awaitExclusive() throws InterruptedException;
 
-    <P1s, P1a, P2, P3, P4> void submitExternal(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Task> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4)
+    <P1> void submitExternal(Consumer<P1> sync, Function<P1, Task> async, P1 p1)
     {
-        if (push(async.apply(p1a, p2, p3, p4)) == null && !isInLoop())
+        if (push(async.apply(p1)) == null && !isInLoop())
             notifyWork();
     }
 }

src/java/org/apache/cassandra/service/accord/execution/AccordCache.java

@@ -15,7 +15,7 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.apache.cassandra.service.accord;
+package org.apache.cassandra.service.accord.execution;
 
 import java.io.IOException;
 import java.nio.ByteBuffer;
@@ -26,6 +26,7 @@ import java.util.IdentityHashMap;
 import java.util.Iterator;
 import java.util.List;
 import java.util.Map;
+import java.util.Objects;
 import java.util.concurrent.CopyOnWriteArrayList;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.locks.Lock;
@@ -44,6 +45,8 @@ import org.slf4j.LoggerFactory;
 
 import accord.api.RoutingKey;
 import accord.local.Command;
+import accord.local.RedundantBefore;
+import accord.local.SafeState;
 import accord.local.cfk.CommandsForKey;
 import accord.local.cfk.Serialize;
 import accord.primitives.Routable;
@@ -64,11 +67,17 @@ import org.apache.cassandra.io.util.DataInputBuffer;
 import org.apache.cassandra.metrics.AccordCacheMetrics;
 import org.apache.cassandra.metrics.LogLinearHistogram;
 import org.apache.cassandra.metrics.ShardedHitRate;
-import org.apache.cassandra.service.accord.AccordCache.Adapter.Shrink;
+import org.apache.cassandra.service.accord.AccordCommandStore;
-import org.apache.cassandra.service.accord.AccordCacheEntry.LoadExecutor;
+import org.apache.cassandra.service.accord.AccordKeyspace;
-import org.apache.cassandra.service.accord.AccordCacheEntry.Status;
+import org.apache.cassandra.service.accord.AccordObjectSizes;
-import org.apache.cassandra.service.accord.AccordSafeCommandsForKey.CommandsForKeyCacheEntry;
+import org.apache.cassandra.service.accord.OrderedKeys;
+import org.apache.cassandra.service.accord.api.TokenKey;
 import org.apache.cassandra.service.accord.events.CacheEvents;
+import org.apache.cassandra.service.accord.execution.AccordCache.Adapter.Shrink;
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry.LoadExecutor;
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry.Loading;
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry.Status;
+import org.apache.cassandra.service.accord.execution.SaferCommandsForKey.CommandsForKeyCacheEntry;
 import org.apache.cassandra.service.accord.journal.CommandChangeWriter;
 import org.apache.cassandra.service.accord.journal.CommandChanges;
 import org.apache.cassandra.service.accord.serializers.CommandSerializers;
@@ -79,9 +88,11 @@ import org.apache.cassandra.utils.ObjectSizes;
 
 import static accord.utils.Invariants.illegalState;
 import static accord.utils.Invariants.require;
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.EVICTED;
+import static org.apache.cassandra.service.accord.execution.AccordCacheEntry.AGE_MASK;
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.LOADED;
+import static org.apache.cassandra.service.accord.execution.AccordCacheEntry.GENERATION_MASK;
-import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.MODIFIED;
+import static org.apache.cassandra.service.accord.execution.AccordCacheEntry.Status.EVICTED;
+import static org.apache.cassandra.service.accord.execution.AccordCacheEntry.Status.LOADED;
+import static org.apache.cassandra.service.accord.execution.AccordCacheEntry.Status.MODIFIED;
 
 /**
  * Cache for AccordCommand and AccordCommandsForKey, available memory is shared between the two object types.
@@ -91,22 +102,22 @@ import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.MODIFI
  *
  * TODO (required): we only iterate over unreferenced entries
  */
-public class AccordCache implements CacheSize
+public class AccordCache
 {
     private static final Logger logger = LoggerFactory.getLogger(AccordCache.class);
     private static final NoSpamLogStatement evictNoEvict = NoSpamLogger.getStatement(logger, "Found and expired {} marked no evict, with age {}, exceeding its expected max age of {}", 1L, TimeUnit.MINUTES);
 
     // Debug mode to verify that loading from journal + system tables results in
     // functionally identical (or superceding) command to the one we've just evicted.
-    private static boolean VALIDATE_LOAD_ON_EVICT = false;
+    private static boolean DEBUG_VALIDATE_LOAD_ON_EVICT = false;
 
     @VisibleForTesting
     public static void validateLoadOnEvict(boolean value)
     {
-        VALIDATE_LOAD_ON_EVICT = value;
+        DEBUG_VALIDATE_LOAD_ON_EVICT = value;
     }
 
-    public interface Adapter<K, V, S>
+    public interface Adapter<K, V, S extends SafeState<V> & SaferState<K, V, S>>
     {
         enum Shrink { EVICT, DONE, PERFORM_WITHOUT_LOCK }
 
@@ -122,10 +133,10 @@ public class AccordCache implements CacheSize
         long estimateHeapSize(V value);
         long estimateShrunkHeapSize(Object shrunk);
         boolean validate(AccordCommandStore commandStore, K key, V value);
-        S safeRef(AccordCacheEntry<K, V> node);
+        S safeRef(AccordCacheEntry<K, V, S> node);
         default Comparator<K> keyComparator() { return null; }
 
-        default AccordCacheEntry<K, V> newEntry(K key, AccordCache.Type<K, V, ?>.Instance owner)
+        default AccordCacheEntry<K, V, S> newEntry(K key, AccordCache.Type<K, V, S>.Instance owner)
         {
             return AccordCacheEntry.createReadyToLoad(key, owner);
         }
@@ -151,8 +162,8 @@ public class AccordCache implements CacheSize
 
     private final List<Type<?, ?, ?>> types = new CopyOnWriteArrayList<>();
     final AccordCacheEntry.SaveExecutor saveExecutor;
-    private final IntrusiveLinkedList<AccordCacheEntry<?,?>> evictQueue = new IntrusiveLinkedList<>();
+    private final IntrusiveLinkedList<AccordCacheEntry<?,?, ?>> evictQueue = new IntrusiveLinkedList<>();
-    private final IntrusiveLinkedList<AccordCacheEntry<?,?>> noEvictQueue = new IntrusiveLinkedList<>();
+    private final IntrusiveLinkedList<AccordCacheEntry<?,?, ?>> noEvictQueue = new IntrusiveLinkedList<>();
 
     private long unreferencedBytes;
     private int unreferenced;
@@ -169,8 +180,8 @@ public class AccordCache implements CacheSize
     }
 
     // note: only affects current contents after lock is released
-    @Override
+    // should NOT be called directly, should be called via AccordExecutor.setCapacity, so it may refresh its derived values
-    public void setCapacity(long sizeInBytes)
+    void setCapacity(long sizeInBytes)
     {
         maxSizeInBytes = sizeInBytes;
         tryShrinkOrEvict = true;
@@ -181,7 +192,6 @@ public class AccordCache implements CacheSize
         this.shrinkingOn = shrinkingOn;
     }
 
-    @Override
     public long capacity()
     {
         return maxSizeInBytes;
@@ -190,18 +200,18 @@ public class AccordCache implements CacheSize
     /**
      * Make sure we don't have any items lingering too long in the no evict queue, to avoid cache memory leaks
      */
-    void processNoEvictQueue()
+    public void processNoEvictQueue()
     {
-        noEvictGeneration = (noEvictGeneration + 1) & 0xffff;
+        noEvictGeneration = (noEvictGeneration + 1) & GENERATION_MASK;
         if (noEvictQueue.isEmpty())
             return;
 
-        Iterator<AccordCacheEntry<?, ?>> iter = noEvictQueue.iterator();
+        Iterator<AccordCacheEntry<?, ?, ?>> iter = noEvictQueue.iterator();
         int skipCount = 3;
         while (skipCount > 0 && iter.hasNext())
         {
-            AccordCacheEntry<?, ?> entry = iter.next();
+            AccordCacheEntry<?, ?, ?> entry = iter.next();
-            int age = (noEvictGeneration - entry.noEvictGeneration()) & 0xffff;
+            int age = (noEvictGeneration - entry.noEvictGeneration()) & GENERATION_MASK;
             if (age >= entry.noEvictMaxAge())
             {
                 evictNoEvict.warn(entry, age, entry.noEvictMaxAge());
@@ -219,21 +229,21 @@ public class AccordCache implements CacheSize
      * Roughly respects LRU semantics when evicting. Might consider prioritising keeping MODIFIED nodes around
      * for longer to maximise the chances of hitting system tables fewer times (or not at all).
      */
-    void tryShrinkOrEvict(Lock lock)
+    public void tryShrinkOrEvict(Lock lock)
     {
         if (!tryShrinkOrEvict)
             return;
 
         while (bytesCached > maxSizeInBytes && !evictQueue.isEmpty())
         {
-            AccordCacheEntry<?, ?> node = evictQueue.peek();
+            AccordCacheEntry<?, ?, ?> node = evictQueue.peek();
             shrinkOrEvict(lock, node);
         }
         tryShrinkOrEvict = false;
     }
 
     @VisibleForTesting
-    private <K, V> void shrinkOrEvict(Lock lock, AccordCacheEntry<K, V> node)
+    private <K, V> void shrinkOrEvict(Lock lock, AccordCacheEntry<K, V, ?> node)
     {
         require(node.references() == 0);
 
@@ -244,7 +254,7 @@ public class AccordCache implements CacheSize
         }
         else
         {
-            IntrusiveLinkedList<AccordCacheEntry<?,?>> queue;
+            IntrusiveLinkedList<AccordCacheEntry<?,?, ?>> queue;
             queue = node.isNoEvict() ? noEvictQueue : evictQueue;
             node.unlink();
             if (shrink == Shrink.DONE)
@@ -272,7 +282,7 @@ public class AccordCache implements CacheSize
     }
 
     @VisibleForTesting
-    public <K, V> void tryEvict(AccordCacheEntry<K, V> node)
+    public <K, V> void tryEvict(AccordCacheEntry<K, V, ?> node)
     {
         require(node.references() == 0);
 
@@ -290,7 +300,6 @@ public class AccordCache implements CacheSize
             case LOADING:
                 node.loading().loading.cancel();
             case WAITING_TO_LOAD:
-                Invariants.paranoid(node.loadingOrWaiting().waiters == null);
             case LOADED:
                 node.unlink();
                 evict(node, true);
@@ -305,7 +314,7 @@ public class AccordCache implements CacheSize
         }
     }
 
-    public void saveWhenReadyExclusive(AccordCacheEntry<?, ?> entry, Runnable onSuccess)
+    public void saveWhenReadyExclusive(AccordCacheEntry<?, ?, ?> entry, Runnable onSuccess)
     {
         if (!entry.isSavingOrWaiting() && !entry.saveWhenReady())
             onSuccess.run();
@@ -313,7 +322,7 @@ public class AccordCache implements CacheSize
             entry.savingOrWaitingToSave().identity.onSuccess(onSuccess);
     }
 
-    private <K> void evict(AccordCacheEntry<K, ?> node, boolean updateUnreferenced)
+    private <K> void evict(AccordCacheEntry<K, ?, ?> node, boolean updateUnreferenced)
     {
         if (logger.isTraceEnabled())
             logger.trace("Evicting {}", node);
@@ -333,29 +342,28 @@ public class AccordCache implements CacheSize
         --parent.size;
 
         // TODO (expected): use listeners
-        if (node.status() == LOADED && VALIDATE_LOAD_ON_EVICT)
+        if (node.status() == LOADED && DEBUG_VALIDATE_LOAD_ON_EVICT)
             owner.validateLoadEvicted(node);
 
-        AccordCacheEntry<K, ?> self = node.owner.remove(node.key());
+        AccordCacheEntry<K, ?, ?> self = node.owner.remove(node.key());
         Invariants.require(self.references() == 0);
         require(self == node, "Leaked node detected; was attempting to remove %s but cache had %s", node, self);
         node.notifyListeners(Listener::onEvict);
         node.evicted();
     }
 
-    <P1, P2, K, V> Collection<AccordTask<?>> load(LoadExecutor<P1, P2> loadExecutor, P1 p1, P2 p2, AccordCacheEntry<K, V> node)
+    <P1, P2, K, V> Loading load(LoadExecutor<P1, P2> loadExecutor, P1 p1, P2 p2, AccordCacheEntry<K, V, ?> node)
     {
-        Type<K, V, ?> parent = node.owner.parent();
+        return node.load(loadExecutor, p1, p2);
-        return node.load(loadExecutor, p1, p2).waiters();
     }
 
-    <K, V> void loaded(AccordCacheEntry<K, V> node, V value)
+    <K, V> void loaded(AccordCacheEntry<K, V, ?> node, V value)
     {
         node.loaded(value);
         node.notifyListeners(Listener::onUpdate);
     }
 
-    <K, V> void failedToLoad(AccordCacheEntry<K, V> node)
+    <K, V> void failedToLoad(AccordCacheEntry<K, V, ?> node)
     {
         Invariants.require(node.references() == 0);
         if (node.isUnqueued())
@@ -368,38 +376,38 @@ public class AccordCache implements CacheSize
         evict(node, true);
     }
 
-    <K, V> void saved(AccordCacheEntry<K, V> node, Object identity, Throwable fail)
+    <K, V> void saved(AccordCacheEntry<K, V, ?> node, Object identity, Throwable fail)
     {
         if (node.saved(identity, fail) && node.references() == 0 && node.isUnqueued())
             evictQueue.addFirst(node); // add to front since we have just saved, so we were eligible for eviction
     }
 
-    public <K, V, S extends AccordSafeState<K, V>> void release(S safeRef, AccordTask<?> owner)
+    public <K, V, S extends SafeState<V> & SaferState<K, V, S>> void release(S safeRef, SafeTask<?> owner)
     {
         safeRef.global().owner.release(safeRef, owner);
     }
 
-    public <K, V, S extends AccordSafeState<K, V>> Type<K, V, S> newType(Class<K> keyClass, Adapter<K, V, S> adapter, AccordCacheMetrics.Shard metrics)
+    public <K, V, S extends SafeState<V> & SaferState<K, V, S>> Type<K, V, S> newType(Class<K> keyClass, Adapter<K, V, S> adapter, AccordCacheMetrics.Shard metrics)
     {
         Type<K, V, S> instance = new Type<>(keyClass, adapter, metrics);
         types.add(instance);
         return instance;
     }
 
-    public <K, V, S extends AccordSafeState<K, V>> Type<K, V, S> newType(
+    public <K, V, S extends SafeState<V> & SaferState<K, V, S>> Type<K, V, S> newType(
         Class<K> keyClass,
         BiFunction<AccordCommandStore, K, V> loadFunction,
         QuadFunction<AccordCommandStore, K, V, Object, Runnable> saveFunction,
         Function<V, V> quickShrink,
         TriFunction<AccordCommandStore, K, V, Boolean> validateFunction,
         ToLongFunction<V> heapEstimator,
-        Function<AccordCacheEntry<K, V>, S> safeRefFactory,
+        Function<AccordCacheEntry<K, V, S>, S> safeRefFactory,
         AccordCacheMetrics.Shard metrics)
     {
         return newType(keyClass, loadFunction, saveFunction, quickShrink, (i, j) -> j, (c, i, j) -> (V)j, validateFunction, heapEstimator, i -> 0, safeRefFactory, metrics);
     }
 
-    public <K, V, S extends AccordSafeState<K, V>> Type<K, V, S> newType(
+    public <K, V, S extends SafeState<V> & SaferState<K, V, S>> Type<K, V, S> newType(
         Class<K> keyClass,
         BiFunction<AccordCommandStore, K, V> loadFunction,
         QuadFunction<AccordCommandStore, K, V, Object, Runnable> saveFunction,
@@ -409,7 +417,7 @@ public class AccordCache implements CacheSize
         TriFunction<AccordCommandStore, K, V, Boolean> validateFunction,
         ToLongFunction<V> heapEstimator,
         ToLongFunction<Object> shrunkHeapEstimator,
-        Function<AccordCacheEntry<K, V>, S> safeRefFactory,
+        Function<AccordCacheEntry<K, V, S>, S> safeRefFactory,
         AccordCacheMetrics.Shard metrics)
     {
         return newType(keyClass, new FunctionalAdapter<>(loadFunction, saveFunction, quickShrink,
@@ -426,18 +434,18 @@ public class AccordCache implements CacheSize
 
     public interface Listener<K, V>
     {
-        default void onAdd(AccordCacheEntry<K, V> state) {}
+        default void onAdd(AccordCacheEntry<K, V, ?> state) {}
-        default void onUpdate(AccordCacheEntry<K, V> state) {}
+        default void onUpdate(AccordCacheEntry<K, V, ?> state) {}
-        default void onEvict(AccordCacheEntry<K, V> state) {}
+        default void onEvict(AccordCacheEntry<K, V, ?> state) {}
     }
 
-    public class Type<K, V, S extends AccordSafeState<K, V>> implements CacheSize
+    public class Type<K, V, S extends SafeState<V> & SaferState<K, V, S>> implements CacheSize
     {
-        public class Instance implements Iterable<AccordCacheEntry<K, V>>
+        public class Instance implements Iterable<AccordCacheEntry<K, V, S>>
         {
             final AccordCommandStore commandStore;
             // TODO (desired): don't need to store key separately as stored in node; ideally use a hash set that allows us to get the current entry
-            private final Map<K, AccordCacheEntry<K, V>> cache = new Object2ObjectHashMap<>();
+            private final Map<K, AccordCacheEntry<K, V, S>> cache = new Object2ObjectHashMap<>();
             private List<Listener<K, V>> listeners = null;
             // TODO (expected): update this after releasing the lock
             private OrderedKeys<K> orderedKeys;
@@ -447,50 +455,51 @@ public class AccordCache implements CacheSize
                 this.commandStore = commandStore;
             }
 
-            public S acquire(K key)
+            public final S acquire(K key)
             {
-                AccordCacheEntry<K, V> node = acquire(key, false);
+                AccordCacheEntry<K, V, S> node = acquire(key, false);
                 return adapter.safeRef(node);
             }
 
-            public S acquireIfLoaded(K key)
+            public final S acquireIfLoadedAndPermitted(K key)
             {
-                AccordCacheEntry<K, V> node = acquire(key, true);
+                AccordCacheEntry<K, V, S> node = acquire(key, true);
                 if (node == null)
                     return null;
                 return adapter.safeRef(node);
             }
 
-            public S acquire(AccordCacheEntry<K, V> node)
+            public final S acquire(AccordCacheEntry<K, V, S> node)
             {
                 Invariants.require(node.owner == this);
                 acquireExisting(node, false);
                 return adapter.safeRef(node);
             }
 
-            public void recordPreAcquired(AccordSafeState<K, V> ref)
+            public final void recordPreAcquired(AccordCacheEntry<K, V, S> entry)
             {
-                Invariants.require(ref.global().owner == this);
+                Invariants.require(entry.owner == this);
-                incrementCacheHits();
+                if (entry.isLoaded()) incrementCacheHits();
+                else incrementCacheMisses();
             }
 
-            private AccordCacheEntry<K, V> acquire(K key, boolean onlyIfLoaded)
+            private AccordCacheEntry<K, V, S> acquire(K key, boolean onlyIfLoadedAndPermitted)
             {
-                AccordCacheEntry<K, V> node = cache.get(key);
+                AccordCacheEntry<K, V, S> node = cache.get(key);
                 return node == null
-                       ? acquireAbsent(key, onlyIfLoaded)
+                       ? acquireAbsent(key, onlyIfLoadedAndPermitted)
-                       : acquireExisting(node, onlyIfLoaded);
+                       : acquireExisting(node, onlyIfLoadedAndPermitted);
             }
 
             /*
              * Can only return a LOADING Node (or null)
              */
-            private AccordCacheEntry<K, V> acquireAbsent(K key, boolean onlyIfLoaded)
+            private AccordCacheEntry<K, V, S> acquireAbsent(K key, boolean onlyIfLoaded)
             {
                 incrementCacheMisses();
                 if (onlyIfLoaded)
                     return null;
-                AccordCacheEntry<K, V> node = adapter.newEntry(key, this);
+                AccordCacheEntry<K, V, S> node = adapter.newEntry(key, this);
                 node.increment();
 
                 Object prev = cache.put(key, node);
@@ -507,7 +516,7 @@ public class AccordCache implements CacheSize
             /*
              * Can't return EVICTED or INITIALIZED
              */
-            private AccordCacheEntry<K, V> acquireExisting(AccordCacheEntry<K, V> node, boolean onlyIfLoaded)
+            private AccordCacheEntry<K, V, S> acquireExisting(AccordCacheEntry<K, V, S> node, boolean onlyIfLoadedAndPermitted)
             {
                 boolean isLoaded = node.isLoaded();
                 if (isLoaded)
@@ -515,8 +524,11 @@ public class AccordCache implements CacheSize
                 else
                     incrementCacheMisses();
 
-                if (onlyIfLoaded && !isLoaded)
+                if (onlyIfLoadedAndPermitted)
-                    return null;
+                {
+                    if (!isLoaded || node.hasFifoOrLocked())
+                        return null;
+                }
 
                 if (node.increment() == 1)
                 {
@@ -528,21 +540,21 @@ public class AccordCache implements CacheSize
                 return node;
             }
 
-            public void release(AccordSafeState<K, V> safeRef, AccordTask<?> owner)
+            public final void release(S safeRef, SafeTask<?> owner)
             {
                 K key = safeRef.global().key();
                 logger.trace("Releasing resources for {}: {}", key, safeRef);
 
-                AccordCacheEntry<K, V> node = cache.get(key);
+                AccordCacheEntry<K, V, ?> node = cache.get(key);
 
-                require(!safeRef.isUnsafe());
+                require(!safeRef.isReleased());
                 require(safeRef.global() != null, "safeRef node is null for %s", key);
                 require(safeRef.global() == node, "safeRef node not in map: %s != %s", safeRef.global(), node);
                 require(node.references() > 0, "references (%d) are zero for %s (%s)", node.references(), key, node);
                 require(node.isUnqueued());
 
                 boolean evict = false;
-                if (safeRef.hasUpdate())
+                if (safeRef.isModified())
                 {
                     V update = safeRef.current();
                     if (update != null)
@@ -556,15 +568,11 @@ public class AccordCache implements CacheSize
                     }
                     node.notifyListeners(Listener::onUpdate);
                 }
-                else if (node.isLoadingOrWaiting())
-                {
-                    node.loadingOrWaiting().remove(owner);
-                }
                 else
                 {
                     evict = node.is(LOADED) && node.isNull();
                 }
-                safeRef.markUnsafe();
+                node.remove(owner, safeRef.setReleased());
 
                 if (node.decrement() == 0)
                 {
@@ -597,9 +605,9 @@ public class AccordCache implements CacheSize
                 tryShrinkOrEvict = true;
             }
 
-            AccordCacheEntry<K, ?> remove(K key)
+            final AccordCacheEntry<K, ?, ?> remove(K key)
             {
-                AccordCacheEntry<K, ?> result = cache.remove(key);
+                AccordCacheEntry<K, ?, ?> result = cache.remove(key);
                 if (orderedKeys != null && result != null)
                     orderedKeys.remove(key);
                 return result;
@@ -610,7 +618,7 @@ public class AccordCache implements CacheSize
                 return Type.this;
             }
 
-            public Iterable<K> keysBetween(K start, boolean startInclusive, K end, boolean endInclusive)
+            public final Iterable<K> keysBetween(K start, boolean startInclusive, K end, boolean endInclusive)
             {
                 if (orderedKeys == null)
                     orderedKeys = new OrderedKeys<>(adapter.keyComparator(), cache.keySet());
@@ -619,15 +627,15 @@ public class AccordCache implements CacheSize
             }
 
             @Override
-            public Iterator<AccordCacheEntry<K, V>> iterator()
+            public final Iterator<AccordCacheEntry<K, V, S>> iterator()
             {
                 return cache.values().iterator();
             }
 
-            void validateLoadEvicted(AccordCacheEntry<?, ?> node)
+            final void validateLoadEvicted(AccordCacheEntry<?, ?, ?> node)
             {
                 @SuppressWarnings("unchecked")
-                AccordCacheEntry<K, V> state = (AccordCacheEntry<K, V>) node;
+                AccordCacheEntry<K, V, ?> state = (AccordCacheEntry<K, V, ?>) node;
                 K key = state.key();
                 V evicted = state.tryGetFull();
                 if (evicted == null)
@@ -650,46 +658,46 @@ public class AccordCache implements CacheSize
             }
 
             @VisibleForTesting
-            public AccordCacheEntry<K, V> getUnsafe(K key)
+            public final AccordCacheEntry<K, V, ?> getUnsafe(K key)
             {
                 return cache.get(key);
             }
 
             @VisibleForTesting
-            public boolean isReferenced(K key)
+            public final boolean isReferenced(K key)
             {
-                AccordCacheEntry<K, V> node = cache.get(key);
+                AccordCacheEntry<K, V, ?> node = cache.get(key);
                 return node != null && node.references() > 0;
             }
 
             @VisibleForTesting
-            boolean keyIsReferenced(Object key, Class<? extends AccordSafeState<?, ?>> valClass)
+            final boolean keyIsReferenced(Object key, Class<? extends SaferState<?, ?, S>> valClass)
             {
-                AccordCacheEntry<?, ?> node = cache.get(key);
+                AccordCacheEntry<?, ?, ?> node = cache.get(key);
                 return node != null && node.references() > 0;
             }
 
             @VisibleForTesting
-            boolean keyIsCached(Object key, Class<? extends AccordSafeState<?, ?>> valClass)
+            final boolean keyIsCached(Object key, Class<? extends SaferState<?, ?, S>> valClass)
             {
-                AccordCacheEntry<?, ?> node = cache.get(key);
+                AccordCacheEntry<?, ?, ?> node = cache.get(key);
                 return node != null;
             }
 
             @VisibleForTesting
-            int references(Object key, Class<? extends AccordSafeState<?, ?>> valClass)
+            final int references(Object key, Class<? extends SaferState<?, ?, S>> valClass)
             {
-                AccordCacheEntry<?, ?> node = cache.get(key);
+                AccordCacheEntry<?, ?, ?> node = cache.get(key);
                 return node != null ? node.references() : 0;
             }
 
-            void notifyListeners(BiConsumer<Listener<K, V>, AccordCacheEntry<K, V>> notify, AccordCacheEntry<K, V> node)
+            final void notifyListeners(BiConsumer<Listener<K, V>, AccordCacheEntry<K, V, ?>> notify, AccordCacheEntry<K, V, ?> node)
             {
                 notifyListeners(listeners, notify, node);
                 notifyListeners(typeListeners, notify, node);
             }
 
-            void notifyListeners(List<Listener<K, V>> listeners, BiConsumer<Listener<K, V>, AccordCacheEntry<K, V>> notify, AccordCacheEntry<K, V> node)
+            final void notifyListeners(List<Listener<K, V>> listeners, BiConsumer<Listener<K, V>, AccordCacheEntry<K, V, ?>> notify, AccordCacheEntry<K, V, ?> node)
             {
                 if (listeners != null)
                 {
@@ -699,20 +707,20 @@ public class AccordCache implements CacheSize
                 }
             }
 
-            public void register(Listener<K, V> l)
+            public final void register(Listener<K, V> l)
             {
                 if (listeners == null)
                     listeners = new ArrayList<>();
                 listeners.add(l);
             }
 
-            public void unregister(Listener<K, V> l)
+            public final void unregister(Listener<K, V> l)
             {
                 if (!tryUnregister(l))
                     throw illegalState("Listener was not registered");
             }
 
-            public boolean tryUnregister(Listener<K, V> l)
+            public final boolean tryUnregister(Listener<K, V> l)
             {
                 if (listeners == null || !listeners.remove(l))
                     return false;
@@ -720,9 +728,23 @@ public class AccordCache implements CacheSize
                     listeners = null;
                 return true;
             }
+
+            final boolean isCommandsForKey()
+            {
+                return getClass() == KeyInstance.class;
+            }
         }
 
-        private final Class<K> keyClass;
+        // KeyInstance exists to provide us slightly easier discrimination about the Type an AccordCacheEntry is associated with
+        public final class KeyInstance extends Instance
+        {
+            public KeyInstance(AccordCommandStore commandStore)
+            {
+                super(commandStore);
+            }
+        }
+
+        private final Class<K> keyClass; // type of key, useful primarily for toString(), but also piggyback for deciding Instance type
         private Adapter<K, V, S> adapter;
         private long bytesCached;
         private int size;
@@ -735,6 +757,8 @@ public class AccordCache implements CacheSize
 
         public Type(Class<K> keyClass, Adapter<K, V, S> adapter, AccordCacheMetrics.Shard metrics)
         {
+            // Integer and String permitted for testing, but the Invariant exists only to enforce that we construct the right kind of Instance
+            Invariants.require(keyClass == RoutingKey.class || keyClass == TxnId.class || keyClass == String.class || keyClass == Integer.class);
             this.keyClass = keyClass;
             this.adapter = adapter;
             this.objectSize = metrics.objectSize;
@@ -750,9 +774,9 @@ public class AccordCache implements CacheSize
         }
 
         // can be safely garbage collected if empty
-        Instance newInstance(AccordCommandStore commandStore)
+        public Instance newInstance(AccordCommandStore commandStore)
         {
-            return new Instance(commandStore);
+            return keyClass == RoutingKey.class ? new KeyInstance(commandStore) : new Instance(commandStore);
         }
 
         private void incrementCacheHits()
@@ -868,17 +892,17 @@ public class AccordCache implements CacheSize
     }
 
     @VisibleForTesting
-    AccordCacheEntry<?, ?> head()
+    AccordCacheEntry<?, ?, ?> head()
     {
-        Iterator<AccordCacheEntry<?, ?>> iter = evictQueue.iterator();
+        Iterator<AccordCacheEntry<?, ?, ?>> iter = evictQueue.iterator();
         return iter.hasNext() ? iter.next() : null;
     }
 
     @VisibleForTesting
-    AccordCacheEntry<?, ?> tail()
+    AccordCacheEntry<?, ?, ?> tail()
     {
-        AccordCacheEntry<?,?> last = null;
+        AccordCacheEntry<?,?, ?> last = null;
-        Iterator<AccordCacheEntry<?, ?>> iter = evictQueue.iterator();
+        Iterator<AccordCacheEntry<?, ?, ?>> iter = evictQueue.iterator();
         while (iter.hasNext())
             last = iter.next();
         return last;
@@ -889,7 +913,7 @@ public class AccordCache implements CacheSize
         return size() == 0;
     }
 
-    Iterable<AccordCacheEntry<?, ?>> evictionQueue()
+    public Iterable<AccordCacheEntry<?, ?, ?>> evictionQueue()
     {
         return evictQueue::iterator;
     }
@@ -920,14 +944,12 @@ public class AccordCache implements CacheSize
         return unreferencedBytes;
     }
 
-    @Override
+    int size()
-    public int size()
     {
         return cacheSize();
     }
 
-    @Override
+    long weightedSize()
-    public long weightedSize()
     {
         return bytesCached;
     }
@@ -938,10 +960,10 @@ public class AccordCache implements CacheSize
             return;
 
         type.register(new AccordCache.Listener<>() {
-            private final IdentityHashMap<AccordCacheEntry<?, ?>, CacheEvents.Evict> pendingEvicts = new IdentityHashMap<>();
+            private final IdentityHashMap<AccordCacheEntry<?, ?, ?>, CacheEvents.Evict> pendingEvicts = new IdentityHashMap<>();
 
             @Override
-            public void onAdd(AccordCacheEntry<K, V> state)
+            public void onAdd(AccordCacheEntry<K, V, ?> state)
             {
                 CacheEvents.Add add = new CacheEvents.Add();
                 CacheEvents.Evict evict = new CacheEvents.Evict();
@@ -958,7 +980,7 @@ public class AccordCache implements CacheSize
             }
 
             @Override
-            public void onEvict(AccordCacheEntry<K, V> state)
+            public void onEvict(AccordCacheEntry<K, V, ?> state)
             {
                 CacheEvents.Evict event = pendingEvicts.remove(state);
                 if (event == null) return;
@@ -968,7 +990,7 @@ public class AccordCache implements CacheSize
         });
     }
 
-    private static void updateMutable(AccordCache.Type<?, ?, ?> type, AccordCacheEntry<?, ?> state, CacheEvents event)
+    private static void updateMutable(AccordCache.Type<?, ?, ?> type, AccordCacheEntry<?, ?, ?> state, CacheEvents event)
     {
         event.status = state.status().name();
 
@@ -988,7 +1010,7 @@ public class AccordCache implements CacheSize
         event.update();
     }
 
-    static class FunctionalAdapter<K, V, S> implements Adapter<K, V, S>
+    static class FunctionalAdapter<K, V, S extends SafeState<V> & SaferState<K, V, S>> implements Adapter<K, V, S>
     {
         final BiFunction<AccordCommandStore, K, V> load;
         final QuadFunction<AccordCommandStore, K, V, Object, Runnable> save;
@@ -998,8 +1020,8 @@ public class AccordCache implements CacheSize
         final TriFunction<AccordCommandStore, K, V, Boolean> validate;
         final ToLongFunction<V> estimateHeapSize;
         final ToLongFunction<Object> estimateShrunkHeapSize;
-        final Function<AccordCacheEntry<K, V>, S> newSafeRef;
+        final Function<AccordCacheEntry<K, V, S>, S> newSafeRef;
-        final BiFunction<K, AccordCache.Type<K, V, ?>.Instance, AccordCacheEntry<K, V>> newNode;
+        final BiFunction<K, AccordCache.Type<K, V, S>.Instance, AccordCacheEntry<K, V, S>> newNode;
 
         FunctionalAdapter(BiFunction<AccordCommandStore, K, V> load,
                           QuadFunction<AccordCommandStore, K, V, Object, Runnable> save,
@@ -1008,8 +1030,8 @@ public class AccordCache implements CacheSize
                           TriFunction<AccordCommandStore, K, V, Boolean> validate,
                           ToLongFunction<V> estimateHeapSize,
                           ToLongFunction<Object> estimateShrunkHeapSize,
-                          Function<AccordCacheEntry<K, V>, S> newSafeRef,
+                          Function<AccordCacheEntry<K, V, S>, S> newSafeRef,
-                          BiFunction<K, Type<K, V, ?>.Instance, AccordCacheEntry<K, V>> newNode)
+                          BiFunction<K, Type<K, V, S>.Instance, AccordCacheEntry<K, V, S>> newNode)
         {
             this.load = load;
             this.save = save;
@@ -1083,13 +1105,13 @@ public class AccordCache implements CacheSize
         }
 
         @Override
-        public S safeRef(AccordCacheEntry<K, V> node)
+        public S safeRef(AccordCacheEntry<K, V, S> node)
         {
             return newSafeRef.apply(node);
         }
 
         @Override
-        public AccordCacheEntry<K, V> newEntry(K key, Type<K, V, ?>.Instance owner)
+        public AccordCacheEntry<K, V, S> newEntry(K key, Type<K, V, S>.Instance owner)
         {
             return newNode.apply(key, owner);
         }
@@ -1101,7 +1123,7 @@ public class AccordCache implements CacheSize
         }
     }
 
-    static class SettableWrapper<K, V, S> extends FunctionalAdapter<K, V, S>
+    static class SettableWrapper<K, V, S extends SafeState<V> & SaferState<K, V, S>> extends FunctionalAdapter<K, V, S>
     {
         volatile BiFunction<AccordCommandStore, K, V> load;
 
@@ -1111,9 +1133,9 @@ public class AccordCache implements CacheSize
             this.load = super.load;
         }
 
-        public static <K, V> Adapter<K, V, ?> loadOnly(BiFunction<AccordCommandStore, K, V> load)
+        public static <K, V, S extends SafeState<V> & SaferState<K, V, S>> Adapter<K, V, S> loadOnly(BiFunction<AccordCommandStore, K, V> load)
         {
-            SettableWrapper<K, V, ?> result = new SettableWrapper<>(new NoOpAdapter<>());
+            SettableWrapper<K, V, S> result = new SettableWrapper<>(new NoOpAdapter<K, V, S>());
             result.load = load;
             return result;
         }
@@ -1125,7 +1147,7 @@ public class AccordCache implements CacheSize
         }
     }
 
-    static class NoOpAdapter<K, V, S> implements Adapter<K, V, S>
+    static class NoOpAdapter<K, V, S extends SafeState<V> & SaferState<K, V, S>> implements Adapter<K, V, S>
     {
         @Override public V load(AccordCommandStore commandStore, K key) { return null; }
         @Override public Runnable save(AccordCommandStore commandStore, K key, @Nullable V value, @Nullable Object shrunk) { return null; }
@@ -1136,10 +1158,10 @@ public class AccordCache implements CacheSize
         @Override public long estimateHeapSize(V value) { return 0; }
         @Override public long estimateShrunkHeapSize(Object shrunk) { return 0; }
         @Override public boolean validate(AccordCommandStore commandStore, K key, V value) { return false; }
-        @Override public S safeRef(AccordCacheEntry<K, V> node) { return null; }
+        @Override public S safeRef(AccordCacheEntry<K, V, S> node) { return null; }
     }
 
-    public static class CommandsForKeyAdapter implements Adapter<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>
+    public static class CommandsForKeyAdapter implements Adapter<RoutingKey, CommandsForKey, SaferCommandsForKey>
     {
         public static final CommandsForKeyAdapter CFK_ADAPTER = new CommandsForKeyAdapter();
         private static int SHRINK_WITHOUT_LOCK = -1;
@@ -1149,7 +1171,13 @@ public class AccordCache implements CacheSize
         @Override
         public CommandsForKey load(AccordCommandStore commandStore, RoutingKey key)
         {
-            return commandStore.loadCommandsForKey(key);
+            CommandsForKey cfk = AccordKeyspace.CommandsForKeyAccessor.load(commandStore.id(), (TokenKey) key);
+            if (cfk == null)
+                return null;
+            RedundantBefore.QuickBounds bounds = commandStore.safeGetRedundantBefore().get(key);
+            if (!Invariants.expect(bounds != null, "No RedundantBefore information found when loading key %s", key))
+                return cfk;
+            return cfk.withCleanCfkBeforeAtLeast(bounds.cleanCfkBefore(), false);
         }
 
         @Override
@@ -1160,7 +1188,7 @@ public class AccordCache implements CacheSize
                 ByteBuffer bb = (ByteBuffer)serialized;
                 serialized = bb.duplicate().position(prefixBytes(bb));
             }
-            return commandStore.saveCommandsForKey(key, value, serialized);
+            return AccordKeyspace.CommandsForKeyAccessor.systemTableUpdater(commandStore.id(), (TokenKey) key, value, serialized);
         }
 
         @Override
@@ -1236,13 +1264,17 @@ public class AccordCache implements CacheSize
         @Override
         public boolean validate(AccordCommandStore commandStore, RoutingKey key, CommandsForKey value)
         {
-            return commandStore.validateCommandsForKey(key, value);
+            if (!Invariants.isParanoid())
+                return true;
+
+            CommandsForKey reloaded = AccordKeyspace.CommandsForKeyAccessor.load(commandStore.id(), (TokenKey) key);
+            return Objects.equals(value, reloaded);
         }
 
         @Override
-        public AccordSafeCommandsForKey safeRef(AccordCacheEntry<RoutingKey, CommandsForKey> node)
+        public SaferCommandsForKey safeRef(AccordCacheEntry<RoutingKey, CommandsForKey, SaferCommandsForKey> node)
         {
-            return new AccordSafeCommandsForKey(node);
+            return new SaferCommandsForKey(node);
         }
 
         @Override
@@ -1252,7 +1284,7 @@ public class AccordCache implements CacheSize
         }
 
         @Override
-        public AccordCacheEntry<RoutingKey, CommandsForKey> newEntry(RoutingKey key, Type<RoutingKey, CommandsForKey, ?>.Instance owner)
+        public AccordCacheEntry<RoutingKey, CommandsForKey, SaferCommandsForKey> newEntry(RoutingKey key, Type<RoutingKey, CommandsForKey, SaferCommandsForKey>.Instance owner)
         {
             CommandsForKeyCacheEntry entry = new CommandsForKeyCacheEntry(key, owner);
             entry.readyToLoad();
@@ -1260,7 +1292,7 @@ public class AccordCache implements CacheSize
         }
     }
 
-    public static class CommandAdapter implements Adapter<TxnId, Command, AccordSafeCommand>
+    public static class CommandAdapter implements Adapter<TxnId, Command, SaferCommand>
     {
         private static final int SHRINK_WITHOUT_LOCK = -1;
 
@@ -1373,23 +1405,27 @@ public class AccordCache implements CacheSize
         @Override
         public boolean validate(AccordCommandStore commandStore, TxnId key, Command value)
         {
-            return commandStore.validateCommand(key, value);
+            if (!Invariants.isParanoid())
+                return true;
+
+            Command reloaded = commandStore.loadCommand(key);
+            return Objects.equals(value, reloaded);
         }
 
         @Override
-        public AccordSafeCommand safeRef(AccordCacheEntry<TxnId, Command> node)
+        public SaferCommand safeRef(AccordCacheEntry<TxnId, Command, SaferCommand> node)
         {
-            return new AccordSafeCommand(node);
+            return new SaferCommand(node);
         }
 
         @Override
-        public AccordCacheEntry<TxnId, Command> newEntry(TxnId txnId, Type<TxnId, Command, ?>.Instance owner)
+        public AccordCacheEntry<TxnId, Command, SaferCommand> newEntry(TxnId txnId, Type<TxnId, Command, SaferCommand>.Instance owner)
         {
-            AccordCacheEntry<TxnId, Command> node = new AccordCacheEntry<>(txnId, owner);
+            AccordCacheEntry<TxnId, Command, SaferCommand> node = new AccordCacheEntry<>(txnId, owner);
             if (txnId.is(Txn.Kind.EphemeralRead))
             {
                 node.initialize(null);
-                int maxAge = (int)Math.min(0xff, 1 + DatabaseDescriptor.getReadRpcTimeout(TimeUnit.SECONDS));
+                int maxAge = (int)Math.min(AGE_MASK, 1 + DatabaseDescriptor.getReadRpcTimeout(TimeUnit.SECONDS));
                 node.markNoEvict(owner.parent().parent().noEvictGeneration, maxAge);
             }
             else

src/java/org/apache/cassandra/service/accord/execution/AccordCacheEntryQueue.java

@@ -0,0 +1,510 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import java.util.Arrays;
+
+import accord.utils.Invariants;
+import accord.utils.SortedArrays;
+import accord.utils.TriConsumer;
+
+import static org.apache.cassandra.service.accord.execution.AccordCacheEntry.RunnableStatus.NEWLY_BLOCKING_RUNNABLE;
+import static org.apache.cassandra.service.accord.execution.AccordCacheEntry.RunnableStatus.NEWLY_RUNNABLE;
+import static org.apache.cassandra.service.accord.execution.AccordCacheEntry.RunnableStatus.NOT_RUNNABLE;
+import static org.apache.cassandra.service.accord.execution.AccordCacheEntry.RunnableStatus.STILL_RUNNABLE;
+
+class AccordCacheEntryQueue
+{
+    static final AccordCacheEntryQueue EMPTY = new AccordCacheEntryQueue(0);
+
+    private static final int DEFAULT_CAPACITY = 4;
+    static final int LOCKED_INDEX = 0;
+    static final int PRIORITY_START_INDEX = LOCKED_INDEX + 1;
+    /**
+     * [priorityHead..priorityTail) stores a priority-sorted list of tasks
+     * (fifoTail...fifoHead] stores a fifo queue that runs ahead of any priority tasks
+     * (fifoTail-unsequencedCount...fifoTail] stores unsequenced tasks that are waiting for a queued incremental task.
+     * This only happens for TxnId cache entries, since they may lockAndHoldQueue. Once the lock is released,
+     * any pending unsequenced tasks are notified and immediately made (irrevocably) runnable for this entry.
+     */
+    SafeTask<?>[] tasks;
+    // TODO (expected): use bytes/shorts for indexes to keep size down, and have an expanded version of the Queue
+    //  with better algorithmic complexity (e.g. Hash -> IntrusivePriorityHeap)
+    int priorityHead, priorityTail, fifoHead, fifoTail;
+    int unsequencedSize;
+
+    public AccordCacheEntryQueue()
+    {
+        this(DEFAULT_CAPACITY);
+    }
+
+    public AccordCacheEntryQueue(int capacity)
+    {
+        tasks = new SafeTask[capacity];
+        priorityHead = priorityTail = PRIORITY_START_INDEX;
+        fifoHead = fifoTail = capacity - 1;
+    }
+
+    AccordCacheEntryQueue(AccordCacheEntryQueue copy)
+    {
+        tasks = copy.tasks.clone();
+        priorityHead = copy.priorityHead;
+        priorityTail = copy.priorityTail;
+        fifoHead = copy.fifoHead;
+        fifoTail = copy.fifoTail;
+    }
+
+    // returns true if no fifo tasks already queue (i.e. so we become head)
+    boolean addFifo(SafeTask<?> task)
+    {
+        ensureCapacity();
+        boolean isHead = fifoHead == fifoTail;
+        if (unsequencedSize > 0) // simply displace the unsequence task, as they're an unordered list
+            tasks[fifoTail - unsequencedSize] = tasks[fifoTail];
+        tasks[fifoTail--] = task;
+        validate();
+        return isHead;
+    }
+
+    boolean addUnsequenced(SafeTask<?> task)
+    {
+        Invariants.require(task.isUnsequenced());
+        ensureCapacity();
+        tasks[fifoTail - unsequencedSize++] = task;
+        return unsequencedSize == 1 && sequencedSize() == 1;
+    }
+
+    boolean isLocked(SafeTask<?> task)
+    {
+        return tasks[LOCKED_INDEX] == task;
+    }
+
+    SafeTask<?> lockedBy()
+    {
+        return tasks[LOCKED_INDEX];
+    }
+
+    boolean removeIfHead(SafeTask<?> task)
+    {
+        return removeIfFifoHead(task) || removeIfPriorityHead(task);
+    }
+
+    boolean removeIfFifoHead(SafeTask<?> task)
+    {
+        if (!hasFifo())
+            return false;
+
+        if (task != tasks[fifoHead])
+            return false;
+        tasks[fifoHead--] = null;
+        return true;
+    }
+
+    boolean removeIfPriorityHead(SafeTask<?> task)
+    {
+        if (!hasPriority())
+            return false;
+
+        if (task != tasks[priorityHead])
+            return false;
+        tasks[priorityHead++] = null;
+        return true;
+    }
+
+    void lock(SafeTask<?> task)
+    {
+        tasks[LOCKED_INDEX] = task;
+    }
+
+    void unlock(SafeTask<?> task)
+    {
+        Invariants.require(tasks[LOCKED_INDEX] == task);
+        tasks[LOCKED_INDEX] = null;
+    }
+
+    // should always return false, as should never be invoked on an empty queue, and returns true only if we're head of the queue
+    boolean addPrioritised(SafeTask<?> task)
+    {
+        ensureCapacity();
+        int insertPos = Arrays.binarySearch(tasks, priorityHead, priorityTail, task, AccordCacheEntryQueue::compare);
+        if (insertPos < 0)
+            insertPos = -1 - insertPos;
+
+        if (priorityHead == PRIORITY_START_INDEX || insertPos > (priorityTail + priorityHead) / 2)
+        {
+            System.arraycopy(tasks, insertPos, tasks, insertPos + 1, priorityTail - insertPos);
+            tasks[insertPos] = task;
+            priorityTail++;
+        }
+        else
+        {
+            System.arraycopy(tasks, priorityHead, tasks, priorityHead - 1, insertPos - priorityHead);
+            tasks[insertPos - 1] = task;
+            priorityHead--;
+        }
+
+        validate();
+        return fifoHead == fifoTail && tasks[priorityHead] == task;
+    }
+
+    // should always return false, as should never be invoked on an empty queue, and returns true only if we're head of the queue
+    void addWaitingToLoad(SafeTask<?> task)
+    {
+        ensureCapacity();
+        tasks[priorityTail++] = task;
+    }
+
+    private boolean hasTailRoom()
+    {
+        if (priorityTail + unsequencedSize <= fifoTail)
+            return true;
+        Invariants.require(priorityTail + unsequencedSize == 1 + fifoTail);
+        return false;
+    }
+
+    private void ensureCapacity()
+    {
+        if (!hasTailRoom())
+        {
+            if (fifoHead == fifoTail && unsequencedSize == 0 && fifoTail < tasks.length - 1)
+                fifoHead = fifoTail = tasks.length - 1;
+            else if (priorityHead == priorityTail && priorityHead > PRIORITY_START_INDEX)
+                priorityHead = priorityTail = PRIORITY_START_INDEX;
+            else if (totalSize() >= (tasks.length - 1) / 2) compact(new SafeTask[tasks.length * 2]);
+            else compact(tasks);
+            Invariants.require(hasTailRoom());
+        }
+    }
+
+    private void compact(SafeTask<?>[] into)
+    {
+        if (priorityHead == priorityTail) priorityHead = priorityTail = PRIORITY_START_INDEX;
+        else
+        {
+            int priorityLength = priorityTail - priorityHead;
+            System.arraycopy(tasks, priorityHead, into, PRIORITY_START_INDEX, priorityLength);
+            int newTail = PRIORITY_START_INDEX + priorityLength;
+            Invariants.require(newTail <= priorityTail);
+            if (into == tasks)
+                Arrays.fill(into, newTail, priorityTail, null);
+            priorityHead = PRIORITY_START_INDEX;
+            priorityTail = newTail;
+        }
+
+        if (fifoHead == fifoTail && unsequencedSize == 0) fifoHead = fifoTail = into.length - 1;
+        else
+        {
+            int fifoLength = fifoHead - fifoTail;
+            int copyLength = fifoLength + unsequencedSize;
+            int copyFrom = (fifoTail - unsequencedSize) + 1;
+            int copyTo = into.length - copyLength;
+            Invariants.require(copyTo >= copyFrom);
+            System.arraycopy(tasks, copyFrom, into, copyTo, copyLength);
+            if (into == tasks)
+                Arrays.fill(into, copyFrom, copyTo, null);
+            fifoHead = into.length - 1;
+            fifoTail = fifoHead - fifoLength;
+        }
+
+        if (tasks != into)
+        {
+            into[LOCKED_INDEX] = tasks[LOCKED_INDEX];
+            tasks = into;
+        }
+        validate();
+    }
+
+    private void validate()
+    {
+        for (int i = PRIORITY_START_INDEX; i < priorityHead; ++i)
+            Invariants.require(tasks[i] == null);
+        for (int i = priorityHead; i < priorityTail; ++i)
+            Invariants.require(tasks[i] != null);
+        for (int i = priorityTail; i <= fifoTail - unsequencedSize; ++i)
+            Invariants.require(tasks[i] == null);
+        for (int i = (fifoTail - unsequencedSize) + 1; i <= fifoHead; ++i)
+            Invariants.require(tasks[i] != null);
+        for (int i = fifoHead + 1; i < tasks.length; ++i)
+            Invariants.require(tasks[i] == null);
+    }
+
+    SafeTask<?> peek()
+    {
+        if (hasFifo()) return tasks[fifoHead];
+        if (hasPriority()) return tasks[priorityHead];
+        return null;
+    }
+
+    SafeTask<?> peekFifo()
+    {
+        return hasFifo() ? tasks[fifoHead] : null;
+    }
+
+    // second task
+    SafeTask<?> peekBehind()
+    {
+        int fifoSize = fifoSize();
+        if (fifoSize > 1)
+            return tasks[fifoHead - 1];
+        int priorityIndex = priorityHead + (1 - fifoSize);
+        if (priorityIndex < priorityTail)
+            return tasks[priorityIndex];
+        return null;
+    }
+
+    boolean hasFifo()
+    {
+        return fifoHead != fifoTail;
+    }
+
+    boolean hasPriority()
+    {
+        return priorityHead != priorityTail;
+    }
+
+    boolean hasUnsequenced()
+    {
+        return unsequencedSize > 0;
+    }
+
+    int sequencedSize()
+    {
+        return prioritySize() + fifoSize();
+    }
+
+    int unsequencedSize()
+    {
+        return unsequencedSize;
+    }
+
+    int totalSize()
+    {
+        return sequencedSize() + unsequencedSize;
+    }
+
+    int prioritySize()
+    {
+        return priorityTail - priorityHead;
+    }
+
+    int fifoSize()
+    {
+        return fifoHead - fifoTail;
+    }
+
+    // true iff was head
+    boolean removeFifoOrPriority(SafeTask<?> task, boolean permitMissing)
+    {
+        int fifoIndex = fifoIndexOf(task);
+        if (fifoIndex >= 0)
+        {
+            if (fifoIndex == fifoHead)
+            {
+                tasks[fifoHead--] = null;
+                validate();
+                return true;
+            }
+            else
+            {
+                if (remove(fifoIndex, fifoTail + 1, fifoHead + 1)) ++fifoTail;
+                else --fifoHead;
+                validate();
+                return false;
+            }
+        }
+
+        int priorityIndex = priorityIndexOf(task);
+        Invariants.require(priorityIndex >= 0 || permitMissing);
+        if (priorityIndex >= 0)
+        {
+            if (priorityIndex == priorityHead)
+            {
+                tasks[priorityHead++] = null;
+                return !hasFifo();
+            }
+
+            if (remove(priorityIndex, priorityHead, priorityTail)) ++priorityHead;
+            else --priorityTail;
+            return false;
+        }
+
+        return false;
+    }
+
+    boolean removeUnsequenced(SafeTask<?> task)
+    {
+        int unsequencedIndex = unsequencedIndexOf(task);
+        if (unsequencedIndex < 0)
+            return false;
+
+        --unsequencedSize;
+        tasks[unsequencedIndex] = tasks[fifoTail - unsequencedSize];
+        tasks[fifoTail - unsequencedSize] = null;
+        return true;
+    }
+
+    // return true IFF was head
+    private boolean removePriority(SafeTask<?> task, boolean permitAbsent)
+    {
+        int i = priorityIndexOf(task);
+        if (i < 0)
+        {
+            Invariants.require(permitAbsent);
+            return false;
+        }
+
+        boolean wasHead = i == priorityHead;
+        if (remove(i, priorityHead, priorityTail)) ++priorityHead;
+        else --priorityTail;
+        return wasHead;
+    }
+
+    // return true if we move the start forwards, false if we moved the end back
+    private boolean remove(int i, int start, int end)
+    {
+        if (i < (start + end) / 2)
+        {
+            System.arraycopy(tasks, start, tasks, start + 1, i - start);
+            tasks[start] = null;
+            return true;
+        }
+        else
+        {
+            System.arraycopy(tasks, i + 1, tasks, i, end - (i + 1));
+            tasks[end - 1] = null;
+            return false;
+        }
+    }
+
+    boolean contains(SafeTask<?> task)
+    {
+        return indexOf(task) >= 0;
+    }
+
+    private int indexOf(SafeTask<?> task)
+    {
+        if (tasks[priorityHead] == task)
+            return priorityHead;
+
+        if (tasks[fifoHead] == task)
+            return fifoHead;
+
+        int i = priorityIndexOf(task);
+        if (i >= 0)
+            return i;
+
+        return fifoIndexOf(task);
+    }
+
+    private int priorityIndexOf(SafeTask<?> task)
+    {
+        if (priorityTail - priorityHead > 16)
+        {
+            if (tasks[priorityHead] == task)
+                return priorityHead;
+
+            return Arrays.binarySearch(tasks, priorityHead + 1, priorityTail, task, AccordCacheEntryQueue::compare);
+        }
+
+        for (int i = priorityHead; i < priorityTail; ++i)
+        {
+            if (tasks[i] == task)
+                return i;
+        }
+        return -1;
+    }
+
+    private int fifoIndexOf(SafeTask<?> task)
+    {
+        for (int i = fifoHead; i > fifoTail; --i)
+        {
+            if (tasks[i] == task)
+                return i;
+        }
+        return -1;
+    }
+
+    private int unsequencedIndexOf(SafeTask<?> task)
+    {
+        for (int i = (fifoTail - unsequencedSize) + 1; i <= fifoTail; ++i)
+        {
+            if (tasks[i] == task)
+                return i;
+        }
+        return -1;
+    }
+
+    <P1, P2> int drainUnsequenced(TriConsumer<SafeTask<?>, P1, P2> forEach, P1 p1, P2 p2)
+    {
+        for (int i = (fifoTail - unsequencedSize) + 1; i <= fifoTail; ++i)
+        {
+            SafeTask<?> task = tasks[i];
+            tasks[i] = null;
+            // should not be reentrant
+            forEach.accept(task, p1, p2);
+        }
+        int count = unsequencedSize;
+        unsequencedSize = 0;
+        return count;
+    }
+
+    AccordCacheEntry.RunnableStatus ensureHeadFifo(SafeTask<?> task)
+    {
+        if (hasFifo())
+        {
+            Invariants.require(tasks[fifoHead] == task);
+            return NOT_RUNNABLE;
+        }
+
+        if (tasks[priorityHead] == task)
+        {
+            tasks[priorityHead++] = null;
+            addFifo(task);
+            return STILL_RUNNABLE;
+        }
+        else
+        {
+            boolean wasPriorityHead = removePriority(task, false);
+            boolean isFifoHead = addFifo(task);
+            if (!isFifoHead)
+                return NOT_RUNNABLE;
+            if (wasPriorityHead)
+                return STILL_RUNNABLE;
+            if (hasPriority() || hasUnsequenced())
+                return NEWLY_BLOCKING_RUNNABLE;
+            return NEWLY_RUNNABLE;
+        }
+    }
+
+    static int compare(SafeTask<?> a, SafeTask<?> b)
+    {
+        Invariants.require(a != null && b != null);
+        int c = Long.compare(a.position, b.position);
+        if (c == 0)
+            c = a.executionContext().executionKind().compareTo(b.executionContext().executionKind());
+        if (c == 0)
+            c = Long.compare(a.createdAt, b.createdAt);
+        return c;
+    }
+
+    public boolean hasQueued()
+    {
+        return hasFifo() || hasPriority();
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/AccordExecutor.java

@@ -0,0 +1,813 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.Lock;
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import java.util.stream.Stream;
+
+import accord.api.Agent;
+import accord.api.ExclusiveAsyncExecutor;
+import accord.api.RoutingKey;
+import accord.impl.AbstractAsyncExecutor;
+import accord.local.Command;
+import accord.local.cfk.CommandsForKey;
+import accord.primitives.TxnId;
+import accord.utils.ArrayBuffers;
+import accord.utils.Invariants;
+import accord.utils.UnhandledEnum;
+import accord.utils.async.AsyncCallbacks.CallAndCallback;
+import accord.utils.async.AsyncCallbacks.RunOrFail;
+import accord.utils.async.AsyncChain;
+import accord.utils.async.AsyncChains;
+import accord.utils.async.Cancellable;
+
+import org.apache.cassandra.cache.CacheSize;
+import org.apache.cassandra.concurrent.DebuggableTask.DebuggableTaskRunner;
+import org.apache.cassandra.concurrent.Shutdownable;
+import org.apache.cassandra.config.AccordConfig;
+import org.apache.cassandra.config.AccordConfig.QueueBalancingModel;
+import org.apache.cassandra.config.AccordConfig.QueuePriorityModel;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.metrics.AccordCacheMetrics;
+import org.apache.cassandra.metrics.AccordExecutorMetrics;
+import org.apache.cassandra.metrics.AccordReplicaMetrics;
+import org.apache.cassandra.metrics.AccordSystemMetrics;
+import org.apache.cassandra.metrics.LogLinearDecayingHistograms;
+import org.apache.cassandra.metrics.LogLinearDecayingHistograms.LogLinearDecayingHistogram;
+import org.apache.cassandra.metrics.ShardedDecayingHistograms;
+import org.apache.cassandra.metrics.ShardedDecayingHistograms.DecayingHistogramsShard;
+import org.apache.cassandra.service.accord.debug.DebugExecution.DebugExecutor;
+import org.apache.cassandra.service.accord.debug.DebugExecution.DebugTask;
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry.LoadExecutor;
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry.SaveExecutor;
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry.UniqueSave;
+import org.apache.cassandra.service.accord.execution.ExclusiveExecutor.ExclusiveExecutorTask;
+import org.apache.cassandra.service.accord.execution.IOTaskWrapper.WrappableIOTask;
+import org.apache.cassandra.service.accord.execution.Task.ExclusiveGroup;
+import org.apache.cassandra.service.accord.execution.Task.ExecutorQueue;
+import org.apache.cassandra.service.accord.execution.Task.GlobalGroup;
+import org.apache.cassandra.utils.Clock;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Condition;
+import org.apache.cassandra.utils.concurrent.Future;
+
+import static org.apache.cassandra.service.accord.debug.DebugExecution.DEBUG_EXECUTION;
+import static org.apache.cassandra.service.accord.execution.AccordCache.CommandAdapter.COMMAND_ADAPTER;
+import static org.apache.cassandra.service.accord.execution.AccordCache.CommandsForKeyAdapter.CFK_ADAPTER;
+import static org.apache.cassandra.service.accord.execution.AccordCache.registerJfrListener;
+import static org.apache.cassandra.service.accord.execution.AccordCacheEntry.Status.EVICTED;
+import static org.apache.cassandra.service.accord.execution.Task.GlobalGroup.LOAD;
+import static org.apache.cassandra.service.accord.execution.Task.GlobalGroup.OTHER;
+import static org.apache.cassandra.service.accord.execution.Task.GlobalGroup.RANGE_LOAD;
+import static org.apache.cassandra.service.accord.execution.Task.GlobalGroup.RANGE_SCAN;
+import static org.apache.cassandra.service.accord.execution.Task.State.EXECUTED;
+import static org.apache.cassandra.service.accord.execution.Task.State.FAILED;
+import static org.apache.cassandra.service.accord.execution.Task.State.UNREGISTERED;
+import static org.apache.cassandra.service.accord.execution.TaskQueueMulti.COUNTER_MASKS;
+import static org.apache.cassandra.service.accord.execution.TaskQueueMulti.overflowBit;
+import static org.apache.cassandra.service.accord.execution.TaskQueueMulti.selectByOverflowBits;
+import static org.apache.cassandra.service.accord.execution.TaskQueueMulti.setOverflowWhenLessEqual;
+
+/**
+ * NOTE: We assume that NO BLOCKING TASKS are submitted to this executor AND WAITED ON by another task executing on this executor.
+ *  (as we do not immediately schedule additional threads for submitted tasks, but schedule new threads only if necessary when the submitting execution completes)
+ */
+public abstract class AccordExecutor implements CacheSize, LoadExecutor<SafeTask<?>, Boolean>, SaveExecutor, Shutdownable, AbstractAsyncExecutor
+{
+    static final QueuePriorityModel PRIORITY_MODEL;
+    static final QueueBalancingModel BALANCING_MODEL;
+    static final long AGE_TO_FIFO;
+    // PRIORITY_FAIR blends two strategies (flow: least fairly serviced; age: earliest-queued work) by deficit
+    // round-robin; weights of BLEND_TOTAL come from a single imbalance ramp (onset..onset+width) trading age->flow.
+    static final int BLEND_SHIFT = 6, BLEND_TOTAL = 1 << BLEND_SHIFT;
+    static final int FLOW_ONSET, FLOW_WIDTH_SHIFT;
+    static final boolean BALANCE_BY_POSITION;
+    static final long GLOBAL_QUEUE_LIMITS, EXCLUSIVE_QUEUE_LIMITS;
+    static final int NONSYNC_MIN_BATCH_SIZE, NONSYNC_MAX_BATCH_SIZE, NONSYNC_BLOCKED_LIMIT;
+    static final boolean NONSYNC_ENABLED;
+    private static final long LOADING_GROUPS = overflowBit(LOAD) | overflowBit(RANGE_LOAD) | overflowBit(RANGE_SCAN);
+
+    static
+    {
+        AccordConfig config = DatabaseDescriptor.getAccord();
+        AGE_TO_FIFO = config.queue_priority_age_to_fifo.to(TimeUnit.MICROSECONDS);
+        PRIORITY_MODEL = config.queue_priority_model != null ? config.queue_priority_model : QueuePriorityModel.HLC_FIFO;
+        BALANCING_MODEL = config.queue_balancing_model != null ? config.queue_balancing_model : QueueBalancingModel.BLENDED_PRIORITY_PHASE_FAIR;
+        FLOW_ONSET  = config.queue_flow_imbalance_onset == null ? 4  : config.queue_flow_imbalance_onset;
+        FLOW_WIDTH_SHIFT  = config.queue_flow_imbalance_width_shift == null ? 5 : config.queue_flow_imbalance_width_shift;
+        NONSYNC_MIN_BATCH_SIZE = config.queue_nonsync_min_batch_size == null ? 16 : config.queue_nonsync_min_batch_size;
+        NONSYNC_MAX_BATCH_SIZE = config.queue_nonsync_max_batch_size == null ? 64 : config.queue_nonsync_max_batch_size;
+        NONSYNC_ENABLED = config.queue_nonsync_enabled == null || config.queue_nonsync_enabled;
+        NONSYNC_BLOCKED_LIMIT = config.queue_nonsync_blocked_limit == null ? 8 : config.queue_nonsync_blocked_limit;
+        Invariants.require(FLOW_ONSET >= 0 && FLOW_WIDTH_SHIFT >= 0);
+        switch (BALANCING_MODEL)
+        {
+            default: throw new UnhandledEnum(BALANCING_MODEL);
+            case PRIORITY_ONLY:
+            case BLENDED_PRIORITY_PHASE_FAIR:
+                BALANCE_BY_POSITION = true;
+                break;
+            case PHASE_ONLY:
+            case PHASE_FAIR:
+                BALANCE_BY_POSITION = false;
+        }
+
+        {
+            // TODO (required): pick default max loads/saves/range loads based on number of threads
+            long global = COUNTER_MASKS, exclusive = COUNTER_MASKS;
+            global ^= (0x7fL ^ 1) << (RANGE_SCAN.ordinal() * 8);
+            if (config.queue_active_limits != null)
+            {
+                long[] limits = parseEnumParams(config.queue_active_limits, "queue_active_limits");
+                global = selectByOverflowBits(setOverflowWhenLessEqual(limits[0], 0), global, limits[0]);
+                exclusive = selectByOverflowBits(setOverflowWhenLessEqual(limits[1], 0), global, limits[1]);
+            }
+            GLOBAL_QUEUE_LIMITS = global;
+            EXCLUSIVE_QUEUE_LIMITS = exclusive;
+        }
+
+    }
+
+    public static final ShardedDecayingHistograms HISTOGRAMS = new ShardedDecayingHistograms();
+
+    public interface AccordExecutorFactory
+    {
+        AccordExecutor get(int executorId, Mode mode, int threads, IntFunction<String> name, Agent agent);
+    }
+
+    public enum Mode { RUN_WITH_LOCK, RUN_WITHOUT_LOCK }
+
+    // WARNING: this is a shared object, so close is NOT idempotent
+    public static final class ExclusiveGlobalCaches extends GlobalCaches implements AutoCloseable
+    {
+        final AccordExecutor executor;
+
+        public ExclusiveGlobalCaches(AccordExecutor executor, AccordCache global, AccordCache.Type<TxnId, Command, SaferCommand> commands, AccordCache.Type<RoutingKey, CommandsForKey, SaferCommandsForKey> commandsForKey)
+        {
+            super(global, commands, commandsForKey);
+            this.executor = executor;
+        }
+
+        @Override
+        public void close()
+        {
+            executor.beforeUnlockExternal();
+            global.tryShrinkOrEvict(executor.lock);
+            executor.unlock(TaskRunner.get());
+        }
+    }
+
+    public static class GlobalCaches
+    {
+        public final AccordCache global;
+        public final AccordCache.Type<TxnId, Command, SaferCommand> commands;
+        public final AccordCache.Type<RoutingKey, CommandsForKey, SaferCommandsForKey> commandsForKey;
+
+        public GlobalCaches(AccordCache global, AccordCache.Type<TxnId, Command, SaferCommand> commands, AccordCache.Type<RoutingKey, CommandsForKey, SaferCommandsForKey> commandsForKey)
+        {
+            this.global = global;
+            this.commands = commands;
+            this.commandsForKey = commandsForKey;
+        }
+    }
+
+    final LogLinearDecayingHistograms histograms;
+    final LogLinearDecayingHistogram elapsedPreparingToRun;
+    final LogLinearDecayingHistogram elapsedWaitingToRun;
+    final LogLinearDecayingHistogram elapsedRunning;
+    final LogLinearDecayingHistogram elapsed;
+    final LogLinearDecayingHistogram keys;
+    public final AccordReplicaMetrics.Shard replicaMetrics;
+
+    private final Lock lock;
+    final Agent agent;
+    public final int executorId;
+    private final AccordCache cache;
+    private final ExclusiveGlobalCaches caches;
+    final AtomicLong uniqueCreatedAt = new AtomicLong();
+    final DebugExecutor debug = DebugExecutor.maybeDebug();
+
+    private long maxWorkingSetSizeInBytes;
+    private long maxWorkingCapacityInBytes;
+
+    final TaskQueueStandalone<SafeTask<?>> loading = new TaskQueueStandalone<>(ExecutorQueue.LOADING);
+    final TaskQueueStandalone<SafeTask<?>> waiting = new TaskQueueStandalone<>(ExecutorQueue.WAITING);
+    final TaskQueueRunnable<Task> runnable = new TaskQueueRunnable<>();
+
+    private final Tranches tranches = new Tranches(this);
+
+    /**
+     * Newly submitted work must take a position >= minPosition, but this condition does not apply to consequences of
+     * previously submitted work; this inherits the originating task's position and tranche.
+     * This is to ensure afterSubmittedAndConsequences functions correctly.
+     */
+    long minPosition = 1;
+    long nextPosition = 1;
+    int tasks;
+
+    private boolean hasPausedLoading;
+
+    private List<Condition> waitingForQuiescence;
+
+    AccordExecutor(Lock lock, int executorId, Agent agent)
+    {
+        this.lock = lock;
+        this.executorId = executorId;
+        this.cache = new AccordCache(this, 0);
+        this.agent = agent;
+
+        final AccordCache.Type<TxnId, Command, SaferCommand> commands;
+        final AccordCache.Type<RoutingKey, CommandsForKey, SaferCommandsForKey> commandsForKey;
+        commands = cache.newType(TxnId.class, COMMAND_ADAPTER, AccordCacheMetrics.CommandsCacheMetrics.newShard(lock));
+        registerJfrListener(executorId, commands, "Command");
+
+        commandsForKey = cache.newType(RoutingKey.class, CFK_ADAPTER, AccordCacheMetrics.CommandsForKeyCacheMetrics.newShard(lock));
+        registerJfrListener(executorId, commandsForKey, "CommandsForKey");
+
+        this.caches = new ExclusiveGlobalCaches(this, cache, commands, commandsForKey);
+
+        DecayingHistogramsShard histogramsShard = HISTOGRAMS.newShard(lock);
+        this.histograms = histogramsShard.unsafeGetInternal();
+        this.elapsedPreparingToRun = AccordExecutorMetrics.INSTANCE.elapsedPreparingToRun.forShard(histogramsShard);
+        this.elapsedWaitingToRun = AccordExecutorMetrics.INSTANCE.elapsedWaitingToRun.forShard(histogramsShard);
+        this.elapsedRunning = AccordExecutorMetrics.INSTANCE.elapsedRunning.forShard(histogramsShard);
+        this.elapsed = AccordExecutorMetrics.INSTANCE.elapsed.forShard(histogramsShard);
+        this.keys = AccordExecutorMetrics.INSTANCE.keys.forShard(histogramsShard);
+        this.replicaMetrics = new AccordReplicaMetrics.Shard(histogramsShard);
+    }
+
+    abstract boolean isInLoop();
+    abstract void beforeUnlockExternal();
+    abstract <P1> void submit(Consumer<P1> sync, Function<P1, Task> async, P1 p1);
+    public abstract boolean hasTasks();
+    public abstract boolean isOwningThread();
+
+    public final Lock unsafeLock()
+    {
+        return lock;
+    }
+
+    public final void lock(TaskRunner self)
+    {
+        long startAt = DEBUG_EXECUTION ? Clock.Global.nanoTime() : 0;
+        if (!self.tryEnterAccordLockedExecutor(this))
+            throw new UnsupportedOperationException("To ensure system performance, it is not permitted to utilise multiple AccordExecutor simultaneously with the same thread");
+        //noinspection LockAcquiredButNotSafelyReleased
+        lock.lock();
+        if (DEBUG_EXECUTION) debug.onEnterLock(startAt);
+    }
+
+    public final void unlock(TaskRunner self)
+    {
+        self.exitAccordLockedExecutor();
+        if (DEBUG_EXECUTION) debug.onExitLock();
+        lock.unlock();
+    }
+
+    public final boolean tryLock(TaskRunner self)
+    {
+        AccordExecutor active = self.accordActiveExecutor();
+        if (active != null && active != this)
+            return false;
+
+       return onTryLock(self, lock.tryLock());
+    }
+
+    final boolean onTryLock(TaskRunner self, boolean result)
+    {
+        if (result && !self.tryEnterAccordLockedExecutor(this))
+        {
+            // shouldn't be possible, we should have checked this already
+            lock.unlock();
+            throw new IllegalStateException();
+        }
+        return result;
+    }
+
+    public ExclusiveGlobalCaches lockCaches()
+    {
+        lock(TaskRunner.get(Thread.currentThread()));
+        return caches;
+    }
+
+    public AccordCache cacheExclusive()
+    {
+        Invariants.require(isOwningThread());
+        return cache;
+    }
+
+    public AccordCache cacheUnsafe()
+    {
+        return cache;
+    }
+
+    final boolean hasAlreadyWaitingToRun()
+    {
+        return runnable.hasWaitingToRun();
+    }
+
+    void updateWaitingToRunExclusive()
+    {
+        // TODO (expected): this should not be invoked on every update of waiting to run
+        maybeUnpauseLoading();
+    }
+
+    // drain only new work; specifically leave anything that would call completeTask queued.
+    // this is to maintain invariants in Tranches.complete, where we may have some consequence of some earlier task
+    // pending but unqueued, so that we have not incremented its tranche count, and in the interim we set the tranche
+    // count to zero
+    void drainUnqueuedNewWorkExclusive()
+    {
+    }
+
+    final Task pollAlreadyWaitingToRunExclusive()
+    {
+        Task next = runnable.poll();
+        if (DEBUG_EXECUTION && next != null) DebugTask.get(next).onPolled();
+        return next;
+    }
+
+    public void waitForQuiescence()
+    {
+        TaskRunner self = TaskRunner.get();
+        Condition condition;
+        lock(self);
+        try
+        {
+            if (tasks == 0)
+                return;
+
+            if (waitingForQuiescence == null)
+                waitingForQuiescence = new ArrayList<>();
+            condition = Condition.newOneTimeCondition();
+            waitingForQuiescence.add(condition);
+        }
+        finally
+        {
+            unlock(self);
+        }
+        condition.awaitThrowUncheckedOnInterrupt();
+    }
+
+    protected void notifyQuiescentExclusive()
+    {
+        if (waitingForQuiescence != null)
+        {
+            waitingForQuiescence.forEach(Condition::signalAll);
+            waitingForQuiescence = null;
+        }
+        tranches.finishAll(nextPosition);
+    }
+
+    public void afterSubmittedAndConsequences(Runnable run)
+    {
+        TaskRunner self = TaskRunner.get();
+
+        lock(self);
+        try
+        {
+            drainUnqueuedNewWorkExclusive();
+            if (tasks == 0)
+            {
+                run.run();
+                return;
+            }
+
+            tranches.registerWait(run);
+        }
+        finally
+        {
+            unlock(self);
+        }
+    }
+
+    private boolean hasNonLoadingWaitingToRun()
+    {
+        return runnable.hasWaitingToRunExcluding(LOADING_GROUPS);
+    }
+
+    final void maybeUnpauseLoading()
+    {
+        if (hasPausedLoading && (cache.weightedSize() < maxWorkingCapacityInBytes || !hasNonLoadingWaitingToRun()))
+        {
+            hasPausedLoading = false;
+            runnable.restart(LOADING_GROUPS);
+        }
+    }
+
+    final void maybePauseLoading()
+    {
+        if (hasPausedLoading)
+            return;
+
+        if (!hasPausedLoading && cache.weightedSize() >= maxWorkingCapacityInBytes && hasNonLoadingWaitingToRun())
+        {
+            AccordSystemMetrics.metrics.pausedExecutorLoading.inc();
+            hasPausedLoading = true;
+            runnable.stop(LOADING_GROUPS);
+        }
+    }
+
+    public ExclusiveExecutor newExclusiveExecutor(int commandStoreId)
+    {
+        return new ExclusiveExecutor(this, commandStoreId);
+    }
+
+    public ExclusiveAsyncExecutor newExclusiveExecutor()
+    {
+        return new ExclusiveExecutor(this);
+    }
+
+    @Override
+    public <K, V> IOTaskLoad<?, ?> load(SafeTask<?> parent, Boolean isForRange, AccordCacheEntry<K, V, ?> entry)
+    {
+        IOTaskLoad<?, ?> result = newLoad(entry, isForRange);
+        result.inherit(parent).submitExclusiveNoExcept();
+        return result;
+    }
+
+    @Override
+    public Cancellable save(AccordCacheEntry<?, ?, ?> entry, UniqueSave identity, Runnable save)
+    {
+        IOTaskSave task = new IOTaskSave(this, entry, identity, save);
+        task.submitExclusiveNoExcept();
+        return task;
+    }
+
+    void submitTask(Task task)
+    {
+        submit(Task::submitExclusiveNoExcept, i -> i, task);
+    }
+
+    public Future<?> submit(Runnable run)
+    {
+        Task inherit = inherit();
+        PlainRunnable task = new PlainRunnable(this, new AsyncPromise<>(), run, OTHER);
+        if (inherit != null) inherit.addConsequence(task);
+        else submitTask(task);
+        return task.result;
+    }
+
+    public void execute(Runnable run)
+    {
+        Task inherit = inherit();
+        PlainRunnable task = new PlainRunnable(this, null, run, OTHER);
+        if (inherit != null) inherit.addConsequence(task);
+        else submitTask(task);
+    }
+
+    @Override
+    public Cancellable execute(RunOrFail runOrFail)
+    {
+        Task inherit = inherit();
+        PlainChain task = new PlainChain(this, runOrFail, null, ExclusiveGroup.OTHER);
+        if (inherit != null) inherit.addConsequence(task);
+        else submitTask(task);
+        return task;
+    }
+
+    public <T> AsyncChain<T> buildDebuggable(Callable<T> call, Object describe)
+    {
+        return new AsyncChains.Head<>()
+        {
+            @Override
+            protected Cancellable start(BiConsumer<? super T, Throwable> callback)
+            {
+                Task inherit = inherit();
+                PlainChainDebuggable task = new PlainChainDebuggable(AccordExecutor.this, new CallAndCallback<>(call, callback), null, describe);
+                if (inherit != null) inherit.addConsequence(task);
+                else submitTask(task);
+                return task;
+            }
+        };
+    }
+
+    public void submitExclusive(Runnable runnable)
+    {
+        new PlainRunnable(this, null, runnable, OTHER).submitExclusiveNoExcept();
+    }
+
+    Cancellable submitExclusive(Task parent, GlobalGroup group, WrappableIOTask wrap)
+    {
+        IOTaskWrapper task = new IOTaskWrapper(this, wrap, group);
+        task.inherit(parent).submitExclusiveNoExcept();
+        return task;
+    }
+
+    Task inherit()
+    {
+        TaskRunner self = TaskRunner.get();
+
+        if (self.accordActiveExecutor() != this)
+            return null;
+
+        Task task = self.accordActiveSelfTask();
+        if (task instanceof ExclusiveExecutorTask)
+            task = ((ExclusiveExecutorTask)task).queue.task;
+        return task;
+    }
+
+    void registerExclusive(Task task)
+    {
+        Invariants.require(isOwningThread());
+        Invariants.require(task.is(UNREGISTERED));
+        ++tasks;
+        if (task.hasInherited())
+        {
+            tranches.addInherited(task.tranche(), task.position);
+        }
+        else
+        {
+            long position = task.position;
+            if (position == 0)
+            {
+                task.position = position = nextPosition++;
+            }
+            else
+            {
+                long delta = nextPosition - position;
+                if (delta >= AGE_TO_FIFO)
+                    task.position = position = nextPosition++;
+                else if (delta <= 0)
+                    nextPosition = position + 1;
+                else if (position < minPosition)
+                    task.position = position = minPosition;
+            }
+            task.setTranche(tranches.addNew(position));
+        }
+    }
+
+    final void completedTaskExclusive(Task task)
+    {
+        Invariants.require(task.compareTo(EXECUTED) >= 0);
+        if (DEBUG_EXECUTION) DebugTask.get(task).onCompleted(debug);
+        unregisterExclusive(task);
+        runnable.cleanup(task);
+        cache.tryShrinkOrEvict(lock);
+        maybeUnpauseLoading();
+    }
+
+    final void unregisterExclusive(Task task)
+    {
+        int tranch = task.tranche();
+        --tasks;
+        tranches.complete(tranch);
+    }
+
+    void onScannedRangesExclusive(SafeTask<?> task, Throwable fail)
+    {
+        // the task may have already been cancelled, in which case we don't need to fail it
+        if (task.state().isDone())
+            return;
+
+        SafeTask<?>.RangeTxnScanner scanner = task.rangeScanner();
+        if (scanner == null && fail == null) fail = new CancellationException();
+        if (fail != null) task.tryFailAndCompleteExclusive(fail, FAILED);
+        else scanner.scannedExclusive();
+    }
+
+    <K, V> void onSavedExclusive(AccordCacheEntry<K, V, ?> state, Object identity, Throwable fail)
+    {
+        cache.saved(state, identity, fail);
+    }
+
+    <K, V> void onLoadedExclusive(AccordCacheEntry<K, V, ?> loaded, V value, Throwable fail)
+    {
+        if (loaded.status() == EVICTED)
+            return;
+
+        try (ArrayBuffers.BufferList<SafeTask<?>> tasks = loaded.drainWaitingToLoad())
+        {
+            if (fail != null)
+            {
+                for (SafeTask<?> task : tasks)
+                    task.tryFailAndCompleteExclusive(fail, FAILED);
+                cache.failedToLoad(loaded);
+            }
+            else
+            {
+                cache.loaded(loaded, value);
+                for (SafeTask<?> task : tasks)
+                    task.onLoadOneExclusive(loaded);
+            }
+        }
+
+        maybePauseLoading();
+    }
+
+    public void executeDirectlyWithLock(Runnable command)
+    {
+        TaskRunner self = TaskRunner.get();
+        lock(self);
+        try
+        {
+            self.setAccordActiveExecutor(this);
+            command.run();
+        }
+        finally
+        {
+            beforeUnlockExternal();
+            self.setAccordActiveExecutor(null);
+            unlock(self);
+        }
+    }
+
+    @Override
+    public void setCapacity(long bytes)
+    {
+        Invariants.require(isOwningThread());
+        cache.setCapacity(bytes);
+        refreshCapacity();
+    }
+
+    public void setWorkingSetSize(long bytes)
+    {
+        Invariants.require(isOwningThread());
+        maxWorkingSetSizeInBytes = bytes;
+        refreshCapacity();
+    }
+
+    private void refreshCapacity()
+    {
+        maxWorkingCapacityInBytes = cache.capacity() + maxWorkingSetSizeInBytes;
+        if (maxWorkingCapacityInBytes < maxWorkingSetSizeInBytes)
+            maxWorkingCapacityInBytes = Long.MAX_VALUE;
+    }
+
+    @Override
+    public long capacity()
+    {
+        return cache.capacity();
+    }
+
+    @Override
+    public int size()
+    {
+        return cache.size();
+    }
+
+    @Override
+    public long weightedSize()
+    {
+        return cache.weightedSize();
+    }
+
+    public int unsafePreparingToRunCount()
+    {
+        return loading.size();
+    }
+
+    public int unsafeWaitingToRunCount()
+    {
+        return runnable.waitingCount();
+    }
+
+    public int unsafeRunningCount()
+    {
+        return runnable.assigned.size();
+    }
+
+    public Stream<? extends DebuggableTaskRunner> active()
+    {
+        return Stream.of();
+    }
+
+    public int executorId()
+    {
+        return executorId;
+    }
+
+    public static <O> IntFunction<O> constant(O out)
+    {
+        return ignore -> out;
+    }
+
+    <K, V> IOTaskLoad<K, V> newLoad(AccordCacheEntry<K, V, ?> entry, boolean isForRange)
+    {
+        return new IOTaskLoad<>(this, entry, isForRange ? RANGE_LOAD : LOAD);
+    }
+
+    public List<TaskInfo> taskSnapshot()
+    {
+        List<TaskInfo> result = new ArrayList<>();
+        TaskRunner self = TaskRunner.get();
+        lock(self);
+        try
+        {
+            addToSnapshot(result, loading, TaskInfo.Status.LOADING, TaskInfo.Status.LOADING);
+            for (TaskQueue queue : runnable.queues)
+            {
+                if (queue != null)
+                    addToSnapshot(result, queue, TaskInfo.Status.WAITING_TO_RUN, TaskInfo.Status.WAITING_TO_RUN);
+            }
+            addToSnapshot(result, runnable.assigned, TaskInfo.Status.RUNNING, TaskInfo.Status.WAITING_TO_RUN);
+        }
+        finally
+        {
+            unlock(self);
+        }
+        result.sort(TaskInfo::compareTo);
+        return result;
+    }
+
+    private static void addToSnapshot(List<TaskInfo> snapshot, TaskQueue<?> queue, TaskInfo.Status ifCurrent, TaskInfo.Status ifQueued)
+    {
+        for (int i = 0 ; i < queue.size() ; ++i)
+        {
+            Task t = queue.getSingle(i);
+            if (t instanceof ExclusiveExecutorTask)
+            {
+                ExclusiveExecutor q = ((ExclusiveExecutorTask) t).queue;
+                snapshot.add(new TaskInfo(ifCurrent, q.commandStoreId, q.task));
+                for (TaskQueue<?> q0 : q.queues)
+                {
+                    if (q0 != null)
+                    {
+                        for (int j = 0 ; j < q0.size() ; ++j)
+                            snapshot.add(new TaskInfo(ifQueued, q.commandStoreId, q0.getSingle(j)));
+                    }
+                }
+            }
+            else
+            {
+                int commmandStoreId = t instanceof SafeTask ? ((SafeTask<?>) t).commandStore.id() : -1;
+                snapshot.add(new TaskInfo(ifCurrent, commmandStoreId, t));
+            }
+        }
+    }
+
+    private static long[] parseEnumParams(String input, String describe)
+    {
+        String[] specs = input.split(";");
+        if (specs.length != 2)
+            throw new IllegalArgumentException("Invalid specifiers in " + describe + "; expect [GlobalGroup];[ExclusiveGroup] but got: " + input);
+        long[] result = new long[2];
+        result[0] = parseEnumParams(GlobalGroup::valueOf, specs[0], describe + " for GlobalGroup");
+        result[1] = parseEnumParams(ExclusiveGroup::valueOf, specs[1], describe + " for ExclusiveGroup");
+        return result;
+    }
+
+    private static long parseEnumParams(Function<String, ? extends Enum<?>> get, String input, String describe)
+    {
+        long result = 0;
+        for (String spec : input.split(","))
+        {
+            if (spec.trim().isEmpty()) continue;
+
+            String[] split = spec.split(":");
+            if (split.length != 2)
+                throw new IllegalArgumentException("Invalid specifier " + spec + " in " + describe + ": " + input);
+
+            try
+            {
+                Enum<?> queue = get.apply(split[0]);
+                long value = Long.parseLong(split[1]);
+                if (value <= 0 || value >= 128)
+                    throw new IllegalArgumentException("Invalid limit " + value + " in queue_active_limits: " + input);
+
+                result |= value << (queue.ordinal() * 8);
+            }
+            catch (Throwable t)
+            {
+                throw new IllegalArgumentException("Invalid queue identifier " + split[0] + " in " + describe + ": " + input);
+            }
+        }
+
+        return result;
+    }
+
+    protected static void prepareRunComplete(TaskRunner self, Task task)
+    {
+        if (task.prepareExclusiveNoExcept())
+        {
+            try { task.runNoExcept(self); }
+            finally { task.completeExclusiveNoExcept(); }
+        }
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/AccordExecutorAsyncSubmit.java

@@ -16,7 +16,7 @@
  * limitations under the License.
  */
 
-package org.apache.cassandra.service.accord;
+package org.apache.cassandra.service.accord.execution;
 
 import java.util.concurrent.TimeUnit;
 import java.util.function.IntFunction;
@@ -26,9 +26,9 @@ import accord.api.Agent;
 import org.apache.cassandra.utils.concurrent.LockWithAsyncSignal;
 
 // WARNING: experimental - needs more testing
-public class AccordExecutorAsyncSubmit extends AccordExecutorAbstractSemiSyncSubmit
+public class AccordExecutorAsyncSubmit extends AbstractSemiSyncSubmit
 {
-    private final AccordExecutorLoops loops;
+    private final Loops loops;
     private final LockWithAsyncSignal lock;
 
     public AccordExecutorAsyncSubmit(int executorId, Mode mode, int threads, IntFunction<String> name, Agent agent)
@@ -40,7 +40,7 @@ public class AccordExecutorAsyncSubmit extends AccordExecutorAbstractSemiSyncSub
     {
         super(lock, executorId, agent);
         this.lock = lock;
-        this.loops = new AccordExecutorLoops(mode, threads, name, this::task);
+        this.loops = new Loops(mode, threads, name, this::task);
     }
 
     @Override
@@ -52,7 +52,7 @@ public class AccordExecutorAsyncSubmit extends AccordExecutorAbstractSemiSyncSub
     }
 
     @Override
-    AccordExecutorLoops loops()
+    Loops loops()
     {
         return loops;
     }
@@ -76,7 +76,7 @@ public class AccordExecutorAsyncSubmit extends AccordExecutorAbstractSemiSyncSub
     }
 
     @Override
-    boolean isOwningThread()
+    public boolean isOwningThread()
     {
         return lock.isOwner(Thread.currentThread());
     }

src/java/org/apache/cassandra/service/accord/execution/AccordExecutorSemiSyncSubmit.java

@@ -16,7 +16,7 @@
  * limitations under the License.
  */
 
-package org.apache.cassandra.service.accord;
+package org.apache.cassandra.service.accord.execution;
 
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.locks.Condition;
@@ -26,12 +26,11 @@ import java.util.function.IntFunction;
 import accord.api.Agent;
 
 // WARNING: experimental - needs more testing
-public class AccordExecutorSemiSyncSubmit extends AccordExecutorAbstractSemiSyncSubmit
+public class AccordExecutorSemiSyncSubmit extends AbstractSemiSyncSubmit
 {
-    private final AccordExecutorLoops loops;
+    private final Loops loops;
     private final ReentrantLock lock;
     private final Condition hasWork;
-    private int waiting;
 
     public AccordExecutorSemiSyncSubmit(int executorId, Mode mode, int threads, IntFunction<String> name, Agent agent)
     {
@@ -43,19 +42,7 @@ public class AccordExecutorSemiSyncSubmit extends AccordExecutorAbstractSemiSync
         super(lock, executorId, agent);
         this.lock = lock;
         this.hasWork = lock.newCondition();
-        this.loops = new AccordExecutorLoops(mode, threads, name, this::task);
+        this.loops = new Loops(mode, threads, name, this::task);
-    }
-
-    @Override
-    void loopYieldExclusive() throws InterruptedException
-    {
-        if (waiting > 0 && hasWaitingToRun())
-        {
-            pauseLoop();
-            hasWork.signal();
-            awaitWork();
-            resumeLoop();
-        }
     }
 
     @Override
@@ -67,13 +54,11 @@ public class AccordExecutorSemiSyncSubmit extends AccordExecutorAbstractSemiSync
 
     private void awaitWork() throws InterruptedException
     {
-        waiting++;
+        hasWork.await();
-        try { hasWork.await(); }
-        finally { waiting--; }
     }
 
     @Override
-    AccordExecutorLoops loops()
+    Loops loops()
     {
         return loops;
     }
@@ -99,7 +84,7 @@ public class AccordExecutorSemiSyncSubmit extends AccordExecutorAbstractSemiSync
     }
 
     @Override
-    boolean isOwningThread()
+    public boolean isOwningThread()
     {
         return lock.isHeldByCurrentThread();
     }

src/java/org/apache/cassandra/service/accord/execution/AccordExecutorSignalLoop.java

@@ -0,0 +1,483 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+
+import javax.annotation.Nullable;
+
+import accord.api.Agent;
+import accord.utils.Invariants;
+
+import org.apache.cassandra.service.accord.debug.DebugExecution.DebugTask;
+import org.apache.cassandra.utils.concurrent.SignalLock;
+
+import static accord.utils.Invariants.nonNull;
+import static org.apache.cassandra.service.accord.debug.DebugExecution.DEBUG_EXECUTION;
+import static org.apache.cassandra.service.accord.execution.Task.State.REGISTERED;
+
+public class AccordExecutorSignalLoop extends AbstractLoop
+{
+    private static class ShutdownException extends RuntimeException {}
+
+    private static final int MAX_LOOPS = 1000; // limit the amount of time we hold the lock for
+    private final SignalLock lock;
+    private final Loops loops;
+    private int readyToRunTarget = 1;
+    private final int readyToRunLimit;
+    // TODO (desired): intrusive queue using Task.next, but a little challenging because we reuse SequentialQueueTask so have ABA problem
+    private final ConcurrentLinkedQueue<Task> readyToRun = new ConcurrentLinkedQueue<>();
+
+    private Task pendingExecutedHead, pendingExecutedTail;
+    private Task pendingNewHead, pendingNewTail;
+    private int pendingCount;
+
+    private boolean shutdown;
+
+    public AccordExecutorSignalLoop(int executorId, Mode mode, int threads, long spinInterval, long stopCheckInterval, TimeUnit units, IntFunction<String> name, Agent agent)
+    {
+        this(new SignalLock(threads, spinInterval, stopCheckInterval, units), executorId, mode, threads, name, agent);
+    }
+
+    public AccordExecutorSignalLoop(SignalLock lock, int executorId, Mode mode, int threads, IntFunction<String> name, Agent agent)
+    {
+        super(lock, executorId, agent);
+        Invariants.require(threads <= SignalLock.MAX_THREADS);
+        this.lock = lock;
+        this.loops = new Loops(mode, threads, name, this::task);
+        this.readyToRunLimit = Math.min(threads * 4, SignalLock.MAX_SIGNAL_COUNT);
+    }
+
+    <P1> void submit(Consumer<P1> sync, Function<P1, Task> async, P1 p1)
+    {
+        Task next = async.apply(p1);
+        Task prev = push(next);
+        if (prev == null)
+            lock.signalLockWork();
+    }
+
+    @Override
+    final void beforeUnlockExternal()
+    {
+    }
+
+    private Task pollReadyToRun()
+    {
+        return readyToRun.poll();
+    }
+
+    private void addReadyToRun(Task task)
+    {
+        readyToRun.add(task);
+    }
+
+    private boolean hasReadyToRun()
+    {
+        return !readyToRun.isEmpty();
+    }
+
+    private LoopTask task(int index, String id, Mode mode)
+    {
+        return new LoopTask(index, id);
+    }
+
+    @Override
+    final void drainUnqueuedNewWorkExclusive()
+    {
+        updatePendingUnqueued();
+        Task requeue = null, requeueTail = null;
+        Task submit = null, submitTail = null;
+        Task cur = pendingNewHead;
+        while (cur != null)
+        {
+            Task next = cur.next;
+            cur.next = null;
+            if (cur.isNewWork())
+            {
+                if (submit == null) submit = cur;
+                else submitTail.next = cur;
+                submitTail = cur;
+                --pendingCount;
+            }
+            else
+            {
+                if (requeue == null) requeue = cur;
+                else requeueTail.next = cur;
+                requeueTail = cur;
+            }
+            cur = next;
+        }
+
+        pendingNewHead = requeue;
+        pendingNewTail = requeueTail;
+        while (submit != null)
+        {
+            Task next = submit.next;
+            submit.next = null;
+            submit.submitExclusiveNoExcept();
+            submit = next;
+        }
+    }
+
+    private boolean enqueueOnePending()
+    {
+        if (pendingCount == 0)
+            return false;
+
+        --pendingCount;
+        if (pendingExecutedHead != null)
+        {
+            Task executed = pendingExecutedHead;
+            pendingExecutedHead = destructiveNext(executed);
+            if (pendingExecutedHead == null)
+                pendingExecutedTail = null;
+            executed.completeExclusiveNoExcept();
+        }
+        else
+        {
+            Task submit = pendingNewHead;
+            pendingNewHead = destructiveNext(submit);
+            if (pendingNewHead == null)
+                pendingNewTail = null;
+            submit.submitExclusiveNoExcept();
+        }
+        return true;
+    }
+
+    private void fetchWorkExclusive()
+    {
+        boolean hasReadyToRun = hasReadyToRun();
+        boolean hadWaitingToRun = hasAlreadyWaitingToRun();
+        {
+            int prevPendingUnqueued = pendingCount;
+            updateAndEnqueuePendingUntilHasWaitingToRun(prevPendingUnqueued / 2);
+        }
+
+        if (hadWaitingToRun && hasReadyToRun)
+        {
+            lock.addAndGetEnabledThreadCount(1);
+        }
+        else if (hadWaitingToRun)
+        {
+            readyToRunTarget = Math.min(readyToRunLimit, readyToRunTarget + (1+readyToRunTarget)/2);
+        }
+        else if (hasReadyToRun && readyToRunTarget > 1)
+        {
+            --readyToRunTarget;
+        }
+
+        boolean hasDrainedSignal = false;
+        int loops = 0;
+        while (true)
+        {
+            long state = lock.state();
+            int signals = SignalLock.asyncSignalCount(state);
+            int waiters = SignalLock.waitingEnabledThreadCount(state);
+            if (signals > 0)
+            {
+                if (++loops > MAX_LOOPS)
+                    return;
+
+                if (signals >= readyToRunTarget)
+                {
+                    if (enqueueOnePending() || (updatePendingUnqueued() && enqueueOnePending())) continue;
+                    else if (hasDrainedSignal)
+                        lock.signalLockWorkExclusive();
+                    return;
+                }
+                else if (waiters > 0 && signals > 1 && SignalLock.activeEnabledThreadCount(state) == 1)
+                {
+                    // ensure at least one other thread is running if there's enough work for it; it will spin up other threads if necessary
+                    lock.propagateAsyncWorkSignals(1);
+                }
+            }
+
+            Task task = pollAlreadyWaitingToRunExclusive();
+            if (task == null)
+            {
+                if (updateAndEnqueuePendingUntilHasWaitingToRun(0))
+                    continue;
+
+                lock.clearLockWork();
+                hasDrainedSignal = true;
+                if (!updateAndEnqueuePendingUntilHasWaitingToRun(0))
+                {
+                    if (tasks == 0)
+                        notifyQuiescentExclusive();
+                    return;
+                }
+            }
+            else if (task.prepareExclusiveNoExcept())
+            {
+                if (DEBUG_EXECUTION) DebugTask.get(task).onPreRun();
+                addReadyToRun(task);
+                boolean incremented = lock.incrementAsyncWork(false);
+                Invariants.require(incremented);
+            }
+        }
+    }
+
+    private boolean updatePendingUnqueued()
+    {
+        if (!hasUnqueued())
+            return false;
+
+        int count = 0;
+        Task addExecutedHead = null, addExecutedTail = null;
+        Task addNewHead = null, addNewTail = null;
+        {
+            Task cur = Task.reverse(acquireUnqueuedExclusive());
+            while (cur != null)
+            {
+                Task next = cur.next;
+                if (cur.compareTo(REGISTERED) < 0)
+                {
+                    if (addNewHead == null) addNewHead = addNewTail = setNextNull(cur);
+                    else addNewHead = reverseOne(addNewHead, cur);
+                }
+                else
+                {
+                    if (addExecutedHead == null) addExecutedHead = addExecutedTail = setNextNull(cur);
+                    else addExecutedHead = reverseOne(addExecutedHead, cur);
+                }
+                ++count;
+                cur = next;
+            }
+        }
+
+        pendingCount += count;
+        if (addExecutedHead != null)
+        {
+            if (pendingExecutedHead == null) pendingExecutedHead = addExecutedHead;
+            else pendingExecutedTail.next = addExecutedHead;
+            pendingExecutedTail = addExecutedTail;
+        }
+        if (addNewHead != null)
+        {
+            if (pendingNewHead == null) pendingNewHead = addNewHead;
+            else pendingNewTail.next = addNewHead;
+            pendingNewTail = addNewTail;
+        }
+        return true;
+    }
+
+    private Task reverseOne(Task prev, Task cur)
+    {
+        cur.next = prev;
+        return cur;
+    }
+
+    private Task setNextNull(Task cur)
+    {
+        cur.next = null;
+        return cur;
+    }
+
+    private boolean updateAndEnqueuePendingUntilHasWaitingToRun(int processAtLeast)
+    {
+        updatePendingUnqueued();
+        int count = 0;
+        while (enqueueOnePending())
+        {
+            if (++count >= processAtLeast && hasAlreadyWaitingToRun())
+                return true;
+        }
+        return hasAlreadyWaitingToRun();
+    }
+
+    class LoopTask extends Loops.LoopTask
+    {
+        final int index;
+
+        LoopTask(int index, String id)
+        {
+            super(id);
+            this.index = index;
+        }
+
+        private Task awaitWork(TaskRunner self)
+        {
+            while (true)
+            {
+                if (lock.awaitAsyncOrLock(index))
+                {
+                    try
+                    {
+                        if (DEBUG_EXECUTION) debug.onEnterLock();
+                        fetchWorkExclusive();
+                    }
+                    catch (Throwable t)
+                    {
+                        unlock(self);
+                        throw t;
+                    }
+
+                    if (!unlockAndAcquire(self))
+                        continue;
+                }
+
+                if (shutdown)
+                    throw new ShutdownException();
+
+                return nonNull(pollReadyToRun());
+            }
+        }
+
+        private Task executedAndMaybeGetWork(TaskRunner self, @Nullable Task executed)
+        {
+            if (lock.tryAcquireAsyncWork())
+            {
+                if (shutdown)
+                    throw new ShutdownException();
+
+                return pushExecutedAndReturn(executed, nonNull(pollReadyToRun()));
+            }
+
+            if (!tryLock(self))
+                return pushExecutedAndReturn(executed, null);
+
+            try
+            {
+                if (executed != null)
+                    executed.completeExclusiveNoExcept();
+                fetchWorkExclusive();
+            }
+            catch (Throwable t)
+            {
+                unlock(self);
+                throw t;
+            }
+
+            if (unlockAndAcquire(self))
+            {
+                if (shutdown)
+                    throw new ShutdownException();
+
+                return nonNull(pollReadyToRun());
+            }
+            return null;
+        }
+
+        private Task pushExecutedAndReturn(Task complete, Task result)
+        {
+            if (push(complete) == null)
+                lock.signalLockWork();
+            return result;
+        }
+
+        final boolean tryLock(TaskRunner self)
+        {
+            if (self.accordLockedExecutor() != null)
+                return false;
+
+            return onTryLock(self, lock.tryLock(index));
+        }
+
+        final boolean unlockAndAcquire(TaskRunner self)
+        {
+            self.exitAccordLockedExecutor();
+            if (DEBUG_EXECUTION) debug.onExitLock();
+            return lock.unlockAndAcquireAsyncWork();
+        }
+
+        @Override
+        public void run()
+        {
+            Thread thread = Thread.currentThread();
+            TaskRunner self = TaskRunner.get(thread);
+            self.setAccordActiveExecutor(AccordExecutorSignalLoop.this);
+            setWrapped(self);
+
+            lock.register(index, thread);
+            Task task = null;
+            while (true)
+            {
+                try
+                {
+                    if (task != null)
+                    {
+                        try { task = executedAndMaybeGetWork(self, task); }
+                        catch (Throwable t) { task = null; throw t; }
+                    }
+                    if (task == null)
+                        task = awaitWork(self);
+
+                    task.runNoExcept(self);
+                }
+                catch (ShutdownException ignore)
+                {
+                    break;
+                }
+                catch (Throwable t)
+                {
+                    agent.onException(t);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void shutdown()
+    {
+        lock.lock();
+        try
+        {
+            shutdown = true;
+            lock.signalAllRegistered();
+        }
+        finally
+        {
+            lock.unlock();
+        }
+    }
+
+    @Override
+    Loops loops()
+    {
+        return loops;
+    }
+
+    @Override
+    boolean isInLoop()
+    {
+        return loops.isInLoop();
+    }
+
+    @Override
+    public boolean isOwningThread()
+    {
+        return lock.isOwner();
+    }
+
+    @Override
+    public boolean isTerminated()
+    {
+        return loops.isTerminated();
+    }
+
+    @Override
+    public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException
+    {
+        return loops.awaitTermination(timeout, unit);
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/AccordExecutorSimple.java

@@ -16,22 +16,22 @@
  * limitations under the License.
  */
 
-package org.apache.cassandra.service.accord;
+package org.apache.cassandra.service.accord.execution;
 
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.Consumer;
+import java.util.function.Function;
 import java.util.function.IntFunction;
 
 import accord.api.Agent;
 import accord.utils.Invariants;
-import accord.utils.QuadFunction;
-import accord.utils.QuintConsumer;
 
 import org.apache.cassandra.concurrent.ExecutorPlus;
 
 import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
 
-class AccordExecutorSimple extends AccordExecutor
+public class AccordExecutorSimple extends AccordExecutor
 {
     final ExecutorPlus executor;
     final ReentrantLock lock;
@@ -82,7 +82,8 @@ class AccordExecutorSimple extends AccordExecutor
 
     protected void run()
     {
-        Thread self = Thread.currentThread();
+        TaskRunner self = TaskRunner.get();
+        self.setAccordActiveExecutor(AccordExecutorSimple.this);
         lock.lock();
         try
         {
@@ -95,12 +96,7 @@ class AccordExecutorSimple extends AccordExecutor
                     return;
                 }
 
-                try { task.preRunExclusive(); task.runInternal(); }
+                prepareRunComplete(self, task);
-                catch (Throwable t) { task.fail(t); }
-                finally
-                {
-                    completeTaskExclusive(task);
-                }
             }
         }
         finally
@@ -112,12 +108,12 @@ class AccordExecutorSimple extends AccordExecutor
     }
 
     @Override
-    <P1s, P1a, P2, P3, P4> void submit(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Task> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4)
+    <P1> void submit(Consumer<P1> sync, Function<P1, Task> async, P1 p1)
     {
         lock.lock();
         try
         {
-            sync.accept(this, p1s, p2, p3, p4);
+            sync.accept(p1);
         }
         finally
         {
@@ -128,8 +124,20 @@ class AccordExecutorSimple extends AccordExecutor
         }
     }
 
+    final boolean hasWaitingToRun()
+    {
+        updateWaitingToRunExclusive();
+        return hasAlreadyWaitingToRun();
+    }
+
+    final Task pollWaitingToRunExclusive()
+    {
+        updateWaitingToRunExclusive();
+        return pollAlreadyWaitingToRunExclusive();
+    }
+
     @Override
-    boolean isOwningThread()
+    public boolean isOwningThread()
     {
         return lock.isHeldByCurrentThread();
     }

src/java/org/apache/cassandra/service/accord/execution/AccordExecutorSyncSubmit.java

@@ -16,20 +16,20 @@
  * limitations under the License.
  */
 
-package org.apache.cassandra.service.accord;
+package org.apache.cassandra.service.accord.execution;
 
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.locks.Condition;
 import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.Consumer;
+import java.util.function.Function;
 import java.util.function.IntFunction;
 
 import accord.api.Agent;
-import accord.utils.QuadFunction;
-import accord.utils.QuintConsumer;
 
-public class AccordExecutorSyncSubmit extends AccordExecutorAbstractLockLoop
+public class AccordExecutorSyncSubmit extends AbstractLockLoop
 {
-    private final AccordExecutorLoops loops;
+    private final Loops loops;
     private final ReentrantLock lock;
     private final Condition hasWork;
 
@@ -48,7 +48,7 @@ public class AccordExecutorSyncSubmit extends AccordExecutorAbstractLockLoop
         super(lock, executorId, agent);
         this.lock = lock;
         this.hasWork = lock.newCondition();
-        this.loops = new AccordExecutorLoops(mode, threads, name, this::task);
+        this.loops = new Loops(mode, threads, name, this::task);
     }
 
     @Override
@@ -58,7 +58,7 @@ public class AccordExecutorSyncSubmit extends AccordExecutorAbstractLockLoop
     }
 
     @Override
-    AccordExecutorLoops loops()
+    Loops loops()
     {
         return loops;
     }
@@ -70,7 +70,7 @@ public class AccordExecutorSyncSubmit extends AccordExecutorAbstractLockLoop
     }
 
     @Override
-    boolean isOwningThread()
+    public boolean isOwningThread()
     {
         return lock.isHeldByCurrentThread();
     }
@@ -89,16 +89,17 @@ public class AccordExecutorSyncSubmit extends AccordExecutorAbstractLockLoop
         hasWork.signal();
     }
 
-    <P1s, P1a, P2, P3, P4> void submitExternal(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Task> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4)
+    <P1> void submitExternal(Consumer<P1> sync, Function<P1, Task> async, P1 p1)
     {
-        lock();
+        TaskRunner self = TaskRunner.get();
+        lock(self);
         try
         {
-            submitExternalExclusive(sync, p1s, p2, p3, p4);
+            submitExternalExclusive(sync, async, p1);
         }
         finally
         {
-            unlock();
+            unlock(self);
         }
     }
 

src/java/org/apache/cassandra/service/accord/execution/CancelTask.java

@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+final class CancelTask extends Task
+{
+    final Task cancel;
+
+    CancelTask(Task cancel)
+    {
+        super(GlobalGroup.OTHER);
+        this.cancel = cancel;
+    }
+
+    @Override
+    void submitExclusiveMayThrow()
+    {
+        cancel.tryCancelExclusive();
+    }
+
+    @Override
+    boolean isNewWork()
+    {
+        return false;
+    }
+
+    @Override
+    public String description()
+    {
+        return "Cancel " + cancel.description();
+    }
+
+    @Override
+    public String briefDescription()
+    {
+        return "Cancel " + cancel.briefDescription();
+    }
+
+    @Override void unqueueIfQueued() { throw new UnsupportedOperationException(); }
+    @Override
+    void maybeCompleteExclusiveMayThrow() { throw new UnsupportedOperationException(); }
+    @Override void tryCancelExclusive() { throw new UnsupportedOperationException(); }
+    @Override void reportFailureMayThrow(Throwable fail) { throw new UnsupportedOperationException(); }
+    @Override boolean runMayThrow() { throw new UnsupportedOperationException(); }
+    @Override public void cancel() { throw new UnsupportedOperationException(); }
+    @Override AccordExecutor executor() { throw new UnsupportedOperationException(); }
+}

src/java/org/apache/cassandra/service/accord/execution/ExclusiveExecutor.java

@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.concurrent.locks.LockSupport;
+
+import accord.api.ExclusiveAsyncExecutor;
+import accord.local.ExecutionContext;
+import accord.utils.IntrusiveHeapNode;
+import accord.utils.Invariants;
+import accord.utils.async.AsyncCallbacks;
+import accord.utils.async.AsyncChain;
+import accord.utils.async.AsyncChains;
+import accord.utils.async.Cancellable;
+
+import org.apache.cassandra.service.accord.debug.DebugExecution;
+import org.apache.cassandra.service.accord.execution.Task.GroupKind;
+
+import static org.apache.cassandra.service.accord.debug.DebugExecution.DEBUG_EXECUTION;
+import static org.apache.cassandra.service.accord.execution.AccordExecutor.EXCLUSIVE_QUEUE_LIMITS;
+import static org.apache.cassandra.service.accord.execution.Task.ExecutorQueue.RUNNABLE;
+import static org.apache.cassandra.service.accord.execution.Task.GlobalGroup.COMMAND_STORE;
+import static org.apache.cassandra.service.accord.execution.Task.State.WAITING_TO_RUN;
+
+public final class ExclusiveExecutor extends TaskQueueMulti<Task> implements ExclusiveAsyncExecutor
+{
+    private static final AtomicReferenceFieldUpdater<ExclusiveExecutor, Thread> ownerUpdater = AtomicReferenceFieldUpdater.newUpdater(ExclusiveExecutor.class, Thread.class, "owner");
+
+    static final class ExclusiveExecutorTask extends Task
+    {
+        final ExclusiveExecutor queue;
+
+        ExclusiveExecutorTask(ExclusiveExecutor queue)
+        {
+            super(COMMAND_STORE);
+            this.queue = queue;
+        }
+
+        @Override AccordExecutor executor() { return queue.executor; }
+        @Override void submitExclusiveMayThrow() { throw new UnsupportedOperationException(); }
+        @Override boolean isNewWork() { throw new UnsupportedOperationException(); }
+        @Override public void cancel() { throw new UnsupportedOperationException(); }
+        @Override boolean runMayThrow() { throw new UnsupportedOperationException(); }
+        @Override void unqueueIfQueued() {}
+        @Override void reportFailureMayThrow(Throwable t) { throw new UnsupportedOperationException(); }
+        @Override void tryCancelExclusive() { throw new UnsupportedOperationException(); }
+
+        boolean prepareTask()
+        {
+            Task task = queue.task;
+            try
+            {
+                task.prepareExclusiveMayThrow();
+                task.setStateExclusive(State.PREPARED);
+                setStateExclusive(State.PREPARED);
+                return true;
+            }
+            catch (Throwable t)
+            {
+                task.setStateExclusive(State.FAILED);
+                task.reportFailureNoExcept(t);
+                maybeCompleteExclusiveMayThrow();
+                return false;
+            }
+        }
+
+        @Override
+        void maybeCompleteExclusiveMayThrow()
+        {
+            try
+            {
+                unsafeSetStateExclusive(WAITING_TO_RUN);
+                executor().runnable.cleanup(this);
+                queue.completeTask();
+            }
+            catch (Throwable t)
+            {
+                onException(t);
+            }
+        }
+
+        @Override
+        public String description()
+        {
+            return queue.task.description();
+        }
+
+        @Override
+        public String briefDescription()
+        {
+            return queue.task.briefDescription();
+        }
+
+        protected boolean isInHeap()
+        {
+            return super.isInHeap();
+        }
+    }
+
+    private final AccordExecutor executor;
+    final int commandStoreId;
+    final ExclusiveExecutorTask selfTask;
+    Task task;
+    volatile Thread owner, waiting;
+    private boolean stopped;
+    private volatile boolean visibleStopped;
+    private boolean terminated;
+
+    final DebugExecution.DebugExclusiveExecutor debug;
+
+    ExclusiveExecutor(AccordExecutor executor)
+    {
+        this(executor, -1);
+    }
+
+    ExclusiveExecutor(AccordExecutor executor, int commandStoreId)
+    {
+        super(RUNNABLE, commandStoreId < 0 ? GroupKind.NONE : GroupKind.EXCLUSIVE, EXCLUSIVE_QUEUE_LIMITS);
+        this.executor = executor;
+        this.commandStoreId = commandStoreId;
+        this.selfTask = new ExclusiveExecutorTask(this);
+        this.debug = DebugExecution.DebugExclusiveExecutor.maybeDebug(executor.debug, commandStoreId);
+    }
+
+    void runTask(TaskRunner self)
+    {
+        Thread thread = Thread.currentThread();
+        if (!ownerUpdater.compareAndSet(this, null, thread))
+        {
+            if (DEBUG_EXECUTION) debug.onWaiting();
+            Invariants.require(waiting == null);
+            waiting = thread;
+            outer:
+            do
+            {
+                while (true)
+                {
+                    Thread owner = this.owner;
+                    if (owner == thread) break outer;
+                    if (owner == null) continue outer;
+                    LockSupport.park();
+                }
+            }
+            while (!ownerUpdater.compareAndSet(this, null, thread));
+            Invariants.require(waiting == thread);
+            waiting = null;
+        }
+
+        try
+        {
+            if (stopped && reject(task))
+                task.rejectAtRuntime(new RejectedExecutionException(commandStoreId + " is terminated. Cannot execute " + task.description()));
+            else
+                task.runNoExcept(self);
+        }
+        finally
+        {
+            // NOTE: we can ONLY safely release owner here due to AccordCacheEntry locking, which remains in place until AccordTask.releaseResourcesExclusive
+            //       this also relies on AccordSafeCommandStore$ExclusiveCaches.acquireIfLoaded returning false when the entry is locked
+            owner = null;
+        }
+    }
+
+    private boolean reject(Task task)
+    {
+        if (!(task instanceof SafeTask<?>))
+            return true;
+
+        ExecutionContext context = ((SafeTask<?>) task).executionContext();
+        return !(terminated ? (context instanceof Unterminatable) : (context instanceof Unstoppable));
+    }
+
+    void completeTask()
+    {
+        try
+        {
+            task.unsetQueue(kind);
+            task.completeExclusiveNoExcept();
+        }
+        finally
+        {
+            active = 0;
+            task = super.pollMulti();
+            if (DEBUG_EXECUTION) debug.onSetTask(task);
+            if (task != null)
+            {
+                selfTask.position = task.position;
+                selfTask.unsafeSetStateExclusive(WAITING_TO_RUN);
+                executor.runnable.enqueue(selfTask, false);
+            }
+        }
+    }
+
+    void enqueue(Task newTask, boolean incrementArrivals)
+    {
+
+        if (task != null)
+        {
+            if (incrementArrivals)
+                executor.runnable.incrementArrivals(selfTask);
+            // TODO (expected): restore some invariant here
+//                Invariants.require(selfTask.isInHeap() || selfTask.is(RUNNING));
+            super.enqueueMulti(newTask, incrementArrivals);
+        }
+        else
+        {
+            Invariants.require(isEmptySingle());
+            if (incrementArrivals)
+                incrementArrivals(newTask);
+            incrementDispatches(newTask);
+            task = newTask;
+            task.setQueue(kind);
+            selfTask.position = newTask.position;
+            selfTask.unsafeSetStateExclusive(WAITING_TO_RUN);
+            executor.runnable.enqueue(selfTask, incrementArrivals);
+            if (DEBUG_EXECUTION) debug.onSetTask(newTask);
+        }
+    }
+
+    @Override
+    void unqueue(Task remove)
+    {
+        if (remove == task) removeCurrentTask(remove);
+        else super.unqueueMulti(remove);
+    }
+
+    boolean tryUnqueueWaiting(Task remove)
+    {
+        if (remove == task) return tryRemoveCurrentTask(remove);
+        else return super.tryUnqueueWaiting(remove);
+    }
+
+    private boolean tryRemoveCurrentTask(IntrusiveHeapNode remove)
+    {
+        if (executor.runnable.isAssigned(selfTask))
+            return false;
+
+        removeCurrentTask(remove);
+        return true;
+    }
+
+    private void removeCurrentTask(IntrusiveHeapNode remove)
+    {
+        Invariants.require(remove == task);
+        // cannot overwrite task while it is being executed - this cannot happen for AccordTask
+        // but can for other tasks that don't track their own state
+
+        decrementDispatches(task);
+        task.unsetQueue(kind);
+        task = pollMulti();
+        if (DEBUG_EXECUTION) debug.onSetTask(task);
+        if (executor.runnable.isWaiting(selfTask))
+        {
+            if (task == null) executor.runnable.unqueue(selfTask);
+            else
+            {
+                selfTask.position = task.position;
+                executor.runnable.requeue(selfTask);
+            }
+        }
+        else
+        {
+            Invariants.expect(false, "%s should have been queued to run as it had the task %s pending, that has now been cancelled", this, remove);
+            if (task != null)
+            {
+                selfTask.position = task.position;
+                selfTask.unsafeSetStateExclusive(WAITING_TO_RUN);
+                executor.runnable.enqueue(selfTask, false);
+            }
+        }
+        Invariants.require(task == null || executor.runnable.isWaiting(selfTask));
+    }
+
+    public boolean inExecutor()
+    {
+        return owner == Thread.currentThread();
+    }
+
+    public boolean stopped()
+    {
+        return visibleStopped;
+    }
+
+    public void stop()
+    {
+        Invariants.require(inExecutor());
+        this.stopped = true;
+        this.visibleStopped = true;
+    }
+
+    public void terminate()
+    {
+        Invariants.require(inExecutor());
+        this.visibleStopped = this.terminated = this.stopped = true;
+    }
+
+    @Override
+    public AsyncChain<Void> chain(Runnable run)
+    {
+        return AsyncChains.chain(this, run);
+    }
+
+    @Override
+    public <V> AsyncChain<V> chain(Callable<V> call)
+    {
+        return AsyncChains.chain(this, call);
+    }
+
+    @Override
+    public <V> AsyncChain<V> flatChain(Callable<? extends AsyncChain<V>> call)
+    {
+        return AsyncChains.flatChain(this, call);
+    }
+
+    @Override
+    public void execute(Runnable run)
+    {
+        Task inherit = executor.inherit();
+        PlainRunnable task = new PlainRunnable(executor, null, run, this, Task.ExclusiveGroup.OTHER);
+        if (inherit != null) inherit.addConsequence(task);
+        else executor.submitTask(task);
+    }
+
+    @Override
+    public Cancellable execute(AsyncCallbacks.RunOrFail runOrFail)
+    {
+        Task inherit = executor.inherit();
+        PlainChain task = new PlainChain(executor, runOrFail, ExclusiveExecutor.this, Task.ExclusiveGroup.OTHER);
+        if (inherit != null) inherit.addConsequence(task);
+        else executor.submitTask(task);
+        return task;
+    }
+
+    @Override
+    public boolean tryExecuteImmediately(Runnable run)
+    {
+        Thread thread = Thread.currentThread();
+        Thread owner = this.owner;
+        if (owner != null && owner != thread)
+            return false;
+
+        TaskRunner self = TaskRunner.get(thread);
+        AccordExecutor active = self.accordActiveExecutor();
+        if (active != null && active != executor)
+            return false; // prevent cross-executor locking/execution
+
+        if (owner == null && !ownerUpdater.compareAndSet(this, null, thread))
+            return false;
+
+        try
+        {
+            if (active == null)
+                self.setAccordActiveExecutor(executor);
+
+            run.run();
+        }
+        catch (Throwable t)
+        {
+            selfTask.onException(t);
+        }
+        finally
+        {
+            if (owner == null)
+            {
+                Thread waiting = this.waiting;
+                Invariants.require(waiting != self);
+                this.owner = waiting;
+                if (waiting == null) // recheck, to ensure happens-before relation with a new waiter that expects any non-null owner to notify it
+                    waiting = this.waiting;
+                if (waiting != null)
+                    LockSupport.unpark(waiting);
+            }
+
+            if (active == null)
+                self.setAccordActiveExecutor(null);
+        }
+        return true;
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/IOTask.java

@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import accord.utils.async.Cancellable;
+
+import org.apache.cassandra.concurrent.DebuggableTask;
+
+// a task that may be submitted to this executor or another
+public abstract class IOTask extends Plain implements Cancellable, DebuggableTask
+{
+    IOTask(AccordExecutor executor, GlobalGroup group)
+    {
+        super(executor, group);
+    }
+
+    @Override
+    ExclusiveExecutor exclusiveExecutor()
+    {
+        return null;
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/IOTaskLoad.java

@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import accord.utils.Invariants;
+
+import static org.apache.cassandra.service.accord.execution.IOTaskLoad.FailureHolder.NOT_STARTED;
+import static org.apache.cassandra.service.accord.execution.Task.GlobalGroup.LOAD;
+import static org.apache.cassandra.service.accord.execution.Task.GlobalGroup.RANGE_LOAD;
+
+public class IOTaskLoad<K, V> extends IOTask
+{
+    static class FailureHolder
+    {
+        static final FailureHolder NOT_STARTED = new FailureHolder(new RuntimeException("Not started"));
+
+        final Throwable fail;
+
+        FailureHolder(Throwable fail)
+        {
+            this.fail = fail;
+        }
+    }
+
+    final AccordCacheEntry<K, V, ?> entry;
+    Object result = NOT_STARTED;
+
+    IOTaskLoad(AccordExecutor executor, AccordCacheEntry<K, V, ?> entry, GlobalGroup group)
+    {
+        super(executor, group);
+        Invariants.require(group == LOAD || group == RANGE_LOAD);
+        this.entry = entry;
+    }
+
+    @Override
+    void maybeCompleteExclusiveMayThrow()
+    {
+        if (!(result instanceof FailureHolder))
+            executor.onLoadedExclusive(entry, (V) result, null);
+        else
+            executor.onLoadedExclusive(entry, null, ((FailureHolder) result).fail);
+        super.maybeCompleteExclusiveMayThrow();
+    }
+
+    @Override
+    public boolean runMayThrow()
+    {
+        result = entry.owner.parent().adapter().load(entry.owner.commandStore, entry.key());
+        return true;
+    }
+
+    @Override
+    void reportFailureMayThrow(Throwable t)
+    {
+        result = new FailureHolder(t);
+    }
+
+    @Override
+    public String description()
+    {
+        return "Load " + entry.key();
+    }
+
+    @Override
+    public String briefDescription()
+    {
+        return description();
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/IOTaskSave.java

@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import static org.apache.cassandra.service.accord.execution.Task.GlobalGroup.SAVE;
+
+class IOTaskSave extends IOTask
+{
+    private static final Throwable NOT_STARTED = new Throwable();
+
+    final AccordCacheEntry<?, ?, ?> entry;
+    final AccordCacheEntry.UniqueSave identity;
+    final Runnable run;
+    Throwable failure = NOT_STARTED;
+
+    IOTaskSave(AccordExecutor executor, AccordCacheEntry<?, ?, ?> entry, AccordCacheEntry.UniqueSave identity, Runnable run)
+    {
+        super(executor, SAVE);
+        this.entry = entry;
+        this.identity = identity;
+        this.run = run;
+    }
+
+    @Override
+    void maybeCompleteExclusiveMayThrow()
+    {
+        executor.onSavedExclusive(entry, identity, failure);
+        super.maybeCompleteExclusiveMayThrow();
+    }
+
+    @Override
+    public boolean runMayThrow()
+    {
+        run.run();
+        failure = null;
+        return true;
+    }
+
+    @Override
+    void reportFailureMayThrow(Throwable t)
+    {
+        failure = t;
+    }
+
+    @Override
+    public String description()
+    {
+        return "Save " + entry;
+    }
+
+    @Override
+    String briefDescription()
+    {
+        return description();
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/IOTaskWrapper.java

@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+class IOTaskWrapper extends IOTask
+{
+    static abstract class WrappableIOTask
+    {
+        abstract void runInternal();
+        abstract void postRunExclusive();
+        abstract void fail(Throwable t);
+        abstract String description();
+    }
+
+    final WrappableIOTask wrapped;
+
+    IOTaskWrapper(AccordExecutor executor, WrappableIOTask wrap, GlobalGroup group)
+    {
+        super(executor, group);
+        this.wrapped = wrap;
+    }
+
+    @Override
+    protected boolean runMayThrow()
+    {
+        wrapped.runInternal();
+        return true;
+    }
+
+    @Override
+    void maybeCompleteExclusiveMayThrow()
+    {
+        wrapped.postRunExclusive();
+        super.maybeCompleteExclusiveMayThrow();
+    }
+
+    @Override
+    public String description()
+    {
+        return wrapped.description();
+    }
+
+    @Override
+    public String briefDescription()
+    {
+        return description();
+    }
+
+    @Override
+    void reportFailureMayThrow(Throwable fail)
+    {
+        wrapped.fail(fail);
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/Loops.java

@@ -16,7 +16,7 @@
  * limitations under the License.
  */
 
-package org.apache.cassandra.service.accord;
+package org.apache.cassandra.service.accord.execution;
 
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
@@ -29,19 +29,19 @@ import accord.utils.Invariants;
 import accord.utils.TriFunction;
 
 import org.apache.cassandra.concurrent.DebuggableTask.DebuggableTaskRunner;
-import org.apache.cassandra.service.accord.AccordExecutor.Mode;
+import org.apache.cassandra.service.accord.execution.AccordExecutor.Mode;
-import org.apache.cassandra.service.accord.AccordExecutor.TaskRunner;
+import org.apache.cassandra.service.accord.execution.TaskRunner.DebuggableWrapper;
 import org.apache.cassandra.utils.concurrent.Condition;
 
 import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
 import static org.apache.cassandra.concurrent.ExecutorFactory.SimulatorThreadTag.INFINITE_LOOP;
 import static org.apache.cassandra.concurrent.ExecutorFactory.SystemThreadTag.NON_DAEMON;
-import static org.apache.cassandra.service.accord.AccordExecutor.Mode.RUN_WITH_LOCK;
+import static org.apache.cassandra.service.accord.execution.AccordExecutor.Mode.RUN_WITH_LOCK;
 import static org.apache.cassandra.utils.Clock.Global.nanoTime;
 
-class AccordExecutorLoops
+class Loops
 {
-    static abstract class LoopTask extends TaskRunner implements Runnable
+    static abstract class LoopTask extends DebuggableWrapper implements Runnable
     {
         final String id;
         LoopTask(String id) { this.id = id; }
@@ -54,7 +54,7 @@ class AccordExecutorLoops
     private final AtomicInteger running = new AtomicInteger();
     private final Condition terminated = Condition.newOneTimeCondition();
 
-    public AccordExecutorLoops(Mode mode, int threads, IntFunction<String> loopName, TriFunction<Integer, String, Mode, LoopTask> loopFactory)
+    public Loops(Mode mode, int threads, IntFunction<String> loopName, TriFunction<Integer, String, Mode, LoopTask> loopFactory)
     {
         Invariants.require(mode == RUN_WITH_LOCK ? threads == 1 : threads >= 1);
         running.addAndGet(threads);

src/java/org/apache/cassandra/service/accord/execution/Plain.java

@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import java.util.concurrent.CancellationException;
+
+import accord.utils.async.Cancellable;
+
+import static org.apache.cassandra.service.accord.execution.Task.State.CANCELLED;
+import static org.apache.cassandra.service.accord.execution.Task.State.WAITING_TO_RUN;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+abstract class Plain extends Task implements Cancellable
+{
+    final AccordExecutor executor;
+
+    Plain(AccordExecutor executor, GlobalGroup group)
+    {
+        super(group);
+        this.executor = executor;
+    }
+
+    Plain(AccordExecutor executor, ExclusiveGroup group)
+    {
+        super(group);
+        this.executor = executor;
+    }
+
+    abstract ExclusiveExecutor exclusiveExecutor();
+
+    @Override
+    public final void cancel()
+    {
+        executor.submit(Task::tryCancelExclusive, CancelTask::new, this);
+    }
+
+    @Override
+    final void tryCancelExclusive()
+    {
+        tryFailAndCompleteExclusive(new CancellationException(), CANCELLED);
+    }
+
+    @Override
+    final void submitExclusiveMayThrow()
+    {
+        executor.registerExclusive(this);
+        setStateExclusive(WAITING_TO_RUN);
+        ExclusiveExecutor exclusiveExecutor = exclusiveExecutor();
+        if (exclusiveExecutor == null) executor.runnable.enqueue(this, true);
+        else exclusiveExecutor.enqueue(this, true);
+    }
+
+    @Override
+    void maybeCompleteExclusiveMayThrow()
+    {
+        if (completeState())
+        {
+            long completeAt = nanoTime();
+            executor.elapsedWaitingToRun.increment(runningAt - createdAt, runningAt);
+            executor.elapsedRunning.increment(completeAt - runningAt, completeAt);
+            executor.elapsed.increment(completeAt - createdAt, completeAt);
+        }
+    }
+
+    @Override
+    void unqueueIfQueued()
+    {
+        if (isQueued())
+        {
+            ExclusiveExecutor exclusiveExecutor = exclusiveExecutor();
+            if (exclusiveExecutor != null) exclusiveExecutor.unqueue(this);
+            else executor.runnable.unqueue(this);
+        }
+    }
+
+    @Override
+    protected boolean isNewWork()
+    {
+        return true;
+    }
+
+    @Override
+    AccordExecutor executor()
+    {
+        return executor;
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/PlainChain.java

@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import javax.annotation.Nullable;
+
+import accord.utils.async.AsyncCallbacks;
+
+class PlainChain extends Plain
+{
+    final AsyncCallbacks.RunOrFail runOrFail;
+    final @Nullable ExclusiveExecutor exclusiveExecutor;
+
+    PlainChain(AccordExecutor executor, AsyncCallbacks.RunOrFail runOrFail, ExclusiveExecutor exclusiveExecutor, ExclusiveGroup group)
+    {
+        super(executor, group);
+        this.runOrFail = runOrFail;
+        this.exclusiveExecutor = exclusiveExecutor;
+    }
+
+    @Override
+    public String description()
+    {
+        return runOrFail.toString();
+    }
+
+    @Override
+    public String briefDescription()
+    {
+        return description();
+    }
+
+    @Override
+    boolean runMayThrow()
+    {
+        runOrFail.run();
+        return true;
+    }
+
+    @Override
+    void reportFailureMayThrow(Throwable fail)
+    {
+        runOrFail.fail(fail);
+    }
+
+    @Override
+    ExclusiveExecutor exclusiveExecutor()
+    {
+        return exclusiveExecutor;
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/PlainChainDebuggable.java

@@ -15,43 +15,29 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.apache.cassandra.service.accord;
 
-import accord.impl.SafeState;
+package org.apache.cassandra.service.accord.execution;
 
-public interface AccordSafeState<K, V> extends SafeState<V>
+import javax.annotation.Nullable;
+
+import accord.utils.Invariants;
+import accord.utils.async.AsyncCallbacks;
+
+import org.apache.cassandra.concurrent.DebuggableTask;
+
+class PlainChainDebuggable extends PlainChain implements DebuggableTask
 {
-    void set(V update);
+    final Object describe;
-    V original();
-    void markUnsafe();
-    boolean isUnsafe();
-    void preExecute();
 
-    AccordCacheEntry<K, V> global();
+    PlainChainDebuggable(AccordExecutor executor, AsyncCallbacks.RunOrFail runOrFail, @Nullable ExclusiveExecutor exclusiveExecutor, Object describe)
-
-    default boolean hasUpdate()
     {
-        return original() != current();
+        super(executor, runOrFail, exclusiveExecutor, ExclusiveGroup.OTHER);
+        this.describe = Invariants.nonNull(describe);
     }
 
-    default void revert()
+    @Override
+    public String description()
     {
-        set(original());
+        return describe.toString();
-    }
-
-    default K key()
-    {
-        return global().key();
-    }
-
-    default Throwable failure()
-    {
-        return global().failure();
-    }
-
-    default void checkNotInvalidated()
-    {
-        if (isUnsafe())
-            throw new IllegalStateException("Cannot access invalidated " + this);
     }
 }

src/java/org/apache/cassandra/service/accord/execution/PlainRunnable.java

@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import javax.annotation.Nullable;
+
+import accord.utils.async.Cancellable;
+
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+
+class PlainRunnable extends Plain implements Cancellable
+{
+    final @Nullable AsyncPromise<Void> result;
+    final Runnable run;
+    final @Nullable ExclusiveExecutor exclusiveExecutor;
+
+    PlainRunnable(AccordExecutor executor, AsyncPromise<Void> result, Runnable run, GlobalGroup group)
+    {
+        super(executor, group);
+        this.result = result;
+        this.run = run;
+        this.exclusiveExecutor = null;
+    }
+
+    PlainRunnable(AccordExecutor executor, AsyncPromise<Void> result, Runnable run, ExclusiveExecutor exclusiveExecutor, ExclusiveGroup group)
+    {
+        super(executor, group);
+        this.result = result;
+        this.run = run;
+        this.exclusiveExecutor = exclusiveExecutor;
+    }
+
+    @Override
+    protected boolean runMayThrow()
+    {
+        run.run();
+        if (result != null)
+        {
+            try { result.trySuccess(null); }
+            catch (Throwable t) { onException(t); }
+        }
+        return true;
+    }
+
+    @Override
+    public String description()
+    {
+        // TODO (expected): ensure this is usefully descriptive, or accept a separate description
+        return run.toString();
+    }
+
+    @Override
+    public String briefDescription()
+    {
+        return description();
+    }
+
+    @Override
+    void reportFailureMayThrow(Throwable t)
+    {
+        if (result != null)
+            result.tryFailure(t);
+    }
+
+    @Override
+    ExclusiveExecutor exclusiveExecutor()
+    {
+        return exclusiveExecutor;
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/SaferCommand.java

@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import java.util.Objects;
+
+import accord.api.Journal;
+import accord.local.Command;
+import accord.local.SafeCommand;
+import accord.primitives.TxnId;
+
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry.LockMode;
+
+public class SaferCommand extends SafeCommand implements SaferState<TxnId, Command, SaferCommand>
+{
+    private final AccordCacheEntry<TxnId, Command, SaferCommand> global;
+    private Command original;
+
+    public SaferCommand(AccordCacheEntry<TxnId, Command, SaferCommand> global)
+    {
+        super(global.key());
+        this.global = global;
+    }
+
+    @Override
+    public boolean equals(Object o)
+    {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+        SaferCommand that = (SaferCommand) o;
+        return Objects.equals(this.original, that.original) && Objects.equals(this.current(), that.current());
+    }
+
+    @Override
+    public int hashCode()
+    {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public String toString()
+    {
+        return "AccordSafeCommand{" +
+               "status=" + statusString() +
+               ", global=" + global +
+               ", original=" + original +
+               ", current=" + current +
+               '}';
+    }
+
+    public AccordCacheEntry<TxnId, Command, SaferCommand> global()
+    {
+        return global;
+    }
+
+    @Override
+    public void postExecute(SafeTask<?> owner)
+    {
+        global.releaseExclusive(this, owner);
+    }
+
+    public Journal.CommandUpdate update()
+    {
+        return new Journal.CommandUpdate(original, current());
+    }
+
+    public void preExecute(SafeTask<?> owner, LockMode lockMode)
+    {
+        requireUninitialised();
+        original = global.lockExclusive(owner, lockMode);
+        current = original;
+        if (current == null)
+            initialise();
+        setSafe();
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/SaferCommandStore.java

@@ -16,15 +16,10 @@
  * limitations under the License.
  */
 
-package org.apache.cassandra.service.accord;
+package org.apache.cassandra.service.accord.execution;
 
-import java.util.Collections;
-import java.util.Map;
-import java.util.Set;
 import java.util.function.Predicate;
 
-import javax.annotation.Nullable;
-
 import com.google.common.annotations.VisibleForTesting;
 
 import accord.api.Agent;
@@ -36,86 +31,76 @@ import accord.impl.AbstractSafeCommandStore;
 import accord.local.Command;
 import accord.local.CommandStores;
 import accord.local.CommandSummaries;
+import accord.local.ExecutionContext;
 import accord.local.NodeCommandStoreService;
 import accord.local.RedundantBefore;
+import accord.local.SafeState;
 import accord.local.cfk.CommandsForKey;
 import accord.local.cfk.SafeCommandsForKey;
+import accord.primitives.AbstractUnseekableKeys;
+import accord.primitives.Routables;
 import accord.primitives.Timestamp;
 import accord.primitives.Txn.Kind.Kinds;
 import accord.primitives.TxnId;
 import accord.primitives.Unseekables;
 import accord.utils.Invariants;
+import accord.utils.UnhandledEnum;
 
 import org.apache.cassandra.metrics.LogLinearDecayingHistograms;
+import org.apache.cassandra.service.accord.AccordCommandStore;
 import org.apache.cassandra.service.accord.AccordCommandStore.ExclusiveCaches;
 import org.apache.cassandra.service.accord.AccordCommandStore.SafeRedundantBefore;
 import org.apache.cassandra.service.accord.AccordDurableOnFlush.ReportDurable;
 import org.apache.cassandra.service.paxos.PaxosState;
 
 import static accord.utils.Invariants.illegalState;
+import static org.apache.cassandra.service.accord.execution.AccordCacheEntry.LockMode.UNQUEUED;
 
-public class AccordSafeCommandStore extends AbstractSafeCommandStore<AccordSafeCommand, AccordSafeCommandsForKey, AccordCommandStore.ExclusiveCaches>
+public final class SaferCommandStore extends AbstractSafeCommandStore<SaferCommand, SaferCommandsForKey, ExclusiveCaches>
 {
-    final AccordTask<?> task;
+    final SafeTask<?> task;
-    private final @Nullable CommandSummaries commandsForRanges;
-    private final AccordCommandStore commandStore;
 
-    private AccordSafeCommandStore(AccordTask<?> task,
+    SaferCommandStore(SafeTask<?> task, ExecutionContext context)
-                                   @Nullable CommandSummaries commandsForRanges,
-                                   AccordCommandStore commandStore)
     {
-        super(task.preLoadContext(), commandStore);
+        super(context);
         this.task = task;
-        this.commandsForRanges = commandsForRanges;
-        this.commandStore = commandStore;
-    }
-
-    @Override
-    public CommandStores.RangesForEpoch ranges()
-    {
-        CommandStores.RangesForEpoch ranges = super.ranges();
-        if (ranges != null)
-            return ranges;
-
-        return commandStore.unsafeGetRangesForEpoch();
-    }
-
-    public static AccordSafeCommandStore create(AccordTask<?> task,
-                                                @Nullable CommandSummaries commandsForRanges,
-                                                AccordCommandStore commandStore)
-    {
-        return new AccordSafeCommandStore(task, commandsForRanges, commandStore);
     }
 
     @VisibleForTesting
-    public Set<RoutingKey> commandsForKeysKeys()
+    public Iterable<SafeCommandsForKey> safeCommandsForKeys()
     {
-        if (task.commandsForKey() == null)
+        return task.refs.values().stream()
-            return Collections.emptySet();
+                        .filter(v -> v instanceof SafeCommandsForKey)
-        return task.commandsForKey().keySet();
+                        .map(v -> (SafeCommandsForKey)v)::iterator;
     }
 
     @Override
-    protected AccordSafeCommand getInternal(TxnId txnId)
+    protected SaferCommand getInternal(TxnId txnId)
     {
-        Map<TxnId, AccordSafeCommand> commands = task.commands();
+        return (SaferCommand) task.refs.get(txnId);
-        if (commands == null)
-            return null;
-        return commands.get(txnId);
     }
 
     @Override
     protected ExclusiveCaches tryGetCaches()
     {
-        return commandStore.tryLockCaches();
+        if (task.isIncremental())
+        {
+            // We could relax this, but requires careful consideration of semantics when:
+            //  - touching keys we have scheduled to process later
+            //  - touching txnIds we may have to logically lock until the whole processing completes to
+            //    ensure persistent state machine updates are consistent with incremental updates to cache
+            return null;
+        }
+        return commandStore().tryLockCaches();
     }
 
-    protected AccordSafeCommand add(AccordSafeCommand safeCommand, ExclusiveCaches caches)
+    protected SaferCommand add(SaferCommand safeCommand, ExclusiveCaches caches)
     {
-        Object check = task.ensureCommands().putIfAbsent(safeCommand.txnId(), safeCommand);
+        Object check = task.refs.putIfAbsent(safeCommand.txnId(), safeCommand);
         if (check == null)
         {
-            safeCommand.preExecute();
+            Invariants.require(!task.isIncremental()); // if isIncremental we'll have to supply holdQueue==true, but for now we forbid it
+            safeCommand.preExecute(task, UNQUEUED);
             return safeCommand;
         }
         else
@@ -131,7 +116,7 @@ public class AccordSafeCommandStore extends AbstractSafeCommandStore<AccordSafeC
         // Field persistence is handled by AccordTask
     }
 
-    protected void persistFieldUpdatesInternal(Runnable onDone)
+    void persistFieldUpdatesInternal(Runnable onDone)
     {
         FieldUpdates updates = fieldUpdates();
         if (updates == null)
@@ -139,26 +124,22 @@ public class AccordSafeCommandStore extends AbstractSafeCommandStore<AccordSafeC
 
         if (updates.newRedundantBefore != null)
         {
-            long ticket = AccordCommandStore.nextSafeRedundantBeforeTicket.incrementAndGet();
+            Runnable reportRedundantBefore = SafeRedundantBefore.updater(commandStore(), updates.newRedundantBefore);
-            SafeRedundantBefore update = new SafeRedundantBefore(ticket, updates.newRedundantBefore);
-            Runnable reportRedundantBefore = () -> {
-                AccordCommandStore.safeRedundantBeforeUpdater.accumulateAndGet(commandStore, update, SafeRedundantBefore::max);
-            };
             Runnable prevOnDone = onDone;
             onDone = prevOnDone == null ? reportRedundantBefore : () -> {
                 try { reportRedundantBefore.run(); }
                 finally { prevOnDone.run(); }
             };
         }
-        commandStore.persistFieldUpdates(updates, onDone);
+        commandStore().persistFieldUpdates(updates, onDone);
     }
 
-    protected AccordSafeCommandsForKey add(AccordSafeCommandsForKey safeCfk, ExclusiveCaches caches)
+    protected SaferCommandsForKey add(SaferCommandsForKey safeCfk, ExclusiveCaches caches)
     {
-        Object check = task.ensureCommandsForKey().putIfAbsent(safeCfk.key(), safeCfk);
+        Object check = task.refs.putIfAbsent(safeCfk.key(), safeCfk);
         if (check == null)
         {
-            safeCfk.preExecute();
+            safeCfk.preExecute(task, UNQUEUED);
             return safeCfk;
         }
         else
@@ -169,19 +150,19 @@ public class AccordSafeCommandStore extends AbstractSafeCommandStore<AccordSafeC
     }
 
     @Override
-    protected AccordSafeCommandsForKey getInternal(RoutingKey key)
+    protected SaferCommandsForKey getInternal(RoutingKey key)
     {
-        Map<RoutingKey, AccordSafeCommandsForKey> commandsForKey = task.commandsForKey();
+        SaferCommandsForKey safeCfk = (SaferCommandsForKey) task.refs.get(key);
-        if (commandsForKey == null)
+        if (safeCfk == null || safeCfk.isUninitialised())
             return null;
-        return commandsForKey.get(key);
+        return safeCfk;
     }
 
     @Override
     public void setRangesForEpoch(CommandStores.RangesForEpoch rangesForEpoch)
     {
         super.setRangesForEpoch(rangesForEpoch);
-        commandStore.updateMinHlc(PaxosState.ballotTracker().getLowBound().unixMicros() + 1);
+        commandStore().updateMinHlc(PaxosState.ballotTracker().getLowBound().unixMicros() + 1);
     }
 
     @Override
@@ -194,13 +175,13 @@ public class AccordSafeCommandStore extends AbstractSafeCommandStore<AccordSafeC
     public void reportDurable(RedundantBefore addRedundantBefore, int flags)
     {
         upsertRedundantBefore(addRedundantBefore);
-        commandStore.maybeTerminated(ReportDurable.isCommandStoreFlush(flags), ReportDurable.isDataStoreFlush(flags));
+        ReportDurable.reportMaybeTerminate(commandStore(), flags);
     }
 
     @Override
     public AccordCommandStore commandStore()
     {
-        return commandStore;
+        return task.commandStore;
     }
 
     @Override
@@ -212,7 +193,7 @@ public class AccordSafeCommandStore extends AbstractSafeCommandStore<AccordSafeC
     @Override
     public Agent agent()
     {
-        return commandStore.agent();
+        return commandStore().agent();
     }
 
     @Override
@@ -224,43 +205,58 @@ public class AccordSafeCommandStore extends AbstractSafeCommandStore<AccordSafeC
     @Override
     public NodeCommandStoreService node()
     {
-        return commandStore.node();
+        return commandStore().node();
     }
 
     public LogLinearDecayingHistograms.Buffer histogramBuffer()
     {
         if (task.histogramBuffer == null)
         {
-            task.histogramBuffer = commandStore.metricsBuffer;
+            task.histogramBuffer = commandStore().metricsBuffer;
             if (task.histogramBuffer == null)
-                task.histogramBuffer = commandStore.metricsBuffer = new LogLinearDecayingHistograms.Buffer(commandStore.executor().histograms);
+                task.histogramBuffer = commandStore().metricsBuffer = new LogLinearDecayingHistograms.Buffer(commandStore().executor().histograms);
-            Invariants.require(task.histogramBuffer.isEmpty());
+            if (!Invariants.expect(task.histogramBuffer.isEmpty()))
+                task.histogramBuffer.clear();
         }
         return task.histogramBuffer;
     }
 
     private boolean visitForKey(Unseekables<?> keysOrRanges, Predicate<CommandsForKey> forEach)
     {
-        Map<RoutingKey, AccordSafeCommandsForKey> commandsForKey = task.commandsForKey;
+        Unseekables<?> unseekables = context.keys();
-        if (commandsForKey == null)
+        switch (unseekables.domain())
-            return true;
-
-        Unseekables<?> skip = context.keys().without(keysOrRanges);
-        for (SafeCommandsForKey safeCfk : commandsForKey.values())
         {
-            if (skip.contains(safeCfk.key()))
+            default: throw new UnhandledEnum(unseekables.domain());
-                continue;
+            case Key:
+                AbstractUnseekableKeys keys = (AbstractUnseekableKeys) context.keys();
+                return Routables.foldl(keys, keysOrRanges, (self, f, key, v, index) -> {
+                    SafeCommandsForKey safeCfk = (SafeCommandsForKey) self.task.refs.get(key);
+                    return f.test(safeCfk.current());
+                }, this, forEach, Boolean.TRUE, cont -> !cont);
 
-            if (!forEach.test(safeCfk.current()))
+            case Range:
-                return false;
+                Unseekables<?> skip = context.keys().without(keysOrRanges);
+                for (SafeState<?> safeState : task.refs.values())
+                {
+                    if (!(safeState instanceof SaferCommandsForKey))
+                        continue;
+
+                    SafeCommandsForKey safeCfk = (SafeCommandsForKey) safeState;
+                    if (skip.contains(safeCfk.key()))
+                        continue;
+
+                    if (!forEach.test(safeCfk.current()))
+                        return false;
+                }
+                return true;
         }
-        return true;
     }
 
     @Override
     public <P1, P2> void visit(Unseekables<?> keysOrRanges, Timestamp startedBefore, Kinds testKind, ActiveCommandVisitor<P1, P2> visitor, P1 p1, P2 p2)
     {
         visitForKey(keysOrRanges, cfk -> { cfk.visit(startedBefore, testKind, visitor, p1, p2); return true; });
+        CommandSummaries commandsForRanges = task.commandsForRanges();
         if (commandsForRanges != null)
             commandsForRanges.visit(keysOrRanges, startedBefore, testKind, visitor, p1, p2);
     }
@@ -268,8 +264,11 @@ public class AccordSafeCommandStore extends AbstractSafeCommandStore<AccordSafeC
     @Override
     public boolean visit(Unseekables<?> keysOrRanges, TxnId testTxnId, Kinds testKind, SupersedingCommandVisitor visit)
     {
-        return visitForKey(keysOrRanges, cfk -> cfk.visit(testTxnId, testKind, visit))
+        if (!visitForKey(keysOrRanges, cfk -> cfk.visit(testTxnId, testKind, visit)))
-               && (commandsForRanges == null || commandsForRanges.visit(keysOrRanges, testTxnId, testKind, visit));
+            return false;
+
+        CommandSummaries commandsForRanges = task.commandsForRanges();
+        return commandsForRanges == null || commandsForRanges.visit(keysOrRanges, testTxnId, testKind, visit);
     }
 
     @Override

src/java/org/apache/cassandra/service/accord/execution/SaferCommandsForKey.java

@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import java.util.Objects;
+
+import accord.api.RoutingKey;
+import accord.local.cfk.CommandsForKey;
+import accord.local.cfk.NotifySink;
+import accord.local.cfk.SafeCommandsForKey;
+
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry.LockMode;
+
+public class SaferCommandsForKey extends SafeCommandsForKey implements SaferState<RoutingKey, CommandsForKey, SaferCommandsForKey>
+{
+    public static class CommandsForKeyCacheEntry extends AccordCacheEntry<RoutingKey, CommandsForKey, SaferCommandsForKey>
+    {
+        private NotifySink overrideSink;
+
+        CommandsForKeyCacheEntry(RoutingKey key, AccordCache.Type<RoutingKey, CommandsForKey, SaferCommandsForKey>.Instance owner)
+        {
+            super(key, owner);
+        }
+    }
+
+    private final AccordCacheEntry<RoutingKey, CommandsForKey, SaferCommandsForKey> global;
+
+    public SaferCommandsForKey(AccordCacheEntry<RoutingKey, CommandsForKey, SaferCommandsForKey> global)
+    {
+        super(global.key());
+        this.global = global;
+    }
+
+    @Override
+    public boolean equals(Object o)
+    {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+        SaferCommandsForKey that = (SaferCommandsForKey) o;
+        return Objects.equals(current, that.current);
+    }
+
+    @Override
+    public int hashCode()
+    {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public String toString()
+    {
+        return "AccordSafeCommandsForKey{" +
+               "state=" + statusString() +
+               ", global=" + global +
+               ", current=" + current +
+               '}';
+    }
+
+    public final AccordCacheEntry<RoutingKey, CommandsForKey, SaferCommandsForKey> global()
+    {
+        return global;
+    }
+
+    @Override
+    public void postExecute(SafeTask<?> owner)
+    {
+        global.releaseExclusive(this, owner);
+    }
+
+    @Override
+    public void overrideSink(NotifySink overrideSink)
+    {
+        ((CommandsForKeyCacheEntry)global).overrideSink = overrideSink;
+    }
+
+    @Override
+    public NotifySink overrideSink()
+    {
+        return ((CommandsForKeyCacheEntry)global).overrideSink;
+    }
+
+    public void preExecute(SafeTask<?> owner, LockMode lockMode)
+    {
+        requireUninitialised();
+        current = global.lockExclusive(owner, lockMode);
+        if (current == null)
+            initialize();
+        setSafe();
+    }
+
+}

src/java/org/apache/cassandra/service/accord/execution/SaferState.java

@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.service.accord.execution;
+
+import accord.local.SafeState;
+
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry.LockMode;
+
+public interface SaferState<K, V, S extends SafeState<V> & SaferState<K, V, S>>
+{
+    AccordCacheEntry<K, V, S> global();
+    void postExecute(SafeTask<?> owner);
+    void preExecute(SafeTask<?> owner, LockMode lockMode);
+
+    static AccordCacheEntry<?, ?, ?> global(SafeState<?> safeState)
+    {
+        return safeState.getClass() == SaferCommand.class ? ((SaferCommand) safeState).global()
+                                                          : ((SaferCommandsForKey) safeState).global();
+    }
+
+    static void postExecute(SafeState<?> safeState, SafeTask<?> owner)
+    {
+        if (safeState.getClass() == SaferCommand.class) ((SaferCommand) safeState).postExecute(owner);
+        else ((SaferCommandsForKey) safeState).postExecute(owner);
+    }
+
+    static void preExecute(SafeState<?> safeState, SafeTask<?> owner, LockMode lockMode)
+    {
+        if (safeState.getClass() == SaferCommand.class) ((SaferCommand) safeState).preExecute(owner, lockMode);
+        else ((SaferCommandsForKey) safeState).preExecute(owner, lockMode);
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/Task.java

@@ -0,0 +1,939 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
+import java.util.concurrent.atomic.AtomicLong;
+
+import accord.local.ExecutionContext;
+import accord.messages.Accept;
+import accord.messages.Commit;
+import accord.messages.MessageType;
+import accord.messages.Request;
+import accord.primitives.Ballot;
+import accord.primitives.SaveStatus;
+import accord.primitives.TxnId;
+import accord.utils.IntrusiveHeapNode;
+import accord.utils.Invariants;
+import accord.utils.TinyEnumSet;
+import accord.utils.UnhandledEnum;
+import accord.utils.async.Cancellable;
+
+import org.apache.cassandra.concurrent.DebuggableTask;
+import org.apache.cassandra.concurrent.ExecutorLocals;
+import org.apache.cassandra.service.accord.debug.DebugExecution.DebugTask;
+import org.apache.cassandra.service.accord.execution.ExclusiveExecutor.ExclusiveExecutorTask;
+import org.apache.cassandra.utils.Closeable;
+import org.apache.cassandra.utils.WithResources;
+
+import static accord.local.ExecutionContext.ExecutionSequence.BY_PRIORITY;
+import static accord.local.ExecutionContext.ExecutionSequence.BY_PRIORITY_ATOMIC;
+import static accord.primitives.Routable.Domain.Range;
+import static org.apache.cassandra.config.AccordConfig.QueuePriorityModel.ORIG_HLC_FIFO;
+import static org.apache.cassandra.service.accord.debug.DebugExecution.DEBUG_EXECUTION;
+import static org.apache.cassandra.service.accord.execution.Task.ExclusiveGroup.APPLY;
+import static org.apache.cassandra.service.accord.execution.Task.ExclusiveGroup.COMMIT;
+import static org.apache.cassandra.service.accord.execution.Task.ExclusiveGroup.STABLE;
+import static org.apache.cassandra.service.accord.execution.Task.RunState.NOT_YET_RUN;
+import static org.apache.cassandra.service.accord.execution.Task.RunState.RUN_FAILED;
+import static org.apache.cassandra.service.accord.execution.Task.RunState.RUN_INCOMPLETE;
+import static org.apache.cassandra.service.accord.execution.Task.RunState.RUNNING;
+import static org.apache.cassandra.service.accord.execution.Task.State.CANCELLED;
+import static org.apache.cassandra.service.accord.execution.Task.State.CANCELLED_UNREGISTERED;
+import static org.apache.cassandra.service.accord.execution.Task.State.EXECUTED;
+import static org.apache.cassandra.service.accord.execution.Task.State.FAILED;
+import static org.apache.cassandra.service.accord.execution.Task.State.LOADING_OPTIONAL;
+import static org.apache.cassandra.service.accord.execution.Task.State.LOADING_REQUIRED;
+import static org.apache.cassandra.service.accord.execution.Task.State.PREPARED;
+import static org.apache.cassandra.service.accord.execution.Task.State.PREPARED_OR_EXECUTED;
+import static org.apache.cassandra.service.accord.execution.Task.State.SCANNING_RANGES;
+import static org.apache.cassandra.service.accord.execution.Task.State.UNREGISTERED;
+import static org.apache.cassandra.service.accord.execution.Task.State.WAITING_ON_OPTIONAL;
+import static org.apache.cassandra.service.accord.execution.Task.State.WAITING_ON_REQUIRED;
+import static org.apache.cassandra.service.accord.execution.Task.State.WAITING_TO_RUN;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+public abstract class Task extends IntrusiveHeapNode implements Cancellable, DebuggableTask
+{
+    private static final int WAITING_ON_OPTIONAL_BIT = 1 << 7;
+    private static final int WAITING_TO_RUN_BIT = 1 << 8;
+    private static final int INCOMPLETE_BIT = 1 << 10;
+
+    enum State
+    {
+        UNREGISTERED(),
+        CANCELLED_UNREGISTERED(UNREGISTERED),
+        REGISTERED(UNREGISTERED),
+        SCANNING_RANGES(REGISTERED),
+        LOADING_REQUIRED(REGISTERED, SCANNING_RANGES),
+        LOADING_OPTIONAL(REGISTERED, SCANNING_RANGES, LOADING_REQUIRED),
+        WAITING_ON_REQUIRED(WAITING_ON_OPTIONAL_BIT | WAITING_TO_RUN_BIT, REGISTERED, SCANNING_RANGES, LOADING_REQUIRED, LOADING_OPTIONAL),
+        WAITING_ON_OPTIONAL(WAITING_TO_RUN_BIT | INCOMPLETE_BIT, REGISTERED, SCANNING_RANGES, LOADING_REQUIRED, LOADING_OPTIONAL, WAITING_ON_REQUIRED),
+        WAITING_TO_RUN(INCOMPLETE_BIT, UNREGISTERED, REGISTERED, SCANNING_RANGES, LOADING_REQUIRED, LOADING_OPTIONAL, WAITING_ON_REQUIRED, WAITING_ON_OPTIONAL),
+        PREPARED(WAITING_TO_RUN),
+        INCOMPLETE(PREPARED),
+        EXECUTED(PREPARED),
+        FAILED(UNREGISTERED, REGISTERED, SCANNING_RANGES, LOADING_REQUIRED, LOADING_OPTIONAL, WAITING_ON_REQUIRED, WAITING_ON_OPTIONAL, WAITING_TO_RUN, PREPARED, INCOMPLETE),
+        CANCELLED(UNREGISTERED, REGISTERED, SCANNING_RANGES, LOADING_REQUIRED, LOADING_OPTIONAL, WAITING_ON_REQUIRED, WAITING_ON_OPTIONAL, WAITING_TO_RUN),
+        ;
+
+        private final int permittedFrom;
+        public static final int WAITING = TinyEnumSet.encode(WAITING_ON_REQUIRED, WAITING_ON_OPTIONAL, WAITING_TO_RUN);
+        public static final int WAITING_OR_PREPARED = WAITING | TinyEnumSet.encode(PREPARED);
+        public static final int PREPARED_OR_EXECUTED = TinyEnumSet.encode(PREPARED, EXECUTED);
+        static final State[] VALUES = values();
+
+        static
+        {
+            // hack to allow us to create loops in our enum transition declarations
+            Invariants.require(INCOMPLETE_BIT == 1 << INCOMPLETE.ordinal());
+            Invariants.require(WAITING_TO_RUN_BIT == 1 << WAITING_TO_RUN.ordinal());
+            Invariants.require(WAITING_ON_OPTIONAL_BIT == 1 << WAITING_ON_OPTIONAL.ordinal());
+        }
+
+        State()
+        {
+            this.permittedFrom = 0;
+        }
+
+        State(State... permittedFroms)
+        {
+            this(0, permittedFroms);
+        }
+
+        State(int additional, State... permittedFroms)
+        {
+            int permittedFrom = additional;
+            for (State state : permittedFroms)
+                permittedFrom |= 1 << state.ordinal();
+            this.permittedFrom = permittedFrom;
+        }
+
+        boolean isPermittedFrom(int prevOrdinal)
+        {
+            return (permittedFrom & (1 << prevOrdinal)) != 0;
+        }
+
+        boolean isDone()
+        {
+            return this.compareTo(EXECUTED) >= 0;
+        }
+
+        boolean hasStarted()
+        {
+            return this.compareTo(PREPARED) >= 0;
+        }
+
+        static State forOrdinal(int ordinal)
+        {
+            return VALUES[ordinal];
+        }
+    }
+
+    enum RunState
+    {
+        NOT_YET_RUN, RUNNING, RUN_INCOMPLETE, RUN_PERSISTING, RUN_SUCCESS, RUN_FAILED;
+
+        private static final RunState[] VALUES = values();
+
+        static RunState forOrdinal(int ordinal)
+        {
+            return VALUES[ordinal];
+        }
+    }
+
+    enum GlobalGroup
+    {
+        COMMAND_STORE,
+        LOAD,
+        SAVE,
+        OTHER,
+        RANGE_LOAD,
+        RANGE_SCAN,
+    }
+
+    enum ExclusiveGroup
+    {
+        APPLY,
+        STABLE,
+        COMMIT,
+        ACCEPT,
+        OTHER,
+        RECOVER,
+        PREACCEPT,
+        RANGE,
+    }
+
+    public enum GroupKind
+    {
+        EXCLUSIVE(ExclusiveGroup.values().length, EXCLUSIVE_GROUP_SHIFT),
+        GLOBAL(GlobalGroup.values().length, GLOBAL_GROUP_SHIFT),
+        NONE(0, 0);
+
+        final int count;
+        final byte shift;
+
+        GroupKind(int count, int shift)
+        {
+            this.count = count;
+            this.shift = (byte) shift;
+        }
+    }
+
+    enum ExecutorQueue
+    {
+        NONE(),
+        LOADING(SCANNING_RANGES, LOADING_OPTIONAL, LOADING_REQUIRED),
+        WAITING(WAITING_ON_OPTIONAL, WAITING_ON_REQUIRED),
+        RUNNABLE(WAITING_TO_RUN);
+
+        private static final ExecutorQueue[] VALUES = values();
+
+        final int permittedStates;
+
+        ExecutorQueue(State ... states)
+        {
+            this.permittedStates = TinyEnumSet.encode(states);
+        }
+
+        public static ExecutorQueue forOrdinal(int ordinal)
+        {
+            return VALUES[ordinal];
+        }
+    }
+
+    private static final int STATE_MASK = 0xf;
+    static final int GROUP_MASK = 0x7;
+    private static final int EXCLUSIVE_GROUP_SHIFT = 4;
+    private static final int GLOBAL_GROUP_SHIFT = 7;
+
+    private static final int NONSYNC_BIT = 1 << 10;
+    private static final int CACHE_QUEUED_BIT = 1 << 11;
+
+    private static final int HAS_TRANCHE_BIT = 1 << 12;
+    private static final int HAS_INHERITED_BIT = 1 << 13;
+    private static final int HAS_INHERITED_RANGE_SCAN_BIT = 1 << 14;
+
+    private static final int EXECUTOR_QUEUE_SHIFT = 16;
+    private static final int EXECUTOR_QUEUE_UNSHIFTED_MASK = 0x3;
+    private static final int EXECUTOR_QUEUE_SHIFTED_MASK = EXECUTOR_QUEUE_UNSHIFTED_MASK << EXECUTOR_QUEUE_SHIFT;
+
+    private static final int INCREMENTAL_SHIFT = 18;
+    private static final int INCREMENTAL_MASK = 0x3 << INCREMENTAL_SHIFT;
+    private static final int INCREMENTAL = 0x1 << INCREMENTAL_SHIFT;
+    private static final int INCREMENTAL_STARTED = 0x2 << INCREMENTAL_SHIFT;
+    private static final int INCREMENTAL_FINISHING = 0x3 << INCREMENTAL_SHIFT;
+
+    private static final int SEQUENCED_SHIFT = 20;
+    private static final int SEQUENCED_MASK = 0x3 << SEQUENCED_SHIFT;
+    private static final int SEQUENCED_PRIORITY = 0x1 << SEQUENCED_SHIFT;
+    private static final int SEQUENCED_ATOMIC = 0x2 << SEQUENCED_SHIFT;
+    private static final int SEQUENCED_ATOMIC_AND_QUEUED = 0x3 << SEQUENCED_SHIFT;
+
+    private static final int TRANCHE_SHIFT = 22;
+    static final int MAX_TRANCHE = 0x3ff;
+
+    static
+    {
+        Invariants.require(SEQUENCED_PRIORITY == BY_PRIORITY.ordinal() << SEQUENCED_SHIFT);
+        Invariants.require(SEQUENCED_ATOMIC == BY_PRIORITY_ATOMIC.ordinal() << SEQUENCED_SHIFT);
+        Invariants.require(ExecutionContext.ExecutionSequence.values().length <= 3);
+    }
+
+    // TODO (desired): quite heavy to pass-through tracing session state we mostly don't use
+    public final WithResources resources;
+    Task next;
+    Task consequences;
+
+    long position;
+    int info;
+
+    public final long createdAt;
+    long runningAt;
+    // TODO (desired): expose via executors vtable
+    private volatile int runState;
+
+    private static final AtomicIntegerFieldUpdater<Task> runStateUpdater = AtomicIntegerFieldUpdater.newUpdater(Task.class, "runState");
+
+    Task(GlobalGroup group)
+    {
+        resources = DebugTask.maybeDebug(ExecutorLocals.propagate(), this);
+        info = init(group, ExclusiveGroup.OTHER);
+        createdAt = nanoTime();
+    }
+
+    Task(ExclusiveGroup group)
+    {
+        resources = DebugTask.maybeDebug(ExecutorLocals.propagate(), this);
+        info = init(GlobalGroup.OTHER, group);
+        createdAt = nanoTime();
+    }
+
+    protected Task(ExecutionContext context, AtomicLong lastCreatedAt)
+    {
+        resources = DebugTask.maybeDebug(ExecutorLocals.propagate(), this);
+        createdAt = lastCreatedAt.accumulateAndGet(nanoTime(), (prev, next) -> next <= prev ? prev + 1 : next);
+        ExclusiveGroup group = ExclusiveGroup.OTHER;
+        TxnId txnId = context.primaryTxnId();
+        if (txnId != null)
+        {
+            if (txnId.is(Range)) group = ExclusiveGroup.RANGE;
+            else
+            {
+                switch (AccordExecutor.PRIORITY_MODEL)
+                {
+                    case HLC_FIFO:
+                    case ORIG_HLC_FIFO:
+                    {
+                        // TODO (expected): port to ExecutionKind; also we aren't consistent about using Ballot
+                        if (context instanceof Request)
+                        {
+                            MessageType type = ((Request) context).type();
+                            if (type instanceof MessageType.StandardMessage)
+                            {
+                                switch ((MessageType.StandardMessage) type)
+                                {
+                                    case APPLY_REQ:
+                                    {
+                                        group = APPLY;
+                                        break;
+                                    }
+                                    case READ_EPHEMERAL_REQ:
+                                    case READ_REQ:
+                                    case STABLE_THEN_READ_REQ:
+                                    {
+                                        group = STABLE;
+                                        break;
+                                    }
+                                    case COMMIT_REQ:
+                                    {
+                                        Commit commit = (Commit) context;
+                                        if (AccordExecutor.PRIORITY_MODEL == ORIG_HLC_FIFO && !commit.ballot.equals(Ballot.ZERO))
+                                            txnId = null;
+                                        if (commit.kind.saveStatus == SaveStatus.Stable) group = STABLE;
+                                        else group = COMMIT;
+                                        break;
+                                    }
+                                    case ACCEPT_REQ:
+                                    {
+                                        Accept accept = (Accept) context;
+                                        if (AccordExecutor.PRIORITY_MODEL == ORIG_HLC_FIFO && !accept.ballot.equals(Ballot.ZERO))
+                                            txnId = null;
+                                        group = ExclusiveGroup.ACCEPT;
+                                        break;
+                                    }
+                                    case GET_EPHEMERAL_READ_DEPS_REQ:
+                                    case PRE_ACCEPT_REQ:
+                                    {
+                                        group = ExclusiveGroup.PREACCEPT;
+                                        break;
+                                    }
+                                    default:
+                                    {
+                                        txnId = null;
+                                    }
+                                }
+                            }
+                        }
+                        else
+                        {
+                            txnId = null;
+                        }
+                        break;
+                    }
+                    case FIFO:
+                    {
+                        txnId = null;
+                        break;
+                    }
+                }
+            }
+        }
+
+        this.info = init(GlobalGroup.OTHER, group);
+        if (txnId != null)
+            this.position = txnId.hlc();
+    }
+
+    public final Task unwrap()
+    {
+        if (this instanceof ExclusiveExecutorTask)
+            return ((ExclusiveExecutorTask) this).queue.task;
+        return this;
+    }
+
+    public DebuggableTask debuggable()
+    {
+        return this;
+    }
+
+    public final long creationTimeNanos()
+    {
+        return createdAt;
+    }
+
+    public final long startTimeNanos()
+    {
+        if (runState() == NOT_YET_RUN)
+            return 0;
+        return runningAt;
+    }
+
+    abstract void submitExclusiveMayThrow();
+    /** Return true if COMPLETED successfully. false indicates more work is being done. Should not throw any exceptions related to reporting success. */
+    abstract boolean runMayThrow();
+    abstract void maybeCompleteExclusiveMayThrow();
+    abstract void tryCancelExclusive();
+    abstract void reportFailureMayThrow(Throwable fail);
+
+    abstract AccordExecutor executor();
+    abstract void unqueueIfQueued();
+    abstract boolean isNewWork();
+    abstract String briefDescription();
+
+    final void submitExclusiveNoExcept()
+    {
+        if (is(UNREGISTERED))
+        {
+            try { submitExclusiveMayThrow(); }
+            catch (Throwable t)
+            {
+                tryFailAndCompleteExclusive(t, State.FAILED);
+                onException(t);
+            }
+        }
+    }
+
+    /**
+     * Prepare to run while holding the state cache lock.
+     * If returns false, prepare failed and the task should be discarded with no further action.
+     */
+    final boolean prepareExclusiveNoExcept()
+    {
+        if (getClass() == ExclusiveExecutorTask.class)
+        {
+            return ((ExclusiveExecutorTask)this).prepareTask();
+        }
+        else
+        {
+            try
+            {
+                prepareExclusiveMayThrow();
+                setStateExclusive(State.PREPARED);
+                return true;
+            }
+            catch (Throwable t)
+            {
+                failAndCompleteExclusive(t, State.FAILED);
+                return false;
+            }
+        }
+    }
+
+    void prepareExclusiveMayThrow()
+    {
+    }
+
+    /**
+     * Run the command; the state cache lock may or may not be held depending on the executor implementation
+     */
+    final void runNoExcept(TaskRunner self)
+    {
+        if (getClass() == ExclusiveExecutorTask.class)
+        {
+            ((ExclusiveExecutorTask)this).queue.runTask(self);
+        }
+        else
+        {
+            onRunning();
+            self.setAccordActiveTask(this);
+            try (Closeable close = resources.get())
+            {
+                if (runMayThrow())
+                    onSuccess();
+                else
+                    Invariants.require(compareTo(RUNNING) > 0);
+            }
+            catch (Throwable t)
+            {
+                setRunState(RunState.RUN_FAILED);
+                reportFailureNoExcept(t);
+            }
+            finally
+            {
+                self.setAccordActiveTask(null);
+            }
+        }
+    }
+
+    final void rejectAtRuntime(Throwable reject)
+    {
+        setRunState(RunState.RUN_FAILED);
+        reportFailureNoExcept(reject);
+    }
+
+    final void completeExclusiveNoExcept()
+    {
+        try
+        {
+            if (DEBUG_EXECUTION) DebugTask.get(this).onComplete();
+            maybeCompleteExclusiveMayThrow();
+        }
+        catch (Throwable t)
+        {
+            onException(t);
+            if (compareTo(EXECUTED) < 0)
+                failExclusive(t, State.FAILED);
+        }
+        finally
+        {
+            if (compareTo(EXECUTED) >= 0)
+            {
+                try { submitConsequencesExclusive(is(EXECUTED)); }
+                catch (Throwable t) { onException(t); }
+                executor().completedTaskExclusive(this);
+            }
+        }
+    }
+
+    final void onException(Throwable t)
+    {
+        try { executor().agent.onException(t); }
+        catch (Throwable t2) { }
+    }
+
+    final void reportFailureNoExcept(Throwable fail)
+    {
+        try { reportFailureMayThrow(fail); }
+        catch (Throwable t)
+        {
+            try { fail.addSuppressed(t); }
+            catch (Throwable t2) { }
+            onException(fail);
+        }
+    }
+
+    // propagate RunState to State
+    // true if task was executed successfully
+    final boolean completeState()
+    {
+        RunState runState = runState();
+        boolean success;
+        switch (runState)
+        {
+            default: throw UnhandledEnum.unknown(runState);
+            case RUN_INCOMPLETE: throw UnhandledEnum.invalid(runState);
+            case NOT_YET_RUN:
+                Invariants.expect(state().isDone());
+                Invariants.expect(consequences == null);
+                success = false;
+                break;
+
+            case RUN_FAILED:
+                if (compareTo(EXECUTED) < 0)
+                    setStateExclusive(State.FAILED);
+                success = false;
+                break;
+
+            case RUNNING:
+            case RUN_PERSISTING:
+            case RUN_SUCCESS:
+                setStateExclusive(EXECUTED);
+                success = true;
+                break;
+        }
+        return success;
+    }
+
+    final void tryFailAndCompleteExclusive(Throwable fail, State newState)
+    {
+        if (is(UNREGISTERED))
+            failExclusive(fail, CANCELLED_UNREGISTERED);
+        else if (compareTo(WAITING_TO_RUN) <= 0)
+            failAndCompleteExclusive(fail, newState);
+    }
+
+    final void failExclusive(Throwable fail, State newState)
+    {
+        unqueueIfQueued();
+        setStateExclusive(newState);
+        reportFailureNoExcept(fail);
+    }
+
+    final void failAndCompleteExclusive(Throwable fail, State newState)
+    {
+        failExclusive(fail, newState);
+        completeExclusiveNoExcept();
+    }
+
+    final void unqueue(TaskQueue expected)
+    {
+        Invariants.require(queuedOrdinal() == expected.kind.ordinal());
+        expected.unqueue(this);
+        info &= ~EXECUTOR_QUEUE_SHIFTED_MASK;
+    }
+
+    final void unsetQueue(ExecutorQueue expected)
+    {
+        Invariants.require(expected.ordinal() == queuedOrdinal());
+        info &= ~EXECUTOR_QUEUE_SHIFTED_MASK;
+    }
+
+    final void onRunning()
+    {
+        Invariants.require(is(PREPARED));
+        Invariants.require(is(NOT_YET_RUN) || is(RUN_INCOMPLETE));
+        runningAt = Math.max(createdAt, nanoTime());
+        setRunState(RunState.RUNNING);
+        if (DEBUG_EXECUTION) ((DebugTask) resources).onRunning();
+    }
+
+    final void onSuccess()
+    {
+        setRunState(RunState.RUN_SUCCESS);
+        if (DEBUG_EXECUTION) ((DebugTask) resources).onRunComplete();
+    }
+
+    final State state()
+    {
+        return State.forOrdinal(stateOrdinal());
+    }
+
+    final RunState runState()
+    {
+        return RunState.forOrdinal(runState);
+    }
+
+    final Enum<?> currentState()
+    {
+        State state = state();
+        if (state == State.PREPARED || state == EXECUTED)
+        {
+            RunState runState = runState();
+            if (runState == NOT_YET_RUN)
+                return state;
+            return runState;
+        }
+        return State.forOrdinal(stateOrdinal());
+    }
+
+    final int stateOrdinal()
+    {
+        return info & STATE_MASK;
+    }
+
+    final boolean is(State state)
+    {
+        return stateOrdinal() == state.ordinal();
+    }
+
+    final int compareTo(State state)
+    {
+        return stateOrdinal() - state.ordinal();
+    }
+
+    final int compareTo(RunState state)
+    {
+        return runState - state.ordinal();
+    }
+
+    final boolean is(RunState state)
+    {
+        return runState == state.ordinal();
+    }
+
+    final boolean isEither(RunState state1, RunState state2)
+    {
+        int runState = this.runState;
+        return runState == state1.ordinal() || runState == state2.ordinal();
+    }
+
+    final boolean isState(int stateBitSet)
+    {
+        return TinyEnumSet.contains(stateBitSet, stateOrdinal());
+    }
+
+    final boolean is(GlobalGroup group)
+    {
+        return globalGroupOrdinal() == group.ordinal();
+    }
+
+    final boolean is(ExclusiveGroup group)
+    {
+        return exclusiveGroupOrdinal() == group.ordinal();
+    }
+
+    final void override(GlobalGroup group)
+    {
+        info = (info & ~(GROUP_MASK << GLOBAL_GROUP_SHIFT)) | (group.ordinal() << GLOBAL_GROUP_SHIFT);
+    }
+
+    final void setStateExclusive(State state)
+    {
+        Invariants.require(state.isPermittedFrom(stateOrdinal()), "%s forbidden from %s", state, this, Task::reportBadStateTransition);
+        unsafeSetStateExclusive(state);
+    }
+
+    final void setRunState(RunState state)
+    {
+        Invariants.require(isState(PREPARED_OR_EXECUTED) || (state == RUN_FAILED && is(FAILED)));
+        setRunState(state.ordinal());
+    }
+
+    final void setRunState(int newRunState)
+    {
+        runStateUpdater.lazySet(this, newRunState);
+    }
+
+    private static String reportBadStateTransition(Task task)
+    {
+        return task.state() + " for " + task.description();
+    }
+
+    final void unsafeSetStateExclusive(State state)
+    {
+        info = (info & ~STATE_MASK) | state.ordinal();
+    }
+
+    final int globalGroupOrdinal()
+    {
+        return (info >>> GLOBAL_GROUP_SHIFT) & GROUP_MASK;
+    }
+
+    final int exclusiveGroupOrdinal()
+    {
+        return (info >>> EXCLUSIVE_GROUP_SHIFT) & GROUP_MASK;
+    }
+
+    private boolean isCompatible(ExecutorQueue queue)
+    {
+        int self = stateOrdinal();
+        return TinyEnumSet.contains(queue.permittedStates, self);
+    }
+
+    final boolean isSync()
+    {
+        return 0 == (info & NONSYNC_BIT);
+    }
+
+    final boolean isNonSync()
+    {
+        return !isSync();
+    }
+
+    final void setNonSyncExclusive()
+    {
+        info |= NONSYNC_BIT;
+    }
+
+    final boolean isIncremental()
+    {
+        return 0 != (info & INCREMENTAL_MASK);
+    }
+
+    final void setIncrementalExclusive()
+    {
+        info |= INCREMENTAL | NONSYNC_BIT;
+    }
+
+    final boolean hasIncrementalStarted()
+    {
+        return (info & INCREMENTAL_MASK) >= INCREMENTAL_STARTED;
+    }
+
+    final void setIncrementalStartedExclusive()
+    {
+        Invariants.require(isIncremental());
+        if (!isIncrementalFinishing())
+            info = (info & ~INCREMENTAL_MASK) | INCREMENTAL_STARTED;
+    }
+
+    final boolean isIncrementalFinishing()
+    {
+        return (info & INCREMENTAL_MASK) >= INCREMENTAL_FINISHING;
+    }
+
+    final void setIncrementalFinishingExclusive()
+    {
+        Invariants.require(isIncremental());
+        info |= INCREMENTAL_FINISHING;
+    }
+
+    final void setSequencedExclusive(ExecutionContext.ExecutionSequence sequence)
+    {
+        Invariants.require(isUnsequenced());
+        info |= sequence.ordinal() << SEQUENCED_SHIFT;
+    }
+
+    final boolean isUnsequenced()
+    {
+        return (info & SEQUENCED_MASK) == 0;
+    }
+
+    final boolean isSequencedByPriority()
+    {
+        return (info & SEQUENCED_MASK) == SEQUENCED_PRIORITY;
+    }
+
+    final boolean isSequencedByPriorityAtomic()
+    {
+        return (info & SEQUENCED_MASK) >= SEQUENCED_ATOMIC;
+    }
+
+    final boolean isCacheQueuedFifo()
+    {
+        return (info & SEQUENCED_MASK) == SEQUENCED_ATOMIC_AND_QUEUED;
+    }
+
+    final boolean isCacheQueued()
+    {
+        return 0 != (info & CACHE_QUEUED_BIT);
+    }
+
+    final boolean isQueued()
+    {
+        return 0 != (info & EXECUTOR_QUEUE_SHIFTED_MASK);
+    }
+
+    final int queuedOrdinal()
+    {
+        return (info >>> EXECUTOR_QUEUE_SHIFT) & EXECUTOR_QUEUE_UNSHIFTED_MASK;
+    }
+
+    final ExecutorQueue queued()
+    {
+        return ExecutorQueue.forOrdinal(queuedOrdinal());
+    }
+
+    final void setQueue(ExecutorQueue queue)
+    {
+        Invariants.require(isCompatible(queue));
+        Invariants.require(!isQueued());
+        info |= queue.ordinal() << EXECUTOR_QUEUE_SHIFT;
+    }
+
+    // supersedes priority, in whichever order they're called
+    final void setCacheQueuedFifoExclusive()
+    {
+        Invariants.require(isSequencedByPriorityAtomic());
+        info |= SEQUENCED_ATOMIC_AND_QUEUED | CACHE_QUEUED_BIT;
+    }
+
+    final void setCacheQueuedExclusive()
+    {
+        info |= CACHE_QUEUED_BIT;
+    }
+
+    final int tranche()
+    {
+        Invariants.require((info & HAS_TRANCHE_BIT) != 0);
+        return info >>> TRANCHE_SHIFT;
+    }
+
+    final void setTranche(int tranche)
+    {
+        Invariants.require(tranche <= MAX_TRANCHE);
+        info = info | (tranche << TRANCHE_SHIFT) | HAS_TRANCHE_BIT;
+    }
+
+    final Task inherit(Task parent)
+    {
+        Invariants.require(!hasInherited());
+        position = parent.position;
+        setInheritedWithTranche(parent.tranche());
+        return this;
+    }
+
+    final void setInheritedWithTranche(int tranche)
+    {
+        Invariants.require(tranche <= MAX_TRANCHE);
+        info = info | (tranche << TRANCHE_SHIFT) | HAS_TRANCHE_BIT | HAS_INHERITED_BIT;
+    }
+
+    final boolean hasInherited()
+    {
+        return (info & HAS_INHERITED_BIT) != 0;
+    }
+
+    final void setInheritedRangeScan()
+    {
+        info = info | HAS_INHERITED_RANGE_SCAN_BIT;
+    }
+
+    final boolean hasInheritedRangeScan()
+    {
+        return (info & HAS_INHERITED_RANGE_SCAN_BIT) != 0;
+    }
+
+    void addConsequence(Task task)
+    {
+        Task prev = consequences;
+        Invariants.require(prev == null || prev.is(UNREGISTERED));
+        task.next = prev;
+        consequences = task;
+    }
+
+    final void submitConsequencesExclusive(boolean success)
+    {
+        if (consequences == null)
+            return;
+
+        Task cur = Task.reverse(consequences);
+        consequences = null;
+
+        while (cur != null)
+        {
+            Task next = cur.next;
+            cur.next = null;
+            if (cur.is(UNREGISTERED))
+            {
+                if (success || !(cur instanceof SafeTask<?> || this instanceof SafeTask<?>))
+                {
+                    cur.inherit(this);
+                    cur.submitExclusiveNoExcept();
+                }
+                else
+                {
+                    cur.failExclusive(new CancellationException("Parent task failed"), CANCELLED);
+                }
+            }
+            cur = next;
+        }
+    }
+
+    static int init(GlobalGroup global, ExclusiveGroup exclusive)
+    {
+        return (global.ordinal() << GLOBAL_GROUP_SHIFT) | (exclusive.ordinal() << EXCLUSIVE_GROUP_SHIFT);
+    }
+
+    static Task reverse(Task unqueued)
+    {
+        Task prev = null;
+        Task cur = unqueued;
+        while (cur != null)
+        {
+            Task next = cur.next;
+            cur.next = prev;
+            prev = cur;
+            cur = next;
+        }
+        return prev;
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/TaskInfo.java

@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import javax.annotation.Nullable;
+
+import accord.local.ExecutionContext;
+
+public class TaskInfo implements Comparable<TaskInfo>
+{
+    // sorted in name order for reporting to virtual tables
+    public enum Status
+    {
+        RUNNING, LOADING, WAITING_TO_RUN
+    }
+
+    final Status status;
+    final int commandStoreId;
+
+    final Task task;
+
+    public TaskInfo(Status status, int commandStoreId, Task task)
+    {
+        this.status = status;
+        this.commandStoreId = commandStoreId;
+        this.task = task;
+    }
+
+    public Status status()
+    {
+        return status;
+    }
+
+    public Integer commandStoreId()
+    {
+        return commandStoreId >= 0 ? commandStoreId : null;
+    }
+
+    public long position()
+    {
+        return task.position;
+    }
+
+    public @Nullable String describe()
+    {
+        if (task instanceof SafeTask)
+            return ((SafeTask<?>) task).executionContext().reason();
+
+        if (task != null)
+            return task.description();
+
+        return null;
+    }
+
+    public @Nullable ExecutionContext preLoadContext()
+    {
+        if (task instanceof SafeTask)
+            return ((SafeTask<?>) task).executionContext();
+        if (task instanceof IOTaskWrapper && ((IOTaskWrapper) task).wrapped instanceof SafeTask.RangeTxnScanner)
+            return ((SafeTask<?>.RangeTxnScanner) ((IOTaskWrapper) task).wrapped).preLoadContext();
+        return null;
+    }
+
+    @Override
+    public int compareTo(TaskInfo that)
+    {
+        int c = this.status.compareTo(that.status);
+        if (c == 0) c = Long.compare(this.position(), that.position());
+        return c;
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/TaskQueue.java

@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import accord.utils.IntrusivePriorityHeap;
+
+import org.apache.cassandra.service.accord.execution.Task.ExecutorQueue;
+
+// has xSingle methods to distinguish from within MultiTaskQueue whether we're invoking the multi or single variation
+class TaskQueue<T extends Task> extends IntrusivePriorityHeap<T>
+{
+    final ExecutorQueue kind;
+
+    public TaskQueue(ExecutorQueue kind)
+    {
+        this.kind = kind;
+    }
+
+    void unqueue(T task)
+    {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public final int compare(T o1, T o2)
+    {
+        return Long.compare(o1.position, o2.position);
+    }
+
+    @Override
+    protected final void ensureHeapified()
+    {
+        super.ensureHeapified();
+    }
+
+    protected final boolean requeueSingle(T requeue)
+    {
+        int oldIndex = updateNode(requeue);
+        if (oldIndex < 0)
+            return false;
+
+        int newIndex = heapIndex(requeue);
+        return Math.min(oldIndex, newIndex) == 0 && heapifiedSize() > 0;
+    }
+
+    final T peekSingle()
+    {
+        ensureHeapified();
+        return peekNode();
+    }
+
+    final T pollSingle()
+    {
+        ensureHeapified();
+        return pollNode();
+    }
+
+    final boolean isEmptySingle()
+    {
+        return isEmptyInternal();
+    }
+
+    final int enqueueSingle(T enqueue)
+    {
+        return insertNode(enqueue);
+    }
+
+    final boolean unqueueSingle(T unqueue)
+    {
+        int heapIndex = heapIndex(unqueue);
+        removeNode(unqueue);
+        return heapIndex == 0;
+    }
+
+    final boolean tryUnqueueSingle(T remove)
+    {
+        return removeNodeIfContains(remove);
+    }
+
+    final T getSingle(int index)
+    {
+        return super.getNode(index);
+    }
+
+    final boolean isQueuedSingle(T test)
+    {
+        return containsNode(test);
+    }
+
+    @Override
+    public String toString()
+    {
+        return kind.toString();
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/TaskQueueMulti.java

@@ -0,0 +1,627 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import accord.utils.Invariants;
+import accord.utils.UnhandledEnum;
+
+import org.apache.cassandra.service.accord.execution.Task.ExecutorQueue;
+import org.apache.cassandra.service.accord.execution.Task.GroupKind;
+
+import static org.apache.cassandra.service.accord.execution.Task.ExecutorQueue.RUNNABLE;
+
+/**
+ * A {@link TaskQueue} sub-divided into up to eight per-group sub-queues (groups are phases for the exclusive
+ * executor, or work classes globally) plus a policy for choosing which sub-queue to serve next. The policy balances
+ * three competing concerns: ordering/priority (run the oldest / highest-priority work), fairness (do not let one
+ * group monopolise the executor), and throughput (do not let an even split leave a busy group's backlog growing
+ * without bound).
+ *
+ * <h2>Packed counters</h2>
+ * Per-group state is held in {@code long}s, one 7-bit lane per group; bit 7 of each byte is a spare "guard" bit used
+ * to run branch-free min/compare across all eight lanes at once (see {@code minCounters}). The lanes are:
+ * <ul>
+ *   <li>{@code positions[]} - the head task's position (HLC, or submission order) per group; drives FIFO/priority.</li>
+ *   <li>{@code recent} - a windowed count of recently <i>served</i> tasks per group (service received).</li>
+ *   <li>{@code arrivals} - a windowed, size-biased count of recent <i>arrivals</i> per group (demand offered).</li>
+ *   <li>{@code current} - tasks currently in flight per group, used to enforce {@code queue_active_limits}.</li>
+ *   <li>{@code hasWork}/{@code dirty} - guard-bit masks: which groups have queued work / need a position refresh.</li>
+ * </ul>
+ *
+ * <h2>Windowing: a bounded memory of the past</h2>
+ * {@code recent} and {@code arrivals} are not running totals. Whenever incrementing {@code recent} tips any lane to
+ * its maximum (0x80), <em>both</em> {@code recent} and {@code arrivals} are halved (see {@code incrementRecents}),
+ * and each lane also saturates at 0x7f. They are therefore an exponentially-decaying window keyed on service/time.
+ * This is what stops a one-off burst of arrivals granting a group a permanent boost: the burst saturates the lane
+ * briefly and then decays away as other work is served.
+ *
+ * <h2>The fair selection counter</h2>
+ * The fair models pick the group with the smallest {@code effective} counter, where per lane
+ * <pre>effective = min(0x7f, max(0, recent - arrivals) + bias)</pre>
+ * <ul>
+ *   <li>{@code max(0, recent - arrivals)} - a group whose arrivals have outpaced its recent service clamps to 0 and
+ *       is therefore preferred; in steady state each group's service tracks its arrival rate, which bounds backlog.
+ *       The clamp is symmetric, so a group cannot bank "credit" during a lull and later use it to monopolise service.</li>
+ * </ul>
+ *
+ * <h2>Choosing a group ({@code minGroup} / {@code pollByBlend})</h2>
+ * <ul>
+ *   <li>{@code PRIORITY_ONLY} - always the lowest {@code positions} (oldest / highest priority), ignoring fairness.</li>
+ *   <li>{@code PHASE_OVERRIDE} - strict phase order: the lowest-index group with work, always.</li>
+ *   <li>{@code PHASE_FAIR} - pure fairness: the smallest {@code effective}; ties broken by index, within a group by
+ *       position. It has no priority fallback, so it is deliberately completing-first under contention.</li>
+ *   <li>{@code PRIORITY_FAIR} (default) - a deficit round-robin blend of two strategies, chosen per poll: <b>flow</b>
+ *       (smallest {@code effective}, i.e. least fairly serviced) and <b>age</b> (oldest {@code positions}, i.e. FIFO).
+ *       The flow weight ramps up with the flow imbalance ({@code max-min} of {@code effective}) over
+ *       {@code queue_flow_imbalance_onset..+width}, trading against age zero-sum; when balanced it is pure age. The
+ *       ramp is smooth (not a hard mode switch), so there is no boundary to oscillate around and no hysteresis is
+ *       needed. Age also drains stale/standing backlogs for free: a backlog's items are the oldest work, so age
+ *       clears them without a separate backlog-aware strategy (the {@code effective} flow counter tracks arrival
+ *       <em>rate</em> and is deliberately blind to standing <em>stock</em>).</li>
+ * </ul>
+ *
+ * <h2>Concurrency limits and eligibility</h2>
+ * A group whose in-flight {@code current} count has reached its {@code queue_active_limits} limit is
+ * {@code saturated} and excluded from selection ({@code disabled}), as is any group with no queued work.
+ *
+ * <h2>Note on {@code positions} freshness</h2>
+ * {@code positions} is refreshed lazily from the sub-queue heads via the {@code dirty} mask; {@code dirty} must use
+ * the same guard-bit layout as {@code hasWork} so the refresh loop in {@code minGroupByPriority} actually runs -
+ * otherwise it silently degenerates to lowest-index-with-work and starves high-index / new-work groups.
+ *
+ * <p>NB it extends {@link TaskQueue} to keep the type hierarchy simple for method dispatch, and for efficiency for
+ * anonymous ExclusiveExecutors which do not use multiple queues, while letting ExclusiveExecutor share a parent
+ * class for both use cases.
+ */
+abstract class TaskQueueMulti<T extends Task> extends TaskQueue<T>
+{
+    private static final TaskQueue[] NO_QUEUES = new TaskQueue[0];
+    private static final long[] NO_POSITIONS = new long[0];
+    static final long COUNTER_OVERFLOWS = 0x8080808080808080L;
+    static final long COUNTER_MASKS = 0x7f7f7f7f7f7f7f7fL;
+    static final long COUNTER_LOWBITS = 0x0101010101010101L;
+
+    final TaskQueue<T>[] queues;
+    final long[] positions;
+    final byte groupShift;
+    final long limits;
+
+    /**
+     * sets overflow bits for a queue that has been stopped
+     */
+    long stopped;
+    /**
+     * sets overflow bits for each counter when it needs its position updated
+     */
+    long dirty;
+    /**
+     * sets overflow bits for each counter when there's associated work
+     */
+    long hasWork;
+    /**
+     * Stores recent dequeue counts for up to 8 sub queues.
+     */
+    long dispatches;
+    // TODO (required): increment arrivals based on internal queue for ExclusiveExecutors
+    //    also: experiment with decaying on arrival schedule rather than poll schedule, since this should respond to work growth more accurately
+    /**
+     * Stores recent enqueue counts for up to 8 sub queues.
+     */
+    long arrivals;
+    /**
+     * Stores currently-active counts for up to 8 sub queues. We can use this to impose limits on specific queues.
+     */
+    long active;
+    /**
+     * deficit-round-robin credits for the two PRIORITY_FAIR strategies (flow/age).
+     */
+    int creditFlow, creditAge;
+
+    int waitingCount;
+
+    TaskQueueMulti(ExecutorQueue kind, GroupKind groups, long limits)
+    {
+        super(kind);
+        this.limits = limits;
+        int queueCount = groups.count;
+        Invariants.require(queueCount <= 8);
+        queues = queueCount == 0 ? NO_QUEUES : new TaskQueue[queueCount];
+        positions = queueCount > 0 && AccordExecutor.BALANCE_BY_POSITION ? new long[queueCount] : NO_POSITIONS;
+        groupShift = groups.shift;
+    }
+
+    final int group(Task task)
+    {
+        if (groupShift == 0)
+            return -1;
+
+        return (task.info >>> groupShift) & Task.GROUP_MASK;
+    }
+
+    void stop(long groupOverflowBits)
+    {
+        stopped |= groupOverflowBits;
+    }
+
+    void restart(long groupOverflowBits)
+    {
+        stopped &= ~groupOverflowBits;
+    }
+
+    final TaskQueue<T> queue(Task task)
+    {
+        int group = group(task);
+        if (group < 0)
+            return this;
+
+        return queue(group);
+    }
+
+    final TaskQueue<T> queue(int group)
+    {
+        TaskQueue<T> queue = queues[group];
+        if (queue == null)
+            queues[group] = queue = new TaskQueue<>(RUNNABLE);
+
+        return queue;
+    }
+
+    private int pollGroup()
+    {
+        if (hasWork == 0)
+            return -1;
+
+        switch (AccordExecutor.BALANCING_MODEL)
+        {
+            default:
+                throw new UnhandledEnum(AccordExecutor.PRIORITY_MODEL);
+            case PRIORITY_ONLY:
+                return pollGroupByPriority();
+            case PHASE_ONLY:
+                return pollGroupByIndex();
+            case PHASE_FAIR:
+                return pollGroupByPhaseFair();
+            case BLENDED_PRIORITY_PHASE_FAIR:
+                return pollGroupByBlended();
+        }
+    }
+
+    private int pollGroupByPriority()
+    {
+        return pollGroupByPriority(unsaturatedWithWork());
+    }
+
+    private int pollGroupByPriority(long enabled)
+    {
+        long refresh = dirty & hasWork;
+        while (refresh != 0)
+        {
+            int bitIndex = Long.numberOfTrailingZeros(refresh);
+            int group = bitIndex / 8;
+            positions[group] = queues[group].peekSingle().position;
+            refresh ^= 1L << bitIndex;
+        }
+        dirty = 0;
+
+        long minPosition = Long.MAX_VALUE;
+        int minGroup = -1;
+        long visit = enabled >>> 7;
+        while (visit != 0)
+        {
+            int bitIndex = Long.numberOfTrailingZeros(visit);
+            int group = bitIndex / 8;
+            long position = positions[group];
+            if (position < minPosition)
+            {
+                minGroup = group;
+                minPosition = position;
+            }
+            visit ^= 1L << bitIndex;
+        }
+
+        return minGroup;
+    }
+
+    private int pollGroupByIndex()
+    {
+        long visit = (hasWork & unsaturated()) >>> 7;
+        if (visit == 0)
+            return -1;
+
+        int bitIndex = Long.numberOfTrailingZeros(visit);
+        return bitIndex / 8;
+    }
+
+    private int pollGroupByPhaseFair()
+    {
+        return minCounterIndex(recentFlowImbalances());
+    }
+
+    // PRIORITY_FAIR selection: a deficit round-robin blend of two strategies, chosen per poll:
+    //   flow -> minCounterIndex(recent - arrivals + bias)  (least fairly serviced)
+    //   age  -> minGroupByPriority()  (earliest-queued work)
+    // wFlow = ramp(flow imbalance F = max-min of the selection counters); wAge = BLEND_TOTAL - wFlow. As flow gets
+    // uneven, polls trade from age to flow zero-sum; when balanced it is pure age (FIFO). A ramp (not a hard
+    // threshold) means there is no mode cliff to oscillate around, so no anti-oscillation penalty is needed. Age is
+    // also what drains a stale/standing backlog -- its items are the oldest -- so no separate stock strategy is needed.
+    private int pollGroupByBlended()
+    {
+        return pollGroupByBlended(saturatedOrWithoutWork());
+    }
+
+    private int pollGroupByBlended(long disabled)
+    {
+        long withoutWork = hasWork ^ COUNTER_OVERFLOWS;
+        long counters = recentFlowImbalances();
+        long minMax = minMaxCounterValue(counters, withoutWork);
+        long min = minMax & 0x7f;
+        long max = minMax >>> 8;
+        int flowImbalance = (int) (max - min);
+
+        int flowWeight = flowWeight(flowImbalance);
+        int priorityWeight = AccordExecutor.BLEND_TOTAL - flowWeight;
+
+        creditFlow += flowWeight;
+        creditAge += priorityWeight;
+
+        if (creditFlow >= creditAge)
+        {
+            creditFlow -= AccordExecutor.BLEND_TOTAL;
+            if (disabled != withoutWork)
+                min = minCounterValue(counters, disabled);
+
+            return minCounterIndex(counters, min, disabled);
+        }
+        else
+        {
+            creditAge -= AccordExecutor.BLEND_TOTAL;
+            return pollGroupByPriority(disabled ^ COUNTER_OVERFLOWS);
+        }
+    }
+
+    private long saturated()
+    {
+        return ((active | COUNTER_OVERFLOWS) - limits) & COUNTER_OVERFLOWS;
+    }
+
+    private long unsaturated()
+    {
+        return saturated() ^ COUNTER_OVERFLOWS;
+    }
+
+    private long unsaturatedWithWork()
+    {
+        return hasWork & unsaturated() & ~stopped;
+    }
+
+    private long saturatedOrWithoutWork()
+    {
+        return (hasWork ^ COUNTER_OVERFLOWS) | saturated() | stopped;
+    }
+
+    // arrivals is a windowed measure of ARRIVAL (incremented on enqueue, size-biased; decays on recent's overflow).
+    // Combined with recent (service) as effective = max(0, recent - arrivals): a queue whose arrivals outpace its
+    // service clamps to 0 and is preferred, so service converges to arrival rate (bounding the busy queue's backlog).
+    private long recentFlowImbalances()
+    {
+        return clampedSubtract(dispatches, arrivals);
+    }
+
+    static long minCounterValue(long counters, long disabled)
+    {
+        long mins = counters;
+        mins |= overflowsToLowMasks(disabled);
+        mins = minCounters(mins, mins >>> 8); // each slot is min of slots [i..i+1]
+        mins = minCounters(mins, mins >>> 16); // each slot is min of slots [i..i+3]
+        mins = minCounters(mins, mins >>> 32); // each slot is min of slots [i..i+7]
+        return mins & 0x7f;
+    }
+
+    static long minMaxCounterValue(long counters, long disabled)
+    {
+        long mins = counters;
+        long maxs = counters ^ COUNTER_MASKS;
+        long overflowMasks = overflowsToLowMasks(disabled);
+        mins |= overflowMasks;
+        maxs |= overflowMasks;
+        mins = minCounters(mins, mins >>> 8) & 0x007f007f007f007fL; // each slot is min of slots [i..i+1]
+        maxs = (minCounters(maxs, maxs << 8) & 0x7f007f007f007f00L); // each slot is min of slots ~[i..i+1]
+        long minmaxs = mins | maxs;
+        minmaxs = minCounters(minmaxs, minmaxs >>> 16); // each slot is min of slots [i..i+3]
+        minmaxs = minCounters(minmaxs, minmaxs >>> 32); // each slot is min of slots [i..i+7]
+        return (minmaxs ^ 0x7f00) & 0x7f7f;
+    }
+
+    /**
+     * If provided two counters (containing 8 7 bit counters each),
+     * returns the minimum of each matching counter
+     */
+    private static long minCounters(long a, long b)
+    {
+        // set overflow bits where a <= b
+        long selecta = setOverflowWhenLessEqual(a, b);
+        return selectByOverflowBits(selecta, a, b);
+    }
+
+    static long setOverflowWhenLessEqual(long a, long b)
+    {
+        return ((b | COUNTER_OVERFLOWS) - a) & COUNTER_OVERFLOWS;
+    }
+
+    // select a if overflow bit is set; b if it is unset
+    static long selectByOverflowBits(long selecta, long a, long b)
+    {
+        selecta = overflowsToLowMasks(selecta);
+        a &= selecta;
+        b &= ~selecta;
+        return a | b;
+    }
+
+    static long overflowsToLowMasks(long v)
+    {
+        return v - (v >>> 7);
+    }
+
+    private static int flowWeight(int flowImbalance)
+    {
+        if (flowImbalance <= AccordExecutor.FLOW_ONSET) return 0;
+        return Math.min(AccordExecutor.BLEND_TOTAL, ((flowImbalance - AccordExecutor.FLOW_ONSET) << AccordExecutor.BLEND_SHIFT) >>> AccordExecutor.FLOW_WIDTH_SHIFT);
+    }
+
+    // per-lane max(0, a - b), carry-free: zero both a and b in lanes where a <= b, then subtract
+    private static long clampedSubtract(long a, long b)
+    {
+        long keep = ~overflowsToLowMasks(setOverflowWhenLessEqual(a, b));
+        return (a & keep) - (b & keep);
+    }
+
+    private int minCounterIndex(long counters)
+    {
+        return minCounterIndex(counters, saturatedOrWithoutWork());
+    }
+
+    private int minCounterIndex(long counters, long disabled)
+    {
+        return minCounterIndex(counters, minCounterValue(counters, disabled), disabled);
+    }
+
+    private int minCounterIndex(long counters, long minCounterValue, long disabled)
+    {
+        long mins = minCounterValue * COUNTER_LOWBITS;
+        long select = ((mins | COUNTER_OVERFLOWS) - counters) & COUNTER_OVERFLOWS;
+        // now unset those overflow bits associated with disabled queues
+        select &= ~disabled;
+        if (select == 0)
+            return -1;
+        return (Long.numberOfTrailingZeros(select) - 7) / 8;
+    }
+
+    final T pollMulti()
+    {
+        int group = pollGroup();
+        if (group < 0)
+        {
+            // group < 0 can mean EITHER we don't have any nested queues OR those queues are either empty or DISABLED
+            T result = pollSingle();
+            if (result != null)
+                --waitingCount;
+            return result;
+        }
+
+        --waitingCount;
+        incrementActive(group);
+        incrementDispatches(group);
+
+        TaskQueue<T> queue = queues[group];
+        T head = queue.pollSingle();
+        // NOTE: must clear dirty when emptied, symmetrically with unqueue(): the fair selection paths
+        // never consume the dirty bit, so a group drained during a fairness episode would otherwise
+        // retain a stale dirty bit and NPE in minGroupByPriority.peekSingle() when balance is restored.
+        if (queue.isEmptySingle())
+        {
+            unsetHasWork(group);
+            unsetDirty(group);
+        }
+        else setDirty(group);
+        return head;
+    }
+
+    final void enqueueMulti(T task, boolean incrementArrivals)
+    {
+        task.setQueue(kind);
+        int group = group(task);
+        if (group < 0)
+        {
+            enqueueSingle(task);
+        }
+        else
+        {
+            TaskQueue<T> queue = queue(group);
+            int result = queue.enqueueSingle(task);
+            if (incrementArrivals)
+                incrementArrivals(group);
+            if (result < 0) setHasWork(group);
+            if (result != 0) setDirty(group);
+        }
+        ++waitingCount;
+    }
+
+    final void requeue(T task)
+    {
+        int group = group(task);
+        if (group < 0) requeueSingle(task);
+        else
+        {
+            TaskQueue<T> queue = queue(group);
+            Invariants.require(queue != null && queue.isQueuedSingle(task));
+            if (queue.requeueSingle(task))
+                setDirty(group);
+        }
+    }
+
+    final void unqueueMulti(T task)
+    {
+        int group = group(task);
+        TaskQueue<T> queue = group < 0 ? this : queue(task);
+        Invariants.require(queue.isQueuedSingle(task));
+        unqueue(task, group, queue);
+    }
+
+    // if there is an active collection, we return false and do not remove ourselves from it
+    boolean tryUnqueueWaiting(T task)
+    {
+        int group = group(task);
+        TaskQueue<T> queue = group < 0 ? this : queue(task);
+        if (!queue.isQueuedSingle(task))
+            return false;
+
+        unqueue(task, group, queue);
+        return true;
+    }
+
+    private void unqueue(T task, int group, TaskQueue<T> queue)
+    {
+        task.unsetQueue(kind);
+        boolean dirty = queue.unqueueSingle(task);
+        --waitingCount;
+        if (group >= 0)
+        {
+            if (queue.isEmptySingle())
+            {
+                unsetHasWork(group);
+                unsetDirty(group);
+            }
+            else if (dirty) setDirty(group);
+        }
+    }
+
+    final void incrementActive(int group)
+    {
+        active += lowBit(group);
+    }
+
+    final void decrementActive(int group)
+    {
+        active -= lowBit(group);
+    }
+
+    final void incrementDispatches(Task task)
+    {
+        int group = group(task);
+        if (group >= 0)
+            incrementDispatches(group);
+    }
+
+    final void incrementDispatches(int group)
+    {
+        dispatches += lowBit(group);
+        if ((dispatches & COUNTER_OVERFLOWS) != 0)
+        {
+            dispatches = (dispatches >>> 1) & COUNTER_MASKS;
+            arrivals = (arrivals >>> 1) & COUNTER_MASKS; // arrivals (arrival) decays on the service/time clock
+        }
+    }
+
+    final void decrementDispatches(Task task)
+    {
+        int group = group(task);
+        if (group >= 0)
+            decrementDispatches(group);
+    }
+
+    final void decrementDispatches(int group)
+    {
+        long lowBit = lowBit(group);
+        dispatches -= lowBit;
+        dispatches += (dispatches >>> 7) & lowBit;
+    }
+
+    final void incrementArrivals(Task task)
+    {
+        int group = group(task);
+        if (group >= 0)
+            incrementArrivals(group);
+    }
+
+    final void incrementArrivals(int group)
+    {
+        int shift = group * 8;
+        long overflowBit = 0x80L << shift;
+        arrivals += 1L << shift;
+        // if we overflow, unset the overflow bit and set all other bits for the counter
+        long overflow = arrivals & overflowBit;
+        arrivals ^= overflow;
+        arrivals |= overflow - (overflow >>> 7);
+    }
+
+    final boolean hasWaitingToRunExcluding(long groupOverflowBits)
+    {
+        return (unsaturatedWithWork() & ~groupOverflowBits) != 0;
+    }
+
+    final void setHasWork(int group)
+    {
+        hasWork |= overflowBit(group);
+    }
+
+    final void unsetHasWork(int group)
+    {
+        hasWork &= ~overflowBit(group);
+    }
+
+    final void setDirty(int group)
+    {
+        dirty |= overflowBit(group);
+    }
+
+    final void unsetDirty(int group)
+    {
+        dirty &= ~overflowBit(group);
+    }
+
+    final boolean hasWaitingToRun()
+    {
+        return unsaturatedWithWork() != 0;
+    }
+
+    final boolean isWaiting(T task)
+    {
+        return queue(task).isQueuedSingle(task);
+    }
+
+    final int waitingCount()
+    {
+        return waitingCount;
+    }
+
+    static long lowBit(int group)
+    {
+        return 1L << (group * 8);
+    }
+
+    static long overflowBit(int group)
+    {
+        return 0x80L << (group * 8);
+    }
+
+    static long overflowBit(Enum<?> group)
+    {
+        return overflowBit(group.ordinal());
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/TaskQueueRunnable.java

@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import static org.apache.cassandra.service.accord.execution.Task.ExecutorQueue.RUNNABLE;
+
+final class TaskQueueRunnable<T extends Task> extends TaskQueueMulti<T>
+{
+    final TaskQueue<T> assigned;
+
+    TaskQueueRunnable()
+    {
+        super(RUNNABLE, Task.GroupKind.GLOBAL, AccordExecutor.GLOBAL_QUEUE_LIMITS);
+        this.assigned = new TaskQueue<>(RUNNABLE);
+    }
+
+    T poll()
+    {
+        T next = pollMulti();
+        if (next == null)
+            return null;
+
+        assigned.enqueueSingle(next);
+        return next;
+    }
+
+    void enqueue(T enqueue, boolean incrementArrivals)
+    {
+        enqueueMulti(enqueue, incrementArrivals);
+    }
+
+    void unqueue(T unqueue)
+    {
+        if (assigned.isQueuedSingle(unqueue))
+        {
+            int group = group(unqueue);
+            if (group >= 0)
+                decrementActive(group);
+
+            unqueue.unsetQueue(kind);
+            assigned.unqueueSingle(unqueue);
+        }
+        else
+        {
+            super.unqueueMulti(unqueue);
+        }
+    }
+
+    int waitingOrAssignedCount()
+    {
+        return waitingCount + assigned.size();
+    }
+
+    boolean hasAssignedOrWaiting()
+    {
+        return waitingCount > 0 || !assigned.isEmptySingle();
+    }
+
+    boolean hasAssigned()
+    {
+        return !assigned.isEmptySingle();
+    }
+
+    boolean isAssigned(T task)
+    {
+        return assigned.isQueuedSingle(task);
+    }
+
+    void cleanup(T task)
+    {
+        if (assigned.tryUnqueueSingle(task))
+        {
+            int group = group(task);
+            if (group >= 0)
+                decrementActive(group);
+            task.unsetQueue(kind);
+        }
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/TaskQueueStandalone.java

@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import org.apache.cassandra.service.accord.execution.Task.ExecutorQueue;
+
+final class TaskQueueStandalone<T extends Task> extends TaskQueue<T>
+{
+    TaskQueueStandalone(ExecutorQueue kind)
+    {
+        super(kind);
+    }
+
+    void enqueue(T enqueue)
+    {
+        enqueue.setQueue(kind);
+        enqueueSingle(enqueue);
+    }
+
+    boolean tryUnqueue(T unqueue)
+    {
+        if (!containsNode(unqueue))
+            return false;
+
+        unqueue(unqueue);
+        return true;
+    }
+
+    void unqueue(T unqueue)
+    {
+        unqueue.unsetQueue(kind);
+        removeNode(unqueue);
+    }
+
+    T peek()
+    {
+        return peekSingle();
+    }
+
+    T poll()
+    {
+        return pollSingle();
+    }
+
+    boolean isEmpty()
+    {
+        return isEmptySingle();
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/TaskRunner.java

@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import org.apache.cassandra.concurrent.CassandraThread;
+import org.apache.cassandra.concurrent.DebuggableTask;
+
+public interface TaskRunner
+{
+    AccordExecutor accordActiveExecutor();
+    void setAccordActiveExecutor(AccordExecutor newExecutor);
+
+    AccordExecutor accordLockedExecutor();
+    boolean tryEnterAccordLockedExecutor(AccordExecutor newLockedExecutor);
+    void exitAccordLockedExecutor();
+
+    // to be called only by the thread itself, so can (eventually) avoid any memory barriers
+    Task accordActiveSelfTask();
+    Task accordActiveTask();
+    void setAccordActiveTask(Task newActiveTask);
+
+    static TaskRunner get()
+    {
+        return get(Thread.currentThread());
+    }
+
+    static TaskRunner get(Thread thread)
+    {
+        return thread instanceof CassandraThread ? (CassandraThread) thread : ThreadLocalTaskRunner.threadLocal.get();
+    }
+
+    final class ThreadLocalTaskRunner implements TaskRunner
+    {
+        private AccordExecutor lockedExecutor;
+        private int lockedExecutorDepth;
+        private AccordExecutor activeExecutor;
+        volatile Task activeTask;
+
+        private static final ThreadLocal<ThreadLocalTaskRunner> threadLocal = ThreadLocal.withInitial(ThreadLocalTaskRunner::new);
+
+        @Override
+        public AccordExecutor accordActiveExecutor()
+        {
+            return activeExecutor;
+        }
+
+        @Override
+        public void setAccordActiveExecutor(AccordExecutor newExecutor)
+        {
+            activeExecutor = newExecutor;
+        }
+
+        @Override
+        public AccordExecutor accordLockedExecutor()
+        {
+            return lockedExecutor;
+        }
+
+        @Override
+        public boolean tryEnterAccordLockedExecutor(AccordExecutor newLockedExecutor)
+        {
+            if (lockedExecutor == null) lockedExecutor = newLockedExecutor;
+            else if (lockedExecutor != newLockedExecutor) return false;
+            ++lockedExecutorDepth;
+            return true;
+        }
+
+        @Override
+        public void exitAccordLockedExecutor()
+        {
+            if (--lockedExecutorDepth == 0)
+                lockedExecutor = null;
+        }
+
+        @Override
+        public void setAccordActiveTask(Task newActiveTask)
+        {
+            activeTask = newActiveTask;
+        }
+
+        @Override
+        public Task accordActiveTask()
+        {
+            return activeTask;
+        }
+
+        @Override
+        public Task accordActiveSelfTask()
+        {
+            return activeTask;
+        }
+    }
+
+    abstract class DebuggableWrapper implements DebuggableTask.DebuggableTaskRunner
+    {
+        private volatile TaskRunner wrapped;
+
+        protected void setWrapped(TaskRunner wrapped)
+        {
+            this.wrapped = wrapped;
+        }
+
+        @Override
+        public DebuggableTask running()
+        {
+            Task running = wrapped.accordActiveTask();
+            return running == null ? null : running.debuggable();
+        }
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/Tranches.java

@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import java.util.ArrayDeque;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import accord.utils.Invariants;
+
+import static org.apache.cassandra.service.accord.execution.Task.MAX_TRANCHE;
+
+/**
+ * Tasks are separated into tranches based on their position.
+ * <p>
+ * When we want to wait for all submitted tasks and their consequences to complete, we create a new tranche
+ * and track the number of tasks still extant for all prior tranches - once these reach zero we can signal
+ * the completion of the work.
+ */
+final class Tranches
+{
+    private static final Logger logger = LoggerFactory.getLogger(Tranches.class);
+
+    class WithDeferred implements Runnable
+    {
+        final Runnable run;
+        final ArrayDeque<Runnable> deferred;
+
+        WithDeferred(Runnable run, Runnable deferred)
+        {
+            this.run = run;
+            this.deferred = new ArrayDeque<>(1);
+            this.deferred.add(deferred);
+        }
+
+        WithDeferred(Runnable run, ArrayDeque<Runnable> deferred)
+        {
+            this.run = run;
+            this.deferred = deferred;
+        }
+
+        @Override
+        public void run()
+        {
+            Runnable register = deferred.poll();
+            if (!deferred.isEmpty())
+            {
+                Invariants.require(runs[firstIndex] != null);
+                Invariants.require(!(runs[firstIndex] instanceof WithDeferred));
+                runs[firstIndex] = new WithDeferred(runs[firstIndex], deferred);
+            }
+            registerWait(register);
+            try
+            {
+                run.run();
+            }
+            catch (Throwable t)
+            {
+                owner.agent.onException(t);
+            }
+        }
+    }
+
+    final AccordExecutor owner;
+
+    int firstTranche;
+    int firstIndex;
+    long[] mins = new long[8];
+    int[] counts = new int[8];
+    Runnable[] runs = new Runnable[8];
+
+    // the next tranche, that all new work is being collected against
+    // (and will move to the array accounting once a new wait is registered)
+    long nextMin;
+    int nextTranche;
+    int nextCount;
+
+    Tranches(AccordExecutor owner)
+    {
+        this.owner = owner;
+    }
+
+    private int trancheToIndex(int tranche)
+    {
+        return firstIndex + trancheToIndexOffset(tranche);
+    }
+
+    private int trancheToIndexOffset(int tranche)
+    {
+        int offset = tranche - firstTranche;
+        if (offset < 0)
+            offset += MAX_TRANCHE + 1;
+        return offset;
+    }
+
+    int size()
+    {
+        return trancheToIndexOffset(nextTranche);
+    }
+
+    int capacity()
+    {
+        return counts.length;
+    }
+
+    int addNew(long position)
+    {
+        Invariants.require(position >= nextMin);
+        ++nextCount;
+        return nextTranche;
+    }
+
+    void addInherited(int tranche, long position)
+    {
+        if (tranche == nextTranche)
+        {
+            Invariants.require(position >= nextMin);
+            ++nextCount;
+        }
+        else
+        {
+            int index = trancheToIndex(tranche);
+            Invariants.require(counts[index] > 0);
+            Invariants.require(mins[index] <= position);
+            ++counts[index];
+        }
+    }
+
+    void complete(int tranche)
+    {
+        if (tranche == nextTranche)
+        {
+            --nextCount;
+        }
+        else
+        {
+
+            if (counts[trancheToIndex(tranche)] == 1)
+                owner.drainUnqueuedNewWorkExclusive(); // make sure we don't have any pending
+
+            if (--counts[trancheToIndex(tranche)] == 0 && tranche == firstTranche)
+            {
+                do
+                {
+                    advance();
+                }
+                while (firstTranche != nextTranche && counts[firstIndex] == 0);
+            }
+
+            if (firstIndex >= counts.length / 2)
+                compact();
+        }
+    }
+
+    public void finishAll(long nextPosition)
+    {
+        while (firstTranche != nextTranche)
+        {
+            logger.warn("{} processed all pending tasks (<{}) but found {} waiting for {}", this,
+                                       nextPosition, counts[firstIndex], size() == 1 ? nextMin : mins[firstIndex + 1]);
+            advance();
+        }
+    }
+
+    private void advance()
+    {
+        Runnable run = runs[firstIndex];
+        runs[firstIndex] = null;
+        ++firstIndex;
+        if (firstTranche == MAX_TRANCHE) firstTranche = 0;
+        else ++firstTranche;
+        try
+        {
+            run.run();
+        }
+        catch (Throwable t)
+        {
+            owner.agent.onException(t);
+        }
+    }
+
+    public void registerWait(Runnable run)
+    {
+        int newNextTranche = (nextTranche + 1) % (MAX_TRANCHE + 1);
+        if (newNextTranche == firstTranche)
+        {
+            Runnable cur = runs[firstIndex];
+            if (cur instanceof WithDeferred) ((WithDeferred) cur).deferred.add(run);
+            else runs[firstIndex] = new WithDeferred(runs[firstIndex], run);
+            return;
+        }
+
+        if ((firstIndex + size()) == capacity())
+            growOrCompact();
+
+        int index = firstIndex + size();
+        mins[index] = nextMin;
+        counts[index] = nextCount;
+        runs[index] = run;
+        nextMin = owner.minPosition = owner.nextPosition;
+        nextCount = 0;
+        nextTranche = newNextTranche;
+    }
+
+    private void compact()
+    {
+        if (size() <= capacity() / 4 && capacity() > 8) resize(capacity() / 2);
+        else compact(mins, counts, runs);
+    }
+
+    private void growOrCompact()
+    {
+        if (size() > capacity() / 2) resize(capacity() * 2);
+        else compact();
+    }
+
+    private void resize(int newSize)
+    {
+        Invariants.require(newSize > 0);
+        long[] newMins = new long[newSize];
+        int[] newCounts = new int[newSize];
+        Runnable[] newRuns = new Runnable[newSize];
+        compact(newMins, newCounts, newRuns);
+        mins = newMins;
+        counts = newCounts;
+        runs = newRuns;
+    }
+
+    private void compact(long[] newMins, int[] newCounts, Runnable[] newRuns)
+    {
+        int size = size();
+        System.arraycopy(mins, firstIndex, newMins, 0, size);
+        System.arraycopy(counts, firstIndex, newCounts, 0, size);
+        System.arraycopy(runs, firstIndex, newRuns, 0, size);
+        firstIndex = 0;
+    }
+}

src/java/org/apache/cassandra/service/accord/execution/Unstoppable.java

@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+import accord.local.ExecutionContext;
+
+// run the task even on a stopped commandStore
+public interface Unstoppable extends ExecutionContext.Empty
+{
+}

src/java/org/apache/cassandra/service/accord/execution/Unterminatable.java

@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord.execution;
+
+// run the task even on a terminated commandStore
+public interface Unterminatable extends Unstoppable
+{
+}

src/java/org/apache/cassandra/service/accord/interop/AccordInteropAdapter.java

@@ -23,6 +23,7 @@ import java.util.function.BiConsumer;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import accord.api.ExclusiveAsyncExecutor;
 import accord.api.Result;
 import accord.api.Update;
 import accord.coordinate.CoordinationAdapter;
@@ -30,7 +31,6 @@ import accord.coordinate.CoordinationAdapter.Adapters.TxnAdapter;
 import accord.coordinate.ExecuteFlag.CoordinationFlags;
 import accord.coordinate.ExecutePath;
 import accord.local.Node;
-import accord.local.SequentialAsyncExecutor;
 import accord.messages.Apply;
 import accord.primitives.Ballot;
 import accord.primitives.Deps;
@@ -47,6 +47,7 @@ import org.apache.cassandra.service.accord.topology.AccordEndpointMapper;
 import org.apache.cassandra.service.accord.txn.AccordUpdate;
 import org.apache.cassandra.service.accord.txn.TxnRead;
 
+import static accord.api.ProtocolModifiers.sendMinimal;
 import static accord.messages.Apply.Kind.Maximal;
 import static accord.messages.Apply.Kind.Minimal;
 
@@ -55,12 +56,13 @@ public class AccordInteropAdapter extends TxnAdapter
     private static final Logger logger = LoggerFactory.getLogger(AccordInteropAdapter.class);
     public static final class AccordInteropFactory extends DefaultFactory
     {
-        final AccordInteropAdapter standard, recovery;
+        final AccordInteropAdapter minimal, maximal, recovery;
 
         public AccordInteropFactory(AccordEndpointMapper endpointMapper)
         {
-            standard = new AccordInteropAdapter(endpointMapper, Minimal);
+            minimal = new AccordInteropAdapter(endpointMapper, Minimal, true);
-            recovery = new AccordInteropAdapter(endpointMapper, Maximal);
+            maximal = new AccordInteropAdapter(endpointMapper, Maximal, true);
+            recovery = new AccordInteropAdapter(endpointMapper, Maximal, false);
         }
 
         @Override
@@ -68,37 +70,37 @@ public class AccordInteropAdapter extends TxnAdapter
         {
             if (txnId.isSyncPoint())
                 return super.get(txnId, step);
-            return (CoordinationAdapter<R>) (step == Kind.Recovery ? recovery : standard);
+            return (CoordinationAdapter<R>) (step == Kind.Recovery ? recovery : sendMinimal() ? minimal : maximal);
         }
     };
 
     private final AccordEndpointMapper endpointMapper;
-    private final Apply.Kind applyKind;
+    private final boolean isUserFacing;
 
-    private AccordInteropAdapter(AccordEndpointMapper endpointMapper, Apply.Kind applyKind)
+    private AccordInteropAdapter(AccordEndpointMapper endpointMapper, Apply.Kind applyKind, boolean isUserFacing)
     {
-        super(Minimal);
+        super(applyKind);
         this.endpointMapper = endpointMapper;
-        this.applyKind = applyKind;
+        this.isUserFacing = isUserFacing;
     }
 
     @Override
-    public void execute(Node node, SequentialAsyncExecutor executor, Topologies any, FullRoute<?> route, Ballot ballot, ExecutePath path, CoordinationFlags flags, TxnId txnId, Txn txn, Timestamp executeAt, Deps stableDeps, Deps sendDeps, BiConsumer<? super Result, Throwable> callback)
+    public void execute(Node node, ExclusiveAsyncExecutor executor, Topologies any, FullRoute<?> route, Ballot ballot, ExecutePath path, CoordinationFlags flags, TxnId txnId, Txn txn, Timestamp executeAt, Deps stableDeps, Deps sendDeps, BiConsumer<? super Result, Throwable> callback)
     {
         if (!doInteropExecute(node, executor, route, ballot, txnId, txn, executeAt, stableDeps, callback))
             super.execute(node, executor, any, route, ballot, path, flags, txnId, txn, executeAt, stableDeps, sendDeps, callback);
     }
 
     @Override
-    public void persist(Node node, SequentialAsyncExecutor executor, Topologies any, Route<?> require, Route<?> sendTo, FullRoute<?> route, Ballot ballot, CoordinationFlags flags, TxnId txnId, Txn txn, Timestamp executeAt, Deps deps, Writes writes, Result result, boolean informDurableOnDone, BiConsumer<? super Result, Throwable> callback)
+    public void persist(Node node, ExclusiveAsyncExecutor executor, Topologies any, Route<?> require, Route<?> sendTo, FullRoute<?> route, Ballot ballot, CoordinationFlags flags, TxnId txnId, Txn txn, Timestamp executeAt, Deps deps, Writes writes, Result result, boolean informDurableOnDone, BiConsumer<? super Result, Throwable> callback)
     {
-        if (applyKind == Minimal && doInteropPersist(node, executor, any, require, sendTo, ballot, txnId, txn, executeAt, deps, writes, result, route, informDurableOnDone, callback))
+        if (isUserFacing && doInteropPersist(node, executor, any, require, sendTo, ballot, txnId, txn, executeAt, deps, writes, result, route, informDurableOnDone, callback))
             return;
 
         super.persist(node, executor, any, require, sendTo, route, ballot, flags, txnId, txn, executeAt, deps, writes, result, informDurableOnDone, callback);
     }
 
-    private boolean doInteropExecute(Node node, SequentialAsyncExecutor executor, FullRoute<?> route, Ballot ballot, TxnId txnId, Txn txn, Timestamp executeAt, Deps deps, BiConsumer<? super Result, Throwable> callback)
+    private boolean doInteropExecute(Node node, ExclusiveAsyncExecutor executor, FullRoute<?> route, Ballot ballot, TxnId txnId, Txn txn, Timestamp executeAt, Deps deps, BiConsumer<? super Result, Throwable> callback)
     {
         // Unrecoverable repair always needs to be run by AccordInteropExecution
         AccordUpdate.Kind updateKind = AccordUpdate.kind(txn.update());
@@ -121,7 +123,7 @@ public class AccordInteropAdapter extends TxnAdapter
         return true;
     }
 
-    private boolean doInteropPersist(Node node, SequentialAsyncExecutor executor, Topologies any, Route<?> require, Route<?> sendTo, Ballot ballot, TxnId txnId, Txn txn, Timestamp executeAt, Deps deps, Writes writes, Result result, FullRoute<?> fullRoute, boolean informDurableOnDone, BiConsumer<? super Result, Throwable> callback)
+    private boolean doInteropPersist(Node node, ExclusiveAsyncExecutor executor, Topologies any, Route<?> require, Route<?> sendTo, Ballot ballot, TxnId txnId, Txn txn, Timestamp executeAt, Deps deps, Writes writes, Result result, FullRoute<?> fullRoute, boolean informDurableOnDone, BiConsumer<? super Result, Throwable> callback)
     {
         Update update = txn.update();
         ConsistencyLevel consistencyLevel = update instanceof AccordUpdate ? ((AccordUpdate) update).cassandraCommitCL() : null;

src/java/org/apache/cassandra/service/accord/interop/AccordInteropExecution.java

@@ -28,12 +28,12 @@ import java.util.concurrent.atomic.AtomicLongFieldUpdater;
 import java.util.function.BiConsumer;
 
 import accord.api.Data;
+import accord.api.ExclusiveAsyncExecutor;
 import accord.api.Result;
 import accord.coordinate.CoordinationAdapter;
 import accord.coordinate.ExecuteFlag.CoordinationFlags;
 import accord.local.Node;
 import accord.local.Node.Id;
-import accord.local.SequentialAsyncExecutor;
 import accord.messages.Commit;
 import accord.messages.Commit.Kind;
 import accord.primitives.AbstractRanges;
@@ -94,6 +94,7 @@ import org.apache.cassandra.service.reads.ReadCoordinator;
 import org.apache.cassandra.tcm.ClusterMetadata;
 import org.apache.cassandra.transport.Dispatcher;
 
+import static accord.api.ProtocolModifiers.sendMinimal;
 import static accord.coordinate.CoordinationAdapter.Factory.Kind.Standard;
 import static accord.primitives.Txn.Kind.Write;
 import static accord.topology.SelectShards.LIVE;
@@ -125,7 +126,7 @@ public class AccordInteropExecution implements ReadCoordinator
     private final Timestamp executeAt;
     private final Deps deps;
     private final BiConsumer<? super Result, Throwable> callback;
-    private final SequentialAsyncExecutor executor;
+    private final ExclusiveAsyncExecutor executor;
     private final ConsistencyLevel consistencyLevel;
     private final AccordEndpointMapper endpointMapper;
 
@@ -141,9 +142,10 @@ public class AccordInteropExecution implements ReadCoordinator
     private volatile long uniqueHlc;
 
     public AccordInteropExecution(Node node, TxnId txnId, Txn txn, AccordUpdate.Kind updateKind, FullRoute<?> route, Ballot ballot, Timestamp executeAt, Deps deps, BiConsumer<? super Result, Throwable> callback,
-                                  SequentialAsyncExecutor executor, ConsistencyLevel consistencyLevel, AccordEndpointMapper endpointMapper) throws TopologyException
+                                  ExclusiveAsyncExecutor executor, ConsistencyLevel consistencyLevel, AccordEndpointMapper endpointMapper) throws TopologyException
     {
         requireArgument(!txn.read().keys().isEmpty() || updateKind == AccordUpdate.Kind.UNRECOVERABLE_REPAIR);
+        // TODO (required): this does not support privileged coordinator optimisation, as must apply local stable before any other message is sent
         this.node = node;
         this.txnId = txnId;
         this.txn = txn;
@@ -217,8 +219,7 @@ public class AccordInteropExecution implements ReadCoordinator
         // TODO (desired): It would be better to use the re-use the command from the transaction but it's fragile
         //  to try and figure out exactly what changed for things like read repair and short read protection
         //  Also this read scope doesn't reflect the contents of this particular read and is larger than it needs to be
-        // TODO (required): understand interop and whether StableFastPath is appropriate
+        AccordInteropStableThenRead commit = new AccordInteropStableThenRead(id, allTopologies, txnId, commitKind(), executeAt, txn, deps, route, message.payload);
-        AccordInteropStableThenRead commit = new AccordInteropStableThenRead(id, allTopologies, txnId, Kind.StableFastPath, executeAt, txn, deps, route, message.payload);
         node.send(id, commit, executor, new AccordInteropRead.ReadCallback(id, to, message, callback, this), null);
     }
 
@@ -376,7 +377,7 @@ public class AccordInteropExecution implements ReadCoordinator
         {
             InetAddressAndPort endpoint = endpointMapper.mappedEndpointOrNull(to);
             if (endpoint != null && !contacted.contains(endpoint))
-                node.send(to, new Commit(Kind.StableFastPath, to, allTopologies, txnId, txn, route, Ballot.ZERO, executeAt, deps), null);
+                node.send(to, new Commit(commitKind(), to, allTopologies, txnId, txn, route, Ballot.ZERO, executeAt, deps), null);
         }
     }
 
@@ -387,7 +388,7 @@ public class AccordInteropExecution implements ReadCoordinator
             for (Node.Id to : allTopologies.nodes())
             {
                 if (!executeTopology.contains(to))
-                    node.send(to, new Commit(Kind.StableFastPath, to, allTopologies, txnId, txn, route, Ballot.ZERO, executeAt, deps), null);
+                    node.send(to, new Commit(commitKind(), to, allTopologies, txnId, txn, route, Ballot.ZERO, executeAt, deps), null);
             }
         }
         AsyncChain<Data> result;
@@ -415,6 +416,11 @@ public class AccordInteropExecution implements ReadCoordinator
         });
     }
 
+    private static Commit.Kind commitKind()
+    {
+        return sendMinimal() ? Kind.StableFastPath : Kind.StableWithTxnAndDeps;
+    }
+
     private AsyncChain<Data> executeUnrecoverableRepairUpdate()
     {
         return AsyncChains.chain(Stage.ACCORD_MIGRATION.executor(), () -> {
@@ -423,7 +429,7 @@ public class AccordInteropExecution implements ReadCoordinator
             // and can be extended similar to MessageType which allows additional types not from Accord to be added
             // This commit won't necessarily execute before the interop read repair message so there could be an insufficient which is fine
             for (Node.Id to : executeTopology.nodes())
-                    node.send(to, new Commit(Kind.StableFastPath, to, allTopologies, txnId, txn, route, Ballot.ZERO, executeAt, deps), null);
+                    node.send(to, new Commit(commitKind(), to, allTopologies, txnId, txn, route, Ballot.ZERO, executeAt, deps), null);
             repairUpdate.runBRR(AccordInteropExecution.this);
             return new TxnData();
         });

src/java/org/apache/cassandra/service/accord/interop/AccordInteropPersist.java

@@ -20,13 +20,13 @@ package org.apache.cassandra.service.accord.interop;
 
 import java.util.function.BiConsumer;
 
+import accord.api.ExclusiveAsyncExecutor;
 import accord.api.Result;
 import accord.coordinate.ExecuteFlag;
 import accord.coordinate.Persist;
 import accord.coordinate.tracking.AllTracker;
 import accord.coordinate.tracking.QuorumTracker;
 import accord.local.Node;
-import accord.local.SequentialAsyncExecutor;
 import accord.messages.Apply;
 import accord.primitives.Ballot;
 import accord.primitives.Deps;
@@ -53,7 +53,7 @@ import static org.apache.cassandra.db.ConsistencyLevel.SERIAL;
  */
 public class AccordInteropPersist extends Persist
 {
-    public AccordInteropPersist(Node node, SequentialAsyncExecutor executor, Topologies topologies, TxnId txnId, Route<?> sendTo, Ballot ballot, Txn txn, Timestamp executeAt, Deps deps, Writes writes, Result result, FullRoute<?> fullRoute, ConsistencyLevel consistencyLevel, ExecuteFlag.CoordinationFlags flags, boolean informDurableOnDone, Apply.Kind applyKind, BiConsumer<? super Result, Throwable> callback)
+    public AccordInteropPersist(Node node, ExclusiveAsyncExecutor executor, Topologies topologies, TxnId txnId, Route<?> sendTo, Ballot ballot, Txn txn, Timestamp executeAt, Deps deps, Writes writes, Result result, FullRoute<?> fullRoute, ConsistencyLevel consistencyLevel, ExecuteFlag.CoordinationFlags flags, boolean informDurableOnDone, Apply.Kind applyKind, BiConsumer<? super Result, Throwable> callback)
     {
         super(node, executor, topologies, txnId, ballot, sendTo, txn, executeAt, deps, writes, result.toPersistable(), fullRoute, flags, informDurableOnDone, AccordInteropApply.FACTORY, applyKind, consistencyLevel == ALL ? AllTracker::new : QuorumTracker::new, (ignore, fail) -> {
             if (fail != null) callback.accept(null, fail);

src/java/org/apache/cassandra/service/accord/journal/AbstractSegmentCompactor.java

@@ -93,6 +93,7 @@ public abstract class AbstractSegmentCompactor<V> implements SegmentCompactor<Jo
     // Only valid in the scope of a single `compact` call
     private JournalKey prevKey;
     private DecoratedKey prevDecoratedKey;
+    private volatile boolean stopped;
 
     @Override
     public Collection<StaticSegment<JournalKey, V>> compact(Collection<StaticSegment<JournalKey, V>> segments)
@@ -129,6 +130,9 @@ public abstract class AbstractSegmentCompactor<V> implements SegmentCompactor<Jo
             KeyOrderReader<JournalKey> reader;
             while ((reader = readers.poll()) != null)
             {
+                if (stopped)
+                    throw new Stopped();
+
                 if (key == null || !reader.key().equals(key))
                 {
                     maybeWritePartition(key, merger, serializer, firstDescriptor, firstOffset);
@@ -232,6 +236,12 @@ public abstract class AbstractSegmentCompactor<V> implements SegmentCompactor<Jo
         }
     }
 
+    @Override
+    public void stop()
+    {
+        stopped = true;
+    }
+
     private static int normalize(int cmp)
     {
         if (cmp == 0)

src/java/org/apache/cassandra/service/accord/journal/JournalRangeIndex.java

@@ -54,13 +54,13 @@ import accord.utils.UnhandledEnum;
 import accord.utils.btree.BTree;
 import accord.utils.btree.IntervalBTree;
 
-import org.apache.cassandra.service.accord.AccordCache;
-import org.apache.cassandra.service.accord.AccordCacheEntry;
 import org.apache.cassandra.service.accord.AccordCommandStore;
 import org.apache.cassandra.service.accord.AccordCommandStore.Caches;
 import org.apache.cassandra.service.accord.RangeIndex;
 import org.apache.cassandra.service.accord.TokenRange;
 import org.apache.cassandra.service.accord.api.TokenKey;
+import org.apache.cassandra.service.accord.execution.AccordCache;
+import org.apache.cassandra.service.accord.execution.AccordCacheEntry;
 
 import static accord.local.CommandSummaries.Relevance.IRRELEVANT;
 import static accord.local.LoadKeysFor.RECOVERY;
@@ -107,7 +107,7 @@ public class JournalRangeIndex extends SemiSyncIntervalTree<Object[]> implements
             }
 
             @Override
-            public void onUpdate(AccordCacheEntry<TxnId, Command> state)
+            public void onUpdate(AccordCacheEntry<TxnId, Command, ?> state)
             {
                 Summary summary = loader.ifRelevant(state);
                 if (summary != null)
@@ -162,7 +162,7 @@ public class JournalRangeIndex extends SemiSyncIntervalTree<Object[]> implements
                             if (isMaybeRelevant(i))
                             {
                                 TxnId txnId = i.txnId;
-                                AccordCacheEntry<TxnId, Command> entry = c.getUnsafe(txnId);
+                                AccordCacheEntry<TxnId, Command, ?> entry = c.getUnsafe(txnId);
                                 Invariants.expect(entry != null, "%s found interval %s but no matching transaction in cache", owner.commandStore, i);
                                 if (entry != null)
                                 {
@@ -199,7 +199,7 @@ public class JournalRangeIndex extends SemiSyncIntervalTree<Object[]> implements
         }
 
         @Override
-        protected void finish(Map<Timestamp, Summary> into)
+        public void finish(Map<Timestamp, Summary> into)
         {
         }
 
@@ -233,7 +233,7 @@ public class JournalRangeIndex extends SemiSyncIntervalTree<Object[]> implements
         }
 
         @Override
-        protected void cleanupExclusive(Caches caches)
+        public void cleanupExclusive(Caches caches)
         {
             if (commandWatcher != null)
             {
@@ -267,7 +267,7 @@ public class JournalRangeIndex extends SemiSyncIntervalTree<Object[]> implements
     }
 
     @Override
-    public void onUpdate(AccordCacheEntry<TxnId, Command> state)
+    public void onUpdate(AccordCacheEntry<TxnId, Command, ?> state)
     {
         TxnId txnId = state.key();
         if (txnId.is(Routable.Domain.Range))
@@ -334,7 +334,7 @@ public class JournalRangeIndex extends SemiSyncIntervalTree<Object[]> implements
     }
 
     @Override
-    public void onEvict(AccordCacheEntry<TxnId, Command> state)
+    public void onEvict(AccordCacheEntry<TxnId, Command, ?> state)
     {
         TxnId txnId = state.key();
         if (txnId.is(Routable.Domain.Range))

src/java/org/apache/cassandra/service/accord/serializers/CommandStoreSerializers.java

@@ -39,6 +39,7 @@ import accord.local.DurableBefore;
 import accord.local.MaxConflicts;
 import accord.local.MaxDecidedRX;
 import accord.local.RedundantBefore;
+import accord.local.RedundantStatus;
 import accord.primitives.Range;
 import accord.primitives.Ranges;
 import accord.primitives.SaveStatus;
@@ -272,11 +273,13 @@ public class CommandStoreSerializers
             }
         }
 
-        private short cast(long v)
+        private int cast(long v)
         {
-            if ((v & ~0xFFFF) != 0)
+            if (v < 0 || v >= (1 << RedundantStatus.Property.WAS_OWNED.ordinal()))
+                throw new IllegalStateException("Should not be serializing WAS_OWNED or NOT_OWNED statuses");
+            if ((v & ~0xFFFF) != 0) // retain this
                 throw new IllegalStateException("Cannot serialize RedundantStatus larger than 0xFFFF. Requires serialization changes.");
-            return (short)v;
+            return (int) v;
         }
 
         @Override
@@ -298,7 +301,7 @@ public class CommandStoreSerializers
                 bounds[i] = CommandSerializers.txnId.deserialize(in);
             int[] statuses = new int[count * 2];
             for (int i = 0 ; i < statuses.length ; ++i)
-                statuses[i] = in.readShort();
+                statuses[i] = in.readShort() & 0xFFFF;
 
             return new RedundantBefore.Bounds(range, startEpoch, endEpoch, bounds, statuses, staleUntilAtLeast);
         }

src/java/org/apache/cassandra/service/accord/serializers/FetchSerializers.java

@@ -25,7 +25,11 @@ import accord.impl.AbstractFetchCoordinator.FetchRequest;
 import accord.impl.AbstractFetchCoordinator.FetchResponse;
 import accord.messages.ReadData.CommitOrReadNack;
 import accord.messages.ReadData.ReadReply;
+import accord.primitives.PartialDeps;
+import accord.primitives.PartialTxn;
 import accord.primitives.Ranges;
+import accord.primitives.TxnId;
+import accord.utils.Invariants;
 import accord.utils.VIntCoding;
 
 import org.apache.cassandra.db.TypeSizes;
@@ -51,18 +55,20 @@ public class FetchSerializers
             out.writeUnsignedVInt(request.executeAtEpoch);
             CommandSerializers.txnId.serialize(request.txnId, out);
             KeySerializers.ranges.serialize((Ranges) request.scope, out);
-            DepsSerializers.partialDeps.serialize(request.partialDeps, out);
+            DepsSerializers.partialDeps.serialize(PartialDeps.NONE, out);
             StreamingTxn.serializer.serialize(request.read, out, version);
         }
 
         @Override
         public FetchRequest deserialize(DataInputPlus in, Version version) throws IOException
         {
-            return new AccordFetchRequest(in.readUnsignedVInt(),
+            long epoch = in.readUnsignedVInt();
-                                          CommandSerializers.txnId.deserialize(in),
+            TxnId txnId = CommandSerializers.txnId.deserialize(in);
-                                          KeySerializers.ranges.deserialize(in),
+            Ranges ranges = KeySerializers.ranges.deserialize(in);
-                                          DepsSerializers.partialDeps.deserialize(in),
+            PartialDeps deps = DepsSerializers.partialDeps.deserialize(in);
-                                          StreamingTxn.serializer.deserialize(in, version));
+            Invariants.require(deps.isEmpty());
+            PartialTxn txn = StreamingTxn.serializer.deserialize(in, version);
+            return new AccordFetchRequest(epoch, txnId, ranges, txn);
         }
 
         @Override
@@ -71,7 +77,7 @@ public class FetchSerializers
             return TypeSizes.sizeofUnsignedVInt(request.executeAtEpoch)
                    + CommandSerializers.txnId.serializedSize(request.txnId)
                    + KeySerializers.ranges.serializedSize((Ranges) request.scope)
-                   + DepsSerializers.partialDeps.serializedSize(request.partialDeps)
+                   + DepsSerializers.partialDeps.serializedSize(PartialDeps.NONE)
                    + StreamingTxn.serializer.serializedSize(request.read, version);
         }
     };

src/java/org/apache/cassandra/service/accord/txn/TxnNamedRead.java

@@ -58,10 +58,10 @@ import org.apache.cassandra.io.util.DataInputPlus;
 import org.apache.cassandra.io.util.DataOutputBuffer;
 import org.apache.cassandra.io.util.DataOutputPlus;
 import org.apache.cassandra.schema.TableId;
-import org.apache.cassandra.service.accord.AccordExecutor;
 import org.apache.cassandra.service.accord.TokenRange;
 import org.apache.cassandra.service.accord.api.PartitionKey;
 import org.apache.cassandra.service.accord.api.TokenKey;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
 import org.apache.cassandra.service.accord.serializers.TableMetadatas;
 import org.apache.cassandra.service.accord.serializers.TableMetadatasAndKeys;
 import org.apache.cassandra.service.accord.serializers.Version;

src/java/org/apache/cassandra/service/accord/txn/TxnRead.java

@@ -55,8 +55,8 @@ import org.apache.cassandra.io.ParameterisedVersionedSerializer;
 import org.apache.cassandra.io.util.DataInputPlus;
 import org.apache.cassandra.io.util.DataOutputPlus;
 import org.apache.cassandra.service.accord.AccordCommandStore;
-import org.apache.cassandra.service.accord.AccordExecutor;
 import org.apache.cassandra.service.accord.api.PartitionKey;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
 import org.apache.cassandra.service.accord.serializers.TableMetadatas;
 import org.apache.cassandra.service.accord.serializers.TableMetadatasAndKeys;
 import org.apache.cassandra.service.accord.serializers.Version;

src/java/org/apache/cassandra/service/accord/txn/TxnWrite.java

@@ -64,8 +64,8 @@ import org.apache.cassandra.io.util.DataOutputBuffer;
 import org.apache.cassandra.io.util.DataOutputPlus;
 import org.apache.cassandra.schema.ColumnMetadata;
 import org.apache.cassandra.service.accord.AccordCommandStore;
-import org.apache.cassandra.service.accord.AccordExecutor;
 import org.apache.cassandra.service.accord.api.PartitionKey;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
 import org.apache.cassandra.service.accord.serializers.TableMetadatas;
 import org.apache.cassandra.service.accord.serializers.TableMetadatasAndKeys;
 import org.apache.cassandra.service.accord.serializers.Version;

src/java/org/apache/cassandra/tcm/sequences/BootstrapAndJoin.java

@@ -74,6 +74,7 @@ import org.apache.cassandra.utils.concurrent.Future;
 import org.apache.cassandra.utils.concurrent.FutureCombiner;
 import org.apache.cassandra.utils.vint.VIntCoding;
 
+import static accord.local.BootstrapReason.GAIN_OWNERSHIP;
 import static com.google.common.collect.ImmutableList.of;
 import static java.util.concurrent.TimeUnit.MILLISECONDS;
 import static org.apache.cassandra.tcm.MultiStepOperation.Kind.JOIN;
@@ -396,7 +397,7 @@ public class BootstrapAndJoin extends MultiStepOperation<Epoch>
 
             Node node = service.node();
             node.commandStores().forAllUnsafe(commandStore -> {
-                AsyncResult<EpochReady> ready = commandStore.resumeBootstrap(node);
+                AsyncResult<EpochReady> ready = commandStore.resumeBootstrap(node, GAIN_OWNERSHIP);
                 bootstraps.add(AccordService.toFuture(ready.flatMap(e -> e.reads)));
             });
         }

src/java/org/apache/cassandra/utils/NoSpamLogger.java

@@ -17,6 +17,11 @@
  */
 package org.apache.cassandra.utils;
 
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicLong;
 import java.util.function.Supplier;
@@ -64,16 +69,18 @@ public class NoSpamLogger
         CLOCK = clock;
     }
 
-    public class NoSpamLogStatement extends AtomicLong
+    public static class NoSpamLogStatement extends AtomicLong
     {
         private static final long serialVersionUID = 1L;
 
+        private final Logger wrapped;
         private final String statement;
         private final long minIntervalNanos;
 
-        public NoSpamLogStatement(String statement, long minIntervalNanos)
+        public NoSpamLogStatement(Logger wrapped, String statement, long minIntervalNanos)
         {
             super(Long.MIN_VALUE);
+            this.wrapped = wrapped;
             this.statement = statement;
             this.minIntervalNanos = minIntervalNanos;
         }
@@ -159,6 +166,154 @@ public class NoSpamLogger
         }
     }
 
+    public static class NoDuplicateSpamLogStatement
+    {
+        private static final long serialVersionUID = 1L;
+        private static final int PRUNE_SIZE = 32;
+
+        private final Logger wrapped;
+        private final String statement;
+        private final long minIntervalNanos;
+        private final ConcurrentHashMap<Long, Long> lastLogged = new ConcurrentHashMap<>();
+        private final AtomicLong nextPruneAt = new AtomicLong();
+
+        public NoDuplicateSpamLogStatement(Logger wrapped, String statement, long minInterval, TimeUnit units)
+        {
+            this(wrapped, statement, units.toNanos(minInterval));
+        }
+
+        public NoDuplicateSpamLogStatement(Logger wrapped, String statement, long minIntervalNanos)
+        {
+            this.wrapped = wrapped;
+            this.statement = statement;
+            this.minIntervalNanos = minIntervalNanos;
+        }
+
+        private boolean shouldLog(long id, long nowNanos)
+        {
+            Long expected = lastLogged.getOrDefault(id, Long.MIN_VALUE);
+            if (nowNanos < expected || !lastLogged.replace(id, expected, nowNanos + minIntervalNanos))
+                return false;
+
+            if (lastLogged.size() >= PRUNE_SIZE)
+            {
+                long pruneAt = nextPruneAt.get();
+                if (nowNanos >= pruneAt && nextPruneAt.compareAndSet(pruneAt, nowNanos + minIntervalNanos))
+                {
+                    for (Map.Entry<Long, Long> e : lastLogged.entrySet())
+                    {
+                        if (nowNanos < e.getValue())
+                            lastLogged.remove(e.getKey(), e.getValue());
+                    }
+                }
+            }
+            return true;
+        }
+
+        public boolean log(Level l, long id, long nowNanos, Object... objects)
+        {
+            if (!shouldLog(id, nowNanos)) return false;
+            return logNoCheck(l, objects);
+        }
+
+        private boolean logNoCheck(Level l, Object... objects)
+        {
+            switch (l)
+            {
+                case DEBUG:
+                    wrapped.debug(statement, objects);
+                    break;
+                case INFO:
+                    wrapped.info(statement, objects);
+                    break;
+                case WARN:
+                    wrapped.warn(statement, objects);
+                    break;
+                case ERROR:
+                    wrapped.error(statement, objects);
+                    break;
+                default:
+                    throw new AssertionError();
+            }
+            return true;
+        }
+
+        public boolean debug(long id, long nowNanos, Object... objects)
+        {
+            return log(Level.DEBUG, id, nowNanos, objects);
+        }
+
+        public boolean debug(long id, Object... objects)
+        {
+            return debug(id, CLOCK.nanoTime(), objects);
+        }
+
+        public boolean info(long id, long nowNanos, Object... objects)
+        {
+            return log(Level.INFO, id, nowNanos, objects);
+        }
+
+        public boolean info(long id, Object... objects)
+        {
+            return info(id, CLOCK.nanoTime(), objects);
+        }
+
+        public boolean warn(long id, long nowNanos, Object... objects)
+        {
+            return log(Level.WARN, id, nowNanos, objects);
+        }
+
+        public boolean warn(long id, Object... objects)
+        {
+            return warn(id, CLOCK.nanoTime(), objects);
+        }
+
+        public boolean error(long id, long nowNanos, Object... objects)
+        {
+            return log(Level.ERROR, id, nowNanos, objects);
+        }
+
+        public boolean error(long id, Object... objects)
+        {
+            return error(id, CLOCK.nanoTime(), objects);
+        }
+
+        public static long exceptionId(Throwable throwable)
+        {
+            return exceptionId(throwable, Collections.newSetFromMap(new IdentityHashMap<>()));
+        }
+
+        private static long exceptionId(Throwable throwable, Set<Throwable> visited)
+        {
+            long id = throwable.getClass().hashCode();
+            for (StackTraceElement ste : throwable.getStackTrace())
+            {
+                id *= 31;
+                id += ste.getClassName().hashCode();
+                id *= 31;
+                id += ste.getLineNumber();
+            }
+
+            for (Throwable suppressed : throwable.getSuppressed())
+            {
+                if (!visited.add(suppressed))
+                    continue;
+
+                id *= 31;
+                id += exceptionId(suppressed, visited);
+            }
+            for (Throwable cause = throwable.getCause() ; cause != null ; cause = cause.getCause())
+            {
+                if (!visited.add(cause))
+                    break;
+
+                id *= 31;
+                id += exceptionId(cause, visited);
+            }
+            return id;
+        }
+    }
+
     private static final NonBlockingHashMap<Logger, NoSpamLogger> wrappedLoggers = new NonBlockingHashMap<>();
 
     @VisibleForTesting
@@ -305,7 +460,7 @@ public class NoSpamLogger
         NoSpamLogStatement statement = lastMessage.get(key);
         if (statement == null)
         {
-            statement = new NoSpamLogStatement(s, minIntervalNanos);
+            statement = new NoSpamLogStatement(wrapped, s, minIntervalNanos);
             NoSpamLogStatement temp = lastMessage.putIfAbsent(key, statement);
             if (temp != null)
                 statement = temp;

src/java/org/apache/cassandra/utils/concurrent/SignalLock.java

@@ -428,7 +428,7 @@ public final class SignalLock implements Lock
         {
             long cur = state;
             int count = asyncSignalCount(cur);
-            if (count == MAX_THREADS)
+            if (count == MAX_SIGNAL_COUNT)
                 return false;
 
             if (signalIfWaiting)

test/burn/org/apache/cassandra/service/accord/AccordExecutorBurnTest.java

@@ -24,6 +24,7 @@ import java.util.function.Function;
 import org.junit.Test;
 
 import org.apache.cassandra.concurrent.ExecutorPlus;
+import org.apache.cassandra.service.accord.execution.AccordExecutor;
 import org.apache.cassandra.utils.concurrent.Semaphore;
 import org.apache.cassandra.utils.concurrent.UncheckedInterruptedException;
 

test/conf/logback-info.xml

@@ -0,0 +1,56 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<configuration debug="false" scan="true" scanPeriod="60 seconds">
+  <define name="cluster_id" class="org.apache.cassandra.distributed.impl.ClusterIDDefiner" />
+  <define name="instance_id" class="org.apache.cassandra.distributed.impl.InstanceIDDefiner" />
+
+  <!-- Shutdown hook ensures that async appender flushes -->
+  <shutdownHook class="ch.qos.logback.core.hook.DefaultShutdownHook"/>
+
+  <appender name="INSTANCEFILE" class="ch.qos.logback.core.FileAppender">
+    <file>./build/test/logs/${cassandra.testtag}/${suitename}/${cluster_id}/${instance_id}/system.log</file>
+    <encoder>
+      <pattern>%-5level [%thread] ${instance_id} %date{"yyyy-MM-dd'T'HH:mm:ss,SSS", UTC} %F:%L - %msg%n</pattern>
+    </encoder>
+    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
+      <level>INFO</level>
+    </filter>
+    <immediateFlush>true</immediateFlush>
+  </appender>
+
+  <appender name="INSTANCESTDERR" target="System.err" class="ch.qos.logback.core.ConsoleAppender">
+    <encoder>
+      <pattern>%-5level %date{"HH:mm:ss,SSS"} %msg%n</pattern>
+    </encoder>
+    <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
+      <level>INFO</level>
+    </filter>
+  </appender>
+
+  <!-- Un-comment to enable TRACE logging for Accord transactions.
+  <logger name="accord" level="TRACE" />
+  <logger name="org.apache.cassandra.service.accord" level="TRACE" />
+  -->
+
+  <root level="INFO">
+    <appender-ref ref="INSTANCEFILE" /> <!-- use blocking to avoid race conditions with appending and searching -->
+    <appender-ref ref="INSTANCESTDERR" />
+  </root>
+</configuration>

test/distributed/org/apache/cassandra/distributed/test/TestBaseImpl.java

@@ -83,7 +83,7 @@ import org.apache.cassandra.distributed.shared.DistributedTestBase;
 import org.apache.cassandra.io.util.DataInputBuffer;
 import org.apache.cassandra.net.Message;
 import org.apache.cassandra.net.Verb;
-import org.apache.cassandra.service.accord.AccordCache;
+import org.apache.cassandra.service.accord.execution.AccordCache;
 
 import static java.lang.System.currentTimeMillis;
 import static java.util.concurrent.TimeUnit.MILLISECONDS;

test/distributed/org/apache/cassandra/distributed/test/accord/AccordBootstrapTest.java

@@ -46,7 +46,7 @@ import org.apache.cassandra.distributed.api.TokenSupplier;
 import org.apache.cassandra.distributed.shared.NetworkTopology;
 import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.service.accord.AccordCommandStore;
-import org.apache.cassandra.service.accord.AccordSafeCommandStore;
+import org.apache.cassandra.service.accord.execution.SaferCommandStore;
 import org.apache.cassandra.service.accord.api.PartitionKey;
 import org.apache.cassandra.streaming.StreamEvent;
 import org.apache.cassandra.streaming.StreamEventHandler;
@@ -262,7 +262,7 @@ public class AccordBootstrapTest extends AccordBootstrapTestBase
                         getBlocking(service().node().commandStores().forEach("Test", RoutingKeys.of(partitionKey.toUnseekable()), Long.MIN_VALUE, Long.MAX_VALUE, safeStore -> {
                             if (safeStore.ranges().currentRanges().contains(partitionKey))
                             {
-                                AccordSafeCommandStore ss = (AccordSafeCommandStore) safeStore;
+                                SaferCommandStore ss = (SaferCommandStore) safeStore;
                                 Assert.assertFalse(ss.bootstrapBeganAt().isEmpty());
                                 Assert.assertFalse(ss.safeToReadAt().isEmpty());
 

test/distributed/org/apache/cassandra/distributed/test/accord/AccordCQLTestBase.java

@@ -81,6 +81,7 @@ import org.apache.cassandra.distributed.api.ICoordinator;
 import org.apache.cassandra.distributed.api.QueryResults;
 import org.apache.cassandra.distributed.api.SimpleQueryResult;
 import org.apache.cassandra.distributed.shared.AssertUtils;
+import org.apache.cassandra.distributed.shared.FutureUtils;
 import org.apache.cassandra.distributed.test.sai.SAIUtil;
 import org.apache.cassandra.distributed.util.QueryResultUtil;
 import org.apache.cassandra.exceptions.InvalidRequestException;
@@ -93,6 +94,7 @@ import org.apache.cassandra.service.consensus.TransactionalMode;
 import org.apache.cassandra.service.consensus.migration.TransactionalMigrationFromMode;
 import org.apache.cassandra.utils.AssertionUtils;
 import org.apache.cassandra.utils.ByteBufferUtil;
+import org.apache.cassandra.utils.Clock;
 import org.apache.cassandra.utils.FailingConsumer;
 import org.apache.cassandra.utils.Pair;
 
@@ -116,8 +118,10 @@ import static org.junit.Assert.fail;
 public abstract class AccordCQLTestBase extends AccordTestBase
 {
     private static final Logger logger = LoggerFactory.getLogger(AccordCQLTestBase.class);
+    private static final int CAS_SIMULATOR_LITE_CONCURRENCY = 20;
 
-    protected AccordCQLTestBase(TransactionalMode transactionalMode) {
+    protected AccordCQLTestBase(TransactionalMode transactionalMode)
+    {
         super(transactionalMode);
     }
 
@@ -131,7 +135,9 @@ public abstract class AccordCQLTestBase extends AccordTestBase
     public static void setupClass() throws IOException
     {
         AccordTestBase.setupCluster(builder -> builder.appendConfig(config -> config.with(GOSSIP, NETWORK, NATIVE_PROTOCOL)
-                                                                                    .set("paxos_variant", PaxosVariant.v2.name())), 2);
+                                                                                    .set("paxos_variant", PaxosVariant.v2.name())
+                                                                                    .set("accord.migration_concurrency", "" + (1 + CAS_SIMULATOR_LITE_CONCURRENCY))
+        ), 2);
         SHARED_CLUSTER.schemaChange("CREATE TYPE " + KEYSPACE + ".person (height int, age int)");
     }
 
@@ -3332,15 +3338,18 @@ public abstract class AccordCQLTestBase extends AccordTestBase
                  coordinator.execute("INSERT INTO " + qualifiedAccordTableName + " (pk, count, seq1, seq2) VALUES (1, 0, '', []) USING TIMESTAMP 0", ConsistencyLevel.ALL);
 
                  ListType<Integer> LIST_TYPE = ListType.getInstance(Int32Type.instance, true);
-                 ExecutorService es = Executors.newCachedThreadPool();
+                 Future<Object[][]>[] futures = new Future[CAS_SIMULATOR_LITE_CONCURRENCY];
-                 List<Future<Object[][]>> futures = new ArrayList<>();
+                 long[] starts = new long[CAS_SIMULATOR_LITE_CONCURRENCY];
-                 for (int ii = 0; ii < 10; ii++)
+                 for (int id = 0; id < CAS_SIMULATOR_LITE_CONCURRENCY; id++)
                  {
-                     int id = ii;
+                     starts[id] = Clock.Global.nanoTime();
-                     futures.add(es.submit(() -> coordinator.execute("UPDATE " + qualifiedAccordTableName + " SET count = count + 1, seq1 = seq1 + ?, seq2 = seq2 + ? WHERE pk = ? IF EXISTS", ConsistencyLevel.ALL, id + ",", ByteBufferUtil.getArray(LIST_TYPE.decompose(singletonList(id))), 1)));
+                     futures[id] = FutureUtils.map(coordinator.asyncExecuteWithResult("UPDATE " + qualifiedAccordTableName + " SET count = count + 1, seq1 = seq1 + ?, seq2 = seq2 + ? WHERE pk = ? IF EXISTS", ConsistencyLevel.ALL, id + ",", ByteBufferUtil.getArray(LIST_TYPE.decompose(singletonList(id))), 1), SimpleQueryResult::toObjectArrays);
+                 }
+                 for (int id = 0; id < CAS_SIMULATOR_LITE_CONCURRENCY; id++)
+                 {
+                     futures[id].get();
+                     System.out.println(String.format("Completed in %.3fs", (Clock.Global.nanoTime() - starts[id]) * 0.000000001));
                  }
-                 for (Future f : futures)
-                     f.get();
 
                  Object[][] result = coordinator.execute("SELECT pk, count, seq1, seq2 FROM  " + qualifiedAccordTableName + " WHERE pk = 1", ConsistencyLevel.SERIAL);
 

test/distributed/org/apache/cassandra/distributed/test/accord/AccordDropTableBase.java

@@ -22,12 +22,11 @@ import com.google.common.base.Throwables;
 
 import org.assertj.core.api.Assertions;
 
-import accord.api.RoutingKey;
 import accord.local.CommandStores;
-import accord.local.LoadKeys;
+import accord.local.ExecutionContext;
 import accord.local.Node;
-import accord.local.PreLoadContext;
 import accord.local.cfk.CommandsForKey;
+import accord.local.cfk.SafeCommandsForKey;
 import accord.primitives.Ranges;
 import accord.primitives.Routable;
 import accord.primitives.Txn;
@@ -42,12 +41,10 @@ import org.apache.cassandra.distributed.test.TestBaseImpl;
 import org.apache.cassandra.net.Verb;
 import org.apache.cassandra.schema.TableId;
 import org.apache.cassandra.service.accord.AccordCommandStore;
-import org.apache.cassandra.service.accord.AccordSafeCommandStore;
+import org.apache.cassandra.service.accord.execution.SaferCommandStore;
-import org.apache.cassandra.service.accord.AccordSafeCommandsForKey;
 import org.apache.cassandra.service.accord.AccordService;
 import org.apache.cassandra.service.accord.TokenRange;
 
-import static accord.local.LoadKeysFor.READ_WRITE;
 import static org.apache.cassandra.config.DatabaseDescriptor.getPartitioner;
 import static org.apache.cassandra.service.accord.AccordService.getBlocking;
 
@@ -135,19 +132,16 @@ public class AccordDropTableBase extends TestBaseImpl
                 TableId tableId = TableId.fromString(s);
                 AccordService accord = (AccordService) AccordService.instance();
                 TxnId syntheticTxnId = new TxnId(TxnId.MAX_EPOCH, 0, Txn.Kind.ExclusiveSyncPoint, Routable.Domain.Range, new Node.Id(1));
-                PreLoadContext ctx = PreLoadContext.contextFor(syntheticTxnId, Ranges.single(TokenRange.fullRange(tableId, getPartitioner())), LoadKeys.SYNC, READ_WRITE, "Test");
+                ExecutionContext ctx = ExecutionContext.unsequencedReadWrite(syntheticTxnId, Ranges.single(TokenRange.fullRange(tableId, getPartitioner())), "Test");
                 CommandStores stores = accord.node().commandStores();
                 for (int storeId : stores.ids())
                 {
                     AccordCommandStore store = (AccordCommandStore) stores.forId(storeId);
                     getBlocking(store.chain(ctx, input -> {
-                        AccordSafeCommandStore safe = (AccordSafeCommandStore) input;
+                        SaferCommandStore safe = (SaferCommandStore) input;
-                        for (RoutingKey key : safe.commandsForKeysKeys())
+                        for (SafeCommandsForKey safeCfk : safe.safeCommandsForKeys())
                         {
-                            AccordSafeCommandsForKey safeCFK = (AccordSafeCommandsForKey) safe.ifLoadedAndInitialised(key);
+                            CommandsForKey cfk = safeCfk.current();
-                            if (safeCFK == null) // we read and found a key, but its null at load time... so ignore it
-                                continue;
-                            CommandsForKey cfk = safeCFK.current();
                             CommandsForKey.TxnInfo minUndecided = cfk.minUndecidedManaged();
                             if (minUndecided != null)
                                 throw new AssertionError("Undecided txn: " + minUndecided);