- consumer and producer threads wait signal without acquiring the lock first
- lock owners may prepare more work than they need, moving some dynamically-adjusted portion of the work from the prioritised lock-managed structures onto a non-blocking queue, so that other threads may consume work from there; the portion is continually micro-adjusted to target some available work whenever the lock is acquired.
- adopts/supports some features of the SEPExecutor:
- threads may auto-adjust the number of running threads based on how much time is collectively spent waiting, to minimise time spent signalling
- consumers may (timed-sleep) poll rather than await a signal, reducing the number of kernel interactions needed; relying on the auto-adjustment to bound the time the scheduler spends waking threads with no work to do
Also Fix:
- Client Result should not be persisted or included in Command object at any time
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for CASSANDRA-21375
- TransactionStatement token aware routing
- Mitigate convoy effect:
- ExecuteTxnBacklog v1 can execute single key transaction backlogs and disseminate the result
- Direct local execution possible for single key transactions
- Use shardAppliedBefore instead of gcBefore to cleanup CFK faster
Also Improve:
- Introduce distributed tracing
- Combine RejectBefore and MaxConflicts
- Self-addressed messages are delivered directly
- Disable AccordSyncPropagator by default, as incompatible with large clusters
- FastPathStrategy.SIMPLE should not modify fast path contents; FastPathStrategy.UP introduced to adopt this behaviour
- Configurable queue prioritisation
- ProtocolModifiers mostly final
- Remove unnecessary condition check when serializing TxnUpdate
Also Fix:
- SequentialExecutor ownership bug
- BTree.XUpdater.reset()
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for CASSANDRA-21361
- Clean Shutdown/Restart
- Rebootstrap to allow nodes to rejoin consensus if they are out of sync, or did not shutdown cleanly
Improve:
- Improve efficiency of BTreeReducingRangeMap
- DurableBefore backed by BTreeReducingRangeMap
- Soft reject new transactions when a replica has a backlog of work
- system_accord_debug.command_store_ops supports starting journal replay
Fix:
- Ensure SequentialExecutor.owner is unset only by the owner
- Detect and avoid taking two AccordExecutor locks simultaneously
- MaxConflicts serializer
- tryToExecuteListening after replay, to handle invalidated dependencies
- TxnNamedRead does not close a RowIterator, leading to native memory leaks
- GetLatestDepsNack serialization
- journal.readLast() did not read last
- DefaultRemoteListeners not correctly synchronised
- RangeTxnScanner cancellation assumes was already started
- Start cfk compaction before replay
- txn_graph should avoid visiting parents twice (possible if two different dependency chains connecting them in the graph)
- Topology.cloneEquivalentWithEpoch should also share nodeLookup and ranges, especially to accelerate computeWaitForEpoch/computeScope
- Do not invoke slowReplicaDelay or slowCoordinatorDelay during restore
- Do not fail if slowReplicaDelay or slowCoordinatorDelay are invoked without knowing the transaction
patch by Benedict; reviewed by Alex Petrov and Ariel Weisberg for CASSANDRA-21355
Adds a compaction implementation utilizing new fixed allocation SSTable reader/writer implementations, and other purpose built code, leading to improved efficiencies.
patch by Nitsan Wakart; reviewed by Branimir Lambov, Dmitry Konstantinov for CASSANDRA-20918
Also Fix Cassandra:
- In memory size calculation for CommandsForKey include Unmanaged
- Accord load out-of-band cleanup should use SafeRedundantBefore
ALso Improve Cassandra:
- Report replay information on begin replay
- Improve AccordService shutdown
- Log command store RedundantBefore on shutdown
- Segment compaction should wait for readOrder barrier to replace segments, for additional safety
- Journal segments should share readOrder with sstables
Also Improve Accord:
- Iterate LocalListeners in order, so can query more effectively on node
- Refine AbstractReplay.minReplay/shouldReplay
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21804
Also Fix:
- Restore MaxDecidedRX on replay
- When catchup_on_start_exit_on_failure == false, should startup on any kind of failure, not only timeout
- lazy vtable LIMIT clause regression
- DurabilityService.onEpochRetired
- Command.validate when uniqueHlc differs
- Avoid unsafe publication of AccordExecutor to scheduledFastTasks
- AccordCache hitRate metric names
- use long for return type of DurationSpec.toNanoseconds
- Repair without all replicas should not request all Accord replicas participate
- ExecuteAtSerializer
- SyncPoints should be coordinated in an epoch that contains the ranges
Also Improve:
- Split Accord startup into local+distributed, ensure we
- Add logging to FetchDurableBefore on startup
- Add randomised testing of AbstractLazyVirtualTable
- Add validation of lazy virtual table key ordering
- Don't send requests to faulty replicas
- shrinkOrEvict large objects without holding lock
- Accord dtest shutdown
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-21042
Merge ConfigurationService with TopologyManager to remove cyclic dependency and duplicated work.
Also:
- Improve visibility of work blocking topology processing
- Ensure we cannot double-count peers when deciding an epoch's distributed readiness
- Remove the possibility of distributed stalls, by processing new topologies as soon as a
contiguous sequence is known locally, regardless of whether anyprior local epoch is ready to
coordinate or has recorded this fact to peers. The notification of local readiness continues
to be processed only once all prior epochs are ready, but we can begin using and readying
later epochs immediately.
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20998
Replace io.airlift:airline dependency with info.picocli:picocli across
all nodetool command implementations. This migration includes:
- Convert 160+ nodetool command classes from airline to picocli
- Add AbstractCommand base class for commands and utility methods
- Add NodetoolCommand top-level class for the cli utility
- Update all command classes to use @Command, @Option, and @Parameters
- Add plain text files to test command help output
- Add mock test classes for improved test coverage and parse validation
- Modify test infrastructure to work with picocli-based commands
- Add new layouts for cli formatting to preserve backwards compatibility
The migration maintains backward compatibility while providing improved
command-line parsing, better help system, and more robust argument
validation through picocli enhanced features.
patch by Maxim Muzafarov; reviewed by Caleb Rackliffe, Dmitry Konstantinov, Stefan Miklosovic for CASSANDRA-17445
- Don't assume stillExecutes applies to remote request - must mark unavailable any keys we don't have the txn definition for
- Don't exit notifyManagedPreBootstrap notify loop early, as could have later transactions with lower txnId
- CoordinateEphemeralRead must retry if insufficient responses from replicas that still own the range
- Burn test terminates while non-recurring tasks pending on command store queues
- Infinite recovery due to not sending InformDurable when partially truncated
- Incorrect participants when invoking removeRedundantDependencies
- Infinite bootstrap loop on retired topology
Improve
- Do not perform linear filters of CommandStores or Topologies
- Some default implementations of forEach/iterator
- TopologyRetiredException messages
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20707
* avoid running python3-dependent tasks when running simulator tasks
* fix a problem with simulated snitch rebase
* add a distinction between short-lived daemon threads and infinite loop ones for cases when we need to simulate user-implemented infinite loops
Patch by Alex Petrov; reviewed by Benedict Elliott Smith for CASSANDRA-20542
- Decouple command serialization from TableMetadata version; introduce ColumnMetadata ids; gracefully handle missing TableId
- DataInputPlus.readLeastSignificantBytes must truncate high bits
- Fix RandomPartitioner accord serialization
- Fast path stable commits must not override recovery propose/commit decisions regarding visibility of a transaction
- RejectBefore must mergeMax, not merge, to ensure we maintain epoch and hlc increasing independently
- Bad commitInvalidate decision
- consistent filtering for touches and stillTouches
- ensure TRUNCATE_BEFORE implies SHARD_APPLIED
- TopologyManager.unsyncedOnly off-by-one error
- DurabilityQueue should not retry SyncPointErased
- handle rare case of no deps but none needed
- not updating CFK synchronously on recovery, which can lead to erroneous recovery decisions for other transactions
- Don't return partial read response when one commandStore rejects the commit
- Filter touches/stillTouches consistently
- WaitingState computeLowEpoch must use hasTouched to handle historic key with no route
Improve:
- Use format parameters to defer building Invariants.requireArgument string
- streamline RedundantStatus/RedundantBefore
- Fix waiting state callback computes different route to initiator
- Invariants.checkX -> Invariants.requireX (to allow complementary Invariants.expectX as appropriate)
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20228
Follow-up to CASSANDRA-20228:
- Fix AccordUpdateParameters to correctly supply List cell paths derived from applyAt
- Fix ExecuteAtSerialize.serialiseNullable
- Fix topologies flush regression caused by markRetired forcing a flush on every call
patch by Benedict; reviewed by Ariel Weisberg for CASSANDRA-20228
- Privileged coordinator. If the coordinator is a replica we can reduce our quorum sizes by including the coordinator's vote.
- with deps: if we include coordinator's preaccept deps we can reliably reduce quorum size by 1, at the expense of recovery sometimes requiring additional phases and waiting for future txns
- with only vote: if we only include the vote we can avoid any additional recovery phases or waiting for future txns, but can reduce our quorum size for only some configurations
- Medium path. If t=t0 at a simple majority we can take just two rounds.
- with additional phase: on recovery, earlier txns must wait for medium path to be disabled (or committed) so we do not accidentally recover a transaction that won't be witnessed
- with unstable deps: on slow path commit, deps not found in accept as committed as unstable and do not affect recovery decisions for earlier transactions
Also improve:
- Recovery of await conditions simply recalculates the rejects flag rather than restarting to ensure faster forward progress
- refactor Command hierarchy
- tweak: don't save Writes for non-write transactions
Also fix:
- isOutOfRange when invoked from callback for some txn < Committed
- handle loadingPruned that is pre-bootstrap on update to bootstrappedAt
- journal replay can overwrite in memory state loaded for a command not yet replayed
- subtle ordering bug in LatestDeps.mergeProposal
- fix occasional class initialisation order bug
- handle race condition on learning of topology
patch by Benedict; reviewed by Aleksey Yeschenko for CASSANDRA-20222
Improve:
- Remove GetMaxConflict; use local MaxConflict collection
Fix:
- Invalidated should retain StoreParticipants so we can update CFK on journal replay
- loading pruned uninitialised commands via CFK: make sure hasTouched contains key so that if invalidated we are notified
- updateExecuteAtLeast should always be higher than TxnId
- Async CFK callbacks treated pruned transactions incorrectly
- node.withEpoch when ExecuteEphemeralRead in futureEpoch
- Deps.without should be key/range aware
- Durably mark bootstrapBeganAt and safeToReadAt in MaxConflicts
- Don't attempt to calculate local deps when DepsErased in GetLatestDeps (command has durably applied to this shard)
- filter StoreParticipants before invoking shouldCleanup
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20183
Includes multiple changes, primary ones:
* Make HarrySimulatorTest work with Accord
* Removed nodes now live in TCM, no need to discover historic epochs in order to find removed nodes
* CommandStore <-> RangesForEpochs mappings required for startup are now stored in journal, and CS can be set up _without_ topology replay
* Topology replay is fully done via journal (where we store topologies themselves), and topology metadata table (where we store redundant/closed information)
* Fixed various bugs related to propagation and staleness
* TCM was previously relied on for "fetching" epoch: we can not rely on it as there's no guarantee we will see a consecutive epoch when grabbing Metadata#current
* Redundant / closed during replay was set with incorrect ranges in 1 of the code paths
* TCM was contacted multiple times for historical epochs, which made startup much longer under some circumstances
Patch by Alex Petrov; reviewed by Benedict Elliott Smith for CASSANDRA-20142
- Fix notifying unmanaged after update redundant before/bootstrap
- Do not infer invalid if we have a single round of replies with minKnown not decided and maxKnown erased - in this case store the knowledge for next request.
- Fix SyncPoint topology selection
- Fix CheckStatusOkFull.with(InvalidIf)
- Fix NotifyWaitingOn
- ExecuteTxn should only contact latest topology for follow-up requests
- DurableBefore.min should not go backwards on new epoch topology, journal replay was not correctly handling PreApplied, partialTxn can be null if not owned
- Fix notify pre-bootstrap that arrives post-bootstrap
- Avoid GC race condition on Propagate where we can incorrectly infer a shard is stale
- Ensure redundantBefore on previously-owned range does not imply redundant before for overlapping queries on still-owned range
- Ensure we don't mark stale unless all of the quorum we contacted had erased, else we may have raced with the agreement and erase
- Fix Invalidate when no route found for FetchData does not report to all requested local epochs
- Fix WAS_OWNED_RETIRED without durableBefore at Universal can lead to assertions with RX that we permit to execute but that have not yet
- Fix initialiseWaitingOn can in some cases transitively notify the command we're updating via maybeCleanup of dependencies, but the command isn't yet updated so isn't ready
- Fix encountering a command that is pre-bootstrap, and for which we have locally 'applied' a supserseding RX, so that we do not know its outcome locally (so we do not cleanup the command), but also it must have been decided - and we should not respond with future dependencies.
- Epoch failures on CoordinatePreAccept should trigger the CoordinatePreAccept failure handler
- Use the shard bound rather than GC bound for fallback dependency
- LatestDeps should be sliced to actual route, so as not to use both PreAccepted AND Stable deps as though Stable
- Fix various callback issues with node.withEpoch and Recover/Propose.isDone
- RecoverWithRoute can encounter a partially truncated transaction where the Deps for one shard are not committed. Must fetch LatestDeps.
- Tighten LatestDeps semantics for Recover
- CommandsForKey: do not restore pruned as APPLIED
- Ensure prune points execute in the epoch in which they are declared
- must merge all fast path votes including those from earlier epochs that may have witnessed a later transaction
- Recoveries that know the transaction is committed a priori should skip the Accept phase
- Maintain GC behaviour for redundant commands that are pre-bootstrap
- don't apply ERASE to CommandsForKey to avoid breaking pruning
- Introduce clearBefore to ProgressLog to more consistently handle cleaning up redundant transactions (and avoid triggering burn test invariants)
- don't replay journal of a bootstrapping node in burn test
- Recover, Accept or Commit reply from epoch that has been retired should be treated as Success rather than Redundant
- Distinguish completely REDUNDANT+PRE_BOOTSTRAP from partially GC_BEFORE and REDUNDANT+PRE_BOOTSTRAP - latter can make stronger inferences based on the GC_BEFORE intersection (could perhaps be treated as simply GC_BEFORE)
- RX must register historical transactions with CFK
- CommandStore.bootstrapper must wait for coordinate sync via same mechanism as sync()
- Don't start topology change for shard where all replicas are already bootstrapping
- Reify executes et al in StoreParticipants
- LocalListeners txn listener reentry may erase the entry entirely
- use registerAt in AbstractRequest for expirations, use correct time for expiresAt in ListAgent
- use txnId.epoch() for pruning, as must be before both txnId and executeAt of prune point for coordinating dependencies
- compute accurate KnownMap when affected by bootstrap or staleness
- upgradeTruncated should calculate Definition and Deps separately
- Invalidate should not sort before Erased when calculating max reply or max knowledge reply
- avoid another infinite loop at end of burn test
- avoid another epoch loading edge case
- pass through low/high epochs to ensure we propagate information to all waiting command stores
- RX must adopt a non-pruned dependency that has a higher TxnId (if is itself behind prune point)
- rejects should also be calculated on COMMITTED started before
- remove Apply Factory wrapper for RX, redundant now we have CoordinationAdapters (and has faulty epoch logic)
- for RX ensure we return maximum writes for each epoch we intersect (same effectively as pruning logic)
- rework updateUnmanaged to improve clarity
- BeginRecovery constructor of LatestDeps should use touches() not owns() for compute localDeps
- BeginRecovery superseding calculation was incorrectly treating startedBefore Committed and Accepted the same, when the point at which a dep should be known differs
- Refactor Command visiting, porting C* integration to accord-core
- RelationMultiMap Builder should resize keys and keyLimits independently
- CommandsForKey Serialization moved to accord-core
- losing ownership of range should trigger re-registration of unmanaged waiting on commit of a no-longer owned txn
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20172
Benedict Elliott Smith