* avoid running python3-dependent tasks when running simulator tasks
* fix a problem with simulated snitch rebase
* add a distinction between short-lived daemon threads and infinite loop ones for cases when we need to simulate user-implemented infinite loops
Patch by Alex Petrov; reviewed by Benedict Elliott Smith for CASSANDRA-20542
Improve:
- InMemoryJournal compaction should simulate files to ensure we compact the same cohorts of records (so shadowing applied correctly)
- Don't update CFK deps if already known
- avoid unnecessary heapification of LogGroupTimers
- begin removal of Guava (mostly unused)
- consider medium path on fast path delayed
- add Seekables.indexOf to support faster serialization
- unboxed Invariant.requires variant
- AsyncChains.addCallback -> invoke
Fix:
- Recovery must wait for earlier transactions on shards that have not yet been accepted, even if we recover a fast Stable record on another shard
- only skip Dep calculation on PreAccept if vote is required by coordinator
- ReadTracker must slice Minimal the first unavailable collection
- If PreLoadContext cannot acquire shared caches, still consider existing contents
- CFK: make sure to insert UNSTABLE into missing array
- Fix failed task stops DelayedCommandStore task queue processing further tasks
- short circuit AbstractRanges.equals()
- Handle edge case where we can take fast/medium path but one of the Deps replies contains a future TxnId
- update executeAtEpoch when retryInFutureEpoch
- Fix incorrect validation in validateMissing (should only validate newInfo is not in missing when in witnessedBy; all other additions should be included)
patch by Benedict; reviewed by David Capwell for CASSANDRA-20522
* Fix short accord simulation test (seed 0x6bea128ae851724b), ConcurrentModificationException
* Increase wait time during closing to avoid Unterminated threads
* Increase timeouts, improve test stability
* More descriptive output from CQL test
* Shorten max CMS delay
* Improve future handling in config service
Patch by Alex Petrov; reviewed by Benedict Elliott Smith for CASSANDRA-20440
- Accord Journal purging was disabled
- remove unique_id from schema keyspace
- avoid String.format in Compactor hot path
- avoid string concatenation on hot path; improve segment compactor partition build efficiency
- Partial compaction should update records in place to ensure truncation of discontiguous compactions do not lead to an incorrect field version being used
- StoreParticipants.touches behaviour for RX was erroneously modified; should touch all non-redundant ranges including those no longer owned
- SetShardDurable should correctly set DurableBefore Majority/Universal based on the Durability parameter
- fix erroneous prunedBefore invariant
- Journal compaction should not rewrite fields shadowed by a newer record
- Don't save updates to ERASED commands
- Simplify CommandChange.getFlags
- fix handling of Durability for Invalidated
- Don't use ApplyAt for GC_BEFORE with partial input, as might be a saveStatus >= ApplyAtKnown but with executeAt < ApplyAtKnown
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20441
- Decouple command serialization from TableMetadata version; introduce ColumnMetadata ids; gracefully handle missing TableId
- DataInputPlus.readLeastSignificantBytes must truncate high bits
- Fix RandomPartitioner accord serialization
- Fast path stable commits must not override recovery propose/commit decisions regarding visibility of a transaction
- RejectBefore must mergeMax, not merge, to ensure we maintain epoch and hlc increasing independently
- Bad commitInvalidate decision
- consistent filtering for touches and stillTouches
- ensure TRUNCATE_BEFORE implies SHARD_APPLIED
- TopologyManager.unsyncedOnly off-by-one error
- DurabilityQueue should not retry SyncPointErased
- handle rare case of no deps but none needed
- not updating CFK synchronously on recovery, which can lead to erroneous recovery decisions for other transactions
- Don't return partial read response when one commandStore rejects the commit
- Filter touches/stillTouches consistently
- WaitingState computeLowEpoch must use hasTouched to handle historic key with no route
Improve:
- Use format parameters to defer building Invariants.requireArgument string
- streamline RedundantStatus/RedundantBefore
Also improve:
- Introduce DurabilityService
- Retire SyncPoint, replace Barrier with Write and RX
- MessageType -> enum, restore GetMaxConflict
- Standardise backoff logic with WaitStrategy
- improve TimeoutStrategy/RetryStrategy specification strings
- Forbid KX, remove directKeyDeps
- Introduce UniqueTimeService, permitting hlc reservations for sync points avoid delay when min TxnId is sufficiently in the past
- Remove ListStore custom purge logic
Also fix:
- RejectBefore should reject on both epoch and hlc
- Do not record sync success for removed nodes
- Support GlobalDurability detecting no command store to run on
- Incorrect ballot constructor
- Serializing 15-bit ballot flags incorrectly
- TopologyManager.hasEpoch deadlock
- Computing withOpenEpochs incorrectly, sometimes stopping one epoch short
- PartitionKey serializer should not depend on schema information that can be erased
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20395
- Bad ArrayBuffers recycling logic
- RX must ensure dependencies TRANSITIVE_VISIBLE
- Permit constructing "antiRange" that spans multiple prefixes
- Not computing range CommandSummary IsDep correctly
- Truncated commands that aren't shard durable could not repopulate CFK on replay, permitting recovery of another command to make an incorrect decision
- NPE on async persist of RX (i.e. supplying no callback)
- NPE in Builder.shouldCleanup when durability is null
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20370
- Only use persisted RedundantBefore for compaction
- RouteIndex should index only touches, not Route
- Flush RangesForEpoch updates to journal immediately, so we do not rely on the command we are processing succeeding
- DurableBefore updates must wait for the epochs to be known locally
- Shard.mustWitnessEpoch to support guaranteeing to witness relevant non-topology schema changes
- We must propagate RedundantBefore RX shard bounds along with epoch syncs
- Prevent a truncated transaction FetchData infinite loop
- GC_BEFORE status being overwritten by bootstrappedAt, permitting old transaction state to be resurrected
- Avoid CFK.maxUniqueHlc read race on bootstrap
- TopologyManager.awaitEpoch could wait for wrong epoch
- Journal fsync thread could miss notifications
Also improve:
- CommandStores uses SearchableRangeList for finding matching stores
- Refactor RedundantBefore to use a sorted array of TxnId/RedundantStatus pairs (to better fix GC_BEFORE issue)
- Accord debug keyspace operates on keyspace/table, and sorts correctly by token
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20361
Improve:
- Introduce pre/accept fast execution flags
- Introduce searchable Deps serialization
- Flatten AccordRoutingKey(s) into single type, using sentinel bits
- Introduce new fast byte-comparable serialization methods for Token and TableId to support above
Fix:
- Fix journal re-serialization logic
- Enable RandomPartitioner for Accord by supporting fixed-width serialization for RouteIndex
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20349
Also fix:
- Topology slicing must declare whether we share/slice node ownership (to assist above)
- CFK.visit removes transitive dependencies too eagerly across epoch change
- apply cleanup to builder consistently, and construct the same value we would produce by purge (so that replay is idempotent)
- Invoke ExecuteTxn.LocalExecute callbacks on originating CommandStore
- misc other minor issues
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20325
Use initializeTopologyUnsafe to for-load the last topology and create shards rather than replaying all topologies.
Patch by Alex Petrov; reviewed by Benedict Elliott Smith for CASSANDRA-20294
- Even if we can decide autonomously that we took the fast path, we must still wait for earlier transactions to decide themselves
- We must update a command that is in CFK.loadingPruned whether or not it is outOfRange
- We must visit pruned commands that are transitive dependencies of RX to ensure dependencies are propagated
- flagsWithoutDomainAndKind() -> flagsWithoutDomainOrKindOrCardinality() [to fix integration regression]
- Don't invoke uniqueNow() twice when allocate nextTxnId
- BTreeReducingRangeMap edge case that corrupts map when merge function does not return one of its inputs
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20292
Also fix:
- Truncate command on first access, without participants
- Use Ballot.ZERO when invoking CFK.insertOutOfRange where appropriate
- Don't supply a command's own route to ProgressLog.waiting to ensure new keys are incorporated
- Ensure progress in CommandsForKey by setting vestigial commands to ERASED
- Add any missing owned keys to StoreParticipants.route to ensure fetch can make progress
- Recovery must wait for earlier not-accepted transactions if either has the privileged coordinator optimisation
- Inclusive SyncPoint used incorrect topologies for propose phase
- Barrier must not register local listener without up-to-date topology information
- Stop home shard truncating a TxnId to vestigial rather than Invalidated so other shards can make progress
Also improve:
- Validate commands are constructed with non-empty participants
- Remove some unnecessary synchronized keywords
- Clear ok messages on PreAccept and Accept to free up memory
- Introduce TxnId.Cardinality flag so we can optimise single key queries
- Update CommandsForKey serialization to better handle larger flag space
- Configurable which Txn.Kind can result in a CommandStore being marked stale
- Process DefaultProgressLog queue synchronously when relevant state is resident in memory
- Remove defunct CollectMaxApplied version of ListStore bootstrap
- Standardise linearizability violation reporting
- Improve CommandStore.execute method naming to reduce chance of misuse
- Prune and address some comments
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20282
- preclude TCM of being behind Accord if newer epoch is reported via withEpoch/fetchTopologyInternal
- improve topology discovery during first boot and replay
- fix races between config service TCM listener reporting topologies, and fetched topologies during
Patch by Alex Petrov, reviewed by Ariel Weisberg and David Capwell for CASSANDRA-20245
- Implement missing parts of protocol optimisations, refine some particulars and remove MEDIUM_PATH_WAIT_ON_RECOVERY
Also fix:
- Deps.txnIds -> Deps.txnIdsWithFlags to make clear unsafety, and validate Command isn't created with flags
- Save a lossy low/high epoch we're waiting on in WaitingState so that we can reconstruct the same Route on callback
- Load any potentially invalidated commands we had in ProgressLog to ensure they are marked invalidated locally and notify any waiters
- Refine Invariant to accept case where a truncated dependency should apply first but both transactions are redundant and only the waiter could not be cleaned up
- only update CFK.maxUniqueHlc using commands we execute
- Don't update uniqueHlc when zero to avoid incompatible transaction invariant check
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20222
TxnNamedRead was incorrectly determining whether a query was a continuation because the comparison was on TokenKey
and when a MinTokenKey was encountered it would discard the paging state. Comparison on token should be sufficient
because multiple queries in the same batch won't share the same left token bound, only the continuation query will.
Patch by Ariel Weisberg; Reviewed by David Capwell for CASSANDRA-20261
With paging it's possible for the range Accord needs to read from to have the start and end token be the same in which case the LHS token needs to be converted to a MinTokenKey
Patch by Ariel Weisberg; Reviewed by David Capwell for CASSANDRA-20262
- Fix waiting state callback computes different route to initiator
- Invariants.checkX -> Invariants.requireX (to allow complementary Invariants.expectX as appropriate)
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20228
Follow-up to CASSANDRA-20228:
- Fix AccordUpdateParameters to correctly supply List cell paths derived from applyAt
- Fix ExecuteAtSerialize.serialiseNullable
- Fix topologies flush regression caused by markRetired forcing a flush on every call
patch by Benedict; reviewed by Ariel Weisberg for CASSANDRA-20228
- detect and save whether RX can be used as a GC bound for HLCs
- have RX durably record if they witnessed any superseding epoch and use this for HLC GC
- rework GC to retain applyAt for writes
Also improve:
- PreCommitted overwrites fast path vote, then loses tie-break with NotAccept, so we don't recover
- track executeAtLeast and uniqueHlc separately from WaitingOn in CommandChange
- Always propose executeAt.hlc() >= txnId.hlc()
- Short-circuit recovery if ApplyAt already known
Also fix:
- Dangling recovery when Commit rejected
- Failed epoch fetch handling in DefaultProgressLog
- CommandsForKey must track Accept executeAt for computing earlierWait collection on recovery
- Report DefaultProgressLog exceptions before retrying
- Fix inverted condition in RecoverWithRoute
- Remove laterNoWait from laterWait collection in recovery
- Invalidate incorrectly selected maxNotTruncated if max had a not truncated component
- WaitingState.fetch not request/propagating prior epochs
- CFK visit active pruning was comparing txn against maxCommittedWriteBefore rather than maxCommittedWriteForEpoch
patch by Benedict; reviewed by Alex Petrov for CASSANDRA-20228
Alex Petrov