split JournalKey in journal table so we can index it
reorder journal fields so we can easily index on route (when present)
use Message.expiresAtNanos for callback expiration
do not notify slow for range barriers
Accord: Do not contact faulty replicas, and promptly report slow replies for preaccept/read. Do not wait for stale or left nodes for durability.
Wake up segment prepared after shutting down allocator, as no new segments will ever be allocated.
Shut down flusher slightly differently: we do not signal from fsync complete, since all blocks should have been fsynced by then, but we will add invariant check to notice runaway threads.
Wait for quiescense
Truncate blocking
Wait for scheduler shutdown before shutting down command store
Shut down accord after shutting down messaging
Truncate caches before replay
* reconstruct CFK, TFK, progressLog
* migrate CommandStore collection state from Accord table to the log
* make memtable writes non-durable; reconstruct memtable state from Writes
Patch by Alex Petrov and Benedict Elliott Smith; reviewed by Benedict Elliott Smith and Alex Petrov for CASSANDRA-19869
(Accord): C* stores table in Range which will cause ranges to be removed from Accord when DROP TABLE is performed
patch by David Capwell, Sam Tunnicliffe; reviewed by Sam Tunnicliffe for CASSANDRA-18675
CEP-15: (Accord) sequence EpochReady.coordinating to allow syncComplete to be learned from newer epochs
patch by David Capwell; reviewed by Alex Petrov, Blake Eggleston for CASSANDRA-19769
Patch by Blake Eggleston; Reviewed by David Capwell for CASSANDRA-19940
Changes:
Increase accord repair range splitting
Streamline table metadata fetching - removes some unnecessary abstraction from the table metadata lookup path
Remote unnecessary set building when building lists of overlapping keys
Add separate recover delay for repair and increase default recover delay
Rebase fixup: when a local keyspace is being open but it isnt present return null so error msg can be provided
Rebase fixup: improved metrics error msg when the exception doesnt match what is expected
Rebase improvement: when we see a timeout or preempt use the new vtable to show the status cross the cluster
Rebase improvement: Cluster.checkForThreadLeaks now groups similar stack traces to make the output less dense
The SimpleProgressLog had a number of problems:
1. It polled for progress with no attempt to determine whether progress could realistically be made, so:
- as the number of pending transactions grew, the proportion of useful work dropped (as many would be unable to make progress without earlier transactions completing)
- each transaction in the chain could recover only on average 1/2 poll interval behind the last transaction to complete
2. It requested full transaction state from every replica on each attempt
3. It maintained a lot of in-memory state
4. Polling happened en-masse, allowing for little per-transaction control
We also separately maintained fairly expensive per-command listener state that negatively affected our command loading and caching.
The new DefaultProgressLog makes use of several new features: LocalListeners, RemoteListeners, Timers and Await messages.
- LocalListeners provide a memory-efficient collection for managing each CommandStore<E2><80><99>s transaction listeners, with dedicated record keeping for inter-transaction relationships.
- RemoteListeners provide a mechanism for request/response pairs that may be separated by longer than the normal Cassandra message timeout, and require minimal state on sender and recipient. This permits replicas to cheaply update their local state machine as soon as distributed information becomes available.
The DefaultProgressLog tracks each transaction with separate timers to handle per-transaction scheduling, backoff etc, and a succinct state machine. To reduce overhead correspondence is preferentially limited to a handful of replicas, and limited to the home shard where appropriate.
patch by Benedict; reviewed by Ariel Weisberg for CASSANDRA-19870
- Remove concept of non-participating home keys; home keys are required to be a participant in the transaction
- Remove covering/covers concept
- Various invalidation/truncation/erase behaviours
patch by Benedict; reviewed by Blake for CASSANDRA-19825
CommandsForKey periodically self-prunes, so as to continue functioning well in-between garbage collections. Once we prune we are left with potentially incomplete information, and have to sometimes load per-command information from disk. But the payoff is ensuring CommandsForKey objects - which drive the majority of the state machine - are kept to a reasonable size.
patch by Benedict; reviewed by Blake Eggleston and David Capwell
Benedict Elliott Smith