* reconstruct CFK, TFK, progressLog
* migrate CommandStore collection state from Accord table to the log
* make memtable writes non-durable; reconstruct memtable state from Writes
Patch by Alex Petrov and Benedict Elliott Smith; reviewed by Benedict Elliott Smith and Alex Petrov for CASSANDRA-19869
(Accord): C* stores table in Range which will cause ranges to be removed from Accord when DROP TABLE is performed
patch by David Capwell, Sam Tunnicliffe; reviewed by Sam Tunnicliffe for CASSANDRA-18675
CEP-15: (Accord) sequence EpochReady.coordinating to allow syncComplete to be learned from newer epochs
patch by David Capwell; reviewed by Alex Petrov, Blake Eggleston for CASSANDRA-19769
Patch by Blake Eggleston; Reviewed by David Capwell for CASSANDRA-19940
Changes:
Increase accord repair range splitting
Streamline table metadata fetching - removes some unnecessary abstraction from the table metadata lookup path
Remote unnecessary set building when building lists of overlapping keys
Add separate recover delay for repair and increase default recover delay
Rebase fixup: when a local keyspace is being open but it isnt present return null so error msg can be provided
Rebase fixup: improved metrics error msg when the exception doesnt match what is expected
Rebase improvement: when we see a timeout or preempt use the new vtable to show the status cross the cluster
Rebase improvement: Cluster.checkForThreadLeaks now groups similar stack traces to make the output less dense
The SimpleProgressLog had a number of problems:
1. It polled for progress with no attempt to determine whether progress could realistically be made, so:
- as the number of pending transactions grew, the proportion of useful work dropped (as many would be unable to make progress without earlier transactions completing)
- each transaction in the chain could recover only on average 1/2 poll interval behind the last transaction to complete
2. It requested full transaction state from every replica on each attempt
3. It maintained a lot of in-memory state
4. Polling happened en-masse, allowing for little per-transaction control
We also separately maintained fairly expensive per-command listener state that negatively affected our command loading and caching.
The new DefaultProgressLog makes use of several new features: LocalListeners, RemoteListeners, Timers and Await messages.
- LocalListeners provide a memory-efficient collection for managing each CommandStore<E2><80><99>s transaction listeners, with dedicated record keeping for inter-transaction relationships.
- RemoteListeners provide a mechanism for request/response pairs that may be separated by longer than the normal Cassandra message timeout, and require minimal state on sender and recipient. This permits replicas to cheaply update their local state machine as soon as distributed information becomes available.
The DefaultProgressLog tracks each transaction with separate timers to handle per-transaction scheduling, backoff etc, and a succinct state machine. To reduce overhead correspondence is preferentially limited to a handful of replicas, and limited to the home shard where appropriate.
patch by Benedict; reviewed by Ariel Weisberg for CASSANDRA-19870
- Remove concept of non-participating home keys; home keys are required to be a participant in the transaction
- Remove covering/covers concept
- Various invalidation/truncation/erase behaviours
patch by Benedict; reviewed by Blake for CASSANDRA-19825
CommandsForKey periodically self-prunes, so as to continue functioning well in-between garbage collections. Once we prune we are left with potentially incomplete information, and have to sometimes load per-command information from disk. But the payoff is ensuring CommandsForKey objects - which drive the majority of the state machine - are kept to a reasonable size.
patch by Benedict; reviewed by Blake Eggleston and David Capwell
Introduce a special kind of non-durable read that provides only per-key linearizable isolation; i.e. strict-serializable isolation for single partition-key reads.
This read creates only a happens-before edge, by collecting dependencies for execution and ensuring that execution happens strictly after these dependencies
have executed, but at no precise time otherwise. So later writes may be witnessed, and if multiple keys are read they may represents different points in time.
patch by Benedict; reviewed by Ariel Weisberg for CASSANDRA-19305
Refactor CommandsForKey for efficiency, and to support transitive dependency elision
patch by Benedict; reviewed by Aleksey Yeshchenko for CASSANDRA-19310
Don't use ImmediateExectuor in JVM dtests to process messages
Fix GuardrailTablesTest
Instance.receiveMessage should use sync
Fix StorageAttachedIndexDDLTest failing due to background Accord compactions
Add back enforceInvariants=false to shouldCleanup in AccordCommandsPurger
Fix CompactionAccordIteratorsTest
Fix empty row check in command/tfk mutation methods
s fc83325fa9 Fix AccordObjectSizes empty sizes
s 44bce6a08c Fix error handling when there are no column families to repair
s f1459540e5 Fix broken nowInSec deserialization
s 91c1befd1c Fix Mutation serializedSize
s b11467c200 Using simulated clock instead of raw nanoTime
s 6412b35924 Accord repair needs to use sentinel tokens
s d19c01ff01 Enable Accord repair with HappyPathFuzzTest
s 2da87f91b8 Implement abort in AccordRepairJob
s e12877c4fa move barrier to accord spec
Patch by Ariel Weisberg; Reviewed by David Capwell for CASSANDRA-19023
s c4bf1533b4 Add more JVM arg checks to the paxos simulation runner
s 1b650a0636 break circular dependency between FileSystemOwnershipCheck and CassandraRelevantProperties
s e0111748dd fix verify with-rng and interrupts
s f8475f2a18 make SimulatedFailureDetector deterministic
s ee77734bbe Deterministic transformation
s 890f23696a Add mechanism forsimulator to intercept password salting
s 5205b99874 Specify Paxos V2 because it is the only linearizable option
Patch by Ariel Weisberg; Reviewed by David Capwell for CASSANDRA-19008
- Avoid serializing full TxnData instances to Accord state tables
patch by Caleb Rackliffe; reviewed by David Capwell, Benedict Elliot Smith, and Ariel Weisberg for CASSANDRA-18355
Accord compaction purgers see random slices of Accord state during compaction (based on randomly selected compaction inputs).
For at least the `durability` column in the `commands` table the tombstone being created when truncating was deleting the latest value since we can get enough information to truncate without actuall yhaving the latest `durability` value.
To fix we can wait to emit a tombstone until we are erasing the entire command row when truncating or truncating with outcome and meanwhile we can drop the extra columns that are no longer needed instead of using a tombstone. We don't need to emit cell tombstones we can drop them from the purger when processing each row.
patch by Ariel Weisberg; reviewed by David Capwell for CASSANDRA-18795
patch by Benedict Elliott Smith; reviewed by Ariel Weisberg, Aleksey Yeschenko, and David Capwell for CASSANDRA-18883
Co-authored-by: Benedict Elliott Smith <benedict@apache.org>
Co-authored-by: Ariel Weisberg <aweisberg@apple.com>
Co-authored-by: Aleksey Yeschenko <aleksey@apache.org>
Co-authored-by: David Capwell <dcapwell@gmail.com>
Patch by Blake Eggleston and Benedict Elliott Smith; Reviewed by David
Capwell for CASSANDRA-17101
CEP-15: Accord TCM integration
Patch by Blake Eggleston; Reviewed by David Capwell for CASSANDRA-18444
* cassandra-5.0:
Fixed multiple single-node SAI query bugs relating to static columns
- Ensure MemtableIndexWriter calculates min/max properly with indexes on partition key elements
- Ensure only rows with live data are indexed
- Ensure min cannot be greater than max in intersection statistics with static keys
- Correct tracking of last key in the searcher in the presence of static keys
- Ensure MemtableIndexWriter calculates min/max properly with indexes on partition key elements
- Ensure only rows with live data are indexed
- Ensure min cannot be greater than max in intersection statistics with static keys
- Correct tracking of last key in the searcher in the presence of static keys
patch by Caleb Rackliffe; reviewed by David Capwell for CASSANDRA-20338
New features in the storage engine need to be appropriately tested in the CQLSSTableWriter code
so they can be used in the Analytics library and others utilizing the writer library.
Add a test for support of Vector data types and constraints to ensure they work with the
CQLSSTableWriter and make any changes necessary if they don’t.
patch by Doug Rohrer; reviewed by Bernardo Botella, Stefan Miklosovic for CASSANDRA-20496
- fix some edge cases for NOT_NULL
- ability to specify constraints when altering a column
- ensure constraint is specified on a column it is bound to
- fix nullity check on map type
- fix satistfiability check on function constraints
patch by Stefan Miklosovic; reviewed by Bernardo Botella for CASSANDRA-20481
Also add test method testVerifyOldTupleSSTables for frozen tuples and dropping of them.
The test data also adds different versions of me format sstables from C* version 3.0.25 and 3.11.11 as they do (unfortunately) differ. Test sstable files have had their sequence generation ids changed to represent the server version used to create them, while this shouldn't be necessary it is for debug purposes when sstable formats have changed between versions.
patch by Roxana Neophytou, Mick Semb Wever; reviewed by Dmitry Konstantinov for CASSANDRA-20485
Co-authored-by: mck <mck@apache.org>
nodetool gcstats output was broken as such / not aligned.
This command was using a completely custom way of displaying the statistics which was fixed.
This patch also supports three output modes: table, yaml, json under -F flag.
There is also a possibility to output the figures in human-friendly format via -H flag.
The patch also adds more JVM statistics related to direct memory besides
already existing "allocated direct memory" being:
- max direct memory
- reserved direct memory
patch by Ling Mao; reviewed by Brad Schoening, Stefan Miklosovic for CASSANDRA-19022
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
* cassandra-5.0:
Fix CommitlogShutdownTest flakiness by testing a real drain logic instead an emulation which is affected by background activities for system tables
* cassandra-4.1:
Fix CommitlogShutdownTest flakiness by testing a real drain logic instead an emulation which is affected by background activities for system tables
* cassandra-4.0:
Fix CommitlogShutdownTest flakiness by testing a real drain logic instead an emulation which is affected by background activities for system tables
Patch by Josh McKenzie and Sam Tunnicliffe; reviewed by Marcus Ericksson for
CASSANDRA-20467
Co-authored-by: Josh McKenzie <jmckenzie@apache.org>
Co-authored-by: Sam Tunnicliffe <samt@apache.org>
Patch by Jordan West, Jon Haddad; Reviewed by David Capwell, Caleb Rackliffe, Dmitry Konstantinov for CASSANDRA-15452
NOTE: This was originally merged up via merge commit but something went wrong, this is the fix commit to bring this back to trunk
Avoid iterator allocations if possible
Handle typical cases (such as a single row, single table writes) more efficiently
Add fast paths for typical scenarios (like absense of views and triggers)
Memorize things which can be computed once
Patch by Dmitry Konstantinov; reviewed by Chris Lohfink, Michael Semb Wever, Vladimir Sitnikov for CASSANDRA-20167
To fix it NetworkTopologyProximity implementation is adjusted using ByteBuddy to make a predictable order of nodes to read (the same idea as in python read repair dtests is used). Also, dynamic snitch is disabled to avoid re-ordering of replicas in a read plan by it. Logging of actual configuration in java dtests is added to simplify troubleshooting (the default logic does not print it because logging is initialized in tests later).
Patch by Dmitry Konstantinov; reviewed by Brandon Williams,Stefan Miklosovic for CASSANDRA-20251
Where MVs are also affected by a table alteration make sure we correctly
set the epoch of the TableMetadata enclosed in the ViewMetadata
Patch by Sam Tunnicliffe and Marcus Eriksson; reviewed by Marcus
Eriksson for CASSANDRA-20318
Co-authored-by: Sam Tunnicliffe <samt@apache.org>
Co-authored-by: Marcus Eriksson <marcuse@apache.org>
patch by Stefan Miklosovic; reviewed by Bernardo Botella, Dinesh Joshi for CASSANDRA-20276
Co-authored-by: Bernardo Botella <bernardo.botella@apple.com>
patch by Caleb Rackliffe; reviewed by David Capwell and Andres de la Peña for CASSANDRA-20313
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Andres de la Peña <a.penya.garcia@gmail.com>
Not unwrapping returnType has consequences of not
returning the correct results when using aggregate in-built functions
(max / min) on clustering columns having descending order.
The safest solution, is to unwrap returnType "at source"
and work with unwrapped type only since then.
patch by Stefan Miklosovic; reviewed by David Capwell for CASSANDRA-20295
* cassandra-5.0:
Avoid possible consistency violations for SAI intersection queries over repaired index matches and multiple non-indexed column matches
* Restrict which permissions can be granted on system keyspaces
* Ensure that GRANT... ON ALL KEYSPACES excludes system keyspaces
* Add system_traces to the always readable set
Patch by Sam Tunnicliffe and Francisco Guerrero; reviewed by Sam
Tunnicliffe and Francisco Guerrero for CASSANDRA-20090
Note: this is a re-application of the commit to cassandra-4.0 only,
to fix a bad merge the first time around.
Co-authored-by: Francisco Guerrero <frankgh@apache.org>
patch by Bernardo Botella; reviewed by Yifan Cai, Stefan Miklosovic, Josh McKenzie, Maxwell Guo, Dmitry Konstantinov, Sam Tunnicliffe for CASSANDRA-19947
Previously users would either have to read the data and check the length
themselves or enable UDF and register UDFs to check the length of
columns. This patch adds a subset of the SQL99 (binary) string
functions: "octet_length" defined on all types and "length" defined on
UTF8 strings.
patch by Joey Lynch; reviewed by Chris Lohfink and Jordan West for CASSANDRA-20102
This commit ports to trunk work already done in the cep-15-accord branch, including parts of CASSANDRA-19955, CASSANDRA-20080, and CASSANDRA-20143.
patch by Caleb Rackliffe; reviewed by David Capwell and Alex Petrov for CASSANDRA-20200
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Ariel Weisberg <aweisberg@apple.com>
Co-authored-by: Alex Petrov <oleksandr.petrov@gmail.com>
Co-authored-by: David Capwell <dcapwell@apache.org>
When snapshot manifest does not exist, no snapshots would be found, because even
they would be loaded, since CASSANDRA-18111, snapshot's presence is logically
determined by the existence of at least one manifest.json file in any data dir. That
would result in snapshots not being shown e.g. in the output of
nodetool listsnapshots.
The fix consists of creating a manifest in each snapshot directory when not present.
When a manifest.json exists, its older format has not contained created_at field (pre-16789)
When this is detected, we will proceed to enrich such manifest by overwriting it
with a manifest of new format.
patch by Stefan Miklosovic; reviewed by Jordan West, Cheng Wang, Bernardo Botella for CASSANDRA-20150
This patch also opportunistically adds a new MBean method for SnapshotManager to
return true snapshot size for a particular snapshot.
patch by Stefan Miklosovic; reviewed by Maxim Muzafarov, Bernardo Botella for CASSANDRA-20149
patch by Caleb Rackliffe; reviewed by David Capwell for CASSANDRA-20198
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: David Capwell <dcapwell@apache.org>
patch by Manish Pillai; reviwed by Stefan Miklosovic, Bernardo Botella, Jordan West for CASSANDRA-20104
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
When hardlinks are created for a snapshot with the name "snapshot"
and then we take a snapshot with the name "Snapshot", macOS platform thinks
that this was already hardlinked because its hardlinking implementation
does not seem to be case-sensitive. The fix consists of checking,
in a case-insensitive manner, if there is already such snapshot,
but only on macOS platform.
The patch also does not create empty directories for a given snapshot
prematurely before checking its presence so there are not dangling empty
directories in case of a failure.
patch by Stefan Miklosovic; reviewed by Bernardo Botella, Caleb Rackliffe for CASSANDRA-20146
Deprecate IEndpointSnitch entirely, to be replaced with new interfaces:
* Locator for endpoint -> location mapping
* InitialLocationProvider to supply the DC & rack for registration
* NodeProximity for sorting endpoints and replicas at query time
For migration/upgrade/deprecation, nodes can still be configured with
an IEndpointSnitch implementation via endpoint_snitch in config, but
we hide this with a facade and only present the new interfaces.
Patch by Sam Tunnicliffe and Marcus Eriksson; reviewed by Sam
Tunnicliffe and Marcus Eriksson for CASSANDRA-19488
Co-authored-by: Marcus Eriksson <marcuse@apache.org>
Co-authored-by: Sam Tunnicliffe <samt@apache.org>
patch by Zhongxiang Zheng; reviewed by Stefan Miklosovic, Maulin Vasavada, Cheng Wang, Jordan West for CASSANDRA-11695
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
Co-authored-by: Sam Tunnicliffe <samt@apache.org>
This removes the usage of index files during compaction and simplifies
and improves the performance of compaction.
patch by Branimir Lambov; reviewed by Sylvain Lebresne for CASSANDRA-20092
Patch by Sam Tunnicliffe and Marcus Eriksson; reviewed by Sam
Tunnicliffe and Marcus Eriksson for CASSANDRA-18961
Co-authored-by: Marcus Eriksson <marcuse@apache.org>
Co-authored-by: Sam Tunnicliffe <samt@apache.org>
Benedict Elliott Smith