This backport differs from CASSANDRA-17812 in such a way that by default the number
of auth request threads is set to 0. That will route all requests to request executor as before this change.
The patch in 5.0 and later sets the default number of auth request threads to 4.
patch by Josh McKenzie; reviewed by Chris Lohfink for CASSANDRA-20057
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
Both CASSANDRA-18120 and CASSANDRA-19946 updated cassandra_latest.yaml but not the corresponding test configs
patch by Mick Semb Wever; reviewed by Brandon Williams, Tiago L. Alves for CASSANDRA-20002
Batchlog endpoint strategy was previously only random placements on other racks. Options now are random_remote, prefer_local, dynamic_remote, and dynamic.
patch by Shayne Hunsaker; reviewed by Mick Semb Wever, Brandon Williams for CASSANDRA-18120
Allows custom implementations of IAuthenticator to return their own
AuthenticateMessage based on ClientState information.
ClientState contains information about driver's name and version which
could be used to determine the contents of the AuthenticateMessage that
is sent back to the clients. This enables, for instance, returning
driver's known authenticator implementations (e.g. DseAuthenticator)
which enables SASL negotiation.
patch by Tiago Alves; reviewed by Mick Semb Wever, Andy Tolbert for CASSANDRA-19984
Uses ParameterizedClass for IAuthorizer, INetworkAuthorizer, and
IRoleManager implementations enabling those to be configurable from
parameters specified directly in cassandra.yaml.
Opportunistically makes consistent the initialization and setting of
all auth-related implementations and removes code duplication.
Notes:
* IInternodeAuthenticator implementations are expected to be set as
default directly in DatabaseDescriptor instead of done via
AuthConfig.applyAuth(). This is assumed in tests and client logic.
For instance ReconnectableSnitchHelperTest fetches this authenticator
before calling daemonInitialize(). Also, BulkLoader fetches this
directly when creating an outbound connection.
* Changing this behavior in BulkLoader causes a cascade of changes.
First, one would need to add AuthConfig.applyAuth() directly in
clientInitialize() and then would need to enable all the additional
classes that this change causes. Long term this might be justified
as if we're depending on the auth logic to be properly set. However,
in the context of this change it was decided to postpone further
changes.
* Standardize auth-related configurations by using a short-form with
the class name instead of the full qualified class name containing
the package name.
patch by Tiago Alves; reviewed by Mick Semb Wever, Stefan Miklosovic for CASSANDRA-19946
This merge just removes LongToken from SkipListMemtable and it replaces it with
a partitioner-agnostic DatabaseDescriptor.getPartitioner().getRandomToken().
The issues in 4.0 / 4.1 are not present in later branches anymore however we
identified that the changes done in 5.0+ are welcome so we included them as well.
Fix estimateRowOverhead in Memtable: ByteBuffer in Partition key was not counted
Fix non-counted size of "columns" field in Columns.unsharedHeapSize
BTree.sizeOfStructureOnHeap wrongly reported a non-zero size for EMPTY_LEAF shared constant value
Fix MemtableSizeTest flakeness by not measuring of allocator and ColumnFamilyStore/Keyspace
Patch by Dmitry Konstantinov; reviewed by Branimir Lambov, Stefan Miklosovic for CASSANDRA-17298
patch by Caleb Rackliffe; reviewed by Zhao Yang, Brandon Williams, and Marcus Eriksson for CASSANDRA-19704
Co-authored-by: Marcus Eriksson <marcuse@apache.org>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
* Add an ability to base _replica_ side queries on the queue time
* Use queue time as a base for message timeouts
* Use native transport deadline for internode messages
* Make sure that local runnables respect transport timeouts and deadlines
* Make sure that remote mutation handler respects message expiration times
Patch by Alex Petrov; reviewed by Caleb Rackliffe for CASSANDRA-19534
Patch by Jon Meredith; reviewed by Josh McKenzie and Berenguer Blasi for CASSANDRA-17815
Co-authored-by: Jon Meredith <jmeredithco@gmail.com>
Co-authored-by: Josh McKenzie <jmckenzie@apache.org>
nodetool import requires keyspace and table on the command line to import SSTables to. This
was not always working as specifying keypace and table while having SSTables located in the
directory structure which differed on keyspace and table (dir and parent dir) just
ignored them.
patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-19401
– ensure test file paths, and their suite names are unique (the latter was broken for cqlshlib and python dtests)
– removal of properties and system-out in test xml happens in CassandraXMLJUnitResultFormatter
– new jenkins agent labels and introduce agents sizes
- ci_summary.html generation script, ref work submitted in apache/cassandra-builds#99
– fix for stress-test and fqltool-test running on small
- ant generate-test-report is limited to only running on individual test types (ci_parser.py provides ci_summary.html for the overview now)
- each cell has a single retry, and the retry will happen on a different agent
- on ci-cassandra the summary stage happens on the builtin, bc copyArtifacts on 15k+ files takes many hours otherwise
- test-burn only needs two splits
- dependency-check is disabled from the lint target until CASSANDRA-19213
- add $DEBUG env var to in-tree scripts, turns on bash debug
- fix FBUtilities' handling of gcp cos_containerd (kernel version comes with a trailing '+' character)
patch by Aleks Volochnev, Mick Semb Wever; reviewed by Aleksandr Volochnev, Josh McKenzie, Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-18594
Co-authored-by: Aleksandr Volochnev <a.volochnev@gmail.com>
Co-authored-by: Mick Semb Wever <mck@apache.org>
Co-authored-by: Josh McKenzie <jmckenzie@apache.org>
Co-authored-by: Artem Chekunov <artem.v.chekunov@gmail.com>
* replace KeyRangeConcatIterator's PriorityQeueu with List
* remove KeyRangeIterator.current and simplify
* remove injected exception and tests - not relevant to the current implementation
* expand randomized testing
* inline getCurrent() -> peek(); rename getCount to getMaxKeys
* redefine skipTo contract to not return a value (which saves unnecessary work when skipTo is called multiple times in a row)
* calling hasNext in skipTo is a pessimization; if the iterator is in DONE state, then skipTo will see it and avoid further effort; if it is not, then we are computing a next value that we're just going to throw away
* fix SingleNodeQueryFailureTest - tests now multi and single index
* rationalize/standardize the way we release SSTableIndexes in QueryController
patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, Ekaterina Dimitrova for CASSANDRA-19428
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Jonathan Ellis <jbellis@apache.org>
Co-authored-by: Piotr Kolaczkowski <pkolaczk@datastax.com>
Co-authored-by: Michael Marshall <michael.marshall@datastax.com>
patch by Caleb Rackliffe; reviewed by David Capwell for CASSANDRA-19461
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: David Capwell <dcapwell@apache.org>
- avoid randomisation to make tests more consistent
- use heap_buffers for VectorDistributedTest for consistency with other tests
patch by Mike Adamson; reviewed by Ekaterina Dimitrova for CASSANDRA-19185
This patch adds a new configuration file, cassandra_latest.yaml, which changes
some of the settings to use new features we usually recommend to users,
including:
- Big Trie-Indexed SSTables
- Trie memtables
- Unified Compaction Strategy
- Storage Attached Index
- Off-heap objects memtables
- Direct-write commit log
- No storage compatibility
These changes are applied in new test targets for unit, in-jvm and python dtests that are
run pre-commit. When changes are made to the new config file, python dtest will use it
directly, but unit and in-jvm dtests require updates, respectively, to latest_diff.yaml
and InstanceConfig.java.
patch by Branimir Lambov; reviewed by Berenguer Blasi for CASSANDRA-18753
- Replica filtring protection now correctly accounts for short reads caused by coordinator filtering of merged rows.
- There is a new flag in ReadCommand that indicates whether the user query it belongs to requires reconciliation.
- Local SAI queries now degrade intersections to unions on unrepaired data if reconciliation is required and multiple mutable columns are restricted.
patch by Caleb Rackliffe; reviewed by Andres de la Peña and Alex Petrov for CASSANDRA-19018
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Alex Petrov <oleksandr.petrov@gmail.com>
patch by Andrés de la Peña; reviewed by David Capwell for CASSANDRA-19336
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
Co-authored-by: David Capwell <dcapwell@apache.org>
When a node is decommissioned, it triggers data transfer to other nodes.
During this transfer process, receiving nodes temporarily hold token ranges in a pending state.
However, the current cleanup process doesn't account for these pending ranges when calculating token ownership,
leading to inadvertent cleanup of data already stored in SSTables.
To address this issue, this patch introduces two changes.
Firstly, it backports CASSANDRA-16418, introducing a preventive check in `StorageService#forceKeyspaceCleanup`.
This check disallows the initiation of cleanup when a node contains any pending ranges for the requested keyspace.
Secondly, it reintroduces a similar condition to test for the existence of pending ranges in `CompactionManager#performCleanup`.
This ensures the safety of this API as well.
Patch by Szymon Miezal; reviewed by Brandon Williams, Jacek Lewandowski for CASSANDRA-18824
Co-authored-by: Szymon Miezal <szymon.miezal@datastax.com>
Co-authored-by: Jacek Lewandowski <lewandowski.jacek@gmail.com>
patch by Caleb Rackliffe; reviewed by Piotr Kołaczkowski and Alex Petrov for CASSANDRA-19278
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Co-authored-by: Piotr Kołaczkowski <pkolaczk@gmail.com>
- Rename and refactor a bit property for overriding storage compatibility mode
- Fix streaming for tools
- consider bulkloader as a client instead of tool
Patch by Jacek Lewandowski; reviewed by Berenguer Blasi, Branimir Lambov for CASSANDRA-19126
This patch adds a converter that allows parsing vector literals
passed via csv files to the COPY command.
patch by Szymon Miezal; reviewed by Andrés de la Peña, Stefan Miklosovic and Maxwell Guo for CASSANDRA-19118
patch by Berenguer Blasi, Benjamin Lerer; reviewed by Benjamin Lerer for CASSANDRA-19050
Co-authored-by: Benjamin Lerer <b.lerer@gmail.com>
Co-authored-by: Berenguer Blasi <berenguerblasi@gmail.com>
This commit fixes Gossiper::hasMajorVersion3Nodes so that it does not return true when all hosts have a known version, no hosts are on a version earlier than 4.0, and there is a 4.x minor version or patch version upgrade in progress. Additionally, this commit improves the clarity of Gossiper::hasMajorVersion3Nodes's name to indicate that it will return true when the cluster has 3.x nodes or if the cluster state is unknown, matching the description in the in-line comment.
patch by Isaac Reath; reviewed by Paulo Motta and Stefan Miklosovic for CASSANDRA-18999
Closes#2967
Change how ByteBuddy injections are handled to avoid ByteBuddy
failures after node restarts
Patch by Mike Adamson; reviewed by Caleb Rackliffe for CASSANDRA-19084
Also, fixes MigrationCoordinatorTest and adds version assertions to Instance.startup
Patch by Jacek Lewandowski; reviewed by Ekaterina Dimitrova for CASSANDRA-18902
This removes the RAMStringIndexer for literal indexes and replaces
it with a SegmentTrieBuffer that replaces BlockBalancedTreeRamBuffer
for literal and numeric indexes.
patch by Mike Adamson; reviewed by Andrés de la Peña, Caleb Rackliffe for CASSANDRA-18598
- Fix comparison in PostingListRangeIterator for updating skip token
- Fix binary search in KeyLookup.clusteredSeekToKey
- Added new on-disk component for storing partition sizes by partition ID
patch by Mike Adamson; reviewed by Caleb Rackliffe, Mick Semb Wever for CASSANDRA-19011
Fixed a bug causing the `OpenDataException` being thrown
when executing the `repair_admin summarize-pending` command.
This patch addresses the problem by including a missing composite
in `PendingStats.toComposite`, ensuring proper data conversion.
patch by Szymon Miezal; reviewed by Stefan Miklosovic and Brandon Williams for CASSANDRA-19014
This is the follow-up commit of CASSANDRA-18935 where we set RPC_READY to false
when transports were shut down in runtime. The problem is that the current logic in StorageProxy.findSuitableEndpoint method,
used for the selection of a leader for counter mutations, is filtering out all endpoints which do not have RPC_READY
set to true. Hence, if there is a deployment of a coordinator node (not joining a ring) and storage nodes which
have transports turned off (e.g. for security reasons), then a coordinator node will select no endpoint as a
counter mutation leader which renders counter mutations impossible.
This change just reverts the original fix which was setting RPC_READY to false when transports were shut down in runtime
(e.g. by nodetool disablebinary).
For trunk (to be 5.1 at time of writing this), there is already TCM in place and the correct fix yet to be implemented
is to decouple from checking RCP_READY state and base it e.g. on JOINED state from TCM perspective. Please follow
CASSANDRA-19103 where this will be addressed.
patch by Stefan Miklosovic; reviewed by Brandon Williams for CASSANDRA-18935
Patch by Amit Pawar and Jacek Lewandowski; reviewed by Branimir Lambov and Maxwell Guo for CASSANDRA-18464
Co-authored-by: Amit Pawar <Amit.Pawar@amd.com>
Co-authored-by: Jacek Lewandowski <lewandowski.jacek@gmail.com>
- This fixes a number of broken tests when the default index is set to SAI
- Composite partition indexes were being filtered prior to row filtering in the
index searcher resulting in incorrect results
- Static and non-static index intersection was failing because static primary keys
were not comparing correctly against non-static primary keys
patch by Mike Adamson; reviewed by Andres de la Peña, Michael Semb Wever for CASSANDRA-19034
- Replace IndexContext with IndexTermType and IndexDefinition
- Move index specific managers, factories and metrics to StorageAttachedIndex
- Refactor Expression to explicitly define indexed and unindexed expressions
patch by Mike Adamson; reviewed by Andres de la Peña, Caleb Rackliffe for CASSANDRA-18166
patch by Andrés de la Peña; reviewed by Caleb Rackliffe for CASSANDRA-19017
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
patch by Zhao Yang and Andrés de la Peña; reviewed by Zhao Yang for CASSANDRA-18926
Co-authored-by: Zhao Yang <zhaoyangsingapore@gmail.com>
Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
HintsMaker was creating hint files which ttl'ed after 10 days which started to fail the tests.
By setting it explicitly to Integer.MAX_VALUE, newly generated hint files will expire in cca 68 years.
Branches 3.0 to trunk introduce the fix to HintsMaker. Branches 5.0 and trunk contain regenerated hint files as well.
Patches for 3.0, 3.11, 4.0 and 4.1 are just adding HintsMaker class to generate hint files.
Patches for 5.0 and trunk are adding tests for commit log upgrading and hints which were generated by HintsMaker.
Patches for 5.0 and trunk are also adding version 1 in HintsDescriptor back as it was removed in CASSANDRA-18314 by mistake.
patch by Stefan Miklosovic; reviewed by Michael Semb Wever for CASSANDRA-19002
The fix makes the DescribeStatement to wait for the in-progress schema transformations to finish before returning the first page. This way, the metadata and schema version encoded in the result set metadata are guaranteed to be consistent.
Patch by Jacek Lewandowski; reviewed by Benjamin Lerer, Ekaterina Dimitrova for CASSANDRA-18921
Removed use of reflection and directly set
relevant property to avoid jdk 17 errors
patch by Mike Adamson; reviewed by Stefan Miklosovic, Michael Semb Wever and Andrés de la Peña for CASSANDRA-18715
- Adds jbellis/jvector (1.0.2) library for DiskANN based indexes on floating point vectors
- Adds ORDER BY ANN OF capability to do ANN search and order the results by score
patch by Mike Adamson; reviewed by Andrés de la Peña, Jonathon Ellis for CASSANDRA-18715
Co-authored-by Jonathon Ellis jbellis@gmail.com
Co-authored-by Zhao Yang zhaoyangsingapore@gmail.com
The Index.Group was being left in the list indexGroups in the SecondaryIndexManager because the incorrect
key was being used to remove it from the map
patch by Mike Adamson; reviewed by Caleb Rackliffe and Zhao Yang for CASSANDRA-18905
Co-authored-by: Zhao Yang <zhaoyangsingapore@gmail.com>
- jvm-dtest-upgrade script running in ci-cassandra.a.o (needs default bash)
- add offline mode (to avoid `git fetch` before building dtest-jars on other branches),
- avoid rebuilding in fqltool-test target, as is standard for test targets.
- python detection and override
- FileTest cleanup (tmp directory was left unreadable and undeletable)
- reuse ~/.ccm (like normal)
- remove support for python 3.6
- remove docker volumes in image (they are added at run time)
- remove superfluousjava version debug
– extra mount for build/tmp as hack to prevent buggy hardlinks on overlay
patch by Mick Semb Wever; reviewed by Brandon Williams, Ekaterina Dimitrova, Josh McKenzie, Maxim Muzafarov for CASSANDRA-18665
In this commit the `org.apache.cassandra.io.sstable.CQLSSTableWriter#build` method synchronizes on the
`Schema.instance` object (instead of the `CQLSSTableWriter.class`) to prevent concurrent schema operations
to fail when the offline tools also updates the schema.
For example, a table creation operation, which modifies the keyspace tables metadata, might end up
missing the update when a concurrent call to the `CQLSSTableWriter#build` method is accessing the
singleton Schema instance.
Patch by Francisco Guerrero, reviewed by Yifan Cai, Maxwell Guo, Alex Petrov for CASSANDRA-18317.
This patch also removes compress-lzf library (leftover from CASSANDRA-12229)
as well as it makes byte-buddy dependencies to be test scoped.
patch by Stefan Miklosovic; reviewed by Michael Semb Wever for CASSANDRA-18877
There are a couple of fixes in this patch. As explained on the ticket, some of the flaky failures are a race of two events that cause the termination of the connection. First is a legitimate close as a result of expected failures. In this case, the server sends the error message and closes the connection. The test expects to receive that message. However, the test was sending more messages that couldn't be received because the server already closed the connection, and they were bounced by the OS, causing immediate connection shutdown on the client side, even before it could receive the error message.
The fix is to stop sending the messages after sending the message, which is expected to cause a failure. Some other accompanying modifications include using Awaitility to wait for specific events and consider the configured maximum number of consecutive failures.
Also added some more logging to help investigate failures in the future.
Patch by Jacek Lewandowski; reviewed by Sam Tunnicliffe for CASSANDRA-16949
This patch removes deprecated chunk_length_kb and sstable_compression in favor of
chunk_length_in_kb and class parameters.
The only deprecated compression parameter left, crc_check_chance (which is configurable outside
of compression parameters as a standalone table parameter) is meant to be treated separately.
patch by Stefan Miklosovic; reviewed by Maxim Muzafarov and Brandon Williams for CASSANDRA-18742
Refactoring prevents the situation where some sstable components, like
data or index, are created before the new sstable is registered with
lifecycle transaction, which leads to a problem such that there is
a short time when incomplete sstable components are present. At the same
time, no transaction file is created, which leads to the possibility
that the sstable can be recognized as completed by various
transaction-aware listers.
Patch by Jacek Lewandowski; reviewed by Branimir Lambov, Mike Adamson for CASSANDRA-18737
- Don't create uber jar for microbenchmarks
- Add async profiler to jmh tests
- Benchmark classes names validation
- Add jmh.args property to make it possible passing extra args to JMH
- Add missing test/anttasks to idea configuration
Patch by Jacek Lewandowski; reviewed by Branimir Lambov, Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-18871
Bereng