This commit (along with the related in-jvm-dtest API change) adds the ability for ResourceLeakTest
to actually detect InstanceClassLoader leaks in 3 loops. In order to find these in CI:
- Pull in the in-jvm dtest API changes in 0.17.0
- Enable the looperEverythingTest to run (but not the others, which remain ignored)
In addition, this commit updates netty to 4.1.113 as the version
of netty previously used (4.1.96) caused a classloader leak (fixed in
Netty 4.1.98)
Patch by Doug Rohrer and Sam Tunnicliffe (in-jvm-dtest fix); reviewed by Mick Semb Wever for CASSANDRA-19783
and CASSANDRA-19239
In docker we want to use the existing populated ~/.ccm/repositories but don't want to write to the container's filesystem (it can leak fd). So we set CCM_CONFIG_DIR to be in the run-python-dtest.XXXXXX tmpdir. Previously this was done in the base (non-docker) script, but this is only needed in docker (and outside of docker we want the normal ~/.ccm usage)
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-19813
- Added debug in testing
- Propagate the $DEBUG env from docker to test scripts
- Be verbose when a pipeline errors that not all tests would have been run
- add timeout to each cell stage (scripts on a node can hang and the only alternative was to abort the whole pipeline)
patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova, Brandon Williams for CASSANDRA-19708
Also
– increase splits for jvm-dtests, container size and docker limits for simulator-dtest
– print exit status in .build/docker/run-tests.sh and .build/docker/_docker_init_tests.sh
– add a summary list of test failures in ci_summary.html
– in ubuntu2004_test.docker make sure /home/cassandra exists and has correct perms (from Marcuse)
– when on ci-cassandra, replace use of copyArtifacts in Jenkinsfile generateTestReports() with manual wget of test files (copyArtifacts is notoriously slow, >4hrs in this case)
– copy ci_summary.html and results_details.tar.xz to nightlies
– capture generateTestReports logs
– stream xz where possible, instead of xz compressing afterwards
– fix ant version in centos7-build.docker
– remove docker login (was meangingless, if credentials exist then docker is logged in)
– prefetch docker images from jfrog (to reduce dockerhub pull rate limits), relates to CASSANDRA-18931
– use scripts from cassandra-builds to clean and report on agents, relates to CASSANDRA-18130
– use groovy elvis operators to avoid NPEs/failures on blank param values
– parallel jfrog pulls and cache it in jfrog too
– pin alpine docker image version
– don't stash .git (was causing AccessDeniedException on some unstash), needed fix so deb/rpm packaging worked on not-git work directories
– durabilityHint needs PERFORMANCE_OPTIMIZED, controller becomes scale bottleneck and breaks without it
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-19558
– ensure test file paths, and their suite names are unique (the latter was broken for cqlshlib and python dtests)
– removal of properties and system-out in test xml happens in CassandraXMLJUnitResultFormatter
– new jenkins agent labels and introduce agents sizes
- ci_summary.html generation script, ref work submitted in apache/cassandra-builds#99
– fix for stress-test and fqltool-test running on small
- ant generate-test-report is limited to only running on individual test types (ci_parser.py provides ci_summary.html for the overview now)
- each cell has a single retry, and the retry will happen on a different agent
- on ci-cassandra the summary stage happens on the builtin, bc copyArtifacts on 15k+ files takes many hours otherwise
- test-burn only needs two splits
- dependency-check is disabled from the lint target until CASSANDRA-19213
- add $DEBUG env var to in-tree scripts, turns on bash debug
- fix FBUtilities' handling of gcp cos_containerd (kernel version comes with a trailing '+' character)
patch by Aleks Volochnev, Mick Semb Wever; reviewed by Aleksandr Volochnev, Josh McKenzie, Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-18594
Co-authored-by: Aleksandr Volochnev <a.volochnev@gmail.com>
Co-authored-by: Mick Semb Wever <mck@apache.org>
Co-authored-by: Josh McKenzie <jmckenzie@apache.org>
Co-authored-by: Artem Chekunov <artem.v.chekunov@gmail.com>
This patch adds a new configuration file, cassandra_latest.yaml, which changes
some of the settings to use new features we usually recommend to users,
including:
- Big Trie-Indexed SSTables
- Trie memtables
- Unified Compaction Strategy
- Storage Attached Index
- Off-heap objects memtables
- Direct-write commit log
- No storage compatibility
These changes are applied in new test targets for unit, in-jvm and python dtests that are
run pre-commit. When changes are made to the new config file, python dtest will use it
directly, but unit and in-jvm dtests require updates, respectively, to latest_diff.yaml
and InstanceConfig.java.
patch by Branimir Lambov; reviewed by Berenguer Blasi for CASSANDRA-18753
patch by Claude Warren; reviewed by Stefan Miklosovic, Jacek Lewandowski, Michael Semb Wever for CASSANDRA-16565
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
Configured OWASP database to use caching and NVD key. Properties can be
configured in ~/.ant/build.properties and define both the key and
the storage location
Patch by Jacek Lewandowski; reviewed by Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-19146
This prevents docker overlay file leakage from containers by keeping write-heavy directories in bind mounts (or volumes). It also offers better performance by avoiding the container filesystem.
patch by Mick Semb Wever; reviewed Brandon Williams for INFRA-25202
An overview of the core components can be found in the included
TransactionalClusterMetadata.md
patch by Alex Petrov, Marcus Eriksson and Sam Tunnicliffe; reviewed by
Alex Petrov, Marcus Eriksson and Sam Tunnicliffe for CASSANDRA-18330
Co-authored-by: Marcus Eriksson <marcuse@apache.org>
Co-authored-by: Alex Petrov <oleksandr.petrov@gmail.com>
Co-authored-by: Sam Tunnicliffe <samt@apache.org>
- Adds jbellis/jvector (1.0.2) library for DiskANN based indexes on floating point vectors
- Adds ORDER BY ANN OF capability to do ANN search and order the results by score
patch by Mike Adamson; reviewed by Andrés de la Peña, Jonathon Ellis for CASSANDRA-18715
Co-authored-by Jonathon Ellis jbellis@gmail.com
Co-authored-by Zhao Yang zhaoyangsingapore@gmail.com
(git -C … will fall back to parent level .git repo, confusing the check)
patch by Mick Semb Wever; reviewed by Štefan Miklošovič for CASSANDRA-18936,CASSANDRA-18665
- jvm-dtest-upgrade script running in ci-cassandra.a.o (needs default bash)
- add offline mode (to avoid `git fetch` before building dtest-jars on other branches),
- avoid rebuilding in fqltool-test target, as is standard for test targets.
- python detection and override
- FileTest cleanup (tmp directory was left unreadable and undeletable)
- reuse ~/.ccm (like normal)
- remove support for python 3.6
- remove docker volumes in image (they are added at run time)
- remove superfluousjava version debug
– extra mount for build/tmp as hack to prevent buggy hardlinks on overlay
patch by Mick Semb Wever; reviewed by Brandon Williams, Ekaterina Dimitrova, Josh McKenzie, Maxim Muzafarov for CASSANDRA-18665
- Don't create uber jar for microbenchmarks
- Add async profiler to jmh tests
- Benchmark classes names validation
- Add jmh.args property to make it possible passing extra args to JMH
- Add missing test/anttasks to idea configuration
Patch by Jacek Lewandowski; reviewed by Branimir Lambov, Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-18871
Modify unit tests so that any files created are rooted in the directory
specified by the "tmp.dir" Ant property. Ant already passes the value of
tmp.dir to the JVM's java.io.tmpdir parameter, so the primary changes are
to make use of Files.createTempDirectory to generate per-test work
directories. Also update the test checkstyle rules to catch any future
usage of hardcoded "/tmp".
Patch by Derek Chen-Becker; reviewed by bereng and brandonwilliams for CASSANDRA-18750
This patch removes the PRIMARY_KEY_TRIE component and adds KeyLookup.Cursor#clusteredSeekToKey() to
search for clustering keys within a partition. To do this a new on-disk component
PARTITION_SIZES has been added that holds the size of each partition in the SSTable.
patch by Mike Adamson; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-18673
This patch also resolves CASSANDRA-18723 by bumping bcpkix-jdk15on of 1.70
to bcpkix-jdk18on of 1.76.
patch by Stefan Miklosovic; reviewed by Jacek Lewandowski CASSANDRA-18729
- Add Bouncycastle dependency
- Upgrade tcnative boringssl
- Add TLSv1.3 to encryption options tests
- Revert defaults after changes in Netty 4.1.75
- Remove Guava 18 from deps - we accidentally ended with Guava 30+ and 18 on the classpath because JimFS includes it as a transient dependency.
Patch by Jacek Lewandowski and Brandon Williams; reviewed by Ekaterina Dimitrova and Berenguer Blasi for CASSANDRA-17992
Co-authored-by: Jacek Lewandowski <lewandowski.jacek@gmail.com>
Co-authored-by: Brandon Williams <driftx@gmail.com>
patch by Ayushi Singh; reviewed by Stefan Miklosovic, Michael Semb Wever and Maxim Muzafarov for CASSANDRA-18624
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
Notes on the upgrade path:
- RamIndexOutput is replaced with ResettableByteBuffersIndexOutput, an extension of ByteBuffersIndexOutput, which was the closest thing to a replacement of RamIndexOutput.
- Lucene exposes the code we needed from DirectReaders more or less directly in DirectReader now, so the old copied code has been deleted.
- Lucene changed its data files to be little endian, but to keep its compatibility story simple it retained BE for the header and footer ints. That's the cause of the changes in SAICodecUtils.
- We could gain a bit of performance making our own code natively little endian but that is too big of a change for this patch.
patch by Jonathan Ellis; reviewed by Andrés de la Peña, Caleb Rackliffe, and Mike Adamson for CASSANDRA-18494
This includes the following elements of the Storage Attached Index:
- Memtable-attached indexes backed by an in-memory trie structure for byte-comparable values
- Query path for the in-memory index
- Index status propagation
- Randomized testing for Memtable-attached indexes
patch my Mike Adamson; reviewed by Caleb Rackliffe and Andres de la Peña for CASSANDRA-18058
Co-authored-by: Mike Adamson <madamson@datastax.com>
Co-authored-by: Caleb Rackliffe <calebrackliffe@gmail.com>
Bump source and target version to 11 for Java UDFs
ICompilationUnit has new default methods
patch by Ekaterina Dimitrova; reviewed by Andres de la Pena and Berenguer Blasi for CASSANRA-18190
Fixes:
- regular expressions when filtering on tests to run, and exit when no match
- safety when removing (cleaning up) directories (${DIST_DIR} and ${TMPDIR})
- pre-condition `ant jar` has been used and not `ant artifacts`
- compatibility with git worktrees (the original git working directory needs to be mounted in its existing path in docker)
- fix tarball for ccm logs
- add `--keep-failed-test-dir` to dtest-upgrade test types
- only do `docker login` if `docker pull` is going to be called
- unset CASSANDRA_HOME when running python dtests
patch by Mick Semb Wever; reviewed by Brandon Williams, Jacek Lewandowski for CASSANDRA-18567, CASSANDRA-18499
See .build/README.md .Build and test scripts cover all CI activities in an CI agnostic manner. Scripts are found in non-docker and dockerised variants.
The non-docker scripts are found first under .build/ .These interact directly with ant, providing additional checks and environment setup. It is intended that what can be pushed down into the base ant build layer should be. The
debian and redhat build scripts are hidden under .build/docker because they are generally not usuable outside of their corresponding linux dists, and we don't want to encourage anything to the contrary. Bring the artifact/deb/rpm
build scripts (and associated docker images) from cassandra-builds repo to the .build directory. These packaging script for debian and redhat are now separated from the artifacts scripts. An additional build script check-code.sh
has been added that is solely responsible for all linter and code checks (includes owasp dependency checker). Only JDK11 and JDK17 are supported (as JDK8 is soon to be dropped). Scripts do not clean (or realclean) so that
scripts can be better pipelined, but `ant jar` was required to be added to circleci and ci-cassandra.a.o cqlsh-tests. Scripts can define a custom BUILD_DIR (build.dir) which can permit running parrallel builds off the same source
(though not all tests support this, or their support for it has atrophied). All build and test artifacts, results, and logs are found under build/ (or the BUILD_DIR). The build local venv and .ccm directories are now also
created under build (or BUILD_DIR). The test venv, logs and results are wiped each run. Use a separate mktemp directory each run, under /tmp (respect /tmp responsibility, and python fails when tmpdir is on a docker volume).
Don't set CASSANDRA_HOME when running python dtests (and fail-fast if `ant artifacts` was used to build instead of `ant jar`).
The docker scripts are found lower down under .build/docker/ .These scripts re-used the non-docker scripts inside containers. The docker images are versioned controlled in the same directory as dockerfiles, and tagged by their
md5sums. When running the docker scripts these docker images will, if not in the local registry, will first be attempted to be pulled from dockerhub and only then fallback to be built locally. This allows the dockerfiles to be
modified in existing patches and be run in existing CI as-is. Docker scripts timeout after one hour. virtualenv-clone is used, re-using the python versioned venvs in the image saves time over creating new ones each run. The inner-spliting of docker containers has been removed. It added a lot of complexity, for the most part duplicating the splitting concept, for little performance gain.
Ant target 'generate-unified-test-report' added, used for aggregating test reports and printing a summary.
patch by Mick Semb Wever; reviewed by Brandon Williams, Josh McKenzie, Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-18133
Mick Semb Wever