Additional fixes
- upgrade ubuntu-test.docker to ubuntu 22.04 (and some small fixes brought forward from CASSANDRA-20997)
- (.jenkins/k8s) changes gke's small node pool from n2-highcpu-8 to e2-highcpu-8 (gke's own pod resources reqs increased)
- (.jenkins/k8s) limit one agent pod per node (when jenkins is deployed in k8s using our helm)
- (.jenkins/k8s) add the cassandra-6.0 job
- pass maven.repo.local sys prop, if set, into accord's gradle build
- check and fail on maven dependencies checksums when downloading
- when on-the-fly building docker images add `--load` (podman compatbility)
- newer python versions can use venv instead of virtualenv
- rename ubuntu2004_test.docker to ubuntu-test.docker (in 5.0)
- (Jenkinsfile) fix retries (which was no longer working when agents/nodes died, since CASSANDRA-20833)
- (Jenkinsfile) @NonCPS where (now) needed
- (Jenkinsfile) fix default value for the branch parameter (after a new helm deploy)
- (.build/run-ci) fix missing parameters on jobs also at helm deploy time
- (.build/run-ci) .git suffixes on repo urls are optional
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov, Stefan Miklosovic for CASSANDRA-21403
Add microbench-test target for quick test run
Unify declaration and add assertions on target types
Fix dirname usages for macos
Parameter for docker_timeout
Add more jmh class ignores (that are broken)
Add disableResume() and better host debug to Jenkinsfile
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov for CASSANDRA-18873
.build/run-ci is a a python script used to create and interact with k8s provisioned ci-cassandra.apache.org clones
See .build/run-ci.d/README.md for docs on usage.
.jenkins/k8s/ contains the k8s jenkins helm chart, JSaC configuration, and docker image for provisioning.
.build/run-ci.d/ contains python requirements and tests for .build/run-ci
patch by Richa Mishra, Nishant Barola, Aleksandr Volochnev, Mick Semb Wever; reviewed by Richa Mishra, Nishant Barola, Aleksandr Volochnev, Brandon Hsieh, Mick Semb Wever, Brandon Williams for CASSANDRA-18145
Co-authored-by: Richa Mishra <richa.mishra@infracloud.io>
Co-authored-by: Nishant Barola <nishant.barola@infracloud.io>
Co-authored-by: Aleksandr Volochnev <a.volochnev@gmail.com>
Co-authored-by: Mick Semb Wever <mck@apache.org>
As part of a broader effort to decouple java driver code from the
server code, this moves sstableloader to its own tools directory.
As sstableloader is also used as a library (CASSANDRA-10637), added
a new artifact 'cassandra-sstableloader' that will get deployed to
maven along with 'cassandra-all'.
While I expect this is likely a niche use case, this will allow users
to continue using BulkExport as a library.
Moves sstableloader-specific targets to its own build.xml in
tools/sstableloader/build.xml.
Also updates IDE project files and circleci to utilize new
sstableloader-specific targets.
patch by Andy Tolbert; reviewed by Stefan Miklosovic and Mick Semb Wever for CASSANDRA-20328
Each test type has a -repeat suffixed companion, used with the `-e REPEATED_TESTS_COUNT` option.
patch by Berenguer Blasi; reviewed by Mick Semb Wever, Brandon Williams for CASSANDRA-18942
In docker we want to use the existing populated ~/.ccm/repositories but don't want to write to the container's filesystem (it can leak fd). So we set CCM_CONFIG_DIR to be in the run-python-dtest.XXXXXX tmpdir. Previously this was done in the base (non-docker) script, but this is only needed in docker (and outside of docker we want the normal ~/.ccm usage)
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-19813
- Added debug in testing
- Propagate the $DEBUG env from docker to test scripts
- Be verbose when a pipeline errors that not all tests would have been run
- add timeout to each cell stage (scripts on a node can hang and the only alternative was to abort the whole pipeline)
patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova, Brandon Williams for CASSANDRA-19708
Also
– increase splits for jvm-dtests, container size and docker limits for simulator-dtest
– print exit status in .build/docker/run-tests.sh and .build/docker/_docker_init_tests.sh
– add a summary list of test failures in ci_summary.html
– in ubuntu2004_test.docker make sure /home/cassandra exists and has correct perms (from Marcuse)
– when on ci-cassandra, replace use of copyArtifacts in Jenkinsfile generateTestReports() with manual wget of test files (copyArtifacts is notoriously slow, >4hrs in this case)
– copy ci_summary.html and results_details.tar.xz to nightlies
– capture generateTestReports logs
– stream xz where possible, instead of xz compressing afterwards
– fix ant version in centos7-build.docker
– remove docker login (was meangingless, if credentials exist then docker is logged in)
– prefetch docker images from jfrog (to reduce dockerhub pull rate limits), relates to CASSANDRA-18931
– use scripts from cassandra-builds to clean and report on agents, relates to CASSANDRA-18130
– use groovy elvis operators to avoid NPEs/failures on blank param values
– parallel jfrog pulls and cache it in jfrog too
– pin alpine docker image version
– don't stash .git (was causing AccessDeniedException on some unstash), needed fix so deb/rpm packaging worked on not-git work directories
– durabilityHint needs PERFORMANCE_OPTIMIZED, controller becomes scale bottleneck and breaks without it
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-19558
– ensure test file paths, and their suite names are unique (the latter was broken for cqlshlib and python dtests)
– removal of properties and system-out in test xml happens in CassandraXMLJUnitResultFormatter
– new jenkins agent labels and introduce agents sizes
- ci_summary.html generation script, ref work submitted in apache/cassandra-builds#99
– fix for stress-test and fqltool-test running on small
- ant generate-test-report is limited to only running on individual test types (ci_parser.py provides ci_summary.html for the overview now)
- each cell has a single retry, and the retry will happen on a different agent
- on ci-cassandra the summary stage happens on the builtin, bc copyArtifacts on 15k+ files takes many hours otherwise
- test-burn only needs two splits
- dependency-check is disabled from the lint target until CASSANDRA-19213
- add $DEBUG env var to in-tree scripts, turns on bash debug
- fix FBUtilities' handling of gcp cos_containerd (kernel version comes with a trailing '+' character)
patch by Aleks Volochnev, Mick Semb Wever; reviewed by Aleksandr Volochnev, Josh McKenzie, Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-18594
Co-authored-by: Aleksandr Volochnev <a.volochnev@gmail.com>
Co-authored-by: Mick Semb Wever <mck@apache.org>
Co-authored-by: Josh McKenzie <jmckenzie@apache.org>
Co-authored-by: Artem Chekunov <artem.v.chekunov@gmail.com>
This patch adds a new configuration file, cassandra_latest.yaml, which changes
some of the settings to use new features we usually recommend to users,
including:
- Big Trie-Indexed SSTables
- Trie memtables
- Unified Compaction Strategy
- Storage Attached Index
- Off-heap objects memtables
- Direct-write commit log
- No storage compatibility
These changes are applied in new test targets for unit, in-jvm and python dtests that are
run pre-commit. When changes are made to the new config file, python dtest will use it
directly, but unit and in-jvm dtests require updates, respectively, to latest_diff.yaml
and InstanceConfig.java.
patch by Branimir Lambov; reviewed by Berenguer Blasi for CASSANDRA-18753
This prevents docker overlay file leakage from containers by keeping write-heavy directories in bind mounts (or volumes). It also offers better performance by avoiding the container filesystem.
patch by Mick Semb Wever; reviewed Brandon Williams for INFRA-25202
- jvm-dtest-upgrade script running in ci-cassandra.a.o (needs default bash)
- add offline mode (to avoid `git fetch` before building dtest-jars on other branches),
- avoid rebuilding in fqltool-test target, as is standard for test targets.
- python detection and override
- FileTest cleanup (tmp directory was left unreadable and undeletable)
- reuse ~/.ccm (like normal)
- remove support for python 3.6
- remove docker volumes in image (they are added at run time)
- remove superfluousjava version debug
– extra mount for build/tmp as hack to prevent buggy hardlinks on overlay
patch by Mick Semb Wever; reviewed by Brandon Williams, Ekaterina Dimitrova, Josh McKenzie, Maxim Muzafarov for CASSANDRA-18665
mck