Additional fixes
- upgrade ubuntu-test.docker to ubuntu 22.04 (and some small fixes brought forward from CASSANDRA-20997)
- (.jenkins/k8s) changes gke's small node pool from n2-highcpu-8 to e2-highcpu-8 (gke's own pod resources reqs increased)
- (.jenkins/k8s) limit one agent pod per node (when jenkins is deployed in k8s using our helm)
- (.jenkins/k8s) add the cassandra-6.0 job
- pass maven.repo.local sys prop, if set, into accord's gradle build
- check and fail on maven dependencies checksums when downloading
- when on-the-fly building docker images add `--load` (podman compatbility)
- newer python versions can use venv instead of virtualenv
- rename ubuntu2004_test.docker to ubuntu-test.docker (in 5.0)
- (Jenkinsfile) fix retries (which was no longer working when agents/nodes died, since CASSANDRA-20833)
- (Jenkinsfile) @NonCPS where (now) needed
- (Jenkinsfile) fix default value for the branch parameter (after a new helm deploy)
- (.build/run-ci) fix missing parameters on jobs also at helm deploy time
- (.build/run-ci) .git suffixes on repo urls are optional
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov, Stefan Miklosovic for CASSANDRA-21403
Add microbench-test target for quick test run
Unify declaration and add assertions on target types
Fix dirname usages for macos
Parameter for docker_timeout
Add more jmh class ignores (that are broken)
Add disableResume() and better host debug to Jenkinsfile
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov for CASSANDRA-18873
Each test type has a -repeat suffixed companion, used with the `-e REPEATED_TESTS_COUNT` option.
patch by Berenguer Blasi; reviewed by Mick Semb Wever, Brandon Williams for CASSANDRA-18942
In docker we want to use the existing populated ~/.ccm/repositories but don't want to write to the container's filesystem (it can leak fd). So we set CCM_CONFIG_DIR to be in the run-python-dtest.XXXXXX tmpdir. Previously this was done in the base (non-docker) script, but this is only needed in docker (and outside of docker we want the normal ~/.ccm usage)
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-19813
- Added debug in testing
- Propagate the $DEBUG env from docker to test scripts
- Be verbose when a pipeline errors that not all tests would have been run
- add timeout to each cell stage (scripts on a node can hang and the only alternative was to abort the whole pipeline)
patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova, Brandon Williams for CASSANDRA-19708
– ensure test file paths, and their suite names are unique (the latter was broken for cqlshlib and python dtests)
– removal of properties and system-out in test xml happens in CassandraXMLJUnitResultFormatter
– new jenkins agent labels and introduce agents sizes
- ci_summary.html generation script, ref work submitted in apache/cassandra-builds#99
– fix for stress-test and fqltool-test running on small
- ant generate-test-report is limited to only running on individual test types (ci_parser.py provides ci_summary.html for the overview now)
- each cell has a single retry, and the retry will happen on a different agent
- on ci-cassandra the summary stage happens on the builtin, bc copyArtifacts on 15k+ files takes many hours otherwise
- test-burn only needs two splits
- dependency-check is disabled from the lint target until CASSANDRA-19213
- add $DEBUG env var to in-tree scripts, turns on bash debug
- fix FBUtilities' handling of gcp cos_containerd (kernel version comes with a trailing '+' character)
patch by Aleks Volochnev, Mick Semb Wever; reviewed by Aleksandr Volochnev, Josh McKenzie, Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-18594
Co-authored-by: Aleksandr Volochnev <a.volochnev@gmail.com>
Co-authored-by: Mick Semb Wever <mck@apache.org>
Co-authored-by: Josh McKenzie <jmckenzie@apache.org>
Co-authored-by: Artem Chekunov <artem.v.chekunov@gmail.com>
This prevents docker overlay file leakage from containers by keeping write-heavy directories in bind mounts (or volumes). It also offers better performance by avoiding the container filesystem.
patch by Mick Semb Wever; reviewed Brandon Williams for INFRA-25202
- jvm-dtest-upgrade script running in ci-cassandra.a.o (needs default bash)
- add offline mode (to avoid `git fetch` before building dtest-jars on other branches),
- avoid rebuilding in fqltool-test target, as is standard for test targets.
- python detection and override
- FileTest cleanup (tmp directory was left unreadable and undeletable)
- reuse ~/.ccm (like normal)
- remove support for python 3.6
- remove docker volumes in image (they are added at run time)
- remove superfluousjava version debug
– extra mount for build/tmp as hack to prevent buggy hardlinks on overlay
patch by Mick Semb Wever; reviewed by Brandon Williams, Ekaterina Dimitrova, Josh McKenzie, Maxim Muzafarov for CASSANDRA-18665
Fixes:
- regular expressions when filtering on tests to run, and exit when no match
- safety when removing (cleaning up) directories (${DIST_DIR} and ${TMPDIR})
- pre-condition `ant jar` has been used and not `ant artifacts`
- compatibility with git worktrees (the original git working directory needs to be mounted in its existing path in docker)
- fix tarball for ccm logs
- add `--keep-failed-test-dir` to dtest-upgrade test types
- only do `docker login` if `docker pull` is going to be called
- unset CASSANDRA_HOME when running python dtests
patch by Mick Semb Wever; reviewed by Brandon Williams, Jacek Lewandowski for CASSANDRA-18567, CASSANDRA-18499
See .build/README.md .Build and test scripts cover all CI activities in an CI agnostic manner. Scripts are found in non-docker and dockerised variants.
The non-docker scripts are found first under .build/ .These interact directly with ant, providing additional checks and environment setup. It is intended that what can be pushed down into the base ant build layer should be. The
debian and redhat build scripts are hidden under .build/docker because they are generally not usuable outside of their corresponding linux dists, and we don't want to encourage anything to the contrary. Bring the artifact/deb/rpm
build scripts (and associated docker images) from cassandra-builds repo to the .build directory. These packaging script for debian and redhat are now separated from the artifacts scripts. An additional build script check-code.sh
has been added that is solely responsible for all linter and code checks (includes owasp dependency checker). Only JDK11 and JDK17 are supported (as JDK8 is soon to be dropped). Scripts do not clean (or realclean) so that
scripts can be better pipelined, but `ant jar` was required to be added to circleci and ci-cassandra.a.o cqlsh-tests. Scripts can define a custom BUILD_DIR (build.dir) which can permit running parrallel builds off the same source
(though not all tests support this, or their support for it has atrophied). All build and test artifacts, results, and logs are found under build/ (or the BUILD_DIR). The build local venv and .ccm directories are now also
created under build (or BUILD_DIR). The test venv, logs and results are wiped each run. Use a separate mktemp directory each run, under /tmp (respect /tmp responsibility, and python fails when tmpdir is on a docker volume).
Don't set CASSANDRA_HOME when running python dtests (and fail-fast if `ant artifacts` was used to build instead of `ant jar`).
The docker scripts are found lower down under .build/docker/ .These scripts re-used the non-docker scripts inside containers. The docker images are versioned controlled in the same directory as dockerfiles, and tagged by their
md5sums. When running the docker scripts these docker images will, if not in the local registry, will first be attempted to be pulled from dockerhub and only then fallback to be built locally. This allows the dockerfiles to be
modified in existing patches and be run in existing CI as-is. Docker scripts timeout after one hour. virtualenv-clone is used, re-using the python versioned venvs in the image saves time over creating new ones each run. The inner-spliting of docker containers has been removed. It added a lot of complexity, for the most part duplicating the splitting concept, for little performance gain.
Ant target 'generate-unified-test-report' added, used for aggregating test reports and printing a summary.
patch by Mick Semb Wever; reviewed by Brandon Williams, Josh McKenzie, Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-18133
mck