* Add ephemeral-storage requests and limits to agents, and log peak disk usage
Agents were evicted at the node level due to lack of disk space. Each pod needs minimum 100GiB, and each template now budgets 80Gi of the ~89Gi that a 100GiB node allocates, with docker's own volume carries a 60Gi sizeLimit. Exceeding either evicts the pod, so logging has been added in each test split before and after the run. All script blocks in Jenkinsfile are now labeled for improved readability in test runs.
* `.build/run-ci` must not silently drop a site's helm values customisations
Long-lived instances like pre-ci.cassandra.apache.org carry values in addition to the jenkins-deployment.yaml, and any `helm upgrade -f jenkins-deployment.yaml` (or `.build/run-ci`) would drop them. Allow these values to be kept in a separate file, passed in with `--values-override`. Before upgrading helm values, we now compare the deployed values against what is about to be applied and require confirmation for anything only the site holds. This catches added keys and dropped list items (but it cannot catch every change).
Also keep the jenkins-home pvc claim on uninstall, so --only-tear-down no longer destroys build history. Add jenkins-test.sh and a gha workflow for test validating changes under .jenkins/
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov for CASSANDRA-21523
The build+test cells run in `ws("workspace/${JOB_NAME}/${BUILD_NUMBER}/${cell.step}/${cell.arch}/jdk-${cell.jdk}/python-${cell.python}")`, so every build writes a new per-build directory.
The `cleanWs()` only cleans the current cell's leaf, and it sat as the last statement in the `ws` block — so any cell hitting error() (failed/timed-out/aborted) exits before reaching it and leaves the full cell's workspace build behind.
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov for CASSANDRA-20436
Additional fixes
- upgrade ubuntu-test.docker to ubuntu 22.04 (and some small fixes brought forward from CASSANDRA-20997)
- (.jenkins/k8s) changes gke's small node pool from n2-highcpu-8 to e2-highcpu-8 (gke's own pod resources reqs increased)
- (.jenkins/k8s) limit one agent pod per node (when jenkins is deployed in k8s using our helm)
- (.jenkins/k8s) add the cassandra-6.0 job
- pass maven.repo.local sys prop, if set, into accord's gradle build
- check and fail on maven dependencies checksums when downloading
- when on-the-fly building docker images add `--load` (podman compatbility)
- newer python versions can use venv instead of virtualenv
- rename ubuntu2004_test.docker to ubuntu-test.docker (in 5.0)
- (Jenkinsfile) fix retries (which was no longer working when agents/nodes died, since CASSANDRA-20833)
- (Jenkinsfile) @NonCPS where (now) needed
- (Jenkinsfile) fix default value for the branch parameter (after a new helm deploy)
- (.build/run-ci) fix missing parameters on jobs also at helm deploy time
- (.build/run-ci) .git suffixes on repo urls are optional
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov, Stefan Miklosovic for CASSANDRA-21403
Add test container memory events printing to check if Linux OOM killer was active
Fix thread dump printing on junit test timeout
Process.pid() API is available since JDK9
patch by Dmitry Konstantinov; reviewed by Michael Semb Wever for CASSANDRA-21172
Add microbench-test target for quick test run
Unify declaration and add assertions on target types
Fix dirname usages for macos
Parameter for docker_timeout
Add more jmh class ignores (that are broken)
Add disableResume() and better host debug to Jenkinsfile
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov for CASSANDRA-18873
Only read the supported and default jdks once. Also add debug when starting the pipeline on the parameters that will be used for building and testing.
ref: https://the-asf.slack.com/archives/CK23JSY2K/p1763714160975359
patch by Mick Semb Wever; reviewed by Sam Tunnicliffe for CASSANDRA-21044
.build/run-ci is a a python script used to create and interact with k8s provisioned ci-cassandra.apache.org clones
See .build/run-ci.d/README.md for docs on usage.
.jenkins/k8s/ contains the k8s jenkins helm chart, JSaC configuration, and docker image for provisioning.
.build/run-ci.d/ contains python requirements and tests for .build/run-ci
patch by Richa Mishra, Nishant Barola, Aleksandr Volochnev, Mick Semb Wever; reviewed by Richa Mishra, Nishant Barola, Aleksandr Volochnev, Brandon Hsieh, Mick Semb Wever, Brandon Williams for CASSANDRA-18145
Co-authored-by: Richa Mishra <richa.mishra@infracloud.io>
Co-authored-by: Nishant Barola <nishant.barola@infracloud.io>
Co-authored-by: Aleksandr Volochnev <a.volochnev@gmail.com>
Co-authored-by: Mick Semb Wever <mck@apache.org>
More splits for dtest-upgrades and all variants of the (unit) "test" type.
Less splits for long-test, dtest-larges, and jvm-dtest-upgrade.
And some debug on xml test files produced in each split.
patch by Mick Semb Wever; reviewed by Brandon Williams, Maxim Muzafarov, Dmitry Konstantinov for CASSANDRA-20311
Each test type has a -repeat suffixed companion, used with the `-e REPEATED_TESTS_COUNT` option.
patch by Berenguer Blasi; reviewed by Mick Semb Wever, Brandon Williams for CASSANDRA-18942
In docker we want to use the existing populated ~/.ccm/repositories but don't want to write to the container's filesystem (it can leak fd). So we set CCM_CONFIG_DIR to be in the run-python-dtest.XXXXXX tmpdir. Previously this was done in the base (non-docker) script, but this is only needed in docker (and outside of docker we want the normal ~/.ccm usage)
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-19813
- Added debug in testing
- Propagate the $DEBUG env from docker to test scripts
- Be verbose when a pipeline errors that not all tests would have been run
- add timeout to each cell stage (scripts on a node can hang and the only alternative was to abort the whole pipeline)
patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova, Brandon Williams for CASSANDRA-19708
Also
– increase splits for jvm-dtests, container size and docker limits for simulator-dtest
– print exit status in .build/docker/run-tests.sh and .build/docker/_docker_init_tests.sh
– add a summary list of test failures in ci_summary.html
– in ubuntu2004_test.docker make sure /home/cassandra exists and has correct perms (from Marcuse)
– when on ci-cassandra, replace use of copyArtifacts in Jenkinsfile generateTestReports() with manual wget of test files (copyArtifacts is notoriously slow, >4hrs in this case)
– copy ci_summary.html and results_details.tar.xz to nightlies
– capture generateTestReports logs
– stream xz where possible, instead of xz compressing afterwards
– fix ant version in centos7-build.docker
– remove docker login (was meangingless, if credentials exist then docker is logged in)
– prefetch docker images from jfrog (to reduce dockerhub pull rate limits), relates to CASSANDRA-18931
– use scripts from cassandra-builds to clean and report on agents, relates to CASSANDRA-18130
– use groovy elvis operators to avoid NPEs/failures on blank param values
– parallel jfrog pulls and cache it in jfrog too
– pin alpine docker image version
– don't stash .git (was causing AccessDeniedException on some unstash), needed fix so deb/rpm packaging worked on not-git work directories
– durabilityHint needs PERFORMANCE_OPTIMIZED, controller becomes scale bottleneck and breaks without it
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-19558
– ensure test file paths, and their suite names are unique (the latter was broken for cqlshlib and python dtests)
– removal of properties and system-out in test xml happens in CassandraXMLJUnitResultFormatter
– new jenkins agent labels and introduce agents sizes
- ci_summary.html generation script, ref work submitted in apache/cassandra-builds#99
– fix for stress-test and fqltool-test running on small
- ant generate-test-report is limited to only running on individual test types (ci_parser.py provides ci_summary.html for the overview now)
- each cell has a single retry, and the retry will happen on a different agent
- on ci-cassandra the summary stage happens on the builtin, bc copyArtifacts on 15k+ files takes many hours otherwise
- test-burn only needs two splits
- dependency-check is disabled from the lint target until CASSANDRA-19213
- add $DEBUG env var to in-tree scripts, turns on bash debug
- fix FBUtilities' handling of gcp cos_containerd (kernel version comes with a trailing '+' character)
patch by Aleks Volochnev, Mick Semb Wever; reviewed by Aleksandr Volochnev, Josh McKenzie, Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-18594
Co-authored-by: Aleksandr Volochnev <a.volochnev@gmail.com>
Co-authored-by: Mick Semb Wever <mck@apache.org>
Co-authored-by: Josh McKenzie <jmckenzie@apache.org>
Co-authored-by: Artem Chekunov <artem.v.chekunov@gmail.com>
This patch adds a new configuration file, cassandra_latest.yaml, which changes
some of the settings to use new features we usually recommend to users,
including:
- Big Trie-Indexed SSTables
- Trie memtables
- Unified Compaction Strategy
- Storage Attached Index
- Off-heap objects memtables
- Direct-write commit log
- No storage compatibility
These changes are applied in new test targets for unit, in-jvm and python dtests that are
run pre-commit. When changes are made to the new config file, python dtest will use it
directly, but unit and in-jvm dtests require updates, respectively, to latest_diff.yaml
and InstanceConfig.java.
patch by Branimir Lambov; reviewed by Berenguer Blasi for CASSANDRA-18753
attempts to alleviate the false-negatives from CI infrastructure faults (disks, network, etc)
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-17664
Mick Semb Wever