* Add ephemeral-storage requests and limits to agents, and log peak disk usage
Agents were evicted at the node level due to lack of disk space. Each pod needs minimum 100GiB, and each template now budgets 80Gi of the ~89Gi that a 100GiB node allocates, with docker's own volume carries a 60Gi sizeLimit. Exceeding either evicts the pod, so logging has been added in each test split before and after the run. All script blocks in Jenkinsfile are now labeled for improved readability in test runs.
* `.build/run-ci` must not silently drop a site's helm values customisations
Long-lived instances like pre-ci.cassandra.apache.org carry values in addition to the jenkins-deployment.yaml, and any `helm upgrade -f jenkins-deployment.yaml` (or `.build/run-ci`) would drop them. Allow these values to be kept in a separate file, passed in with `--values-override`. Before upgrading helm values, we now compare the deployed values against what is about to be applied and require confirmation for anything only the site holds. This catches added keys and dropped list items (but it cannot catch every change).
Also keep the jenkins-home pvc claim on uninstall, so --only-tear-down no longer destroys build history. Add jenkins-test.sh and a gha workflow for test validating changes under .jenkins/
patch by Mick Semb Wever; reviewed by Dmitry Konstantinov for CASSANDRA-21523
See .build/README.md .Build and test scripts cover all CI activities in an CI agnostic manner. Scripts are found in non-docker and dockerised variants.
The non-docker scripts are found first under .build/ .These interact directly with ant, providing additional checks and environment setup. It is intended that what can be pushed down into the base ant build layer should be. The
debian and redhat build scripts are hidden under .build/docker because they are generally not usuable outside of their corresponding linux dists, and we don't want to encourage anything to the contrary. Bring the artifact/deb/rpm
build scripts (and associated docker images) from cassandra-builds repo to the .build directory. These packaging script for debian and redhat are now separated from the artifacts scripts. An additional build script check-code.sh
has been added that is solely responsible for all linter and code checks (includes owasp dependency checker). Only JDK11 and JDK17 are supported (as JDK8 is soon to be dropped). Scripts do not clean (or realclean) so that
scripts can be better pipelined, but `ant jar` was required to be added to circleci and ci-cassandra.a.o cqlsh-tests. Scripts can define a custom BUILD_DIR (build.dir) which can permit running parrallel builds off the same source
(though not all tests support this, or their support for it has atrophied). All build and test artifacts, results, and logs are found under build/ (or the BUILD_DIR). The build local venv and .ccm directories are now also
created under build (or BUILD_DIR). The test venv, logs and results are wiped each run. Use a separate mktemp directory each run, under /tmp (respect /tmp responsibility, and python fails when tmpdir is on a docker volume).
Don't set CASSANDRA_HOME when running python dtests (and fail-fast if `ant artifacts` was used to build instead of `ant jar`).
The docker scripts are found lower down under .build/docker/ .These scripts re-used the non-docker scripts inside containers. The docker images are versioned controlled in the same directory as dockerfiles, and tagged by their
md5sums. When running the docker scripts these docker images will, if not in the local registry, will first be attempted to be pulled from dockerhub and only then fallback to be built locally. This allows the dockerfiles to be
modified in existing patches and be run in existing CI as-is. Docker scripts timeout after one hour. virtualenv-clone is used, re-using the python versioned venvs in the image saves time over creating new ones each run. The inner-spliting of docker containers has been removed. It added a lot of complexity, for the most part duplicating the splitting concept, for little performance gain.
Ant target 'generate-unified-test-report' added, used for aggregating test reports and printing a summary.
patch by Mick Semb Wever; reviewed by Brandon Williams, Josh McKenzie, Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-18133
This work encompasses a range of improvements to Paxos, summarised as:
- The introduction of Paxos Repair for ensuring consistency during range movements
- The reduction of network round-trips by a factor of 2x for reads and writes
patch by Benedict Elliott Smith and Blake Eggleston; reviewed by Alex Petrov, Aleksey Yeschenko and Sam Tunnicliffe for CASSANDRA-17164
Patch by Dmitry Potepalov; reviewed by bereng and brandonwilliams for
CASSANDRA-17367
Encryption settings were silently lost in BulkLoadConnectionFactory.
Forced the bulk loader test to do some actual streaming besides just
handshaking on the native port, the test does not pass without the
change in BulkLoadConnectionFactory.
Patch by Dmitry Potepalov; reviewed by bereng and brandonwilliams for
CASSANDRA-17367
Encryption settings were silently lost in BulkLoadConnectionFactory.
Forced the bulk loader test to do some actual streaming besides just
handshaking on the native port, the test does not pass without the
change in BulkLoadConnectionFactory.
Fix the DB descriptorRefTest which failed on the previous commit
patch by Ekaterina Dimitrova; reviewed by Caleb Rackliffe, David Capwell, Michael Semb Wever and Benjamin Lerer for CASSANDRA-15234
This commit sets up a new documentation structure and format:
* The directory struture changes from a Sphinx project to an Antora module layout.
* The formatting of the content changes from reStructuredText to AsciiDoc.
The documentation must now be built and published with Antora. Initially only from the cassandra-website repository.
This change was done to make maintaining versioned website documentation easier. As Antora is designed to generate versioned documentation.
The old directory structure was:
<ROOT>
- doc/
- cql3/
- CQL.css
- CQL.textile
- source/
- _static/
- _templates/
- _theme/
- _util/
- <other directory sections>/
- conf.py
- index.rst
- <other *.rst pages>
- make.bat
- Makefile
- README.md
- SASI.md
- <*.spec files>
- <generation scripts>
The new directory structure organises the documentation into modules:
<ROOT>
- doc/
- cql3
- modules/
- cassandra/
- assets/
- examples/
- pages/
- partials/
- nav.adoc
- ROOT/
- pages/
- nav.adoc
- scripts/
- antora.yaml
- README.md
- SASI.md
patch by Lorina Poland, Anthony Grasso; reviewed by Anthony Grasso, Mick Semb Wever for CASSANDRA-16763
Co-authored-by: Anthony Grasso <anthony@thelastpickle.com>
- This removes lib/ from version control.
- The generated pom files are now the source of truth to the project's depdendencies, and the lib/ folder re-created during the build from compile scope dependencies excluding provided and system scopes. Project dependencies are first downloaded to the maven local repository at `~/.m2/repository`.
- Maven is now required to be installed to build, that is the `mvn` command needs to be found in the path.
- The new resolver-ant-tasks targets have been encapsulated into the `build/build-resolver.xml` ant file.
- Most classpaths in build.xml are replaced with resolver:resolve definitions.
- The build-deps-pom generated pom file is now only used for declaring test dependencies.
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-16557
- This removes lib/ from version control.
- The generated pom files are now the source of truth to the project's depdendencies, and the lib/ folder re-created during the build from compile scope dependencies excluding provided and system scopes. Project dependencies are first downloaded to the maven local repository at `~/.m2/repository`.
- Maven is now required to be installed to build, that is the `mvn` command needs to be found in the path.
- The new resolver-ant-tasks targets have been encapsulated into the `build/build-resolver.xml` ant file.
- Most classpaths in build.xml are replaced with resolver:resolve definitions.
- The build-deps-pom generated pom file is now only used for declaring test dependencies.
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-16557
* Don't run rat check if git fails or not available
* Check for presence of rat.txt before running eclipse-warnings task
(to ensure that the report is always generated by CI runs)
Patch by Sam Tunnicliffe; reviewed by Mick Semb Wever for CASSANDRA-16633
Mick Semb Wever