In docker we want to use the existing populated ~/.ccm/repositories but don't want to write to the container's filesystem (it can leak fd). So we set CCM_CONFIG_DIR to be in the run-python-dtest.XXXXXX tmpdir. Previously this was done in the base (non-docker) script, but this is only needed in docker (and outside of docker we want the normal ~/.ccm usage)
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-19813
- Added debug in testing
- Propagate the $DEBUG env from docker to test scripts
- Be verbose when a pipeline errors that not all tests would have been run
- add timeout to each cell stage (scripts on a node can hang and the only alternative was to abort the whole pipeline)
patch by Mick Semb Wever; reviewed by Ekaterina Dimitrova, Brandon Williams for CASSANDRA-19708
Also
– increase splits for jvm-dtests, container size and docker limits for simulator-dtest
– print exit status in .build/docker/run-tests.sh and .build/docker/_docker_init_tests.sh
– add a summary list of test failures in ci_summary.html
– in ubuntu2004_test.docker make sure /home/cassandra exists and has correct perms (from Marcuse)
– when on ci-cassandra, replace use of copyArtifacts in Jenkinsfile generateTestReports() with manual wget of test files (copyArtifacts is notoriously slow, >4hrs in this case)
– copy ci_summary.html and results_details.tar.xz to nightlies
– capture generateTestReports logs
– stream xz where possible, instead of xz compressing afterwards
– fix ant version in centos7-build.docker
– remove docker login (was meangingless, if credentials exist then docker is logged in)
– prefetch docker images from jfrog (to reduce dockerhub pull rate limits), relates to CASSANDRA-18931
– use scripts from cassandra-builds to clean and report on agents, relates to CASSANDRA-18130
– use groovy elvis operators to avoid NPEs/failures on blank param values
– parallel jfrog pulls and cache it in jfrog too
– pin alpine docker image version
– don't stash .git (was causing AccessDeniedException on some unstash), needed fix so deb/rpm packaging worked on not-git work directories
– durabilityHint needs PERFORMANCE_OPTIMIZED, controller becomes scale bottleneck and breaks without it
patch by Mick Semb Wever; reviewed by Brandon Williams for CASSANDRA-19558
– ensure test file paths, and their suite names are unique (the latter was broken for cqlshlib and python dtests)
– removal of properties and system-out in test xml happens in CassandraXMLJUnitResultFormatter
– new jenkins agent labels and introduce agents sizes
- ci_summary.html generation script, ref work submitted in apache/cassandra-builds#99
– fix for stress-test and fqltool-test running on small
- ant generate-test-report is limited to only running on individual test types (ci_parser.py provides ci_summary.html for the overview now)
- each cell has a single retry, and the retry will happen on a different agent
- on ci-cassandra the summary stage happens on the builtin, bc copyArtifacts on 15k+ files takes many hours otherwise
- test-burn only needs two splits
- dependency-check is disabled from the lint target until CASSANDRA-19213
- add $DEBUG env var to in-tree scripts, turns on bash debug
- fix FBUtilities' handling of gcp cos_containerd (kernel version comes with a trailing '+' character)
patch by Aleks Volochnev, Mick Semb Wever; reviewed by Aleksandr Volochnev, Josh McKenzie, Maxim Muzafarov, Stefan Miklosovic for CASSANDRA-18594
Co-authored-by: Aleksandr Volochnev <a.volochnev@gmail.com>
Co-authored-by: Mick Semb Wever <mck@apache.org>
Co-authored-by: Josh McKenzie <jmckenzie@apache.org>
Co-authored-by: Artem Chekunov <artem.v.chekunov@gmail.com>
This patch adds a new configuration file, cassandra_latest.yaml, which changes
some of the settings to use new features we usually recommend to users,
including:
- Big Trie-Indexed SSTables
- Trie memtables
- Unified Compaction Strategy
- Storage Attached Index
- Off-heap objects memtables
- Direct-write commit log
- No storage compatibility
These changes are applied in new test targets for unit, in-jvm and python dtests that are
run pre-commit. When changes are made to the new config file, python dtest will use it
directly, but unit and in-jvm dtests require updates, respectively, to latest_diff.yaml
and InstanceConfig.java.
patch by Branimir Lambov; reviewed by Berenguer Blasi for CASSANDRA-18753
patch by Claude Warren; reviewed by Stefan Miklosovic, Jacek Lewandowski, Michael Semb Wever for CASSANDRA-16565
Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
Josh McKenzie