cassandra/.circleci

CircleCI config files

This directory contains the configuration for CircleCI continous integration platform. The file config.yml is the configuration file that is read by CircleCI. This file is automatically generated by the generate.sh script from the config_template.yml file. Experimental JDK17 configuration is added for test purposes to enable easier testing while working on JDK17 related issues. The file config_11_and_17.yml is the configuration file that we can copy over config.yml to test JDK11+17 workflows. This file is automatically generated by the generate_11_and_17.sh script from the config_template_11_and_17.yml file.

The provided config.yml file uses low resources so users of the CircleCI free tier can use it. Additionally, there are two versions of this file using different resources so users who have access to premium CircleCI resources can use larger instances and more parallelism. These files are config.yml.FREE and config.yml.PAID. The default config.yml file is just a copy of config.yml.FREE. For the JDK11+17 workflows we have equivalent files - config_11_and_17.yml.FREE and config_11_and_17.yml.PAID.

Switching to higher resource settings

This directory contains generated files for free and paid resource settings. Switch between them by copying the correct file to config.yml and committing the result. For example, for using higher resources benefiting from CircleCI's paid tier:

cp .circleci/config.yml.PAID .circleci/config.yml

respectively for JDK11+17:

cp .circleci/config_11_and_17.yml.PAID .circleci/config.yml

And for using lower resources comaptible with CircleCI's free tier:

cp .circleci/config.yml.FREE .circleci/config.yml

respectively for JDK11+17:

cp .circleci/config_11_and_17.yml.FREE .circleci/config.yml

Alternatively, you can run the generate.sh script with the flags -f/-p to regenerate the config.yml file from config_template.yml using free or paid resources. For JDK11 and 17 you can run the generate_11_and_17.sh script with the flags -f/-p to regenerate the config.yml file from config_template_11_and_17.yml using free or paid resources. The two scripts validate and apply any changes to the config_template.ymland config_template_11_and_17.yml files, and they require the CircleCI CLI to be installed.

Setting environment variables

Both config_template.yml and config.yml files contain a set of environment variables defining things like what dtest repo and branch to use, what tests could be repeatedly run, etc. Same applies for config_template_11_and_17.yml and config_11_and_17.yml.

These environment variables can be directly edited in the config.yml file, although if you do this you should take into account that the entire set of env vars is repeated on every job.

A probably better approach is editing them in config_template.yml or respectively config_template_11_and_17.yml and then regenerate the config.yml and config_11_and_17.yml files using the generate.sh or respectively the generate_11_and_17.yml script. You can also directly pass environment variable values to the generate.sh and generate_11_and_17.sh scripts with the -e flag. For example, to set the dtest repo and branch with PAID config you can run:

generate.sh -p \
  -e DTEST_REPO=https://github.com/adelapena/cassandra-dtest.git \
  -e DTEST_BRANCH=CASSANDRA-8272

Running tests in a loop

Running the generate.sh or generate_11_and_17.sh script will use git diff to find the new or modified tests. The scripts will then create jobs to run each of these new or modified tests for a certain number of times, to verify that they are stable. You can use environment variables to specify the number of iterations of each type of test:

generate.sh -p \
  -e REPEATED_UTESTS_COUNT=500 \
  -e REPEATED_UTESTS_FQLTOOL_COUNT=500 \
  -e REPEATED_UTESTS_LONG_COUNT=100 \
  -e REPEATED_UTESTS_STRESS_COUNT=500 \
  -e REPEATED_SIMULATOR_DTESTS_COUNT=500 \
  -e REPEATED_JVM_DTESTS_COUNT=500 \
  -e REPEATED_JVM_UPGRADE_DTESTS_COUNT=500 \
  -e REPEATED_DTESTS_COUNT=500 \
  -e REPEATED_LARGE_DTESTS_COUNT=100 \
  -e REPEATED_UPGRADE_DTESTS_COUNT=25 \
  -e REPEATED_ANT_TEST_COUNT=500

You can also specify whether the iteration should fail on the first test failure:

generate.sh -p -e REPEATED_TESTS_STOP_ON_FAILURE=false

In addition to the automatically detected tests, it's also possible to provide lists of specific tests to be repeated:

generate.sh -p \
  -e REPEATED_UTESTS=org.apache.cassandra.cql3.ViewTest,org.apache.cassandra.db.CellTest \
  -e REPEATED_UTESTS_FQLTOOL=org.apache.cassandra.fqltool.FQLCompareTest \
  -e REPEATED_UTESTS_LONG=org.apache.cassandra.io.sstable.CQLSSTableWriterLongTest#testWideRow \
  -e REPEATED_UTESTS_STRESS=org.apache.cassandra.stress.generate.DistributionGaussianTest \
  -e REPEATED_SIMULATOR_DTESTS=org.apache.cassandra.simulator.test.TrivialSimulationTest \
  -e REPEATED_DTESTS=cql_test.py,consistency_test.py::TestAvailability::test_simple_strategy \
  -e REPEATED_LARGE_DTESTS=replace_address_test.py::TestReplaceAddress::test_replace_stopped_node \
  -e REPEATED_JVM_DTESTS=org.apache.cassandra.distributed.test.PagingTest#testPaging \
  -e REPEATED_UPGRADE_DTESTS=upgrade_tests/cql_tests.py \
  -e REPEATED_JVM_UPGRADE_DTESTS=org.apache.cassandra.distributed.upgrade.GroupByTest

If you need to repeat a particular set of tests and avoid automatic detection of changed tests use the -s command line option.

For particular Ant test targets that are not included in the regular test suites, you can use the run_repeated_utest job:

generate.sh -p \
  -e REPEATED_ANT_TEST_TARGET=test-cdc \
  -e REPEATED_ANT_TEST_CLASS=org.apache.cassandra.cql3.ViewTest \
  -e REPEATED_ANT_TEST_METHODS=testCompoundPartitionKey,testStaticTable \
  -e REPEATED_ANT_TEST_VNODES=false \
  -e REPEATED_ANT_TEST_COUNT=500

Putting all together, you can have runs as complex as:

generate.sh -p \
  -e REPEATED_TESTS_STOP_ON_FAILURE=true \
  -e REPEATED_UTESTS=org.apache.cassandra.cql3.ViewTest,org.apache.cassandra.db.CellTest \
  -e REPEATED_UTESTS_COUNT=500 \
  -e REPEATED_UTESTS_FQLTOOL=org.apache.cassandra.fqltool.FQLCompareTest \
  -e REPEATED_UTESTS_FQLTOOL_COUNT=500 \
  -e REPEATED_UTESTS_LONG=org.apache.cassandra.io.sstable.CQLSSTableWriterLongTest#testWideRow \
  -e REPEATED_UTESTS_LONG_COUNT=100 \
  -e REPEATED_UTESTS_STRESS=org.apache.cassandra.stress.generate.DistributionGaussianTest \
  -e REPEATED_UTESTS_STRESS_COUNT=500 \
  -e REPEATED_SIMULATOR_DTESTS=org.apache.cassandra.simulator.test.TrivialSimulationTest \
  -e REPEATED_SIMULATOR_DTESTS_COUNT=500 \
  -e REPEATED_DTESTS=cql_test.py,consistency_test.py::TestAvailability::test_simple_strategy \
  -e REPEATED_DTESTS_COUNT=500 \
  -e REPEATED_LARGE_DTESTS=replace_address_test.py,materialized_views_test.py \
  -e REPEATED_LARGE_DTESTS_COUNT=100 \
  -e REPEATED_JVM_DTESTS=org.apache.cassandra.distributed.test.PagingTest#testPaging \
  -e REPEATED_JVM_DTESTS_COUNT=500 \
  -e REPEATED_UPGRADE_DTESTS=upgrade_tests/cql_tests.py \
  -e REPEATED_UPGRADE_DTESTS_COUNT=25 \
  -e REPEATED_JVM_UPGRADE_DTESTS=org.apache.cassandra.distributed.upgrade.GroupByTest \
  -e REPEATED_JVM_UPGRADE_DTESTS_COUNT=500 \
  -e REPEATED_ANT_TEST_TARGET=test-cdc \
  -e REPEATED_ANT_TEST_CLASS=org.apache.cassandra.cql3.ViewTest \
  -e REPEATED_ANT_TEST_METHODS=testCompoundPartitionKey,testStaticTable \
  -e REPEATED_ANT_TEST_VNODES=false \
  -e REPEATED_ANT_TEST_COUNT=500

WARNING: The Python large DTests test_network_topology_strategy and test_network_topology_strategy_each_quorum require XLarge containers in order to be run in a loop. Otherwise, you would see NO HOST AVAILABLE if you run them with the default medium config and Large containers.

Updating the config

For configuration changes meant to be permanent in the Apache repo you should never edit the config.yml file manually. Instead, you should edit the config_template.yml file and then regenerate the config.yml, config.yml.FREE and config.yml.PAID files by runnining the generate.sh script with -a flag. For using this script you need to install the CircleCI CLI. Same applies for the equivalent JDK11+17 config files and generate_11_and_17.sh.

As for temporal changes done while working in a patch, such as pointing to you dtest repo or running a test repeatedly, you can either directly edit config.yml or edit config_template.yml and then regenerate config.yml with the generate.sh script using a -f/-p flag. When this flag is used only the config.yml will be generated. Same workflow applies to the respective JDK11+17 CircleCI configuration files.

Please note that any previous swapping or edition of the generated files will be overriden by running generate.sh/generate_11_and_17.sh with -a argument, returning config.yml to the default FREE. So if you previously swapped your config.yml to PAID you would need to either swap it again or use the -f/-p script flags.

Read below for details how to generate the files manually without the generate.sh script:

1. make your edits to config_template.yml - let it stay at free tier settings
2. generate a valid FREE file: circleci config process config_template.yml > config.yml.FREE
3. add the Apache license header to the newly created FREE file: cat license.yml config.yml.FREE > config.yml.FREE.new && mv config.yml.FREE.new config.yml.FREE
4. then apply the PAID patch to config_template.yml; patch -o config_template.yml.PAID config_template.yml config_template.yml.PAID.patch (this creates a new file config_template.yml.PAID instead of in-place patching config_template.yml). Note that if the patch no longer applies to config_template.yml a new patch file is needed, do this by manually creating and editing config_template.yml.PAID, and create the patch file based on the diff (don't commit it though).
5. generate the PAID file: circleci config process config_template.yml.PAID > config.yml.PAID
6. remove the temporary patched PAID file: rm config_template.yml.PAID
7. add the Apache license header to the newly created PAID file: cat license.yml config.yml.PAID > config.yml.PAID.new && mv config.yml.PAID.new config.yml.PAID
8. finally, remember to update the config.yml

The process is equivalent for generate_11_and_17.sh and the respective JDK11+17 configuration files.