cassandra/.circleci

CircleCI config files

This directory contains the configuration for CircleCI continous integration platform. The file config.yml is the configuration file that is read by CircleCI. This file is automatically generated by the generate.sh script from the config_template.yml file.

The provided config.yml file uses low resources so users of the CircleCI free tier can use it. Additionally, there are two versions of this file using different resources so users who have access to premium CircleCI resources can use larger instances and more parallelism. These files are config.yml.FREE and config.yml.PAID. The default config.yml file is just a copy of config.yml.FREE.

Switching to higher resource settings

This directory contains generated files for free and paid resource settings. Switch between them by copying the correct file to config.yml and committing the result. For example, for using higher resources benefiting from CircleCI's paid tier:

cp .circleci/config.yml.PAID .circleci/config.yml

And for using lower resources comaptible with CircleCI's free tier:

cp .circleci/config.yml.FREE .circleci/config.yml

Alternatively, you can run the generate.sh script with the flags -f/-p to regenerate the config.yml file from config_template.yml using free or paid resources. This script validates and applies any changes to the config_template.yml file, and it requires the CircleCI CLI to be installed.

Setting environment variables

Both config_template.yml and config.yml files contain a set of environment variables defining things like what dtest repo and branch to use, what tests could be repeatedly run, etc.

These environment variables can be directly edited in the config.yml file, although if you do this you should take into account that the entire set of env vars is repeated on every job.

A probably better approach is editing them in config_template.yml and then regenerate the config.yml file using the generate.sh script. You can also directly pass environment variable values to the generate.sh script with the -e flag. For example, to set the dtest repo and branch with MIDRES config you can run:

generate.sh -p \
  -e DTEST_REPO=https://github.com/adelapena/cassandra-dtest.git \
  -e DTEST_BRANCH=CASSANDRA-8272

Running tests in a loop

Running the generate.sh script with use git diff to find the new or modified tests. The script will then create jobs to run each of these new or modified tests for a certain number of times, to verify that they are stable. You can use environment variables to specify the number of iterations of each type of test:

generate.sh -p \
  -e REPEATED_UTESTS_COUNT=500 \
  -e REPEATED_UTESTS_LONG_COUNT=100 \
  -e REPEATED_UTESTS_STRESS_COUNT=500 \
  -e REPEATED_SIMULATOR_DTESTS_COUNT=500 \
  -e REPEATED_JVM_DTESTS_COUNT=500 \
  -e REPEATED_JVM_UPGRADE_DTESTS_COUNT=500 \
  -e REPEATED_DTESTS_COUNT=500 \
  -e REPEATED_LARGE_DTESTS_COUNT=100 \
  -e REPEATED_UPGRADE_DTESTS_COUNT=25 \
  -e REPEATED_ANT_TEST_COUNT=500

You can also specify whether the iteration should fail on the first test failure:

generate.sh -p -e REPEATED_TESTS_STOP_ON_FAILURE=false

In addition to the automatically detected tests, it's also possible to provide lists of specific tests to be repeated:

generate.sh -p \
  -e REPEATED_UTESTS=org.apache.cassandra.cql3.ViewTest,org.apache.cassandra.db.CellTest \
  -e REPEATED_UTESTS_LONG=org.apache.cassandra.io.sstable.CQLSSTableWriterLongTest#testWideRow \
  -e REPEATED_UTESTS_STRESS=org.apache.cassandra.stress.generate.DistributionGaussianTest \
  -e REPEATED_DTESTS=cql_test.py,consistency_test.py::TestAvailability::test_simple_strategy \
  -e REPEATED_LARGE_DTESTS=replace_address_test.py::TestReplaceAddress::test_replace_stopped_node \
  -e REPEATED_JVM_DTESTS=org.apache.cassandra.distributed.test.PagingTest#testPaging \
  -e REPEATED_UPGRADE_DTESTS=upgrade_tests/cql_tests.py \
  -e REPEATED_JVM_UPGRADE_DTESTS=org.apache.cassandra.distributed.upgrade.PagingTest

If you need to repeat a particular set of tests and avoid automatic detection of changed tests use the -s command line option.

For particular Ant test targets that are not included in the regular test suites, you can use the run_repeated_utest job:

generate.sh -p \
  -e REPEATED_ANT_TEST_TARGET=test-cdc \
  -e REPEATED_ANT_TEST_CLASS=org.apache.cassandra.cql3.ViewTest \
  -e REPEATED_ANT_TEST_METHODS=testCompoundPartitionKey,testStaticTable \
  -e REPEATED_ANT_TEST_COUNT=500

Putting all together, you can have runs as complex as:

generate.sh -p \
  -e REPEATED_TESTS_STOP_ON_FAILURE=true \
  -e REPEATED_UTESTS=org.apache.cassandra.cql3.ViewTest,org.apache.cassandra.db.CellTest \
  -e REPEATED_UTESTS_COUNT=500 \
  -e REPEATED_UTESTS_LONG=org.apache.cassandra.io.sstable.CQLSSTableWriterLongTest#testWideRow \
  -e REPEATED_UTESTS_LONG_COUNT=100 \
  -e REPEATED_UTESTS_STRESS=org.apache.cassandra.stress.generate.DistributionGaussianTest \
  -e REPEATED_UTESTS_STRESS_COUNT=500 \
  -e REPEATED_DTESTS=cql_test.py,consistency_test.py::TestAvailability::test_simple_strategy \
  -e REPEATED_DTESTS_COUNT=500 \
  -e REPEATED_LARGE_DTESTS=replace_address_test.py,materialized_views_test.py \
  -e REPEATED_LARGE_DTESTS_COUNT=100 \
  -e REPEATED_JVM_DTESTS=org.apache.cassandra.distributed.test.PagingTest#testPaging \
  -e REPEATED_JVM_DTESTS_COUNT=500 \
  -e REPEATED_UPGRADE_DTESTS=upgrade_tests/cql_tests.py \
  -e REPEATED_UPGRADE_DTESTS_COUNT=25 \
  -e REPEATED_JVM_UPGRADE_DTESTS=org.apache.cassandra.distributed.upgrade.GroupByTest \
  -e REPEATED_JVM_UPGRADE_DTESTS_COUNT=500 \
  -e REPEATED_ANT_TEST_TARGET=test-cdc \
  -e REPEATED_ANT_TEST_CLASS=org.apache.cassandra.cql3.ViewTest \
  -e REPEATED_ANT_TEST_METHODS=testCompoundPartitionKey,testStaticTable \
  -e REPEATED_ANT_TEST_COUNT=500

WARNING: The Python large DTests test_network_topology_strategy and test_network_topology_strategy_each_quorum require XLarge containers in order to be run in a loop. Otherwise, you would see NO HOST AVAILABLE if you run them with the default medium config and Large containers.

Updating the config

For configuration changes meant to be permanent in the Apache repo you should never edit the config.yml file manually. Instead, you should edit the config_template.yml file and then regenerate the config.yml, config.yml.FREE and config.yml.PAID files by runnining the generate.sh script with -a flag. For using this script you need to install the CircleCI CLI.

As for temporal changes done while working in a patch, such as pointing to you dtest repo or running a test repeatedly, you can either directly edit config.yml or edit config_template.yml and then regenerate config.yml with the generate.sh script using a -f/-p flag. When this flag is used only the config.yml will be generated.

Please note that any previous swapping or edition of the generated files will be overriden by running generate.sh with -a argument, returning config.yml to the default FREE. So if you previously swapped your config.yml to MIDRES you would need to either swap it again or use the -f/-p script flags.

Read below for details how to generate the files manually without the generate.sh script:

1. make your edits to config_template.yml - let it stay at free tier settings
2. generate a valid FREE file: circleci config process config_template.yml > config.yml.FREE
3. add the Apache license header to the newly created FREE file: cat license.yml config.yml.FREE > config.yml.FREE.new && mv config.yml.FREE.new config.yml.FREE
4. then apply the PAID patch to config_template.yml; patch -o config_template.yml.PAID config_template.yml config_template.yml.PAID.patch (this creates a new file config_template.yml.PAID instead of in-place patching config_template.yml). Note that if the patch no longer applies to config_template.yml a new patch file is needed, do this by manually creating and editing config_template.yml.PAID, and create the patch file based on the diff (don't commit it though).
5. generate the PAID file: circleci config process config_template.yml.PAID > config.yml.PAID
6. remove the temporary patched PAID file: rm config_template.yml.PAID
7. add the Apache license header to the newly created PAID file: cat license.yml config.yml.PAID > config.yml.PAID.new && mv config.yml.PAID.new config.yml.PAID
8. finally, remember to update the config.yml