* make it obvious which gke benchmark script is experimental
* add cfg file for new kokoro jobs with a more fitting name
* update performance readme after renaming _v2 benchmark job
* fix typo
* Reorganize docs for end2end benchmarks
* Update README.md
* Restore heading "gRPC OSS benchmarks" (#27646)
This heading is used in cross-references from other repositories and sites. Also fix formatting.
Co-authored-by: Paulo Castello da Costa <6579971+paulosjca@users.noreply.github.com>
- Support multiline strings in template generation (as already supported in config generation, supporting roundtrip).
- Ignore substitution of variables that are set by the controller at runtime (`DRIVER_PORT`, `KILL_AFTER`, `POD_TIMEOUT`).
* Update load test template and config generation.
This change includes the following features and fixes:
* Add a script to generate load test examples.
* Update template generation logic to support round trip from configs to templates (handling of repeated clients and servers for the same language and of named clients and servers in source configs).
* Integrate safe language names from scenario config.
* Update template and config formatting (now that we generate in round trip).
* Fix shellcheck lint warnings.
* Update README.md.
This change adds the following features and improvements to load test generation:
* Multiple client instances are supported by setting the option `--instances_per_client`.
* `python_asyncio` is supported as a separate language.
* `php` is renamed to `php7` to match the scenario language names.
* Client and server names are shortened by the introduction of an index (`0`, `1`, ... with an optional prefix). This index replaces the uuid added to pod names when no component name is specified.
* Driver pool and image substitution are added to template generation.
* Empty entries are removed from the final configuration.
NOTE: Controller is currently not respecting driver settings from configuration, so driver is still named with a uuid. This will be addressed separately.
This example crashes the driver process, since `bq_upload_result.py` expects
the `big_query_table` argument to have the format `<dataset name>.<table name>`.
The project ID is already encoded in the script.
```
+ python3 /src/code/tools/run_tests/performance/bq_upload_result.py --bq_result_table=grpc-testing.e2e_benchmarks.experimental_results_hork_x5
Traceback (most recent call last):
File "/src/code/tools/run_tests/performance/bq_upload_result.py", line 177, in <module>
dataset_id, table_id = args.bq_result_table.split('.', 2)
ValueError: too many values to unpack (expected 2)
```
This change fixes typos and makes minor changes to the format of the generated yaml files for LoadTest configuration (embedded scenarios end in newline, and element lists are indented). The templates generated earlier are also updated.
This commit includes the following changes:
1. A new load test template generator (loadtest_template.py) is added. The template generator combines existing configurations or templates for several languages into a single template that can be used to generate configurations for different languages or combinations of languages.
2. A basic template generated from the example tests in grpc/test-infra (loadtest_template_basic_all_languages.yaml) is added.
3. The load test config generator is updated to use the combined template.
4. An example run consisting of a single test (generated from the combined template) is added and set up to run continuously.
* LoadTest generator for OSS benchmarks framework.
This change adds a LoadTest configuration generator for the OSS
benchmarks framework. The output of the generator is a multipart
YAML file that specifies uniquely named LoadTest resources that
can be applied to a kubernetes cluster.
For the benchmarks framework, see https://github.com/github/test-infra.
The tools/run_tests/performance/README.md file provides an example to
run workers in Java. However, this example does not work out of the
box, because the Java build requires an Android SDK. This change adds a
Gradle flag `-PskipAndroid=true` to the existing example, allowing it to
work as intended.
Signed-off-by: Ben Reed <benreed@google.com>
Jan Tattermusch