Commit Graph

197 Commits

Author SHA1 Message Date
Sergii Tkachenko de6ed9ba9f
[Python] Migrate from yapf to black (#33138)
- Switched  from yapf to black
- Reconfigure isort for black
- Resolve black/pylint idiosyncrasies 

Note: I used `--experimental-string-processing` because black was
producing "implicit string concatenation", similar to what described
here: https://github.com/psf/black/issues/1837. While currently this
feature is experimental, it will be enabled by default:
https://github.com/psf/black/issues/2188. After running black with the
new string processing so that the generated code merges these `"hello" "
world"` strings concatenations, then I removed
`--experimental-string-processing` for stability, and regenerated the
code again.

To the reviewer: don't even try to open "Files Changed" tab 😄 It's
better to review commit-by-commit, and ignore `run black and isort`.
2023-06-09 15:08:55 -07:00
Jan Tattermusch 07a75427bd
Simplify running bazel with structured test results on CI (#29353)
* add bazel_report_helper.py

* simplify bazel invocations in selected builds

* introduce __main__

* update gitignore

* introduce sleep constant

* add type annotations

* use f-strings

* Revert "use f-strings"

This reverts commit f970d6a40b.
2022-04-12 11:31:41 +02:00
Jan Tattermusch a3be072605
Switch .py shebang to python3 (#28963)
* switch shebang to python3

* prepare_build_macos: install latest available patch for each python
2022-03-14 10:41:30 +01:00
Craig Tiller 5d81683faa
Add per-call memory bloat report to CI (#29037)
* Add per-call memory bloat report to CI

* fix

* test

* Automated change: Fix sanity tests

* Update grpc_bloat_diff_in_docker.sh

* Update memory_diff.py

* review feedback

* prep for new job

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2022-03-12 11:56:48 +01:00
Craig Tiller 27662173e7
Clip long summaries (#28401) 2021-12-28 21:03:15 +00:00
Jan Tattermusch 82abb46ed3
improve tty output of run_tests.py and other scripts (#28308) 2021-12-09 07:23:02 +01:00
Jan Tattermusch 3a024ea40c
Avoid extremely slow kokoro job finalization by moving workspace to a non-synced directory on kokoro workers. (#28259)
* support "move and respawn" for kokoro jobs

* delete_nonartifacts.sh no longer needed

* bazel distribtest dont publish any artifacts

* try the altsrc respawn for selected jobs

* use altsrc for all windows CI jobs

* use altsrc for all macos CI jobs

* only move files under src/github

* fix uploading artifacts on windows
2021-12-03 19:41:14 +01:00
Richard Belleville 2eb686e8ad
Fix rbe_upload SSL issue (#27982)
* Attempt to fix rbe_upload SSL issue

* Fix comparison
2021-11-09 14:58:59 -08:00
Richard Belleville 73dcfd375d
Fix relative imports for Python 3 (#27971)
* Fix relative imports for Python 3

* Remove space

* isort
2021-11-08 15:37:32 -08:00
Richard Belleville 03b46b228e
Fix RBE upload (#27969) 2021-11-08 12:55:26 -08:00
Richard Belleville 59693b7e72
Migrate Infrastructure Scripts to Python 3 (#27135)
* Run 2to3 on tools directory

* Delete github_stats_tracking

* Re-run 2to3

* Remove unused script

* Remove unused script

* Remove unused line count utility

* Yapf. Isort

* Remove accidentally included file

* Migrate tools/distrib directory to python 3

* Remove unnecessary shebang

* Restore line_count directory

* Immediately convert subprocess.check_output output to string

* Take care of Python 2 shebangs

* Invoke scripts using a Python 3 interpreter

* Yapf. Isort

* Try installing Python 3 first

* See if we have any Python 3 versions installed

* Add Python 3.7 to Windows path

* Try adding a symlink

* Try to symlink differently

* Install six for Python 3

* Run run_interop_tests with python 3

* Try installing six in python3.7 explicitly

* Revert "Try installing six in python3.7 explicitly"

This reverts commit 2cf60d72f3.

* And debug some more

* Fix issue with jobset.py

* Add debug for CI failure

* Revert microbenchmark changes
2021-11-08 08:55:25 -08:00
Craig Tiller 8cbf308ce2
Use PUT not POST to avoid duplicate bloat labels (#27952) 2021-11-05 13:56:31 -07:00
Craig Tiller 59ed9d09a2
Reland bloat labelling code. (#27926)
* Revert "Revert "Assign a label for level of bloat diff (#27880)" (#27925)"

This reverts commit f2b96000a7.

* fix
2021-11-03 09:57:13 -07:00
Craig Tiller f2b96000a7
Revert "Assign a label for level of bloat diff (#27880)" (#27925)
This reverts commit 882cc1259e.
2021-11-03 08:14:19 -07:00
Craig Tiller 882cc1259e
Assign a label for level of bloat diff (#27880)
* upgrade bloaty

* Update check_submodules.sh

* fix

* label bloat

* set labels based on severity

* measure severity of changes

* Automated change: Fix sanity tests

* test

* test

* Update bloat_diff.py

* Update check_on_pr.py

* Automated change: Fix sanity tests

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2021-11-02 22:39:15 -07:00
Richard Belleville 2c889a9921
Fix stdout handling in jobset.py (#27493)
* Fix stdout handling in jobset.py

* Apparently ascii was a bad assumption

* Or how about this

* Just ignore non-utf8 characters
2021-10-04 16:00:40 -07:00
Jan Tattermusch c1bf4e1a2a
Revert "Enable Alpine portability test to run under Python 3. (#27464)" (#27485)
This reverts commit 54d4e8f68b.
2021-09-27 10:53:34 -07:00
Richard Belleville 54d4e8f68b
Enable Alpine portability test to run under Python 3. (#27464)
* Enable Alpine portability test to run under Python 3.

* Fix other scripts
2021-09-24 10:12:19 -07:00
Richard Belleville eada6c6e56
Update Alpine Test Image to Use Python 3.8 (#27415)
* Upgrade Alpine to Python 3.8

* Upgrade docker image to use 3.8
2021-09-20 19:38:36 -07:00
Richard Belleville b6cc72f129
Revert "Run 2to3 on tools directory (#26002)" (#27183)
This reverts commit a388361c3d.
2021-08-30 12:29:20 -07:00
Richard Belleville a388361c3d
Run 2to3 on tools directory (#26002)
* Run 2to3 on tools directory

* Delete github_stats_tracking

* Re-run 2to3

* Remove unused script

* Remove unused script

* Remove unused line count utility

* Yapf. Isort

* Remove accidentally included file

* Restore line_count directory
2021-08-30 09:25:21 -07:00
yifeizhuang 48b194f737
release v1.40.0: build gcr images for release, fix python2.7 (#27045) 2021-08-18 08:22:11 -07:00
Lidi Zheng 2231c2ba77
Introduce Python import sorting to our sanity test suite (#26768)
* Add isort_code.sh to sanity tests

* Run tools/distrib/isort_code.sh

* Fine tune the import order for relative imports

* Make pylint and project generation happy

* Fix a few corner cases

* Use --check instead of --diff

* The import order impacts test result somehow

* Make isort print diff and check output at the same time

* Let tools/run_tests/python_utils be firstparty library

* Run isort against latest HEAD
2021-07-26 12:31:21 -07:00
Sergii Tkachenko 368b9077e2
xds-k8s kokoro buildscripts: exclude from tests suites (#26098) 2021-04-27 19:26:53 -04:00
AJ Heller 6ff042277e
Fix invalid use of python's traceback.print_exc (#25743)
The exception is being interpreted as a `limit` (int) to cap the number
of stack trace entries to print.

https://docs.python.org/3/library/traceback.html#traceback.print_exc
2021-03-17 12:04:15 -07:00
AJ Heller 5139a012e7
Migrate the bm_diff benchmarks to python3 (#25619)
* Migrate the bm_diff benchmarks to python3

Includes a requirements.txt pinned at ~2017 versions, when this script
was first written.

* Replace p2 with p3 scipy/numpy dependencies.

* py2->3 for benchmark setup scripts

* upgrade pip to resolve python3 cryptography/setuptools-rust problem

* re-add jobset import (accidentally removed)

* re-add six's urllib import. This file is still used in py2 tests

* force py3 in run_if_c_cpp_modified.sh

* Fix another instance of subprocess.check_output binary mode

* Use the requirements.txt for CI perf environment setup

* Try to upgrade PyJWT. (v2.0.0 was problematic, #25053)

v2.x makes encode return strs from jwt.encode in both py2 and py3.
Previously, py3 would return bytes, and py2 a str.

* upgate cryptography lib version requirements for jwt.

* Wrap pip requirements specifier in quotes '>=x,<y'

* Decode subprocess output once instead of for every line

* Revert "Decode subprocess output once instead of for every line"

This reverts commit 28d14026431622ac7afd3535a8d7118e5be96628.

py2 doesn't support the `text` argument to subprocess.check_output.

* Address reviewer requests

* Pin a valid scipy version

* Remove scipy and tabulate dependencies from macos tests
2021-03-09 15:44:00 +01:00
Lidi Zheng 39cc797025
[xDS Proto] Modernize buildgen scripts (#25512)
* [xDS Proto] Modernize buildgen scripts

* This PR only covers some buildgen scripts that will be used by xDS
proto
* This PR also improves the debuggability of some scripts
* Merge with master

* Adopt reviewer's advices
2021-03-02 11:25:01 -08:00
Yash Tibrewal 9526a55776 Remove disrespectful term 2021-02-08 15:23:18 -08:00
Sergii Tkachenko 2258cee7b2
xds-k8s exclude fix: dirs end with a slash 2021-01-19 22:42:57 -05:00
Sergii Tkachenko 83501b822d
xds-k8s driver: exclude from tests suites 2021-01-19 22:36:58 -05:00
Jan Tattermusch d446ec036f
Merge pull request #24861 from dapeng-mi/master
Bug fix: decode bytes object to string for subprocess.check_output ou…
2021-01-15 17:09:10 +01:00
Sergii Tkachenko 8db79e2e71 Python style: reformat code using updated yapf 0.30.0 2021-01-13 14:00:12 -05:00
dapeng-mi 965aa55c9e Bug fix: add __lt__ method for JobSpec class.
Another exception is found when running "./tools/run_tests/run_tests.py
 -l c++ -c dbg" test in python3 environment. The exception complains
the "<" method is not supported when calling sorted() function to sort
the JobSpec class.

Here is the exception.

./tools/run_tests/run_tests.py -l c++ -c dbg
PASSED: make [time=1.6sec, retries=0:0]

Omitting EPOLLEXCLUSIVE tests

2020-12-01 10:49:42,122 detected port server running version 21
2020-12-01 10:49:42,182 my port server is version 21
Traceback (most recent call last):
  File "./tools/run_tests/run_tests.py", line 1919, in <module>
    build_only=args.build_only)
  File "./tools/run_tests/run_tests.py", line 1800, in _build_and_run
    one_run = set(spec for language in languages
  File "./tools/run_tests/run_tests.py", line 1801, in <genexpr>
    for spec in language.test_specs()
  File "./tools/run_tests/run_tests.py", line 425, in test_specs
    return sorted(out)
TypeError: '<' not supported between instances of 'JobSpec' and 'JobSpec'

Signed-off-by: dapeng-mi <clark_mdp@163.com>
2020-12-04 09:57:57 +08:00
Lidi Zheng 511af22c78 Hopefully fixed 27 build on winserver2016
* Download the working msys64 and use it to compile 27
* Remove the cygwin detection override logic
* Only install Python interpreters in jobs running on winserver2016
* Fix the batch script substring check
* Increase the timeout for related jobs
* TIL that there is an allow list for Kokoro env vars
2020-11-23 14:17:58 -08:00
Lidi Zheng 0e275f3d2b Change to explicitly encode 2020-11-13 12:07:45 -08:00
Lidi Zheng e191e98f37 Unify the encoding to ascii 2020-11-13 12:07:44 -08:00
Lidi Zheng 892e7bacb1 Explicitly set encoding to ascii to prevent TypeError in 3.7+ 2020-11-13 12:07:44 -08:00
Esun Kim eeb2b0f3a9 Fix pagination in upload_rbe_results 2020-10-19 11:26:30 -07:00
zhungxd e4cdcb7b25 fix wrong path 2020-09-09 19:41:32 +08:00
Jan Tattermusch 3456636150
Merge pull request #23455 from jtattermusch/port_server_python3
Make port server python3 compatible
2020-07-14 20:21:53 +02:00
Jan Tattermusch e329d1ddd0 bytes(string, encoding) is not supported in python2
results in error "TypeError: str() takes at most 1 argument (2 given)"
2020-07-14 17:46:27 +02:00
Jan Tattermusch 375b8551f7 update bigquery partition expiration 2020-07-13 15:04:53 +02:00
Jan Tattermusch 4847f85367 make port server python3 compatible 2020-07-13 11:35:07 +02:00
Vijay Pai 9d82fcc1a2
Fix test filter for C++ include files 2020-06-16 08:38:04 -07:00
Jan Tattermusch 6ad89b6369 also upload className in upload_rbe_results.py 2020-03-29 11:01:21 -04:00
Eric Gribkoff 9e1110cb11 Split logs by test case and run client separately for each test 2020-03-22 23:56:44 -07:00
Jan Tattermusch fea074384f sometimes not all RBE results are uploaded 2020-03-16 07:32:20 -04:00
Jan Tattermusch e20ed51e6f easier debugging for upload_rbe_results.py 2020-03-16 07:32:20 -04:00
Esun Kim 7be821ac47 Yapf all target python sources 2020-01-03 09:25:10 -08:00
chentanjun 05b3eeff77 fix-up some spelling mistakes 2019-09-25 23:29:47 +08:00