Commit Graph

95 Commits

Author SHA1 Message Date
Sergii Tkachenko de6ed9ba9f
[Python] Migrate from yapf to black (#33138)
- Switched  from yapf to black
- Reconfigure isort for black
- Resolve black/pylint idiosyncrasies 

Note: I used `--experimental-string-processing` because black was
producing "implicit string concatenation", similar to what described
here: https://github.com/psf/black/issues/1837. While currently this
feature is experimental, it will be enabled by default:
https://github.com/psf/black/issues/2188. After running black with the
new string processing so that the generated code merges these `"hello" "
world"` strings concatenations, then I removed
`--experimental-string-processing` for stability, and regenerated the
code again.

To the reviewer: don't even try to open "Files Changed" tab 😄 It's
better to review commit-by-commit, and ignore `run black and isort`.
2023-06-09 15:08:55 -07:00
Jan Tattermusch 84c091315b
increase timeout for windows C++ distribtests (#32669)
After switching to grpc-win2019 kokoro image, the windows C++
distribtest got slightly slower (from about ~1500sec to ~1650sec), which
got us too close to the 1800sec time limit and thus we've seen the
windows distribtests occasionally time out.

I'd still like to look into speeding up the distribtests, but for now we
don't want frequest test timeouts in the CI.

example failure: 

https://fusion2.corp.google.com/ci;ids=1923284992/kokoro/prod:grpc%2Fcore%2Fmaster%2Fwindows%2Fgrpc_distribtests_standalone/activity/996fd2cf-bef3-4490-85da-a52765d84127/log

```
2023-03-20 23:43:44,740 TIMEOUT: distribtest.cpp_windows_x86_cmake_as_externalproject [pid=5984, time=1800.1sec]
```
2023-03-22 08:51:26 +01:00
Esun Kim 564a46291b
Update all Debian:9 to Debian:10 (#31917) 2023-01-23 10:50:41 -08:00
apolcyn 86d314a4cb
Undrop ruby 2.6 (#32160) 2023-01-20 09:36:59 -08:00
apolcyn c1a4b526fb
[ruby]: add pre-compiled binaries for ruby 3.2; drop them for ruby 2.6 (#32089)
[ruby]: add pre-compiled binaries for ruby 3.2; drop them for ruby 2.6
2023-01-19 15:32:26 -08:00
Esun Kim 5fb4c2364d
Clean-up pkgconfig (#31660)
* Clean-up pkgconfig

* Removed run_distrib_test_cmake_module_install_pkgconfig.sh

* Proper re2

* Clean up pkgconfig.sh test script
2022-11-22 10:51:17 -08:00
Jan Tattermusch 3cf3ed7d59
Revert "Revert "Add back C# distribtests for Grpc.Tools"" (#31436)
* Revert "Revert "Add back C# distribtests for Grpc.Tools (#31399)" (#31412)"

This reverts commit dbc45208e2.

* avoid clash with internal protos
2022-10-27 09:59:06 +02:00
Alisha Nanda dbc45208e2
Revert "Add back C# distribtests for Grpc.Tools (#31399)" (#31412)
This reverts commit 50bad293c9.
2022-10-20 12:08:29 -07:00
Jan Tattermusch 50bad293c9
Add back C# distribtests for Grpc.Tools (#31399)
* remove remains of grpc_csharp_ext artifact build completely

* add back part of C# distribtests

* redirect C# linux distribtests

* comment out some distribtests

* switch updateversion to Grpc.tools

* fix .proto

* fix distribtest complile

* temporarily comment out service in .proto

* cleanup

* improvements

* reenable mac distribtests

* fixes
2022-10-20 16:42:29 +02:00
Jan Tattermusch 2d7ebdb5fd
avoid frequent PHP mac distribtest timeouts (#30847) 2022-09-06 19:59:11 +02:00
apolcyn 8b941bb648
Drop support for ruby 2.5 (#30699)
* Drop ruby 2.5 support
2022-08-25 21:39:31 -07:00
Richard Belleville 4472304d63
Drop Python 3.6 From Distribtests (#30562)
* First attempt at fixing distribtest

* Add dev ubuntu 20.04 image

* Plase the copyright overlords
2022-08-12 15:05:16 -07:00
Jan Tattermusch e8404698d5
Remove C# implementation (individual packages will continue to be maintained through v2.46.x patches or moved to grpc-dotnet) (#29225)
* remove examples/csharp

* remove reference to examples/csharp

* remove C# references from doc/

* remove src/csharp

* update test_creds locations

* remove C# distribtests

* remove buildgen templates for files under src/csharp

* remove grpc_csharp_ext C build target

* regenerate projects

* update tools/run_tests/README.md

* change many C# test/build scripts to noop

* unbreak run_tests.py -l csharp

* unbreak C# build_package targets

* unbreak C# distribtests

* remove C# pre-build steps from prepare_build_windows.bat

* remove src/csharp from clang_format

* remove csharp and csharpcoreclr from run_interop_tests.py

* remove C# interop test docker images

* Revert "remove src/csharp"

This reverts commit 97f2b6d707.

* remove src/csharp, but keep Grpc.Tools

* add back templates/src/csharp/build/dependencies.props.template

* remove everything but Grpc.Tools and Grpc.Tools.Tests from sln

* Revert "unbreak run_tests.py -l csharp"

This reverts commit 529b5e2ec1.

* revive Grpc.Tools.Tests through run_tests.py

* Revert "unbreak C# build_package targets"

This reverts commit 79d8d83c8b.

* experiment with build_nuget.sh fixes

* simplify create_fake_native_artifacts.sh

* building of Grpc.Tools in package_targets.py

* move shared files from Grpc.Core to Grpc.Tools

* yapf format: distribtest_targets.py

* Revert "remove C# pre-build steps from prepare_build_windows.bat"

This reverts commit bd38956a2b.

* update C# README.md

* disable C# in gke OSS benchmarks
2022-04-28 12:00:34 +02:00
Jan Tattermusch a3be072605
Switch .py shebang to python3 (#28963)
* switch shebang to python3

* prepare_build_macos: install latest available patch for each python
2022-03-14 10:41:30 +01:00
Esun Kim 2a699b9a09
Removed jessie-based distribtest (#29029)
* Removed jessie-based distribtest

* Update by review
2022-03-08 08:13:13 -08:00
Lidi Zheng cd4261b946
Support musllinux binary wheels on x64 and x86 (#28092)
* Support musllinux binary wheels

* Skip aarach64 for now
* Consolidate the difference of mktemp
* Extend linux artifact building time && install bash for distribtest
* Stop using grpc.tools, use grpc_tools
* Update the README to use grpc_tools
* Force static link libc++ for alpine binaries
* Rebase recent build script changes

* Install ccache for musllinux distribtest images

* Revert timeout change to grpc_build_artifacts
2022-02-10 14:14:56 -08:00
Jan Tattermusch c8c1774ec4
Revert "Revert "Cleanup run docker machinery in run_tests.py, task_runner.py and elsewhere (#28704)" (#28741)" (#28748)
* Revert "Revert "Cleanup run docker machinery in run_tests.py, task_runner.py and elsewhere (#28704)" (#28741)"

This reverts commit 9a79d44e9b.

* fix python distribtest failure on fedora34
2022-02-01 11:33:56 +01:00
Yash Tibrewal 9a79d44e9b
Revert "Cleanup run docker machinery in run_tests.py, task_runner.py and elsewhere (#28704)" (#28741)
This reverts commit be723121fc.
2022-01-28 11:42:12 -08:00
Jan Tattermusch be723121fc
Cleanup run docker machinery in run_tests.py, task_runner.py and elsewhere (#28704)
* attempt

* slim down build_docker_and_run_tests.sh

* HOST_GIT_ROOT is unused

* config and arch variable no need to propagate to run_tests.py

* unify unused LOCAL_GIT_ROOT to EXTERNAL_GIT_ROOT

* mounting gcloud config seems useless

* reorder docker args

* cleanup

* allow passing args to build_docker_and_run_tests.sh

* converge build_docker_and_run_tests.sh and build_and_run_docker.sh

* more convergence

* convergence of docker runners

* finalize convergence of run_docker scripts

* GRPC_TEST_REPORT_BASE_DIR might not exist

* adjust report copying

* make report_dir and output_dir readable

* alpine linux does not support cp -t
2022-01-28 11:46:36 +01:00
Jan Tattermusch 7924a5fde7
add task_runner.py --inner_jobs support for C++ distribtests (#28594)
* add support for task_runner.py --inner_jobs for C++ distribtests

* fine tune linux c++ distribtests parallelism

* yapf code
2022-01-18 18:45:13 +01:00
apolcyn 95b313de82
Remove ruby 2.4 support (#28522)
* changes to remove ruby 2.4 support

* some fixes

* remove opensuse since eol

* fixes

* remove unused x86 dockerfile

* stop building for ruby 2.4

* update build scripts

* fix scripts

* fix sanity

Co-authored-by: Jan Tattermusch <jtattermusch@google.com>
2022-01-11 13:30:58 +01:00
Jan Tattermusch f23e1d9e85
task_runner.py improvements (foundations for future speedup) (#28480)
* add --dry_run support for task_runner.py

* dont sort task_runner targets before running

* fixup dry run

* support task_runner.py --inner_jobs

* pass inner_jobs to build_jobspec for task_runner.py

* support inner_jobs for C# artifacts

* support inner_jobs for protoc artifacts

* inner jobs support for src/csharp/experimental/build_native_ext_for_android.sh

* address review feedback
2022-01-11 10:26:32 +01:00
Lidi Zheng 7cfd399dd6
Reduce the workload of presubmit artifact builds and distribtests (#28228)
* Reduce the workload of presubmit artifact builds and distribtests

* Create and redirect Kokoro config
* Remove duplicated scripts and use env var config

* Clean-up Kokoro cfg

* [DO-NOT-SUBMIT] Add ad-hoc TASK_RUNNER_EXTRA_FILTERS

* Revert "[DO-NOT-SUBMIT] Add ad-hoc TASK_RUNNER_EXTRA_FILTERS"

This reverts commit 9436ac54ac.
2021-12-06 10:03:31 -08:00
Richard Belleville 59693b7e72
Migrate Infrastructure Scripts to Python 3 (#27135)
* Run 2to3 on tools directory

* Delete github_stats_tracking

* Re-run 2to3

* Remove unused script

* Remove unused script

* Remove unused line count utility

* Yapf. Isort

* Remove accidentally included file

* Migrate tools/distrib directory to python 3

* Remove unnecessary shebang

* Restore line_count directory

* Immediately convert subprocess.check_output output to string

* Take care of Python 2 shebangs

* Invoke scripts using a Python 3 interpreter

* Yapf. Isort

* Try installing Python 3 first

* See if we have any Python 3 versions installed

* Add Python 3.7 to Windows path

* Try adding a symlink

* Try to symlink differently

* Install six for Python 3

* Run run_interop_tests with python 3

* Try installing six in python3.7 explicitly

* Revert "Try installing six in python3.7 explicitly"

This reverts commit 2cf60d72f3.

* And debug some more

* Fix issue with jobset.py

* Add debug for CI failure

* Revert microbenchmark changes
2021-11-08 08:55:25 -08:00
Lidi Zheng 9978223a26
Add python_requires field to grpcio-* packages (#27495)
- Update the run_tests.py and distrib tests
- Update the Dockerfiles
2021-09-28 16:01:15 -07:00
Lidi Zheng 4729f6fdcc
Add Python 3.10 drop 3.5 (#26074)
* Add Python 3.10.0rc1 binary wheels

* Drop Python 3.5 artifacts

* Document the drop of 3.5

* Fix the wrong pip pointer

* Update manylinux2014 to a newer version, remove 3.5 distribtest

* Update manylinux aarch64 to see if the absl error go away

* Use the preferred alias

* Allow different wheel library to produce different tag order

* Remove unused shell var and log produced wheels

* Use copy instead of move

* Make bash happy about the wildcard

* Upgrade the debian image to use 3.5+ Python

* Polish the comments for the Dockerfiles
2021-09-02 14:02:53 -07:00
Jan Tattermusch ce5b4e949f
Faster and streamlined PHP distribtests (for linux and macos) (#27101)
* faster and streamlined PHP distribtests

* add kokoro job for PHP distribtests

* php distribtests job for macos

* add trailing newline
2021-08-31 09:23:13 +02:00
Richard Belleville b6cc72f129
Revert "Run 2to3 on tools directory (#26002)" (#27183)
This reverts commit a388361c3d.
2021-08-30 12:29:20 -07:00
Richard Belleville a388361c3d
Run 2to3 on tools directory (#26002)
* Run 2to3 on tools directory

* Delete github_stats_tracking

* Re-run 2to3

* Remove unused script

* Remove unused script

* Remove unused line count utility

* Yapf. Isort

* Remove accidentally included file

* Restore line_count directory
2021-08-30 09:25:21 -07:00
Lidi Zheng 5820e152cd
Remove Python 2.7 binary wheel generations (#26691)
* Remove Python 2.7 binary wheels

* Make sure pip won't pick-up newer versions

* Attempt to fix the distribtest

* Fallback to virtualenv==16.7.9 for 34 and 35

* Remove 34 from binary wheel distribtests
2021-07-23 08:20:39 -07:00
Jan Tattermusch d61db49b48
Add python3.8 aarch64 wheel distribtest (#26136)
* add python3.8 aarch64 distribtest

* register qemu emulator for linux distribtests
2021-05-03 08:16:11 +02:00
Esun Kim 160cb1b215
Changed RPI build test to general ARM build test (#25580)
* Change rpi build test to arm test
* Having a separate docker and targetting aarch64
2021-03-02 08:46:03 -08:00
Alexander Polcyn b94f09aaf4 Drop support for ruby 2.3 2021-01-19 17:25:47 -08:00
Sergii Tkachenko 8db79e2e71 Python style: reformat code using updated yapf 0.30.0 2021-01-13 14:00:12 -05:00
Alexander Polcyn 4fa05591e3 skil ruby 3.0 distrib test until protobuf adds support 2020-12-30 15:52:15 -08:00
Alexander Polcyn f5fae54331 Remove no longer supported centos6 distrib test 2020-12-30 11:48:31 -08:00
Alexander Polcyn 039936387e Add a ruby 3.0.0 distrib test 2020-12-29 20:28:46 -08:00
Jan Tattermusch 6c43b2c98a add netcoreapp3.1 C# distribtest 2020-10-27 16:30:01 +01:00
Jan Tattermusch e274e7b93a add C# distribtest for .NET 5 2020-10-27 16:30:00 +01:00
Jan Tattermusch e7f5fe1605 dont include x86 grpc_csharp_ext for linux and mac 2020-10-22 05:48:44 -04:00
root 49bad76da3 replace label name php to php7 2020-10-02 17:55:31 +00:00
root 455e2c9847 replace jessie to stretch, using name PHP7 instead of PHP 2020-10-02 17:13:36 +00:00
Jan Tattermusch 37c980d98a improve python3 compatibility of the task_runner.py script 2020-08-28 10:14:17 -04:00
Esun Kim c0047f46b5 Upgrade Ubuntu 14.04 -> 18.04 2020-08-13 19:20:42 -07:00
Alexander Polcyn 825dc5731f Add a distrib test for source-only ruby packages 2020-07-28 21:33:10 -07:00
Jan Tattermusch cf21444b3c increase timeout for PHP distribtests 2020-07-20 18:15:44 +02:00
Jan Tattermusch 4f96118724 move routeguide c++ distribtest under cmake_pkgconfig 2020-07-15 08:56:46 +02:00
Lars Kanis 4c667476a0 Add ruby-2.7 to build process of binary gems for Windows and Linux
With rake-compiler-dock-1.0.1 there's no need to use a derived Dockerfile.
The standard RCD image can be used instead.
The only issue is with conflicting declarations of gettimeofday(), which can be easily worked around by patching win32.h before the build.

Fixes #21514
2020-02-16 21:25:38 +01:00
Esun Kim 377fcc9bba Update docker images for Ruby 2020-02-13 10:19:47 -08:00
Jan Tattermusch 87f7674248 cleanup C++ distribtests 2020-02-04 10:14:46 -05:00