Commit Graph

121 Commits

Author SHA1 Message Date
Esun Kim 076f719286
Revert "Revert "[Test] Dedup vs2019 test in the Windows test" (#33393)" (#33394) 2023-06-09 16:10:29 -07:00
Esun Kim e9e5dc0cf6
Revert "[Test] Dedup vs2019 test in the Windows test" (#33393)
Reverts grpc/grpc#33385

Temporary revert to unblock https://github.com/grpc/grpc/pull/33138
2023-06-09 15:08:20 -07:00
Esun Kim 6b3949593d
[Test] Dedup vs2019 test in the Windows test (#33385)
`cmake_ninja_vs2019` and `default` are using the same
`cmake_ninja_vs2019` so having two tests are waste so this is removing
`cmake_ninja_vs2019` leaving `default` which does `cmake_ninja_vs2019`.

This change can cut the space consumption by half and with 250GB disc, 

- Pre-test: 267,770,322,944 bytes free 
- Post-test: 134,499,295,232 bytes free
2023-06-09 13:52:58 -07:00
Esun Kim 3bd6c38650
[Infra] Use Bazel 6 and drop Bazel 4 (Part 2) (#32910)
Oops I missed important changes from
https://github.com/grpc/grpc/pull/32712. And it turned out that there
are two problems that I couldn't fix at this point.
- Windows Bazel RBE Linker Error: This may be caused by how new Bazel 6
invokes build tools chain but it's not clear. I put workaround to use
Bazel 5 by using `OVERRIDE_BAZEL_VERSION=5.4.1`
- Rule `rules_pods` to fetch CronetFramework from CocoaPod has
incompatibility with sort of built-in apple toolchain.
(https://github.com/bazel-xcode/PodToBUILD/issues/232): I couldn't find
a workaround to fix this so I ended up disabling all tests depending
this target.
2023-04-21 15:53:52 -07:00
Jan Tattermusch 3cf3ed7d59
Revert "Revert "Add back C# distribtests for Grpc.Tools"" (#31436)
* Revert "Revert "Add back C# distribtests for Grpc.Tools (#31399)" (#31412)"

This reverts commit dbc45208e2.

* avoid clash with internal protos
2022-10-27 09:59:06 +02:00
Alisha Nanda dbc45208e2
Revert "Add back C# distribtests for Grpc.Tools (#31399)" (#31412)
This reverts commit 50bad293c9.
2022-10-20 12:08:29 -07:00
Jan Tattermusch 50bad293c9
Add back C# distribtests for Grpc.Tools (#31399)
* remove remains of grpc_csharp_ext artifact build completely

* add back part of C# distribtests

* redirect C# linux distribtests

* comment out some distribtests

* switch updateversion to Grpc.tools

* fix .proto

* fix distribtest complile

* temporarily comment out service in .proto

* cleanup

* improvements

* reenable mac distribtests

* fixes
2022-10-20 16:42:29 +02:00
Jan Tattermusch 2f5d06af6b
make legacy artifact -> packages -> distribtest jobs a noop (#31124) 2022-09-27 21:01:32 +02:00
Jan Tattermusch 686345acb5
avoid flake in C# Grpc.Tools tests by running tests sequentially (#30691) 2022-08-22 17:05:46 +02:00
Jan Tattermusch ad2ca50712
Simplify building release packages through per-language grpc_distribtests_* jobs. (#29699)
* add release version of grpc_distribtests_* jobs .cfg files

* make grpc_distribtests_ruby more aligned with other single-job distribtests

* dont hide packages from build_artifacts step for python an php
2022-05-18 13:09:15 +02:00
Jan Tattermusch da1977f98c
Cleanup configuration of bazel RBE jobs (#29301)
* cleanup bazel_rbe .cfg and .sh files

* upload sponge_log.xml artifacts for selected bazel jobs

* use move_src_tree_and_respawn_itself_rc for bazel RBE tests on linux

* fix wrong config
2022-04-13 13:14:00 +02:00
Jan Tattermusch 07a75427bd
Simplify running bazel with structured test results on CI (#29353)
* add bazel_report_helper.py

* simplify bazel invocations in selected builds

* introduce __main__

* update gitignore

* introduce sleep constant

* add type annotations

* use f-strings

* Revert "use f-strings"

This reverts commit f970d6a40b.
2022-04-12 11:31:41 +02:00
Jan Tattermusch 59d9749c71
fix csharp single-job distribtest on grpc-win2016 (#29183) 2022-03-22 20:00:13 +01:00
Jan Tattermusch c8430e6ac4
add grpc_distribtests_python for windows (#29153) 2022-03-18 20:07:32 +01:00
Jan Tattermusch 409c732813
C# windows single-job distribtest (#29152)
* add windows grpc_distribtest_csharp

* add kokoro cfg files
2022-03-18 16:34:31 +01:00
Jan Tattermusch 24dfe21cad
activate ccache for selected win builds (#28735) 2022-03-10 19:50:34 +01:00
Jan Tattermusch 8024eed4c5
Print debug info about prepare_build_windows.bat timing (#28872)
* prepare build windows timing

* C# deps not need to be installed for C# native artifact on win
2022-03-07 12:16:46 +01:00
Jan Tattermusch fea1f3de33
A more reliable way of installing bazel on windows workers (#28835)
* more reliable bazel installation for win bazel RBE builds

* win RBE build already using the same credentials

* also recognize MSYS_NT returned by kokoro workers
2022-02-15 19:41:29 +01:00
Jan Tattermusch 9d73b3e85a
Use ninja for all grpc_csharp_ext builds on windows (#28841)
* cleanup C# win artifact build

* build C# basictests on win with ninja

* specify default parallelism for C# win artifact

* honor GRPC_PYTHON_BUILD_EXT_COMPILER_JOBS in build_artifact_python.bat

* set --inner_jobs for windows grpc_build_artifact job

* fixup C# build picking env variables from C core build

* get rid of no longer needed NativeDependenciesConfiguration C# setting
2022-02-11 18:58:04 +01:00
Jan Tattermusch e324bf5eee
Speed up windows C/C++ builds by using cmake Ninja generator (#28833)
* support building C/C++ with ninja

* make cmake_ninja_vs2015 build the default for C/C++

* fix vcvarsall location for vs2017
2022-02-11 09:59:51 +01:00
Jan Tattermusch c0564aa21f
Speedup grpc_portability_build_only windows kokoro job (#28460)
* only run windows C++ portability build only tests on win

* remove .cfg files for no longer used kokoro windows jobs
2022-01-04 20:29:46 +01:00
Jan Tattermusch 7dfdf3d679
Delete no-longer-used kokoro jobs .cfg files. (#28465)
* remove .cfg files for non-existent kokoro jobs

* more cleanup
2022-01-04 20:26:25 +01:00
Jan Tattermusch c796d5786b
Reintroduce #28396 (Cleanup of windows CI .bat scripts) (#28457)
* Revert "Revert "Cleanup of windows CI .bat scripts (#28396)""

This reverts commit 530187f865.

* fixup
2022-01-04 18:30:05 +01:00
Craig Tiller a7dcceb365
Revert "Cleanup of windows CI .bat scripts (#28396)" (#28446)
This reverts commit 44e7be448d.
2021-12-29 12:03:22 -08:00
Jan Tattermusch 44e7be448d
Cleanup of windows CI .bat scripts (#28396)
* cleanup of windows CI .bat scripts

* fixup

* fixup

* fixup
2021-12-29 12:15:48 +01:00
Jan Tattermusch 769369ad3f
fix invocation of respawn scripts on windows (#28366) 2021-12-19 18:50:19 +01:00
Jan Tattermusch 8afb68238b
Fix windows RBE results uploading to bigquery (#28365) 2021-12-19 18:43:04 +01:00
Lidi Zheng 7cfd399dd6
Reduce the workload of presubmit artifact builds and distribtests (#28228)
* Reduce the workload of presubmit artifact builds and distribtests

* Create and redirect Kokoro config
* Remove duplicated scripts and use env var config

* Clean-up Kokoro cfg

* [DO-NOT-SUBMIT] Add ad-hoc TASK_RUNNER_EXTRA_FILTERS

* Revert "[DO-NOT-SUBMIT] Add ad-hoc TASK_RUNNER_EXTRA_FILTERS"

This reverts commit 9436ac54ac.
2021-12-06 10:03:31 -08:00
Jan Tattermusch 3a024ea40c
Avoid extremely slow kokoro job finalization by moving workspace to a non-synced directory on kokoro workers. (#28259)
* support "move and respawn" for kokoro jobs

* delete_nonartifacts.sh no longer needed

* bazel distribtest dont publish any artifacts

* try the altsrc respawn for selected jobs

* use altsrc for all windows CI jobs

* use altsrc for all macos CI jobs

* only move files under src/github

* fix uploading artifacts on windows
2021-12-03 19:41:14 +01:00
Esun Kim 63d48ee8d8
Use more parallelism to windows portability tests (#28179)
* Use more parallelism to windows portability tests

* Added /MP

* Changed it to gRPC_BUILD_MSVC_MP_COUNT
2021-11-24 08:41:23 -08:00
Richard Belleville 8f50d574fb
Add note about starting port server out of band (#28012) 2021-11-12 10:04:12 -08:00
Richard Belleville 05b695c95a
Start port server separately (#28005) 2021-11-11 10:29:31 +01:00
Richard Belleville 59693b7e72
Migrate Infrastructure Scripts to Python 3 (#27135)
* Run 2to3 on tools directory

* Delete github_stats_tracking

* Re-run 2to3

* Remove unused script

* Remove unused script

* Remove unused line count utility

* Yapf. Isort

* Remove accidentally included file

* Migrate tools/distrib directory to python 3

* Remove unnecessary shebang

* Restore line_count directory

* Immediately convert subprocess.check_output output to string

* Take care of Python 2 shebangs

* Invoke scripts using a Python 3 interpreter

* Yapf. Isort

* Try installing Python 3 first

* See if we have any Python 3 versions installed

* Add Python 3.7 to Windows path

* Try adding a symlink

* Try to symlink differently

* Install six for Python 3

* Run run_interop_tests with python 3

* Try installing six in python3.7 explicitly

* Revert "Try installing six in python3.7 explicitly"

This reverts commit 2cf60d72f3.

* And debug some more

* Fix issue with jobset.py

* Add debug for CI failure

* Revert microbenchmark changes
2021-11-08 08:55:25 -08:00
Lidi Zheng 99af2bdf3f
Extend the Windows Python test timeout to 3hr (#27843) 2021-10-27 13:20:50 -07:00
Jan Tattermusch 9b2b9976c6
increase windows C# basictests job timeout (#27483) 2021-09-27 18:30:28 +02:00
Jan Tattermusch 0042c2b86f
Upgrade bazel to 4.2.1 (LTS), upgrade bazel toolchain to 4.1.0 (#27410)
* update bazel to 4.2.1

* add bazel 4.2.1 to update_mirror.sh

* regenerate dockerfiles

* upgrade bazel toolchains to 4.1.0

* use create_rbe_exec_properties_dict instead of the deprecated create_exec_properties_dict

* RBE ubsan doesnt require the crosstool_top hack anymore

* update rules_apple and build_bazel_apple_support
2021-09-23 16:20:25 +02:00
Jan Tattermusch c4aa6af0a0
Increase timeout for windows portability test (on master) (#27210)
* remove no longer use kokoro job config

* increase timeout of windows grpc_portability job
2021-09-01 19:44:10 +02:00
Esun Kim f6bcb40b7f
Use T drive for windows bazel RBE build (#26895) 2021-08-05 16:31:46 -07:00
AJ Heller b516dbfa21
Increase artifact build timeouts for problem targets (#25695)
Going back 2 weeks, nightly packages have failed to build due to
artifact build timeouts. This change increases timeouts for:

Python Linux Extra: 5hrs -> 7hrs
Python Manylinux: 1hr -> 2hrs
CSharp iOS: 45min -> 1hr
CSharp MacOS x64: 30m -> 45m

This is a stopgap measure to get nightly packages deployed more regularly;
March has seen 3 nightlies published in 11 days.

I've cherry picked some of the artifact build timeout errors below.

2021.03.10
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-10 05:03:40,560 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=7384, time=18000.4sec]
2021-03-10 05:03:40,646 TIMEOUT: build_artifact.python_linux_extra_armv7_2.7 [pid=7375, time=18000.5sec]
2021-03-10 05:03:40,728 TIMEOUT: build_artifact.python_linux_extra_armv7_3.5 [pid=7378, time=18000.6sec]

2021.03.08
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-08 05:03:18,542 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=6853, time=18000.5sec]
2021-03-08 05:03:18,639 TIMEOUT: build_artifact.python_linux_extra_armv7_2.7 [pid=6844, time=18000.6sec]
2021-03-08 05:03:18,741 TIMEOUT: build_artifact.python_linux_extra_armv7_3.5 [pid=6847, time=18000.7sec]

2021.03.07
grpc/core/master/macos/grpc_build_artifacts:
2021-03-07 00:58:02,492 TIMEOUT: build_artifact.csharp_ext_macos_x64 [pid=58673, time=1805.5sec]
2021-03-07 01:13:03,065 TIMEOUT: build_artifact.csharp_ext_macos_ios [pid=58672, time=2706.1sec]

Bumping iOS again from https://github.com/grpc/grpc/pull/25049

grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-07 05:03:36,061 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=8867, time=18002.9sec]

2021.03.06
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-06 05:03:12,474 TIMEOUT: build_artifact.python_linux_extra_armv7_3.5 [pid=8307, time=18000.6sec]

2021.03.05
grpc/core/master/linux/grpc_build_artifacts_extra_release
2021-03-05 05:03:48,611 TIMEOUT: build_artifact.python_linux_extra_armv7_3.6 [pid=7075, time=18008.6sec]

2021.02.27
grpc/core/master/linux/grpc_build_artifacts
2021-02-27 01:09:57,943 TIMEOUT: build_artifact.python_manylinux2010_x64_cp27-cp27mu [pid=21983, time=3608.9sec]

2021.02.23
grpc/core/master/windows/grpc_build_artifacts
```
2021-02-23 01:48:23,729 SUCCESS: All targets built successfully.

T:\src\github\grpc>set RUNTESTS_EXITCODE=0

T:\src\github\grpc>bash tools/internal_ci/helper_scripts/delete_nonartifacts.sh
++ dirname tools/internal_ci/helper_scripts/delete_nonartifacts.sh
+ cd tools/internal_ci/helper_scripts/../../..
+ find . -type f -not -iname '*sponge_log.*' -not -path './reports/*' -not -path './artifacts/*' -not -path './tools/internal_ci/*' -exec rm -f '{}' +

ERROR: Aborting VM command due to timeout of 10800 seconds
```
2021-03-12 09:25:26 -08:00
Jan Tattermusch bf21b2b3a0 update bazel version in other places too 2020-12-15 17:29:49 +01:00
Lidi Zheng 511af22c78 Hopefully fixed 27 build on winserver2016
* Download the working msys64 and use it to compile 27
* Remove the cygwin detection override logic
* Only install Python interpreters in jobs running on winserver2016
* Fix the batch script substring check
* Increase the timeout for related jobs
* TIL that there is an allow list for Kokoro env vars
2020-11-23 14:17:58 -08:00
Jan Tattermusch e58f0129d2 Stop relying on rbe-windows-credentials.json 2020-11-09 15:42:30 +01:00
Lidi Zheng 85d0d548c4 Increase the timeout of Python basic tests on Windows 2020-11-06 09:50:50 -08:00
Jan Tattermusch 858797ae77 disable grpc_build_packages.bat 2020-10-21 15:04:53 +02:00
Lidi Zheng a0a69de4df Improve Win CI script with fail fast 2020-09-15 14:56:37 -07:00
Jan Tattermusch 424d33415c upgrade bazel to 2.2.0 2020-08-20 16:48:56 +02:00
Lidi Zheng db4ce94ebb
Revert "Revert "Release Python3.8 wheels for Windows"" 2019-12-02 10:59:03 -08:00
Jan Tattermusch 42039d09b7
Revert "Release Python3.8 wheels for Windows" 2019-12-02 18:01:42 +01:00
Lidi Zheng ed054d03a1 Fix the hack, hope the test still passes 2019-11-26 11:10:57 -08:00
Lidi Zheng 6b92b4af05 Attempt to drop support for Python 3.4 2019-10-30 14:06:22 -07:00