Commit Graph

189 Commits

Author SHA1 Message Date
Jan Tattermusch b9f0c019d0
[cmake] Improved stdalign.h workaround for windows build (#32777)
Improved the workaround from https://github.com/grpc/grpc/pull/32764 as
suggested in
https://github.com/grpc/grpc/pull/32764#issuecomment-1491794750.

Once merged, I'll backport to 1.54.x and 1.53.x together with
https://github.com/grpc/grpc/pull/32764 (these have switched to VS2019)
2023-04-05 08:20:20 +00:00
Jan Tattermusch 569d0076b3
[cmake] Workaround missing stdalign.h in windows portability test (#32764)
Tentative fix for b/275694647. Also see
https://github.com/grpc/grpc/pull/32662

Adhoc windows/grpc_portability run:
http://sponge/bb75e14c-fc77-45bb-89bd-4c4685746490

Once merged, I'll backport to 1.54.x and 1.53.x (these have switched to
VS2019)
2023-03-31 09:00:39 -07:00
Jan Tattermusch 43c3d1f932
Switch all VS2017 builds to VS2019 (#32662)
(part of removing support for VS2017)

Also see https://github.com/grpc/grpc/pull/32649
Also see https://github.com/grpc/grpc/pull/32615

The switch to grpc-win2019 windows workers has already happened:
(cl/517400022).

Once this PR lands, I'll backport to 1.53.x branch as well (since that
release removes the VS2017 support).
2023-03-23 16:44:39 +01:00
Jan Tattermusch 5873ba96dc
Ensure compatibility with the new custom kokoro win2019 image (#32649)
Fix incompatibilities identified when running adhoc runs on the new
custom win2019 image.
After merging this, it should be possible to switch to the new image
without breaking any tests.

- for most fixes I added a comment that explains why they're necessary.
- the new image won't have VS2015 installed, so I'm switching the protoc
artifact build to VS2017

This PR will need to be backported to older release branches to ensure
the windows tests continue working on those branches as well (IMHO I
haven't made any changes that would be difficult to backport and I tried
to keeps the diff as small as possible to avoid issues when
backporting).

After we switch to the new image (and all the windows tests are green),
we can incrementally move the builds that are still using VS2017 to
VS2019.
2023-03-20 16:03:50 +01:00
Richard Belleville e41e41e209
Remove protobuf 4.22.0 from testing (#32442)
This version broke backward compatibility in `plugin_pb2.py`, which is
presumably a relatively minor regression, since we have not yet heard
any complaints about it. This PR:

- Excludes `4.22.0` from installation
- _Includes_ protobuf pre-releases into testing so this can be caught
more quickly in the future.

When bad prereleases are caught, we can exclude them from testing in a
similar manner to this PR. We may eventually want to invest into a
system where we can define these bad versions centrally.
2023-02-22 09:40:48 -08:00
apolcyn c1a4b526fb
[ruby]: add pre-compiled binaries for ruby 3.2; drop them for ruby 2.6 (#32089)
[ruby]: add pre-compiled binaries for ruby 3.2; drop them for ruby 2.6
2023-01-19 15:32:26 -08:00
Esun Kim 4b491e3274
Use 64bit linker on windows even when creating 32bit targets (#31237)
* Try CMAKE_VS_PLATFORM_TOOLSET_HOST_ARCHITECTURE

* Target

* fix vcvarsall.bat argument

Co-authored-by: Jan Tattermusch <jtattermusch@google.com>
2022-10-11 10:49:57 -07:00
Jan Tattermusch 108444e9f7
Fix python tests on kokoro macos monterey (#31126)
* fix python build on kokoro macos monterey

* avoid using realpath

* fix grpc_distribtests_python on macos monterey

* try upgrade setuptools
2022-10-10 17:45:31 +02:00
Yash Tibrewal 41ec08c69a
Update third_party/protobuf to 3.21.4 (#30377)
* Update third_party/protobuf

* run tools/distrib/python/make_grpcio_tools.py

* regenerate protos for ruby, php

* update build_handwritten.yaml

* regenerate projects

* Build - Use :well_known_type_protos instead of :well_known_protos

* Fix target

* Update upb

* Update Python for Protobuf 4.21 (#140)

* Update protobuf dependency on grpcio-tools

* Off by one

* Drop python 3.6 support

* Try upgrading pip

* And in the other script

* Try to figure out if we're compatible with abi3

* See what we've already got installed

* Update the requirements.txt file I didn't know existed

* And here too

* See what's installed

* Let's try that again

* Remove

* Try to confirm version

* Let me see the generated code

* Fix non-Bazel test runner

* Work for all test directories

* Regenerate example protos

* Clean up

* Generate .pyi files

* Fix type checking and linting

* Exclude pyi files from isort

* Upgrade to 3.21.4

* Update iwyu to get around messy protobuf IWYU rules

Co-authored-by: Richard Belleville <gnossen@gmail.com>
2022-08-04 09:39:49 -07:00
Richard Belleville dec31be045
Fix Race Condition During grpcio-tests Install on CI (#30093)
* Tryonly running fetch_build_eggs once

* Run pip freeze

* Use the correct pip

* Try explicitly printing stacktrace

* Try to pin/upgrade the wheel package

* Try running only 3.8

* Show the initial state of the egg directory

* Whoops

* More permissive

* What does this button do?

* Try just 3.9

* Try 3.10

* Back to 3.8

* Try pre-installing protobuf

* Clean up

* Clean up better
2022-06-28 11:22:53 -07:00
Esun Kim 0afcc37589
MSVC 2015 -> 2017 (#29754)
* MSVC 2015 -> 2017

* Update by review
2022-05-24 14:09:55 -07:00
Esun Kim ec58a346f9
Changed C99 to C11 (#29687)
* Changed C99 to C11

* Remove gnu99 option

* Switch to gnu11 for xcode
2022-05-20 14:03:04 -07:00
Jan Tattermusch e8404698d5
Remove C# implementation (individual packages will continue to be maintained through v2.46.x patches or moved to grpc-dotnet) (#29225)
* remove examples/csharp

* remove reference to examples/csharp

* remove C# references from doc/

* remove src/csharp

* update test_creds locations

* remove C# distribtests

* remove buildgen templates for files under src/csharp

* remove grpc_csharp_ext C build target

* regenerate projects

* update tools/run_tests/README.md

* change many C# test/build scripts to noop

* unbreak run_tests.py -l csharp

* unbreak C# build_package targets

* unbreak C# distribtests

* remove C# pre-build steps from prepare_build_windows.bat

* remove src/csharp from clang_format

* remove csharp and csharpcoreclr from run_interop_tests.py

* remove C# interop test docker images

* Revert "remove src/csharp"

This reverts commit 97f2b6d707.

* remove src/csharp, but keep Grpc.Tools

* add back templates/src/csharp/build/dependencies.props.template

* remove everything but Grpc.Tools and Grpc.Tools.Tests from sln

* Revert "unbreak run_tests.py -l csharp"

This reverts commit 529b5e2ec1.

* revive Grpc.Tools.Tests through run_tests.py

* Revert "unbreak C# build_package targets"

This reverts commit 79d8d83c8b.

* experiment with build_nuget.sh fixes

* simplify create_fake_native_artifacts.sh

* building of Grpc.Tools in package_targets.py

* move shared files from Grpc.Core to Grpc.Tools

* yapf format: distribtest_targets.py

* Revert "remove C# pre-build steps from prepare_build_windows.bat"

This reverts commit bd38956a2b.

* update C# README.md

* disable C# in gke OSS benchmarks
2022-04-28 12:00:34 +02:00
Craig Tiller 766cd6b573
Remove epollex poller (#29160)
* start delete

* remove build.gn... again

* remove more

* Automated change: Fix sanity tests

* fixes

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2022-03-21 10:27:52 -07:00
Jan Tattermusch 6f75e54e4f
run_tests_matrix.py: run workspace jobs with python3 (#28125) 2022-03-14 12:12:51 +01:00
Jan Tattermusch 9d73b3e85a
Use ninja for all grpc_csharp_ext builds on windows (#28841)
* cleanup C# win artifact build

* build C# basictests on win with ninja

* specify default parallelism for C# win artifact

* honor GRPC_PYTHON_BUILD_EXT_COMPILER_JOBS in build_artifact_python.bat

* set --inner_jobs for windows grpc_build_artifact job

* fixup C# build picking env variables from C core build

* get rid of no longer needed NativeDependenciesConfiguration C# setting
2022-02-11 18:58:04 +01:00
Jan Tattermusch e324bf5eee
Speed up windows C/C++ builds by using cmake Ninja generator (#28833)
* support building C/C++ with ninja

* make cmake_ninja_vs2015 build the default for C/C++

* fix vcvarsall location for vs2017
2022-02-11 09:59:51 +01:00
Jan Tattermusch 0ec0479ded
run_tests.py cleanup and simplification (#28808)
* only support cmake for CLanguage in run_tests.py

* add support for run_tests.py build step environ

* switch C/C++ run_tests.py build to build_cxx script

* CLanguage cleanup

* build C# entirely with build_csharp script

* move entire PHP build to build_php.sh

* fixup C# build on linux and mac

* run_dep_checks Makefile target is deprecated

* get rid of the "makefile" logic in run_tests.py

* fixup C# build on linux and mac

* XML_REPORT env variable is useless for --use_docker runs

* add a TODO

* move "main" functionality towards end of run_tests.py

* use self.args instead of global

* yapf format

* remove the no longer useful --update_submodules features of run_tests.py

* fix check_epollexclusive check in run_tests.py
2022-02-08 18:41:56 +01:00
Jan Tattermusch f23f1bb51f
Use ccache for selected C++ and python builds (with redis server as cache) (#28661)
* add cmake support for ccache

* cleanup: use --env-file for docker run invocations

* make python build compatible with using ccache

* enable building using ccache in selected kokoro jobs

* print ccache stats and the end of run_tests.py
2022-01-25 19:18:58 +01:00
Jan Tattermusch fda6158600
Build ruby artifacts in parallel (#28243)
* build ruby artifacts in parallel

* fine tune grpc_distribtests_ruby.sh parallelism

* fine tune linux/grpc_build_artifacts.sh parallelism

* cleanup in bundle_install_wrapper

* address review comments
2022-01-19 18:28:27 +01:00
Lidi Zheng 7069770ee0
Merge the 3 repeating Python binary compilations (Attempt 2) (#28543)
* Revert "Revert "Merge the 3 repeating Python binary compilations (#28500)" (#28539)"

This reverts commit 0554cbee9c.

* Update hardcoded Python path

* Repect existing images and only use new path for newer releases

* fix interop_matrix testcases

Co-authored-by: Jan Tattermusch <jtattermusch@google.com>
2022-01-14 09:25:33 -08:00
Jan Tattermusch 0554cbee9c
Revert "Merge the 3 repeating Python binary compilations (#28500)" (#28539)
This reverts commit 2d4f3c5600.
2022-01-12 18:31:02 +01:00
Lidi Zheng 2d4f3c5600
Merge the 3 repeating Python binary compilations (#28500)
* Merge the 3 repeating Python binary compilations

* Restore grpcio_metadata.py

* run_tests.py is running on <3.6

* Restore the Windows gevent version pin
2022-01-11 17:14:22 -08:00
apolcyn 3c368a5b2c
Don't gem install rake-compiler on macos setup scripts (#28415)
* Don't gem install rake-compiler on macos setup scripts
2021-12-28 09:57:33 -08:00
Lidi Zheng 83e7cb24fe
Fix the setuptools distutils overriding issue (#28393)
* Fix the distutil install issue

* Try to move setuptools install forward

* Try pin the distribution tools

* Enforce local distutil

* Fix script order of definition

* Add comment

* Try to fix the setuptool monkey patch

* Clean-up imports for commands.py

* Pin 59.6.0 instead of 59.7.0 (incomplete release)
2021-12-21 12:27:01 -08:00
Jan Tattermusch 205d7569ea
Big dockerimage cleanup (#28010)
* use python3 for running run_tests.py

* update dockerimage templates

* update dockerfiles

* fix bazel build in the bazel docker image

* adjust run_tests.py to renamed php docker image

* add previously forgotten new templates

* add missing images

* php dockerfile remove whitespaces

* restore accidentally deleted binder_transport_apk image

* add back cxx_buster_x64 image

* add back php zts stretch image

* add back binder_transport_apk image

* fix bazel builds by temporarily adding back python2-dev

* check_redundant_namespace_qualifiers.py to python3

* fix clang-tidy

* make run_tests_matrix.py use python3

* fixup cxx_debian9 templates

* regenerate

* stop running dummy grpc-node tests for protobuf-at-head and boringssl-at-head jobs

* remove cxx_buster image (replaced by cxx_debian9)

* add back python_alpine_x64 image

* run_tests_in_workspace using python3

* unbreak python macos basictests by partially reverting run_tests_matrix to python2

* fix cxx_gcc_X docker images
2021-12-01 12:56:26 +01:00
Jan Tattermusch ae65bb8f7f
windows: speedup build of C# extension and protoc artifacts (#28225) 2021-12-01 10:56:14 +01:00
Menghan Li 3e34e3aac7
xds/testing: print timestamp of commands in xds tests cripts to help debug timeout failures (#27000)
* xds/testing: print timestamp of commands in xds tests cripts to help debug timeout failures

There were recently many kokoro timeout, after job running for 6 hours.
In those, the run_xds_tests.py only ran less then 2 hours.

There was no enough timestamp to find what was taking so long (either
run_xds_tests.py is stuck, or something before it took too long). This
should add more timestamps to help debug.

* print script start time

* format
2021-08-12 17:12:08 -07:00
Jan Tattermusch 156edd687e
set xds user agent name everywhere where grpc_csharp_ext is built (#26337)
* set xds user agent name everywhere where grpc_csharp_ext is built

* regenerate projects
2021-06-14 08:25:14 +02:00
Lidi Zheng 25f5399b41
Add grpcio-admin Python package (#26166)
* Add grpcio-admin Python package

* Polish package content

* Make Linux artifact build happy

* Improve documentation
2021-05-05 13:20:50 -07:00
Lidi Zheng dc63d6a53e
Add CSDS API to Python (#26114)
* Add grpcio-csds pacakge

* Remove unused file

* Fix the proto import path issue

* Update the CSDS package and xds-protos for PY2

* Make tests happy

* Fix Bazel proto dependency

* Add Python2 tests for CSDS
2021-05-03 11:54:34 -07:00
Lidi Zheng 0b8cb0227b
Upgrade setuptools and ProtoBuf Python in prep_xds.sh (#26029)
* Upgrade

* Trick Kokoro to run xDS test

* Restore grpc_python_bazel_test.cfg

* Update protobuf

* Add extra layer of protection
2021-04-20 17:47:21 -07:00
Lidi Zheng 3625d9f75e
Update the Python dependency for xDS interop test (#26024)
`xds-protos` includes all the generated Python files for Envoy protos, which is required by Envoy.
2021-04-20 12:36:15 -07:00
Richard Belleville 1437dc9816
Fix pypa link (#25647)
* Fix pypa link

* And 3.5 as well
2021-03-08 17:26:52 -08:00
Donna Dionne a734650728 Modify build_python.sh because get-pip.py has moved. 2021-01-25 12:49:39 -08:00
Jan Tattermusch 113f5e89b5 build_ruby.sh: build Debug protoc on config=dbg 2020-12-12 18:07:10 +01:00
Jan Tattermusch c5c123efc9 add cmake to ruby dockerfiles 2020-12-09 11:54:45 +01:00
Jan Tattermusch a35cd4f89d build protoc artifacts with cmake 2020-12-09 11:54:44 +01:00
Lidi Zheng 5527e0f328 Simplify the check with existing if 2020-09-29 13:28:45 -07:00
Lidi Zheng 0bd1c6ae28 Check $VENV_PYTHON instead of python 2020-09-29 13:27:46 -07:00
Lidi Zheng c45f50581b Don't install enum34 for non-2.7 Python 2020-09-29 13:25:56 -07:00
Eric Gribkoff a6a7755c3f Pin grpcio and grpc-tools to 1.31.0
Can remove when https://github.com/grpc/grpc/pull/24094 is in a patch
release
2020-09-09 16:07:49 -07:00
Lidi Zheng c8fa1c9978 Pin setuptools in one more location 2020-08-31 12:26:24 -07:00
Menghan Li a7dc27d4e9 xds testing: check serving status is set correctly
And print what status was set. This helps debugging when the test fails
2020-08-20 23:16:54 -07:00
Jan Tattermusch c67db2e8eb fix pre_build_cmake.bat 2020-08-03 19:18:39 +02:00
Jan Tattermusch 31fa9a3530 switch run_tests.py C/C++ tests to cmake 2020-08-03 19:18:39 +02:00
Alexander Polcyn 9b4ef62cb9 Split up run_ruby_end2end_tests.sh into more granular test targets 2020-07-27 15:07:23 -07:00
Alexander Polcyn 95387716dc Make sure call creds user callbacks can't kill the ruby call credentials thread 2020-06-26 18:29:22 -07:00
Alexander Polcyn b88c0a22c7 Fix use-after-free in ruby call creds 2020-06-26 13:26:54 -07:00
Lidi Zheng 18054754c5 Install rsa==4.0 to avoid the egg cache race issue 2020-06-15 14:19:54 -07:00