Adds experimental fork support to gRPC/Ruby
Works towards https://github.com/grpc/grpc/issues/8798 (see caveats for why this wasn't marked fixed yet)
Works towards https://github.com/grpc/grpc/issues/33578 (see caveats for why this wasn't marked fixed yet)
This leverages existing `pthread_atfork` based C-core support for
forking that python/php use, but there's a bit extra involved mainly
because gRPC/Ruby has additional background threads.
New tests under `src/ruby/end2end` show example usage.
Based on https://github.com/grpc/grpc/pull/33495
Caveats:
- Bidi streams are not yet supported (bidi streams spawn background
threads which are not yet fork safe)
- Servers not supported
- Only linux supported
- Switched from yapf to black
- Reconfigure isort for black
- Resolve black/pylint idiosyncrasies
Note: I used `--experimental-string-processing` because black was
producing "implicit string concatenation", similar to what described
here: https://github.com/psf/black/issues/1837. While currently this
feature is experimental, it will be enabled by default:
https://github.com/psf/black/issues/2188. After running black with the
new string processing so that the generated code merges these `"hello" "
world"` strings concatenations, then I removed
`--experimental-string-processing` for stability, and regenerated the
code again.
To the reviewer: don't even try to open "Files Changed" tab 😄 It's
better to review commit-by-commit, and ignore `run black and isort`.
Fix at-head tests (this is a missing piece of
https://github.com/grpc/grpc/pull/32905) with the following error;
```
/var/local/git/grpc/tools/run_tests/helper_scripts/build_python.sh: line 126: python3.8: command not found
```
Followup for https://github.com/grpc/grpc/pull/32649 (which disabled the
tests mentioned below).
Also sets correct path for tests build by ninja on windows, so that they
don't get skipped.
Once merged, I'll backport to 1.54.x and 1.53.x
The original issue with tests being skipped.
```
+ python3 workspace_c_windows_dbg_native/tools/run_tests/run_tests.py -t -j 8 -x run_tests/c_windows_dbg_native/sponge_log.xml --report_suite_name c_windows_dbg_native -l c -c dbg --iomgr_platform native --bq_result_table aggregate_results --measure_cpu_costs
2023-03-20 07:56:53,523 START: tools\run_tests\helper_scripts\build_cxx.bat
2023-03-20 08:04:51,388 PASSED: tools\run_tests\helper_scripts\build_cxx.bat [time=477.9sec, retries=0:0; cpu_cost=0.0; estimated=1.0]
2023-03-20 08:04:52,434 detected port server running version 21
2023-03-20 08:04:52,672 my port server is version 21
2023-03-20 08:04:52,703 SUCCESS: All tests passed
WARNING: binary not found, skipping cmake/build/Debug/bad_server_response_test.exe
WARNING: binary not found, skipping cmake/build/Debug/connection_refused_test.exe
WARNING: binary not found, skipping cmake/build/Debug/goaway_server_test.exe
WARNING: binary not found, skipping cmake/build/Debug/invalid_call_argument_test.exe
WARNING: binary not found, skipping cmake/build/Debug/multiple_server_queues_test.exe
WARNING: binary not found, skipping cmake/build/Debug/no_server_test.exe
WARNING: binary not found, skipping cmake/build/Debug/pollset_windows_starvation_test.exe
WARNING: binary not found, skipping cmake/build/Debug/public_headers_must_be_c89.exe
```
(part of removing support for VS2017)
Also see https://github.com/grpc/grpc/pull/32649
Also see https://github.com/grpc/grpc/pull/32615
The switch to grpc-win2019 windows workers has already happened:
(cl/517400022).
Once this PR lands, I'll backport to 1.53.x branch as well (since that
release removes the VS2017 support).
Fix incompatibilities identified when running adhoc runs on the new
custom win2019 image.
After merging this, it should be possible to switch to the new image
without breaking any tests.
- for most fixes I added a comment that explains why they're necessary.
- the new image won't have VS2015 installed, so I'm switching the protoc
artifact build to VS2017
This PR will need to be backported to older release branches to ensure
the windows tests continue working on those branches as well (IMHO I
haven't made any changes that would be difficult to backport and I tried
to keeps the diff as small as possible to avoid issues when
backporting).
After we switch to the new image (and all the windows tests are green),
we can incrementally move the builds that are still using VS2017 to
VS2019.
This test has been occasionally failing on CI with "Bus Error" crashes
while requiring the grpc shared library.
These crashes have been unreproducible locally. Let's continue debugging
(b/266212253) but skip this on CI.
Make remaining objC jobs compatible with kokoro monterey workers and
prepare for boringssl upgrade.
The changes here are taken from https://github.com/grpc/grpc/pull/32357,
but they should be merged in a separate PR
(we need the changes to be able to upgrade to monterey anyway and
there's no reason to make the boringssl upgrade PR more complicated by
bundling more fixes into it).
I've checked that the grpc_basictests_objc_examples and
grpc_ios_binary_size are green if switched to monterey.
Unfortunately it's hard to make grpc_basictests_objc_examples pass on
both monterey and mojave, so I suggest merging this PR at the same time
as CL to upgrade the kokoro jobs to monterey.
- that way both PR and continuous runs will remain green
- older branches would need a backport anyway
---------
Co-authored-by: Hannah Shi <hannahshisfb@gmail.com>
Cleanup and remove ios cpp test cronet
To test manually:
./tools/bazel test //src/objective-c/tests:CppCronetTests
@sampajano
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
* Update third_party/protobuf
* run tools/distrib/python/make_grpcio_tools.py
* regenerate protos for ruby, php
* update build_handwritten.yaml
* regenerate projects
* Build - Use :well_known_type_protos instead of :well_known_protos
* Fix target
* Update upb
* Update Python for Protobuf 4.21 (#140)
* Update protobuf dependency on grpcio-tools
* Off by one
* Drop python 3.6 support
* Try upgrading pip
* And in the other script
* Try to figure out if we're compatible with abi3
* See what we've already got installed
* Update the requirements.txt file I didn't know existed
* And here too
* See what's installed
* Let's try that again
* Remove
* Try to confirm version
* Let me see the generated code
* Fix non-Bazel test runner
* Work for all test directories
* Regenerate example protos
* Clean up
* Generate .pyi files
* Fix type checking and linting
* Exclude pyi files from isort
* Upgrade to 3.21.4
* Update iwyu to get around messy protobuf IWYU rules
Co-authored-by: Richard Belleville <gnossen@gmail.com>
* stop running objc plugin tests with run_tests.py
* stop running bazel examples as part of run_tests.py
* stop running UnitTests and MacTests as part part of run_tests.py
* objC bazel test: run InteropTests and MacTests
* objC bazel test: run TvTests
* Revert "objC bazel test: run TvTests"
This reverts commit 46a81991a9.
* avoid running flaky objC InteropTests under bazel
* add TODOs for objc run_tests.py tasks
* add a note about GRPC_VERBOSITY=debug being set
* increase timeout for ios-buildtest-example-tvOS-sample test task
* only support cmake for CLanguage in run_tests.py
* add support for run_tests.py build step environ
* switch C/C++ run_tests.py build to build_cxx script
* CLanguage cleanup
* build C# entirely with build_csharp script
* move entire PHP build to build_php.sh
* fixup C# build on linux and mac
* run_dep_checks Makefile target is deprecated
* get rid of the "makefile" logic in run_tests.py
* fixup C# build on linux and mac
* XML_REPORT env variable is useless for --use_docker runs
* add a TODO
* move "main" functionality towards end of run_tests.py
* use self.args instead of global
* yapf format
* remove the no longer useful --update_submodules features of run_tests.py
* fix check_epollexclusive check in run_tests.py
* Revert "Revert "Cleanup run docker machinery in run_tests.py, task_runner.py and elsewhere (#28704)" (#28741)"
This reverts commit 9a79d44e9b.
* fix python distribtest failure on fedora34
* attempt
* slim down build_docker_and_run_tests.sh
* HOST_GIT_ROOT is unused
* config and arch variable no need to propagate to run_tests.py
* unify unused LOCAL_GIT_ROOT to EXTERNAL_GIT_ROOT
* mounting gcloud config seems useless
* reorder docker args
* cleanup
* allow passing args to build_docker_and_run_tests.sh
* converge build_docker_and_run_tests.sh and build_and_run_docker.sh
* more convergence
* convergence of docker runners
* finalize convergence of run_docker scripts
* GRPC_TEST_REPORT_BASE_DIR might not exist
* adjust report copying
* make report_dir and output_dir readable
* alpine linux does not support cp -t
* upgrade ruby docker images to debian 11
* upgrade C# docker images to debian11
* update sanity dockerimage to debian11
* upgrade cxx interop to debian11
* add ccache to python interop images
* enable use of ccache for interop tests
* adjust run_tests.py to new docker images
* add ccache to rake-compiler-dock docker images
* improve prepare_ccache_symlinks_rc
* enable use of ccache in rake-compiler-dock docker containers
* add ccache support for python_manylinux2014_aarch64
* add ccache support for python_linux_armv7
* deduplicate python3.9 install
* ccache for crosscompiled darwin gems is broken
* fix bash -l resetting of PATH in grpc_artifact_python_linux_armv7
* try speeding up linux portability test
* remove gcc8.3 portability test
* c-ares is the default resolver on linux
* dont set parallel run_tests jobs too high
* original parallelism might lead to better results
apolcyn