Commit Graph

54619 Commits

Author SHA1 Message Date
veblush fb93164443 Automated change: Fix sanity tests 2024-03-06 16:36:53 +00:00
Esun Kim 02276f4774
[Misc] Added third_party/utf8_range to .bazelignore (#36054)
This is to clam down bazel which is eager to build everything in the
repo.
2024-03-06 08:31:48 -08:00
Alexander Polcyn 1491233a80 Expose LB policy test lib SubchannelState::state_tracker_ to tests
PiperOrigin-RevId: 613029941
2024-03-05 18:07:09 -08:00
Alexander Polcyn 419a070078 Allow configuring server_notify_client_when_started field in xds e2e test lib
While we're here, also change the Echo service handler to mutate clients before handling the RPC, mainly so that it can be reasoned about in cases involving client cancellation.

PiperOrigin-RevId: 613029541
2024-03-05 17:57:46 -08:00
AJ Heller 25ef96c4ad [test] Enable callback benchmarks (#36055)
Closes #36055

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36055 from drfloob:enable-callback-benchmarks 42283a52c9
PiperOrigin-RevId: 612975697
2024-03-05 14:47:25 -08:00
dmaclach c332854964 Use clock_gettime on Apple platforms (#35923)
Apple platforms have supported clock_gettime for several years now (since 2016).

Simplify implementation and remove required api call to `mach_absolute_time`.

<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35923

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35923 from dmaclach:patch-4 45f670d10e
PiperOrigin-RevId: 612965724
2024-03-05 14:17:34 -08:00
Esun Kim 52fc023bae [CI] Upgraded cocoapods to 1.12.0 (#36053)
This is needed to accommodate the recent Protobuf v26 change requiring cocoapod 1.12 or later.

Closes #36053

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36053 from veblush:cocoapod-1.12 ab94e124a6
PiperOrigin-RevId: 612942897
2024-03-05 13:08:19 -08:00
Alexander Polcyn 6f6f0df2ba Make TestServiceSignaller more generically useful
There are tests in another change I'm working on for which we need to do things like:

1) waiting for N RPCs to concurrently pile up on a server

2) doing 1) multiple times sequentially on the same server

PiperOrigin-RevId: 612934305
2024-03-05 12:40:05 -08:00
Mark D. Roth db9ca10b35 [xDS] add optional locality label to per-call metrics (#36049)
Closes #36049

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36049 from markdroth:xds_locality_label_on_per_call_metrics 9c5aeb7fb1
PiperOrigin-RevId: 612648213
2024-03-04 17:43:49 -08:00
Yash Tibrewal 692b7fcb7f
[bazel] Add OTel C++ submodule to bazelignore list (#36052) 2024-03-04 17:29:50 -08:00
Xuan Wang 672d8abdca [Python Otel] Allow start observability without context manager (#35932)
Allow start observability globally with a new API `start_open_telemetry_observability`.

<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35932

PiperOrigin-RevId: 612639020
2024-03-04 17:08:15 -08:00
Craig Tiller 427c8a89e9 [chaotic-good] Add a microbenchmark for ping pong round trips (#36050)
also:
- remove tail recursion from promise endpoint read completion (actually overflowed stack!)
- remove retry filter from benchmark - we probably don't want this long term, but for now nobody else is using this benchmark and our use case doesn't use grpc retries so.... good enough

Closes #36050

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36050 from ctiller:cgbm 65b1c26767
PiperOrigin-RevId: 612577071
2024-03-04 14:07:28 -08:00
Mark D. Roth c43f1a63b0 [RLS] implement non-per-call metrics (#36001)
Closes #36001

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36001 from markdroth:non_per_call_metrics_rls b713980f21
PiperOrigin-RevId: 612532005
2024-03-04 11:58:31 -08:00
Mark D. Roth 0213523907 [build] move channel out of grpc_base (#35924)
This adds the following new targets:
- `channel`: A virtual interface for a channel.
- `legacy_channel`: A channel implementation that supports the filter stack and call v2.
- `channel_create`: A standalone function to create a channel.
- `server_interface`: A base class with a few accessor methods used in surface/call.cc.
- `server`: The actual server implementation.
- `api_trace`, `call_tracer`, `server_call_tracer_filter`, `call_finalization`: These were split out of `grpc_base` to avoid various dependency problems.
- `compression`: This is a combination of the previously existing `compression_internal` target and the compression code that was part of `grpc_base`.

Closes #35924

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35924 from markdroth:channel_interface 94a7fffddb
PiperOrigin-RevId: 612512438
2024-03-04 11:07:17 -08:00
Yash Tibrewal 4ce4f4ea3a [OTel] Add OTel C++ as a submodule (#36032)
Closes #36032

PiperOrigin-RevId: 611657802
2024-02-29 18:06:02 -08:00
tony 5043eb4d65 [csharp] Fix csharp_ubuntu2204_x64 docker image (#36019)
Fix the docker image for C# distribution tests - csharp_ubuntu2204_x64

The package that installed the .NET 7.0 SDK is missing setting up links and thus the `dotnet` command cannot find the SDK. Setting this links has been added to the dockerfile.

**NOTE**: I'm unable to update `tools/dockerfile/distribtest/csharp_ubuntu2204_x64.current_version` as I don't have access to the artifact registry.

Closes #36019

PiperOrigin-RevId: 611590679
2024-02-29 14:10:36 -08:00
AJ Heller 73c141cb57 Add `byte_buffer` dependency back to gRPC iomgr target.
PiperOrigin-RevId: 611518672
2024-02-29 10:33:42 -08:00
Mark D. Roth de3e6f8234 [pick_first] implement non-per-call metrics (#35984)
As per gRFC A78 (https://github.com/grpc/proposal/pull/419).

Closes #35984

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35984 from markdroth:non_per_call_metrics_pf 02db004d5a
PiperOrigin-RevId: 611475162
2024-02-29 08:12:55 -08:00
Tanvi Jagtap e39bd50716 [grpc] Redacting unknown metadata types in debug logs. (#36006)
Redacting unknown metadata types in debug logs.

<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #36006

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36006 from tanvi-jagtap:tjagtap_redact_01 94c5738bfe
PiperOrigin-RevId: 611334483
2024-02-28 21:18:41 -08:00
Bernhard M. Wiedemann d379dc24c5
[Build Reproducibility] Sort input file list (#35687)
Sort input file list
so that the `.so` file builds in a reproducible way
in spite of non-deterministic filesystem readdir order and
http://bugs.python.org/issue30461

See https://reproducible-builds.org/ for why this is good.

This patch was done while working on reproducible builds for openSUSE.




<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->

---------

Co-authored-by: Richard Belleville <rbellevi@google.com>
2024-02-28 17:05:07 -08:00
Xuan Wang 9b424b5dd4 [Python o11y] Fix Python O11Y artifacts name (#35965)
Update observability to PYPI failed because the artifact name is not correct.

This PR:
* Fix the artifacts name.
* Add step to test observability artifacts in `test_packages.sh`.
* Added `-fno-ipa-cp` compile flag.
  * We're seeing `inlining failed in call to always_inline 'vsnprintf': function body can be overwritten at link time` errors when building from source using musl libc.
  * Based on [investigation](https://gitlab.alpinelinux.org/alpine/aports/-/issues/8626), it's because we're using `-flto` flag.
  * One solution is to [disable fortify by adding this flag](https://gitlab.alpinelinux.org/alpine/aports/-/issues/8626).

After this PR, the observability artifacts have the correct name:
* [Distribution Tests Python Linux](https://pantheon.corp.google.com/storage/browser/grpc-testing-kokoro-prod/test_result_public/prod/grpc/core/pull_request/linux/grpc_distribtests_python/28687/20240223-105306/github/grpc/artifacts;tab=objects?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&e=13802955&mods=-logs_tg_prod&prefix=&forceOnObjectsSortingFiltering=false)

Also tested that the artifacts build in this PR can be successfully uploaded to testpypi:
* https://test.pypi.org/project/grpcio-observability/1.63.0.dev0

<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35965

PiperOrigin-RevId: 611268154
2024-02-28 16:33:28 -08:00
Craig Tiller 493cad8ef2 [call-v3] Grab payload handling from registered method (#36017)
We were skipping reading a request payload, which led to a serialization failure, which led to UNIMPLEMENTED errors.

Closes #36017

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36017 from ctiller:goodness 606545295f
PiperOrigin-RevId: 611254339
2024-02-28 15:47:54 -08:00
AJ Heller 00c01f395c [EventEngine] Do not accept connections if the listener is shut down (#35990)
There's a race with the iomgr/EventEngine shims on listener shutdown, specifically for the PosixEventEngine implementation. This showed up as TSAN failures, such as in https://source.cloud.google.com/results/invocations/539def09-0647-4508-960a-f22b76e76dae/targets/%2F%2Ftest%2Fcore%2Fend2end:retry_cancel_after_first_attempt_starts_test@poller%3Depoll1/log. This PR alters the lifetime of the grpc_tcp_server object and its constituent members, letting the EventEngine listener manage it when EE listeners are enabled. Previously, some member objects were destroyed before the tcp server itself was destroyed.

Notes:
 * This is best reviewed once with whitespace diff turned off. Blocks was added to manage mutex lifetimes.
 * BUILD file changes are result of `fix_build_deps.py :iomgr`

Closes #35990

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35990 from drfloob:fix/proxy-listener-accept d35eed5d16
PiperOrigin-RevId: 611149345
2024-02-28 10:35:37 -08:00
Gregory Cooke d6089c336c [Security - CrlProvider] Add AKID to CertificateInfo (#35931)
This PR adds the Authority Key Identifier to CertificateInfo. This value _can be_ important in finding the right CRLs to use if there are Issuer name overlaps or a more complicated CA setup with multiple signing keys.

We should observe no behavior change in our `CrlProvider` implementations, this is just adding an important field for users who implement it themselves.

Closes #35931

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35931 from gtcooke94:AkidCheck dd048a53b6
PiperOrigin-RevId: 611143198
2024-02-28 10:19:05 -08:00
Yousuk Seung d4b5e8d11d Move work_serializer_dispatch dependency to rollouts (#36013)
<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #36013

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36013 from yousukseung:work-serializer 8fdf679309
PiperOrigin-RevId: 610938472
2024-02-27 18:25:19 -08:00
Zach Reyes 0d69f8dc74 Add gRPC-Go version 1.62.0 to interop matrix (#35966)
Interop run: https://fusion2.corp.google.com/ci/kokoro/prod:grpc%2Fcore%2Fexperimental%2Flinux%2Fgrpc_interop_matrix_adhoc/activity/99858b52-009a-404b-a9a7-1546e239d5d9/summary
<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35966

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35966 from zasweq:go-release-interop 3fa9483786
PiperOrigin-RevId: 610848138
2024-02-27 13:08:12 -08:00
Alisha Nanda fa2bae473a [metadata] Expose metadata limits and annotation info to filter stack (#35970)
<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35970

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35970 from ananda1066:client_metadata b62a9ae146
PiperOrigin-RevId: 610830228
2024-02-27 12:07:35 -08:00
Vignesh Babu 338fc05c21 Add stats to tcp_posix.cc to log commonly seen I/O errnos.
PiperOrigin-RevId: 610799993
2024-02-27 10:43:44 -08:00
apolcyn 9054988785 [csharp] Use a stable/supported C# runtime in ubuntu 22.04 distribtest docker images (#35955)
The current .NET SDK in these tests is EOL, and we've been seeing strange SSL errors transiently.

Update to a supported version and remove cruft in the docker image that was there to install the older version.

TODO: submitting first with the ubuntu 22.04 image test skipped, need to follow-up with a fix and unskip the test

Closes #35955

PiperOrigin-RevId: 610798209
2024-02-27 10:34:33 -08:00
Richard Belleville 12897ab0fb [Interop] Make C++ xDS Interop Client and Server Responsive to SIGTERM (#34518)
This PR adds `tini` to the C++ interop client and server to make them responsive to SIGTERM.

Closes #34518

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/34518 from gnossen:ssa_interop_server a762b27178
PiperOrigin-RevId: 610766377
2024-02-27 08:52:21 -08:00
Mark D. Roth 40577dd585 [WRR] implement non-per-call metrics (#35977)
As per gRFC A78 (https://github.com/grpc/proposal/pull/419).

Note that these new metrics are populated only if the `wrr_delegate_to_pick_first` experiment is enabled, which is the case by default.

Closes #35977

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35977 from markdroth:non_per_call_metrics_wrr 7acea32fc4
PiperOrigin-RevId: 610760874
2024-02-27 08:32:09 -08:00
Craig Tiller 571da7be78 Make channel idle filter more aggressive in cleaning up calls
If call objects leak we never declare a channel idle, which can lead to fd leaks in other processes.

PiperOrigin-RevId: 610613450
2024-02-26 20:33:43 -08:00
Yash Tibrewal 8b4079f4dd [shutdown] Add logs when shutdown is done (#36002)
Needed for some shutdown debugging. I was thinking whether to use `fprintf` to stderr directly instead of using `gpr_log`, the reason being that grpc has shutdown and it may be bad to use `gpr_log` after shutdown. Currently atleast, we do not modify the `gpr_log` function on `init`/`shutdown` so it seems fine to continue using it. Additionally, given that this is just for own internal debugging purposes, I don't want to sprinkle `fprintf` around if it's not necessary.

Closes #36002

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36002 from yashykt:LogOnGrpcShutdown 61b9ad573a
PiperOrigin-RevId: 610515417
2024-02-26 13:51:25 -08:00
Eugene Ostroukhov 70af2c2fda [xds] xDS client per data plane target (#35730)
Closes #35730

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35730 from eugeneo:tasks/multiple-xds-clients 30b459658e
PiperOrigin-RevId: 610511488
2024-02-26 13:38:21 -08:00
Gregory Cooke a1b7e1e523 [Security] Crl helpers (#35963)
Re-applies after the revert
Fixes the Openssl 1.0.2 portability errors

Closes #35963

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35963 from gtcooke94:CrlHelpers 9cfb401a47
PiperOrigin-RevId: 610503595
2024-02-26 13:12:36 -08:00
Yousuk Seung 641213ee58 [tracing] Remove debugging annotations
PiperOrigin-RevId: 610469149
2024-02-26 11:22:52 -08:00
Jan Katins d4afc993ed
[python] Fix typo in code comment (#35981)
Found this while reading the code.
2024-02-26 09:22:44 -08:00
Craig Tiller d2efbfe449 Log gRPC experiments at INFO
Often we need to debug something after a crash and this set of breadcrumbs would be hugely beneficial.

PiperOrigin-RevId: 610285566
2024-02-25 21:37:06 -08:00
Craig Tiller 4ac804bec9 [call-v3] Correctly detect v3 filters even if the immediate base class is not ImplementCallFilter<> (#35997)
Fixes max_age filter not getting a v3 init_call implementation

Closes #35997

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35997 from ctiller:naughty df17e57e91
PiperOrigin-RevId: 610194630
2024-02-25 10:12:07 -08:00
Yousuk Seung ed065878e0 [experiments] enable work_serializer_dispatch (posix) (#35992)
<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35992

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35992 from yousukseung:work-serializer-windows 7b21f46dde
PiperOrigin-RevId: 610042481
2024-02-24 13:22:54 -08:00
Yousuk Seung 77879c4be7 Clarify caller should call dtor with New()
PiperOrigin-RevId: 609863609
2024-02-23 16:12:55 -08:00
Yousuk Seung ad06808bf6 [tracing] Pass index in stream for traced byte ranges
PiperOrigin-RevId: 609821314
2024-02-23 13:35:33 -08:00
Xuan Wang ca36ff84ba [Python o11y] Fix readme (#35988)
Fix `readme.rst`.

Tested locally by running `python setup.py doc`, no more errors from observability.

Only error is related to grpc_status:
```
WARNING: autodoc: failed to import module 'rpc_status' from module 'grpc_status'; the following exception was raised:
cannot import name 'status_pb2' from 'google.rpc' (unknown location)
```

<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35988

PiperOrigin-RevId: 609820414
2024-02-23 13:26:20 -08:00
AJ Heller 6c29a8720e [test] Remove passthru_endpiont and its microbenchmarks (#35986)
Closes #35986

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35986 from drfloob:rm-passthru-endpoint 12c491f7f6
PiperOrigin-RevId: 609800868
2024-02-23 12:13:24 -08:00
Björn Svensson 1ad14040ef [EventEngine] Set DSCP on listener socket (#35983)
Prepare the listener socket with the configured DSCP value to mark outgoing packets correctly.
This is already done correctly for clients.

During implementation and delivery of the DSCP feature both experiments `event_engine_listener` and `event_engine_client` were default `OFF`. Unfortunately the `event_engine_listener` experiment was not enabled correctly during verification, which resulted in that the working `iomgr` codepath was used instead.
This PR adds the missing action which I now have verified correctly.

Fixes #35954

Closes #35983

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35983 from Nordix:fix-dscp 72969db88e
PiperOrigin-RevId: 609770886
2024-02-23 10:38:30 -08:00
Craig Tiller 3d32d7a15d [chaotic-good] Extend connection timeouts (#35937)
5 seconds is really too small when we've got dozens of connections being established on a busy system. We need a deadline, but it can be significantly looser.

Closes #35937

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35937 from ctiller:timeout1 dc26a82985
PiperOrigin-RevId: 609587934
2024-02-22 20:12:09 -08:00
Richard Belleville 2ad297344d [Documentation] Make Bazel requirements installable directly via pip (#35973)
This fixes the documentation generation tooling.

Closes #35973

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35973 from gnossen:requirements_bazel_fix 1e78fab008
PiperOrigin-RevId: 609544722
2024-02-22 16:56:22 -08:00
Yijie Ma fb4c043803 [Metrics] gRPC Non-Per-Call Metrics framework implementation (#35871)
<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35871

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35871 from yijiem:grpc-metrics 86ebe484ae
PiperOrigin-RevId: 609533796
2024-02-22 16:17:01 -08:00
Eugene Ostroukhov 5838f6840a [release] Add 1.62 to interop matrix (#35972)
Closes #35972

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35972 from eugeneo:release1.62/interop-matrix 1bc8e0f4c7
PiperOrigin-RevId: 609514657
2024-02-22 15:15:27 -08:00
AJ Heller fc412cd390 [test] Add comment about workaround (re #35969) (#35971)
Followup from #35969

This also fixes a few scenarios we had not caught earlier.

Closes #35971

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35971 from drfloob:doc 32de2d7033
PiperOrigin-RevId: 609509072
2024-02-22 14:58:27 -08:00