Commit Graph

6435 Commits

Author SHA1 Message Date
Yash Tibrewal 387c894117 [CSM] Remove experimental CSM PluginOption API in favor of CsmObservability API (#35812)
Also update interop tests to use `CsmObservability` API

Closes #35812

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35812 from yashykt:UpdateCsmInterop dc99764aea
PiperOrigin-RevId: 604726881
2024-02-06 12:20:23 -08:00
Mark D. Roth 10e83973e7 [reorg] move resolver code to src/core/resolver (#35804)
This new directory combines code from the following locations:
- src/core/ext/filters/client_channel/resolver
- src/core/lib/resolver

Closes #35804

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35804 from markdroth:client_channel_resolver_reorg2 30660e6b00
PiperOrigin-RevId: 604665835
2024-02-06 09:07:56 -08:00
Mark D. Roth 148f59c15a [reorg] move LB policy code to src/core/load_balancing (#35786)
This new directory combines code from the following locations:
- src/core/ext/filters/client_channel/lb_policy
- src/core/lib/load_balancing

Closes #35786

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35786 from markdroth:client_channel_resolver_reorg 98554efb98
PiperOrigin-RevId: 604351832
2024-02-05 10:05:35 -08:00
Yash Tibrewal 20e5e0cb29 [CSM] Modify CsmObservability to CsmOpenTelemetryPluginOption internally (#35803)
Changes -
* `CsmObservability` API will now use the `CsmOpenTelemetryPluginOption` internally. After this change, `CsmObservability` will enable observability for all channels and servers. (Earlier, `CsmObservability` only enabled observability for CSM-enabled channels and servers.) CSM labels will still be added just for CSM-enabled channels and servers.
* Also, we no longer need the ability to set `LabelInjector` on the `OpenTelemetryPluginBuilder` directly. Instead, we always use `PluginOption` to inject the `LabelInjector`. This simplifies the code as well.

Note that `SetTargetSelector` and `SetServerSelector` APIs on the `OpenTelemetryPluginBuilderImpl` are not being deleted yet since we might need them shortly. This is also why `OpenTelemetryPluginBuilderImpl` is not being deleted right now.

Closes #35803

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35803 from yashykt:CsmO11yApisUsePluginOption cf3d65900d
PiperOrigin-RevId: 604323898
2024-02-05 08:21:58 -08:00
Stanley Cheung 48961e8a2c [PSM Interop] Add a payload to xds interop client when sending RPCs (#35545)
When testing CSM Observability, we discovered that the c++ xds interop client is not sending any payload with the `UnaryCall` RPCs so most of the metrics will have a value of 0.

Adding a payload to the xds interop client here.

We need this fix so that we can verify that the metrics are recording the right number of bytes being sent / received. So we need a non-trivial payload to be sent with the `UnaryCall` RPC between the xds interop client and server.

Closes #35545

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35545 from stanley-cheung:xds-client-payload 11be4e6f4f
PiperOrigin-RevId: 601596246
2024-01-25 16:12:45 -08:00
Craig Tiller ab795b0adc Revert "[EventEngine] Enable the EventEngine DNS Resolver on Posix (#35573)" (#35667)
This reverts commit 6318e9e7e9.

<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35667

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35667 from ctiller:a 032999b51e
PiperOrigin-RevId: 601495207
2024-01-25 10:46:47 -08:00
Yash Tibrewal 76c45b98d1 [otel] Return absl::Status as a return from BuildAndRegisterGlobal (#35659)
Just to be future-proof, I'm amending the `void` return status of `BuildAndRegisterGlobal` in `OpenTelemetryPluginBuilder` to absl::Status.

This will be backported to 1.61 as well.

Closes #35659

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35659 from yashykt:UpdateOtelApiToAddStatus 07d3f41b8a
PiperOrigin-RevId: 601458408
2024-01-25 08:40:16 -08:00
Yash Tibrewal 984daf98d7 [CSM o11y] Re-experimentalize CSM OTel Plugin Option (#35660)
We are no longer sure about this API, so re-experimentalizing it.

This PR will be backported to 1.61 as well.

Closes #35660

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35660 from yashykt:ReexperimentalizeCsmPluginOption 4f114a54d9
PiperOrigin-RevId: 601378856
2024-01-25 01:57:24 -08:00
Yijie Ma 6318e9e7e9 [EventEngine] Enable the EventEngine DNS Resolver on Posix (#35573)
<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35573

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35573 from yijiem:enable-oss-ee-dns-posix-real 017b99312f
PiperOrigin-RevId: 601245249
2024-01-24 15:04:27 -08:00
Yash Tibrewal 4cea5d0cba [xDS Interop] Use XdsServerBuilder when testing CSM Observability (#35634)
[prod:grpc/core/master/linux/grpc_xds_k8s_lb](https://fusion2.corp.google.com/ci/kokoro/prod:grpc%2Fcore%2Fmaster%2Flinux%2Fgrpc_xds_k8s_lb/activity/6c2577f0-965b-4f22-ac2a-5454a0f77444/log)

Closes #35634

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35634 from yashykt:CsmInteropTrial a8213cd10a
PiperOrigin-RevId: 601010836
2024-01-23 22:33:57 -08:00
Yash Tibrewal f51f14e70c [otel] Re-structure otel_plugin_test initialization (#35631)
Closes #35631

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35631 from yashykt:UpdateOtelPluginTest ba1a9cecc2
PiperOrigin-RevId: 600950962
2024-01-23 17:11:09 -08:00
Yijie Ma 16b71d91d8 [CSM O11Y] Fix issue when CSM optional labels are present in server metrics (#35633)
<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35633

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35633 from yijiem:labels-injector-patch 6876243943
PiperOrigin-RevId: 600931754
2024-01-23 15:55:49 -08:00
David Chamberlin 5b724c09c5 [tls] Add copy constructor for TlsCredentialsOptions (#35499)
<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35499

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35499 from dawidcha:cred_opts_copy_constr 330165930f
PiperOrigin-RevId: 599977221
2024-01-19 17:21:50 -08:00
Esun Kim bc04417440 [Clean-up] Fix Abseil-usage (#35590)
Some minor fixes for https://github.com/grpc/grpc/pull/35589

Closes #35590

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35590 from veblush:abseil-fix a89b800739
PiperOrigin-RevId: 599524969
2024-01-18 09:09:35 -08:00
Yash Tibrewal 8231340aef [CSM] Add support for GCE resources (#35371)
Add support for GCE resources in CSM Observability.

Additionally, fix a bug where we were not adding the remote workload's canonical service label for unknown resource types.

Also, if zone and region are both specified, zone takes precedence.

Closes #35371

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35371 from yashykt:GceSupportToCsm e3064d8c3c
PiperOrigin-RevId: 597989825
2024-01-12 16:30:16 -08:00
ericsalo a6823b6e97 Fix deprecation warnings from upb (#35510)
Replace "upb:collections" with "upb:message"
Replace "upb/collections" with "upb/message"
Replace "upb:upb" with "upb:mem" and/or "upb:base"
Replace "upb/upb.hpp" with "upb/mem/arena.hpp" and/or "upb/base/status.hpp"

Closes #35510

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35510 from ericsalo:master 112392a050
PiperOrigin-RevId: 597872730
2024-01-12 10:15:56 -08:00
Yash Tibrewal c5a8e5af64 [OTel] Add back server_selector that got deleted by a merge fiasco (#35532)
It looks like this ended up getting deleted in https://github.com/grpc/grpc/pull/34350 probably when merging.

Also, the `Init` method in the otel test library is getting unwieldy. I'm going to send out a follow-up PR to convert this into a builder instead.

Closes #35532

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35532 from yashykt:OTelPluginBuilderFix 372bf26338
PiperOrigin-RevId: 597846622
2024-01-12 08:33:17 -08:00
Yash Tibrewal acc1ad1b2b [CSM] De-experimentalize CSM OTel Plugin Option (#35526)
Closes #35526

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35526 from yashykt:DeexperimentalizeCsmPluginOption 441eccb857
PiperOrigin-RevId: 597675383
2024-01-11 15:55:26 -08:00
Yash Tibrewal 5880e05e9d [OTel] Fix clang-tidy caused by merge fiasco (#35529)
Closes #35529

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35529 from yashykt:FixMasterClangTidy eb44716ffa
PiperOrigin-RevId: 597660074
2024-01-11 14:50:54 -08:00
Yijie Ma 77ad5a786e [CSM O11Y] CSM Service Label Plumbing from LB Policies to CallAttemptTracer (#35210)
<!--

If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the appropriate
lang label.

-->

Closes #35210

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35210 from yijiem:csm-service-label 6a6a7d1774
PiperOrigin-RevId: 597641393
2024-01-11 13:37:44 -08:00
Yash Tibrewal d2cc24f189 [OTel] De-experimentalize API (#35509)
Closes #35509

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35509 from yashykt:DeeexperimentalizeOTelPlugin f5edf94f2c
PiperOrigin-RevId: 597613307
2024-01-11 11:44:57 -08:00
Yash Tibrewal 6741e2064d [xds interop] Use new CSM Observability APIs (#35508)
Closes #35508

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35508 from yashykt:XdsInteropNewCsm 09d408c082
PiperOrigin-RevId: 597588033
2024-01-11 10:15:04 -08:00
Mark D. Roth aa326c947c [xDS] fix aggregate cluster design (#35313)
Implements the aggregate cluster changes described in gRFC A75 (https://github.com/grpc/proposal/pull/405).

Closes #35313

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35313 from markdroth:xds_aggregate_cluster_fix 85cfd70c59
PiperOrigin-RevId: 597326624
2024-01-10 12:40:56 -08:00
Eugene Ostroukhov e73b76a7da [Test] Fix bug in waiting for the Orca OOB report (#35467)
Make sure there is no unnecessary delays when there are multiple reports in the queue.

This change also adds a test for the custom LB policy.

Closes #35467

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35467 from eugeneo:tasks/orca-test-timeout-316026521 4aab50a118
PiperOrigin-RevId: 597007131
2024-01-09 12:21:45 -08:00
Yash Tibrewal 5888738c5a [OTel] Add a PluginOption API (#35434)
Closes #35434

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35434 from yashykt:OTelPluginOption 1db870bed4
PiperOrigin-RevId: 596966190
2024-01-09 18:28:43 +00:00
Gregory Cooke b5c5b7a264 [OpenSSL] Disable another test (#35461)
This accidentally slipped through the cracks on https://github.com/grpc/grpc/pull/35444

Closes #35461

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35461 from gtcooke94:fix_ossl102_part2 c8b526cefa
PiperOrigin-RevId: 596135991
2024-01-05 19:42:36 -08:00
Craig Tiller 2694f87c23 [channel] Remove test only feature (#35465)
It's not clear to me that this one unit test of very marginal importance warrants 8 bytes per channel.

Closes #35465

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35465 from ctiller:we-dont-need-this-really e7ee62ccb2
PiperOrigin-RevId: 596091614
2024-01-05 15:03:33 -08:00
Gregory Cooke 61a841b208 Disable failing openssl102 tests (#35354)
There are a select few tests that are failing when building with OpenSSL102 - disable them until we can fix.

Closes #35354

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35354 from gtcooke94:fix_ossl_102 8708d6ce86
PiperOrigin-RevId: 595761932
2024-01-04 11:43:09 -08:00
Mark D. Roth c7101d0867 [xDS] move CDS and EDS watchers into xds resolver (#35011)
Implements gRFC A74 (https://github.com/grpc/proposal/pull/404).

Closes #35011

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35011 from markdroth:xds_watchers_in_xds_resolver a39f71f37f
PiperOrigin-RevId: 595134549
2024-01-02 09:20:06 -08:00
Yash Tibrewal c12a5645f7 [OTel] Experimental API for metrics (#35348)
Provide a public experimental API and bazel compatible build target for OpenTelemetry metrics.

Details -
* New `OpenTelemetryPluginBuilder` class that provides the API specified in https://github.com/grpc/proposal/blob/master/A66-otel-stats.md
* The existing `grpc::internal::OpenTelemetryPluginBuilder` class is moved to `grpc::internal::OpenTelemetryPluginBuilderImpl` for disambiguation.
* Renamed `OTel` in some instances to `OpenTelemetry` for consistency.

Closes #35348

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35348 from yashykt:OTelPublicApi e32328825e
PiperOrigin-RevId: 594271246
2023-12-28 09:45:29 -08:00
Esun Kim 594d370819 [Build] Prepared for Bazel 7 (#35374)
This is a prerequisite change to start supporting Bazel 7. Changes are

- Disabled bzlmod which Bazel 7 begins to enable by default. This eventually needs to be done to support bzlmod but not now.
- Upgraded some bazel rule dependencies which are required to support Bazel 7.
- Using Python 3 explcitly as Bazel 7 begins to reject Python 2.

Note that this isn't enough to enable Bazel 7 by default and another PR will follow for that.

Closes #35374

PiperOrigin-RevId: 592931675
2023-12-21 13:02:38 -08:00
Mark D. Roth 3e785d395d [RefCounted and friends] Fix type safety of ref-counted types.
Previously, `RefCountedPtr<>` and `WeakRefCountedPtr<>` incorrectly allowed
implicit casting of any type to any other type.  This hadn't caused a
problem until recently, but now that it has, we need to fix it.  I have
fixed this by changing these smart pointer types to allow type
conversions only when the type used is convertible to the type of the
smart pointer.  This means that if `Subclass` inherits from `Base`, then
we can set a `RefCountedPtr<BaseClass>` to a value of type
`RefCountedPtr<Subclass>`, but we cannot do the reverse.

We had been (ab)using this bug to make it more convenient to deal with
down-casting in subclasses of ref-counted types.  For example, because
`Resolver` inherits from `InternallyRefCounted<Resolver>`, calling
`Ref()` on a subclass of `Resolver` will return `RefCountedPtr<Resolver>`
rather than returning the subclass's type.  The ability to implicitly
convert to the subclass type made this a bit easier to deal with.  Now
that that ability is gone, we need a different way of dealing with that
problem.

I considered several ways of dealing with this, but none of them are
quite as ergonomic as I would ideally like.  For now, I've settled on
requiring callers to explicitly down-cast as needed, although I have
provided some utility functions to make this slightly easier:

- `RefCounted<>`, `InternallyRefCounted<>`, and `DualRefCounted<>` all
  provide a templated `RefAsSubclass<>()` method that will return a new
  ref as a subclass.  The type used with `RefAsSubclass()` must be a
  subclass of the type passed to `RefCounted<>`, `InternallyRefCounted<>`,
  or `DualRefCounted<>`.
- In addition, `DualRefCounted<>` provides a templated `WeakRefAsSubclass<T>()`
  method.  This is the same as `RefAsSubclass()`, except that it returns
  a weak ref instead of a strong ref.
- In `RefCountedPtr<>`, I have added a new `Ref()` method that takes
  debug tracing parameters.  This can be used instead of calling `Ref()`
  on the underlying object in cases where the caller already has a
  `RefCountedPtr<>` and is calling `Ref()` only to specify the debug
  tracing parameters.  Using this method on `RefCountedPtr<>` is more
  ergonomic, because the smart pointer is already using the right
  subclass, so no down-casting is needed.
- In `WeakRefCountedPtr<>`, I have added a new `WeakRef()` method that
  takes debug tracing parameters.  This is the same as the new `Ref()`
  method on `RefCountedPtr<>`.
- In both `RefCountedPtr<>` and `WeakRefCountedPtr<>`, I have added a
  templated `TakeAsSubclass<>()` method that takes the ref out of the
  smart pointer and returns a new smart pointer of the down-casted type.
  Just as with the `RefAsSubclass()` method above, the type used with
  `TakeAsSubclass()` must be a subclass of the type passed to
  `RefCountedPtr<>` or `WeakRefCountedPtr<>`.

Note that I have *not* provided an `AsSubclass<>()` variant of the
`RefIfNonZero()` methods.  Those methods are used relatively rarely, so
it's not as important for them to be quite so ergonomic.  Callers of
these methods that need to down-cast can use
`RefIfNonZero().TakeAsSubclass<>()`.

PiperOrigin-RevId: 592327447
2023-12-19 13:27:56 -08:00
Craig Tiller 4e666c740b Internal change
PiperOrigin-RevId: 590780731
2023-12-13 20:06:30 -08:00
Tanvi Jagtap 8019903a44 Internal configuration cleanup
PiperOrigin-RevId: 590528536
2023-12-14 03:42:30 +00:00
Gregory Cooke a5c71e132e [Security - TLS] DirectoryReloaderCrlProvider construction time bug fix (#35247)
The `DirectoryReloaderProvider` currently segfaults on construction if grpc_init() is not called before construction. This is because when creating the `DirectoryReloaderCrlProvider` we [call GetDefaultEventEngine](a58f3f2df5/src/core/lib/security/credentials/tls/grpc_tls_crl_provider.cc (L152)), and getting the default event engine requires that `grpc_init` is called.

This PR adds a test that catches the segfault and adds `grpc_init` and `grpc_shutdown` to the ctor and dtor of `DirectoryReloaderCrlProvider` so that the test passes.

Closes #35247

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35247 from gtcooke94:crl_provider_init_fix 25f3dc7f27
PiperOrigin-RevId: 589885254
2023-12-11 11:25:49 -08:00
Luwei Ge dd12460018 [tls] Add set min/max TLS version APIs to TLS credentials APIs. (#34861)
Address #28382. This is a recreation of #31368 except e2e tests are not handled here (yet).

Closes #34861

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/34861 from rockspore:tls_version f9a1215ac1
PiperOrigin-RevId: 589847110
2023-12-11 09:27:44 -08:00
gRPC Team Bot 667def7505 Internal change
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35043 from gtcooke94:deprecate_old_crl_apis 003057a93c
PiperOrigin-RevId: 585744149
2023-11-27 21:41:47 +00:00
Mark D. Roth 5f41fde4f2 [xDS] add test for mTLS for aggregate clusters (#34927)
Closes #34927

COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/34927 from markdroth:xds_mtls_aggregate_cluster_fix 6aa956997f
PiperOrigin-RevId: 582838553
2023-11-15 16:14:04 -08:00
Yijie Ma 720d7a0653
[EventEngine] WindowsDNSResolver Implementation (#34400)
Design is documented at
[go/windows-dns-resolver-issue](http://go/windows-dns-resolver-issue)
(note that the design doc is slightly outdated regarding the shared
ownership model of the virtual socket that was implemented in
13bd2b404e).

Passed `//test/cpp/naming:resolver_component_tests_runner_invoker` and
`//test/cpp/naming:cancel_ares_query_test`:
```
C:\Users\yijiem\projects\grpc>bazel --output_base=C:\bazel6 test --dynamic_mode=off --verbose_failures --test_env=GRPC_EXPERIMENTS=event_engine_dns --test_env=GRPC_VERBOSITY=debug --test_env=GRPC_TRACE=cares_resolver --enable_runfiles=yes --nocache_test_results //test/cpp/naming:resolver_component_tests_runner_invoker
INFO: Analyzed target //test/cpp/naming:resolver_component_tests_runner_invoker (1 packages loaded, 8 targets configured).
INFO: Found 1 test target...
INFO: From Compiling src/core/lib/event_engine/windows/windows_engine.cc:
C:\bazel6\execroot\com_github_grpc_grpc\src/core/lib/channel/channel_args.h(287): warning C4312: 'reinterpret_cast': conversion from 'int' to 'void *' of greater size
Target //test/cpp/naming:resolver_component_tests_runner_invoker up-to-date:
  bazel-bin/test/cpp/naming/resolver_component_tests_runner_invoker.exe
INFO: Elapsed time: 230.374s, Critical Path: 228.54s
INFO: 9 processes: 2 internal, 7 local.
INFO: Build completed successfully, 9 total actions
//test/cpp/naming:resolver_component_tests_runner_invoker                PASSED in 221.2s

Executed 1 out of 1 test: 1 test passes.
```

```
C:\Users\yijiem\projects\grpc>bazel --output_base=C:\bazel6 test --dynamic_mode=off --verbose_failures --test_env=GRPC_EXPERIMENTS=event_engine_dns --test_env=GRPC_VERBOSITY=debug --test_env=GRPC_TRACE=cares_resolver --enable_runfiles=yes --nocache_test_results //test/cpp/naming:cancel_ares_query_test
INFO: Analyzed target //test/cpp/naming:cancel_ares_query_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
Target //test/cpp/naming:cancel_ares_query_test up-to-date:
  bazel-bin/test/cpp/naming/cancel_ares_query_test.exe
INFO: Elapsed time: 49.656s, Critical Path: 48.00s
INFO: 6 processes: 2 internal, 4 local.
INFO: Build completed successfully, 6 total actions
//test/cpp/naming:cancel_ares_query_test                                 PASSED in 43.0s

Executed 1 out of 1 test: 1 test passes.
```

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-11-09 14:22:31 -08:00
Yijie Ma d3828ebfbd
[Fork] Add ObjectGroupForkHandler fork-handling system (#33733)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->

---------

Co-authored-by: Bradley Hess <bdhess@google.com>
Co-authored-by: AJ Heller <hork@google.com>
2023-11-07 17:48:13 -08:00
Mark D. Roth 1e111df8be
[grpclb_e2e_test] more test improvements and flake fixes (#34896)
- Fix deadlock in load reporting tests.
- Add timeout to `WaitForLoadReport()`. (Note: this required changing
from `grpc::internal::Mutex` and friends to `grpc_core::Mutex` and
friends.)
- Fix balancer stream shutdown machinery.
- Change `ServerThread` to be a class instead of a struct.
2023-11-07 14:47:15 -08:00
Mark D. Roth 8a000f45f8
[grpclb and fake resolver] clean up e2e tests and simplify fake resolver (#34887)
Changes to fake resolver:
- Add `WaitForReresolutionRequest()` method to fake resolver response
generator to allow tests to tell when re-resolution has been requested.
- Change fake resolver response generator API to have only one mechanism
for injecting results, regardless of whether the result is an error or
whether it's triggered by a re-resolution.

Changes to grpclb_end2end_test:
- Change balancer interface such that instead of setting a list of
responses with fixed delays, the test can control exactly when each
response is set.
- Change balancer impl to always send the initial LB response, as
expected by the grpclb protocol.
- Change balancer impl to always read load reports, even if load
reporting is not expected to be enabled. (The latter case will still
cause the test to fail.) Reads are done in a different thread than
writes.
- Allow each test to directly control how many backends and balancers
are started and the client load reporting interval, so that (a) we don't
waste resources starting servers we don't need and (b) there is no need
to arbitrarily split tests across different test classes.
- Add timeouts to `WaitForAllBackends()` functionality, so that tests
will fail with a useful error rather than timing out.
- Improved ergonomics of various helper functions in the test framework.

In the process of making these changes, I found a couple of bugs:
- A bug in pick_first, which I fixed in #34885.
- A bug in grpclb, in which we were using the wrong condition to decide
whether to propagate a re-resolution request from the child policy,
which I've fixed in this PR. (This bug probably originated way back in
#18344.)

This should address a lot of the flakes seen in grpclb_e2e_test
recently.
2023-11-06 14:42:43 -08:00
Craig Tiller 78825f79ca
[event-engine] Fix channelz tests to hermetically clean up callbacks (#34883)
EventEngine experiments, especially with `work_serializer_dispatch` tend
to cause callbacks to occur later than we've previously seen, so tests
that verify global data structures tend to become flakier when these are
introduced.

Here, the fix is waiting for EventEngine to be closed before starting
the new test.

Whilst here, make some adjustments to the test for better readability on
what's going on:
- if we fail a request to an echo service, we do not actually expect the
messages to match, so don't report that
- if we expect a value of 1 or 2, AnyOf is a better tool: it will report
the actual value too

---------

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-11-06 07:00:04 -08:00
Matthew Stevenson 52f9e011f3
[tls] Allow skipping server cert verification when no default roots are present. (#34859)
This PR fixes a bug identified in #29667, where the TLS channel
credentials still require a trust bundle even if the user has explicitly
opted to not verify the server certificate. This PR is based on #29810.
2023-11-03 10:23:10 -07:00
Stanley Cheung ff2c0313e3
[PSM Interop] Add flag to enable CSM Observability in c++ image (#34866)
Roll forward of #34832, 3rd attempt. Will run a grpc import cherry-pick
to make sure the next import is going to be clean.
2023-11-02 14:59:16 -07:00
Stanley Cheung 9303b86010
Revert "[PSM Interop] Add flag to enable CSM Observability in c++ image" (#34858)
Reverts grpc/grpc#34840
2023-11-02 10:42:07 -07:00
Stanley Cheung a62c2607a6
[PSM Interop] Add flag to enable CSM Observability in c++ image (#34840)
Roll forward of #34832 (reverted in #34837).

Need #34836 to be  merged first.
2023-11-01 18:52:57 -07:00
Vignesh Babu c4b9a9a8fc
Revert "[PSM Interop] Add flag to enable CSM Observability in c++ image" (#34837)
Reverts grpc/grpc#34832. Broke import
2023-10-31 18:27:29 -07:00
Stanley Cheung 433cfa99a5
[PSM Interop] Add flag to enable CSM Observability in c++ image (#34832)
Add the flag `enable_csm_observability` to the c++ PSM interop testing
image, such that when enabled from the PSM interop testing framework,
the C++ client/server app will enable the CSM Observability plugin.
2023-10-31 14:42:51 -07:00
Mark D. Roth e8f2e27b2c
[memory test] reland: add benchmarks with xDS enabled (#34820)
Relands #34785, which was reverted in #34818.

The first commit is the revert. The second commit removes the gtest
dependency from the xds_server library, which should address the
testonly problem internally.
2023-10-31 12:56:40 -07:00
Mark D. Roth b1901eb732
Revert "[memory test] add benchmarks with xDS enabled" (#34818)
Reverts grpc/grpc#34785
2023-10-27 15:25:43 -07:00
Mark D. Roth e0a5190a5e
[memory test] add benchmarks with xDS enabled (#34785)
Initial results:

```
default: call/client: 15038.38208
default: call/server: 12873.64608
default: call/xds_client: 15850.04544
default: call/xds_server: 13177.15968
default: channel/client: 27747.9424
default: channel/server: 19939.328
default: channel/xds_client: 45272.2688
default: channel/xds_server: 20805.632
```

b/297028827
2023-10-27 13:52:05 -07:00
Matthew Stevenson 07985907f2
[tls] Fix ownership bugs in TlsCredentialsOptions and grpc_tls_credentials_options. (#34758)
Currently it is very easy to use the `TlsCredentialsOptions` in such a
way that it produces a memory leak. For example, the code block
```
{
  TlsCredentialsOptions options;
}
```
produces a memory leak. This PR fixes up the ownership bugs in this
class and its `grpc_tls_credentials_options`, the C-core analogue.
2023-10-27 08:09:04 -07:00
Mark D. Roth 066d5dd2bb
[xDS e2e tests] move xDS bootstrap and resource helpers into their own library (#34773)
This paves the way for reusing these helpers in a standalone xDS server
for memory usage tests.
2023-10-24 12:33:00 -07:00
Mark D. Roth 30a24a37d1
[e2e tests] refactor code for determining local IP address (#34769) 2023-10-23 17:27:21 -07:00
Yash Tibrewal 77824d8093
[chttp2] Fix outgoing data stats (#34693)
Earlier, the grpc message-length prefix for outgoing data messages was
incorrectly being counted towards `data_bytes` instead of
`framing_bytes`. This PR fixes it.

Note that the incoming stats collection properly attributes the grpc
message-length prefix to `framing_bytes`.

This change will affect all stats plugins (OpenCensus and OpenTelemetry)
that make use of this information for metrics.
2023-10-23 17:04:48 -07:00
Vignesh Babu 299b4fe3fd
[flakiness] Fix regex comparison which causes client_lb test to flake at a high rate internally when event engine is enabled. (#34607) 2023-10-23 09:05:01 -07:00
Yousuk Seung e3ff5f4166
[tracing] Add TcpTracer and HttpAnnotation (#34592)
Add TcpTracer interface for TCP instrumentation. It takes no gRPC
dependencies for use in external TCP implementations. Also add
HttpAnnotation for HTTP transport instrumentation using CallTracer.




<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-10-20 16:04:16 -07:00
Gregory Cooke 9969d820b5
[TLS - Revocation] Crl Provider (#34715)
This reverts commit 7af5efcfd3.
2023-10-19 11:33:41 -07:00
Gregory Cooke 74c1de6512
[Security] Reapply Move ownership of tsi_ssl_client_handshaker_factory (#34726)
This reverts commit 601aaf80b7, which
results in rolling forward #34408.

The 601aaf reversion happened because of a deadlock found in Python. The
root cause ended up being an issue with the Python wrapper and was fixed
in #34712 , so this can be rolled forward again
2023-10-18 17:59:44 -04:00
Craig Tiller a0c1027bb3
[transport] Move transport interface to C++ (#34618)
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-10-18 13:13:23 -07:00
apolcyn 7af5efcfd3
Revert "[TLS - Revocation] Crl Provider (#33786)" (#34713)
This reverts commit 0f0396ae92.
2023-10-17 13:53:36 -07:00
Craig Tiller 975184f04b
[channel-stack] Make ordering explicit (#32852)
Ditch the old priority scheme for ordering filters, instead explicitly
mark up before/after constraints.

---------

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-10-17 13:52:58 -07:00
Gregory Cooke 0f0396ae92
[TLS - Revocation] Crl Provider (#33786)
The basic APIs for the CRL Reloading features.
This adds external types to represent CRL Providers, CRLs, and
CertificateInfo.
Internally we will use `CrlImpl` - this layer is needed to hide OpenSSL
details from the user.

GRFC - https://github.com/grpc/proposal/pull/382

Things Done
* Add external API for `CrlProvider`, `Crl`, `CertInfo` (`CertInfo` is
used during CRL lookup rather than passing the entire certificate).
* Add code paths in `ssl_transport_security` to utilize CRL providers
* Add `StaticCrlProvider`
* Refactor `crl_ssl_transport_security_test.cc` so it is more extensible
and can be used with providers
2023-10-17 13:03:52 -04:00
Yash Tibrewal 5fd09c1aff
[CSM C++] Fix behavior when peer does not send metadata (#34692)
@stanley-cheung noticed a bug where CSM labels were not being added on
metrics if the peer was not also CSM Observability enabled.

This PR fixes the behavior to add in the local labels in this case, as
well as add the remote workload type label with the value of unknown.
2023-10-16 18:13:36 -07:00
nanahpang 601aaf80b7
Revert "[Security] Move ownership of tsi_ssl_client_handshaker_factory to grpc_ssl_credentials, version 2." (#34701)
Reverts grpc/grpc#34408
2023-10-16 18:18:44 +00:00
Craig Tiller c9df0ca470
[cpp] Reland removal of channel_filter code (#34598)
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-10-12 09:44:11 -07:00
Yash Tibrewal 7a97b35d9b
[CSM] Google Cloud C++ libraries now allow bazel layering checks (#34664)
With the latest cloud C++ release - 
1) The opentelemetry bazel target is no longer experimental.
2) layering checks now work with bazel
2023-10-11 17:14:59 -07:00
Yijie Ma bae0c705aa
[Deps] Update to Clang-16 (#34492)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-10-11 16:26:32 -07:00
Mark D. Roth bb6a6faa69
[SSA] support multiple addresses per endpoint (#34472) 2023-10-06 15:23:33 -07:00
Eugene Ostroukhov 3ed72c0c60
[reflection] Add v1 reflection (#34599) 2023-10-06 11:40:31 -07:00
Craig Tiller 3b0916fc0a
Revert "[reflection] Add v1 reflection" (#34594)
Reverts grpc/grpc#34535
2023-10-04 19:06:58 -07:00
Matthew Stevenson 539e912ef8
[ssl] Do not crash if ssl session cache capacity is zero. (#34539)
This behavior is dangerous because we will crash when the cache is
created, which is not necessarily on application startup and is likely
when you first try to establish an SSL connection. Instead, we log an
error. If the SSL library attempts to put a session ticket in the cache
it will fail to do so, but everything else will continue as normal. In
particular, we will always seamlessly fall back to a full SSL handshake.

Along the way, we also ensure that you cannot put a null `SSL_SESSION`
into the cache, which would lead to a segfault when it is fetched from
the cache.
2023-10-04 12:45:09 -07:00
Eugene Ostroukhov 5817f0f6c1
[reflection] Add v1 reflection (#34535) 2023-10-04 11:09:59 -07:00
Mark D. Roth 7044af2691
[tests] fix flakes caused by dualstack changes (#34575)
Fix chttp2 too_many_pings test to use only one of IPv4 or IPv6,
depending on test environment.

Also fix dumb reversed conditional bug in some other tests that was
accidentally introduced in #34426.
2023-10-02 13:48:43 -07:00
Mark D. Roth 883ec58237
[client_lb_e2e_test] fix flake in RR HealthChecking test (#34572) 2023-10-02 13:08:32 -07:00
Mark D. Roth 24f995538c
[xDS] add support for multiple addresses per endpoint (#34506)
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
2023-09-29 09:46:45 -07:00
Craig Tiller 7b17e4aa92
Revert "[cpp] Remove legacy channel filter C++ wrapper" (#34541)
Reverts grpc/grpc#34522
2023-09-28 22:29:53 -07:00
Craig Tiller 727c92a15c
[cpp] Remove legacy channel filter C++ wrapper (#34522)
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-09-28 20:59:56 -07:00
Mark D. Roth 7a06614f95
[resolver and LB policy APIs] reland: change address list to support multiple addresses per endpoint (#34531)
Re-land #33567, which was reverted in #34527.

First commit is a pure revert, second commit is a small fix needed to
avoid breaking internal callers.
2023-09-28 15:40:05 -07:00
Mark D. Roth 41f26de3b6
Revert "[resolver and LB policy APIs] change address list to support multiple addresses per endpoint" (#34527)
Reverts grpc/grpc#33567 due to import problems.
2023-09-28 12:15:35 -07:00
Mark D. Roth fd2e8c9462
[resolver and LB policy APIs] change address list to support multiple addresses per endpoint (#33567)
More changes as part of the dualstack design:
- Change resolver and LB policy APIs to support multiple addresses per
endpoint. Specifically, replace `ServerAddress` with
`EndpointAddresses`, which encodes more than one address. Per-address
channel args are retained at the same level, so they are now
per-endpoint. For now, `EndpointAddress` provides a single-address ctor
and a single-address accessor for backward compatibility, so
`ServerAdress` is an alias for `EndpointAddresses`; eventually, this
alias and the single-address methods will be removed.
- Add an `EndpointAddressSet` class, which represents an unordered set
of addresses to be used as a map key. This will be used in a number of
LB policies that need to store per-endpoint state.
- Change the LB policy API's `ChannelControlHelper::CreateSubchannel()`
method to take the address and per-endpoint channel args as separate
parameters, so that we don't need to construct a legacy `ServerAddress`
object as we create a new subchannel for each address in the endpoint.
- Change pick_first to flatten the address list.
- Change ring_hash to use `EndpointAddressSet` as the key for its
endpoint map, and to use the first address of the endpoint as the hash
key.
- Change WRR to use `EndpointAddressSet` as the key for its endpoint
weight map.

Note that support for multiple addresses per endpoint is guarded in RR
by the existing `round_robin_delegate_to_pick_fist` experiment and in
WRR by the existing `wrr_delegate_to_pick_first` experiment.

This PR does *not* include support for multiple addresses per endpoint
for the outlier_detection or xds_override_host LB policies; those will
come in subsequent PRs.
2023-09-28 09:32:36 -07:00
Mark D. Roth 835775e347
[pick_first] implement Happy Eyeballs (#34426) 2023-09-27 11:10:53 -07:00
Craig Tiller 59d886cb5c
[fuzzing] Expose random number generator to some fuzzers (#34415)
Expand our fuzzing capabilities by allowing fuzzers to choose the bits
that go into random number distribution generators.

---------

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-09-25 10:17:58 -07:00
Yash Tibrewal a0e3794f4b
[OTel C++] Method attribute filtering (#34350)
Implements method attribute filtering as defined by the gRFC
https://github.com/grpc/proposal/pull/380
2023-09-22 23:48:01 +00:00
Yash Tibrewal 06b55bdaa4
[RegisteredMethod] Set information on initial metadata about whether method is registered or not (#34432)
Summary - 
On the server-side, we are changing the point at which we decide whether
a method is registered or not from the surface to the transport at the
point where we are done receiving initial metadata and before we invoke
the recv_initial_metadata_ready closures from the filters. The main
motivation for this is to allow filters to check whether the incoming
method is a registered or not. The exact use-case is for observability
where we only want to record the method if it is registered. We store
the information about the registered method in the initial metadata.

On the client-side, we also set information about whether the method is
registered or not in the outgoing initial metadata.

Since we are effectively changing the lookup point of the registered
method, there are slight concerns of this being a potentially breaking
change, so we are guarding this with an experiment to be safe.

Changes - 
* Transport API changes -
* Along with `accept_stream_fn`, a new callback
`registered_method_matcher_cb` will be sent down as a transport op on
the server side. When initial metadata is received on the server side,
this callback is invoked. This happens before invoking the
`recv_initial_metadata_ready` closure.
* Metadata changes -
* We add a new non-serializable metadata trait `GrpcRegisteredMethod()`.
On the client-side, the value is a uintptr_t with a value of 1 if the
call has a registered/known method, or 0, if it's not known. On the
server side, the value is a (ChannelRegisteredMethod*). This metadata
information can be used throughout the stack to check whether a call is
registered or not.
* Server Changes -
* When a new transport connection is accepted, the server sets
`registered_method_matcher_cb` along with `accept_stream_fn`. This
function checks whether the method is registered or not and sets the
RegisteredMethod matcher in the metadata for use later.
* Client Changes -
* Set the metadata on call creation on whether the method is registered
or not.
2023-09-22 13:13:16 -07:00
Eugene Ostroukhov 2f78fffa37
[xds ssa] Remove environment variable protection for stateful affinity (#34435) 2023-09-22 13:06:51 -07:00
Craig Tiller 1dabdfbe6f
[per-cpu] Change up the cpu caching mechanism (#34421)
Lets us sever the dependency between stats & exec ctx (finally).

More work likely needs to go into the *mechanism* used here (I'm not a
fan of the per thread index), but that's also something we can address
later.
2023-09-22 11:23:11 -07:00
Yash Tibrewal 1091cc3211
[CSM] Update labels (#34412)
Changes - 
* Remove `csm.remote_workload_pod_name` and
`csm.remote_workload_container_name`.
* Add `csm.remote_workload_name`, the value for which is sent through
MetadataExchange, from the `CSM_WORKLOAD_NAME` env var. (Note that this
is not added in local labels.)
* Add a local `csm.canonical_service` (@markdroth, please verify the key
that we want here) that is read from `CSM_CANONICAL_SERVICE_NAME` env
var, and we continue to send it over via MetadataExchange
2023-09-21 16:02:03 -07:00
Gregory Cooke aa17285f8e
[Security] Move ownership of tsi_ssl_client_handshaker_factory to grpc_ssl_credentials, version 2. (#34408)
Revert the reversion of the SSL_CTX_new change (#34355 reverted #34180 )
with a fix.

There was an issue with using `strcpy` on a `new[] string` in the
constructor of `ssl_credentials`. An ASAN test caught this in some CI
down the line - `ERROR: AddressSanitizer: alloc-dealloc-mismatch
(operator new [] vs free)`

That `strcpy` call was changed to `grp_strdup` which duplicates a string
in a way that can be freed by `gpr_free` and should resolve the ASAN
failure.
2023-09-21 10:07:18 -07:00
Yash Tibrewal 112fffcdb4
[OTel] Minor test cleanup (#34353)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-09-21 09:23:47 -07:00
Eugene Ostroukhov 98ac00d7d2
[interop test] Fix the test (#34424) 2023-09-20 11:53:08 -07:00
Eugene Ostroukhov 490f6a3ee9
[test interop] Add HookService to the maintenence server (#34413)
This pull request adds another hook service on the maintenance server.
This will enable clients to gradually migrate from the standalone hook
server.

Changes:
1. Hook service can now be used separately.
2. Copied latest protos and updated the hook service to new API.
3. Added the hook service to the maintenance server.
2023-09-19 18:06:31 -07:00
Stanley Cheung fc159a6901
[Observability Testing] register prometheus exporter (#34380)
Working towards testing against CSM Observability. Added ability to
register a prometheus exporter with our Opentelemetry plugin. This will
allow our metrics to be available at the standard prometheus port
`:9464`.
2023-09-19 16:54:58 -07:00
AJ Heller 3707b42bec
[reland][EventEngine] Move combiner executor usage to EventEngine (#34396)
Relands #31713
2023-09-19 14:41:17 -07:00
Mark D. Roth 0bffb766ff
[grpclb e2e test] increase RPC deadlines to fix flakes (#34403) 2023-09-19 11:29:34 -07:00
apolcyn 87eed73a47
[dns] unskip c-ares tests on arm (#34232)
Now that we have https://github.com/grpc/grpc/pull/33942 (and another
follow-up [fix](https://github.com/grpc/grpc/pull/34185)), I think the
issue from https://github.com/grpc/grpc/issues/25289 is likely fixed
2023-09-18 12:18:29 -07:00
Craig Tiller 0375a585e2
[work-serializer] Fix synchronous test assumption (#34392)
This test assumed synchronous work serializer execution (or at least
faster async than we always get)... make a trivial change to keep the
test semantics but allow for the implementation to be more async.
2023-09-18 09:27:49 -07:00
Yash Tibrewal d670ffa92c
[CSM] Create an experimental target (#34381) 2023-09-15 17:51:33 -07:00
Craig Tiller 86b931c354
[work-serializer] Dispatch on run experiment (relanding) (#34372)
Reverts grpc/grpc#34371
2023-09-15 12:46:04 -07:00