Commit Graph

4441 Commits

Author SHA1 Message Date
Yijie Ma bae0c705aa
[Deps] Update to Clang-16 (#34492)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-10-11 16:26:32 -07:00
Romain Geissler @ Amadeus c38837a741
[C++] Use GOOGLE_PROTOBUF_VERSION instead of PROTOBUF_VERSION. (#34635)
Indeed it seems that on Protobuf side, PROTOBUF_VERSION is defined in
port_def.inc which always comes with a matching include of
port_undef.inc which explicitly undef all macros, among them
PROTOBUF_VERSION. GOOGLE_PROTOBUF_VERSION doesn't suffer from this
issue.

This is a followup of #33646.
2023-10-10 15:06:34 -07:00
Eugene Ostroukhov 3ed72c0c60
[reflection] Add v1 reflection (#34599) 2023-10-06 11:40:31 -07:00
Craig Tiller 3b0916fc0a
Revert "[reflection] Add v1 reflection" (#34594)
Reverts grpc/grpc#34535
2023-10-04 19:06:58 -07:00
Eugene Ostroukhov 5817f0f6c1
[reflection] Add v1 reflection (#34535) 2023-10-04 11:09:59 -07:00
Esun Kim 4fa02accc1
[Cord] Fix MakeCordFromSlice memory bug (#34549)
Fix `MakeCordFromSlice` function to fail to carry an inlined slice which
can give a corrupted result when deserializing a Cord.
2023-09-29 14:23:16 -07:00
Mark D. Roth 835775e347
[pick_first] implement Happy Eyeballs (#34426) 2023-09-27 11:10:53 -07:00
Richard Belleville 3bdd972c4a
[Release] Bump master version to 1.60 (gjallarhorn) (#34488)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-09-26 13:10:20 -07:00
Yash Tibrewal a0e3794f4b
[OTel C++] Method attribute filtering (#34350)
Implements method attribute filtering as defined by the gRFC
https://github.com/grpc/proposal/pull/380
2023-09-22 23:48:01 +00:00
Yash Tibrewal b9b208d6da
[CSM] Remove enable/disable metrics API (#34453)
There's an ongoing discussion on whether we should have API to disable
default metrics. Removing this API till we have a decision on that.

I'm keeping the internal API for enabling/disabling metrics on the OTel
plugin for now, just not exposing it publicly.
2023-09-22 16:45:53 -07:00
Yash Tibrewal ddc682c6c1
[CSM] Plumbing fixes (#34440)
Changes -
* CsmObservability doesn't need `SetTargetSelector`. Removed it.
* Added missing plumbing of `ServiceMeshLabelsInjector` in
`CsmObservability` to actually do the metadata exchange.

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-09-22 12:03:30 -07:00
Roman-Byshliaha-Bose 49bb52d7f4
[build] Add detection of QNX platform (#34418) 2023-09-20 11:24:24 -07:00
Yash Tibrewal d670ffa92c
[CSM] Create an experimental target (#34381) 2023-09-15 17:51:33 -07:00
Yash Tibrewal b038da5072
[CSM] Second attempt: Add a server selector based on channel args (#34376)
This reverts commit 2db446aa9a.




<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-09-15 15:28:52 -07:00
nanahpang 2db446aa9a
Revert "[CSM] Add a server selector based on channel args" (#34375)
Reverts grpc/grpc#34312
2023-09-15 11:28:20 -07:00
Yash Tibrewal c145b7910e
[CSM] Add a server selector based on channel args (#34312)
I've added channel args to `CreateNewServerCallTracer` on the
`ServerCallTracerFactory`.
The motivation is for CSM Observability where the OTel plugin will be
configured to only do stats on servers which are xDS enabled, so I plan
to check this via channel args.
In the future, with the new scopes for metrics, I think I'll be able to
change this to only check once per server or server connection instead
of per call.

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-09-15 10:19:46 -07:00
Romain Geissler @ Amadeus 0bc07c957e
[C++] Fix clang's -Winconsistent-missing-override in proto reader/writer. (#33646)
The "override" is not added on purpose to remain compatible with
Protobuf < 22.x, as already written in the comment on top of these two
functions.

CC @veblush as the author of this code.

Note: I am personally not super enthousiastic about this change. As an
alternative, I can propose to selectively add the `override` keyword,
based on the value of the `PROTOBUF_VERSION` macro (comparing it to
`4022000`). Tell me if you prefer this version instead.
2023-09-12 12:47:55 -07:00
Romain Geissler @ Amadeus d0d826750f
[C++] Stop using std::aligned_storage. (#34110)
Indeed this is now deprecated since C++23.

Fix #32848.
2023-09-12 12:46:41 -07:00
h-vetinari 2d1595c9af
[Build] Shared builds on windows (#34103)
Towards https://github.com/grpc/grpc/issues/33032,

Reopen after botched force-push in #33175 that then got "merged" and
cannot be reopened anymore.

More context in that PR.

---------

Co-authored-by: Isuru Fernando <isuruf@gmail.com>
Co-authored-by: David Chamberlin <david.chamberlin@ln.email.gs.com>
2023-09-12 09:12:36 -07:00
Craig Tiller 81db191c2b
[public-iwyu] Get channel_arg_names.h redirected to a public header (#34213) 2023-08-31 13:02:10 -07:00
AJ Heller aaf6c3475c
[EventEngine] Update Cancel contract on closure deletion timeline (#34167)
CC @ctiller @Vignesh2208
2023-08-25 11:12:03 -07:00
Esun Kim a90f30008d
[Release] Bump version to 1.59.0-dev (on master branch) (#34144)
Change was created by the release automation script. See
go/grpc-release.
2023-08-23 15:53:32 -07:00
AJ Heller 724ef464b2
[docs] Document Alarm thread-safety guarantees (#34092) 2023-08-17 10:52:36 -07:00
AJ Heller 0d5dc5c45b
[EventEngine] C++ Alarm migration and PosixEventEngine performance enhancements (#34056)
This PR is mainly a set of improvements that allow the C++ Alarm to be
migrated away from legacy iomgr. It cannot be landed without significant
speedup, due to third-parties relying on a fast path for immediate timer
execution with deadlines <= now.

Previous EventEngine performance of bm_alarm, compared to baseline iomgr
timers: *0.014%*
This PR: *2.5%*

Regarding previous failures to land this change: The cloud libraries
team agreed to reduce the amount of stress in their alarm stress test
https://github.com/googleapis/google-cloud-cpp/pull/12378
2023-08-16 11:54:04 -07:00
Craig Tiller 6b2de0fa4b
[chttp2] Use RefCountedPtr for grpc_chttp2_transport (#33746)
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-07-28 11:24:22 -07:00
Craig Tiller 3717ff04ba
[chttp2] Split ping policy from transport (#33703)
Why: Cleanup for chttp2_transport ahead of promise conversion - lots of
logic has become interleaved throughout chttp2, so some effort to
isolate logic out is warranted ahead of that conversion.

What: Split configuration and policy tracking for each of ping rate
throttling and abuse detection into their own modules. Add tests for
them.

Incidentally: Split channel args into their own header so that we can
split the policy stuff into separate build targets.

---------

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-07-26 09:19:35 -07:00
AJ Heller 42b0d01e68
[Release] Bump version to 1.58.0-dev (on master branch) (#33825)
Change was created by the release automation script. See go/grpc-release
2023-07-24 10:09:08 -07:00
Yijie Ma a7bf07e86a
[EventEngine] PosixEventEngine DNS Resolver (#32701)
This PR implements a c-ares based DNS resolver for EventEngine with the
reference from the original
[grpc_ares_wrapper.h](../blob/master/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.h).
The PosixEventEngine DNSResolver is implemented on top of that. Tests
which use the client channel resolver API
([resolver.h](../blob/master/src/core/lib/resolver/resolver.h#L54)) are
ported, namely the
[resolver_component_test.cc](../blob/master/test/cpp/naming/resolver_component_test.cc)
and the
[cancel_ares_query_test.cc](../blob/master/test/cpp/naming/cancel_ares_query_test.cc).
The WindowsEventEngine DNSResolver will use the same EventEngine's
grpc_ares_wrapper and will be worked on next.

The
[resolve_address_test.cc](https://github.com/grpc/grpc/blob/master/test/core/iomgr/resolve_address_test.cc)
which uses the iomgr
[DNSResolver](../blob/master/src/core/lib/iomgr/resolve_address.h#L44)
API has been ported to EventEngine's dns_test.cc. That leaves only 2
tests which use iomgr's API, notably the
[dns_resolver_cooldown_test.cc](../blob/master/test/core/client_channel/resolvers/dns_resolver_cooldown_test.cc)
and the
[goaway_server_test.cc](../blob/master/test/core/end2end/goaway_server_test.cc)
which probably need to be restructured to use EventEngine DNSResolver
(for one thing they override the original grpc_ares_wrapper's free
functions). I will try to tackle these in the next step.
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-07-21 13:24:16 -07:00
Yijie Ma 73605f4eac
[EventEngine] Change `GetDNSResolver` to return `absl::StatusOr<std::unique_ptr<DNSResolver>>` (#33744)
Based on the discussion at:
595a75cc5d..e3b402a8fa (r1244325752)


<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-07-19 14:23:26 -07:00
Craig Tiller 76bd606bae
[build] Make GRPC_MUST_USE_RESULT default backed by nodiscard (#33742)
Going forward `[[nodiscard]]` is the portable way to spell this;
requires yanking a bunch of usage from after the param list to before.

We should further refine the GRPC_MUST_USE_RESULT macro to make it work
uniformly for any compilers that it doesn't today (most likely by making
it expand to nothing).

---------

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-07-18 15:23:46 -07:00
Romain Geissler @ Amadeus 899bf72004
[C++] Fix two compilation warnings in grpcpp/support/proto_buffer_reader.h. (#33635)
Detected with gcc 13:
```
In file included from /data/mwrep/res/osp/Grpc/23-0-0-0/include/grpcpp/impl/proto_utils.h:31,
                 from ./include/generated/gacms.object.grpc.pb.h:18,
                 from ./include/generated/gacms.object.grpc.pb.cc:6:
/data/mwrep/res/osp/Grpc/23-0-0-0/include/grpcpp/support/proto_buffer_reader.h: In member function 'virtual bool grpc::ProtoBufferReader::ReadCord(absl::lts_20230125::Cord*, int)': /data/mwrep/res/osp/Grpc/23-0-0-0/include/grpcpp/support/proto_buffer_reader.h:157:24: error: comparison of integer expressions of different signedness: 'uint64_t' {aka 'long unsigned int'} and 'int' [-Werror=sign-compare]
  157 |       if (slice_length <= count) {
      |           ~~~~~~~~~~~~~^~~~~~~~
/data/mwrep/res/osp/Grpc/23-0-0-0/include/grpcpp/support/proto_buffer_reader.h: In lambda function:
/data/mwrep/res/osp/Grpc/23-0-0-0/include/grpcpp/support/proto_buffer_reader.h:191:35: warning: unused parameter 'view' [-Wunused-parameter]
  191 |         [slice](absl::string_view view) { grpc_slice_unref(slice); });
      |                 ~~~~~~~~~~~~~~~~~~^~~~
cc1plus: all warnings being treated as errors
```
2023-07-10 17:42:23 -07:00
Bjorn Svensson ac874c2c83
[core] Add a channel argument to set DSCP on streams (#28322)
This adds a new channel argument `GRPC_ARG_DSCP` which allows users to
create classified gRPC streams with a
Differentiated Services Code Point (DSCP) marking on the IP frames.

The channel argument is handled on both clients and servers, but
currently only on posix based systems.

Fixes #17225

**Background**:
In addition to what is already described is #17225, when gRPC is used in
telco systems there is often a need to classify streams of importance.
There can be multiple hops between two endpoints (e.g. between 2 telecom
operators) and some streams that are more important than others (e.g.
emergency call related or similar). By marking the IP packets using DSCP
the aware routers can make a sound decision of the prioritization.

This PR propose to use DSCP as the configuration value since its common
for both IPv4/IPv6, an alternative would be to use a config name that
includes TOS and Traffic Class.
There might be more needed regarding documentation and end2end testing,
but there I need some advice.

**References**
https://datatracker.ietf.org/doc/html/rfc2474
https://www.iana.org/assignments/dscp-registry/dscp-registry.xhtml

<!--

Your pull request will be routed to the following person by default for
triaging.
If you know who should review your pull request, please remove the
mentioning below.

-->

@yashykt
2023-06-30 16:10:21 -07:00
Matthew Stevenson 278978d6f0
[tls] Remove use of SSL_CTX_set_client_CA_list for TLS server credentials. (#33558)
This PR does the following: for the TLS server credentials, stops
calling `SSL_CTX_set_client_CA_list` by default in
`ssl_transport_security.cc`, and gives users a knob to re-enable calling
this API.

## What does the `SSL_CTX_set_client_CA_list` API do?

When this API is called, a gRPC TLS server sends the following data in
the ServerHello: for each certificate in the server's trust bundle, the
CA name in the certificate.

This API does not change the set of certificates trusted by the server
in any way. Rather, it is just providing a hint to the client about what
client certificate should be sent to the server.

## Why are we removing the use of `SSL_CTX_set_client_CA_list` by
default for the TLS server credentials?

Removing the use of this API by default has 2 benefits:
1. Calling this API makes gRPC TLS unusable for servers with a
sufficiently large trust bundle. Indeed, if the server trust bundle is
too large, then the server will always fail to build the ServerHello.
2. Calling this API is introducing a huge amount of overhead (1000s of
bytes) to each ServerHello, so removing this feature will improve
connection establishment latency for all users of the TLS server
credentials.
2023-06-29 09:48:15 -07:00
Yijie Ma 6e95cebbd0
[EventEngine] Simplify `EventEngine::DNSResolver` API (#33459)
This change simplifies `EventEngine::DNSResolver`'s API based on the
proposal:
[go/event-engine-dns-resolver-api-changes](http://go/event-engine-dns-resolver-api-changes).
Note that this API change + the implementation described in
[go/event-engine-dns-resolver-implementation](http://go/event-engine-dns-resolver-implementation)
has already been tested against our main test suites and are passing
them.


<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-06-22 12:19:20 -07:00
Yousuk Seung c03cd744b2
[WRR] Prefer application_utilization to cpu_utilization (#33355)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-06-07 17:38:29 -07:00
Yijie Ma 5a0678001b
[Release] Bump version to 1.57.0-dev (on master branch) (#33311)
Change was created by the release automation script. See
go/grpc-release.
2023-06-01 10:55:31 -07:00
Mehrdad Afshari 20023faa2e
[API] Document gotcha in AsyncNotifyWhenDone behavior (#33208)
As the [issue](https://github.com/grpc/grpc/issues/10136) documents, the
behavior of AsyncNotifyWhenDone is documented as:

"The comment on `AsyncNotifyWhenDone` states "Has to be called before
the rpc starts" but it seems that if the request tag is returned with
ok=false (i.e. because the CQ is shutting down) then the async done tag
is never received. Instead, I expect the async done tag to be received
regardless of whether or not an incoming call request was successfully
received."

The TODO item is marked closed as stale, and it seems unlikely this will
be resolved, without breaking
existing users whose code is written under the assumption that the tag
is not seen if the call never starts, so it may be time to documented
the idiosyncratic corner case and make it the expected behavior.
2023-05-23 15:16:58 -07:00
Luwei Ge 0d70c632fb
[Audit Logging] Second attempt: rbac service config parsing with audit logging (#33183)
This is basically the same as #33145 except that the ctor `Rules()`
cannot be default but have to explicitly set a default audit condition.
2023-05-18 14:27:26 -07:00
Eugene Ostroukhov 922b18f71b
Revert "[Audit Logging] Audit logging config translation by rbac service config parser" (#33178)
Reverts grpc/grpc#33145

It causes internal breakages.
2023-05-18 10:58:16 -07:00
Luwei Ge f60d0c7247
[Audit Logging] Audit logging config translation by rbac service config parser (#33145)
This translates the service config from HTTP RBAC filter into the rbac
policy, which is used to construct authorization engines.
2023-05-17 15:15:30 -07:00
Mark D. Roth 1432fe4e4c
[JSON] make API public but experimental (#32987)
This makes the JSON API visible as part of the C-core API, but in the
`experimental` namespace. It will be used as part of various
experimental APIs that we will be introducing in the near future, such
as the audit logging API.
2023-05-09 08:05:24 -07:00
Yijie Ma 7df0e11755
[EventEngine] Change TXT lookup result type to std::vector<std::string> (#33030)
One TXT lookup query can return multiple TXT records (see the following
example). `EventEngine::DNSResolver` should return all of them to let
the caller (e.g. `event_engine_client_channel_resolver`) decide which
one they would use.

```
$ dig TXT wikipedia.org

; <<>> DiG 9.18.12-1+build1-Debian <<>> TXT wikipedia.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49626
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;wikipedia.org.                 IN      TXT

;; ANSWER SECTION:
wikipedia.org.          600     IN      TXT     "google-site-verification=AMHkgs-4ViEvIJf5znZle-BSE2EPNFqM1nDJGRyn2qk"
wikipedia.org.          600     IN      TXT     "yandex-verification: 35c08d23099dc863"
wikipedia.org.          600     IN      TXT     "v=spf1 include:wikimedia.org ~all"
```

Note that this change also deviates us from the iomgr's DNSResolver API
which uses std::string as the result type.


<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-05-08 11:12:29 -07:00
AJ Heller 18c1cc5a51
[EventEngine] Update contract for Listener's on_shutdown execution (#33003)
See `event_engine.h` for the contract change. All other changes are
cleanup.

I confirmed that both the Posix and Windows implementations comply with
this already.

On Windows, the `WindowsEventEngineListener` will only call
`on_shutdown` after all `SinglePortSocketListener`s have been destroyed,
which ensures that no `on_accept` callback will be executed, even if
there is still trailing overlapped activity on the listening socket.

On Posix, the `PosixEngineListenerImpl` will only call `on_shutdown`
after all `AsyncConnectionAcceptor`s have been destroyed, which ensures
`EventHandle::OrphanHandle` has been called. The `OrphanHandle` contract
indicates that all existing notify closures must have already run. The
implementation looks to comply, so if it does not, that's a bug.
3aae08d25e/src/core/lib/event_engine/posix_engine/event_poller.h (L48-L50)
2023-05-03 13:34:01 -07:00
Yousuk Seung 8b02295e58
[xDS] Accept cpu_utilization over 100% (#32954)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-05-02 03:25:17 +00:00
Luwei Ge dcfc5d6904
[Audit Logging] Logger and factory APIs in C-Core and C++. (#32750)
Audit logging APIs for both built-in loggers and third-party logger
implementations.

C++ uses using decls referring to C-Core APIs.

---------

Co-authored-by: rockspore <rockspore@users.noreply.github.com>
2023-04-26 06:44:14 -07:00
Yash Tibrewal dc075539e7
[Release] Bump version to 1.56.0-dev (on master branch) (#32918)
Change was created by the release automation script. See go/grpc-release
2023-04-24 10:47:57 -07:00
Yash Tibrewal fe4147a4a9
[Documentation] Update docs for TryCancel() on ServerContext (#32889)
Fix https://github.com/grpc/grpc/issues/32638

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-04-21 09:24:43 -07:00
Esun Kim c523bdac1e
[C++] Added a cord support to gRPC protobuf serializer (#32617)
As Protobuf is going to support Cord to reduce memory copy when
[de]serializing Cord fields, gRPC is going to leverage it. This
implementation is based on the internal one but it's slightly modified
to use the public APIs of Cord. only
2023-04-19 08:55:25 -07:00
AJ Heller c57c27ff90
[tcp] Add channel argument for SO_RVCBUF size. (#32887)
Example usage:

```
ServerBuilder builder;
builder.AddChannelArgument(GRPC_ARG_TCP_RECEIVE_BUFFER_SIZE, 1024*1024);
```
2023-04-18 20:03:08 -07:00
Jan Tattermusch 0f1afec5a8
[protobuf] Upgrade third_party/protobuf to 22.x (#32606)
The very non-trivial upgrade of third_party/protobuf to 22.x
This PR strives to be as small as possible and many changes that were
compatible with protobuf 21.x and didn't have to be merged atomically
with the upgrade were already merged.
Due to the complexity of the upgrade, this PR wasn't created
automatically by a tool, but manually. Subsequent upgraded of
third_party/protobuf with our OSS release script should work again once
this change is merged.

This is best reviewed commit-by-commit, I tried to group changes in
logical areas.

Notable changes:
- the upgrade of third_party/protobuf submodule, the bazel protobuf
dependency itself
- upgrade of UPB dependency to 22.x (in the past, we used to always
upgrade upb to "main", but upb now has release branch as well). UPB
needs to be upgraded atomically with protobuf since there's a de-facto
circular dependency (new protobuf depends on new upb, which depends on
new protobuf for codegen).
- some protobuf and upb bazel rules are now aliases, so `
extract_metadata_from_bazel_xml.py` and `gen_upb_api_from_bazel_xml.py`
had to be modified to be able to follow aliases and reach the actual
aliased targets.
- some protobuf public headers were renamed, so especially
`src/compiler` needed to be updated to use the new headers.
- protobuf and upb now both depend on utf8_range project, so since we
bundle upb with grpc in some languages, we now have to bundle utf8_range
as well (hence changes in build for python, PHP, objC, cmake etc).
- protoc now depends on absl and utf8_range (previously protobuf had
absl dependency, but not for the codegen part), so python's
make_grpcio_tools.py required partial rewrite to be able to handle those
dependencies in the grpcio_tools build.
- many updates and fixes required for C++ distribtests (currently they
all pass, but we'll probably need to follow up, make protobuf's and
grpc's handling of dependencies more aligned and revisit the
distribtests)
- bunch of other changes mostly due to overhaul of protobuf's and upb's
internal build layout.

TODOs:
- [DONE] make sure IWYU and clang_tidy_code pass
- create a list of followups (e.g. work to reenable the few tests I had
to disable and to remove workaround I had to use)
- [DONE in cl/523706129] figure out problem(s) with internal import

---------

Co-authored-by: Craig Tiller <ctiller@google.com>
2023-04-12 18:43:58 +02:00
Yash Tibrewal bdd1ac4d1d
[GcpObservability C++] De-experimentalize API (#32715)
This PR aims to de-experimentalize the APIs for GCP Observability. 

We would have ideally wanted public feedback before declaring the APIs
stable, but we need stable APIs for GA.

Changes made after API review with @markdroth @veblush, @ctiller and the
entire Core/C++ team -
* The old experimental APIs `grpc::experimental::GcpObservabilityInit`
and `grpc::experimental::GcpObservabilityClose` are now deprecated and
will be deleted after v.1.55 release.
* The new API gets rid of the Close method and follows the RAII idiom
with a single `grpc::GcpObservability::Init()` call that returns an
`GcpObservability` object, the lifetime of which controls when
observability data is flushed.
* The `GcpObservability` class could in the future add more methods. For
example, a debug method that shows the current configuration.
* Document that GcpObservability initialization and flushing (on
`GcpObservability` destruction) are blocking calls.
* Document that gRPC is still usable if GcpObservability initialization
failed. (Added a test to prove the same).
* Since we don't have a good way to flush stats and tracing with
OpenCensus, the examples required users to sleep for 25 seconds. This
sleep is now part of `GcpObservability` destruction.

Additional Implementation details -
* `GcpObservability::Init` is now marked with `GRPC_MUST_USE_RESULT` to
make sure that the results are used. We ideally want users to store it,
but this is better than nothing.
* Added a note on GCP Observability lifetime guarantees.

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-04-05 21:45:21 +00:00
AJ Heller 49b9dfa644
Revert "[EventEngine] Revert "Revert "EventEngine::RunAt: C++ Alarm (#30024)" (#30147)"" (#32783)
Reverts grpc/grpc#32461. One new benchmark showed performance problems.

Co-authored-by: Yijie Ma <yijiem.main@gmail.com>
2023-04-03 13:55:26 -07:00
Stanley Cheung ec1d75bb0a
[Release process] Bump version to 1.55.0-dev (on master branch) (#32743)
Change was created by the release automation script. See
go/grpc-release.
2023-03-29 11:03:07 -07:00
Yousuk Seung c02b3e695c
xDS: Include orca named_metrics in LRS load reports (#32690)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-03-28 12:04:01 -07:00
Yousuk Seung 16c03db9ac
Revert "Revert "WRR: Support EPS" (#32723)" (#32725)
This reverts commit 7bd9267f32.




<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-03-27 16:49:50 -07:00
Alisha Nanda 19d06a78ec
Add random early rejection for metadata (#32600)
(hopefully last try)

Add new channel arg GRPC_ARG_ABSOLUTE_MAX_METADATA_SIZE as hard limit
for metadata. Change GRPC_ARG_MAX_METADATA_SIZE to be a soft limit.
Behavior is as follows:

Hard limit
(1) if hard limit is explicitly set, this will be used.
(2) if hard limit is not explicitly set, maximum of default and soft
limit * 1.25 (if soft limit is set) will be used.

Soft limit
(1) if soft limit is explicitly set, this will be used.
(2) if soft limit is not explicitly set, maximum of default and hard
limit * 0.8 (if hard limit is set) will be used.

Requests between soft and hard limit will be rejected randomly, requests
above hard limit will be rejected.
2023-03-27 14:13:20 -07:00
Esun Kim 7bd9267f32
Revert "WRR: Support EPS" (#32723)
Reverts grpc/grpc#32657
2023-03-27 11:53:17 -07:00
Yousuk Seung 4429066516
WRR: Support EPS (#32657)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-03-24 16:23:51 -07:00
AJ Heller 3fe678e306
[EventEngine] Add EventEngine::*Handle equality operators (#32695)
This allows us to replace `absl::optional<TaskHandle>` with checks
against the invalid handle.

This PR also replaces the differently-named invalid handle instances
with a uniform way of accessing static invalid instances across all
handle types, which aids a bit in testing.
2023-03-23 15:31:24 -07:00
Yash Tibrewal bf1a26cde8
GcpObservability: Add client api latency (#32645)
This PR adds the view `grpc.io/client/api_latency` for GCP Observability
which aims to collect the end-to-end time taken by a call.

Changes made to support this -
1) A global interceptor factory registration is created for stats
plugins.
2) OpenCensus plugin now provides a new interceptor that's responsible
for collecting the new latency.
3) Gcp Observability registers this plugin.
4) A new OpenCensus measurement and view is created for api latency.
Note that this is internal as of now, since it's not clear if it should
be exposed as public experimental API. Leaving that decision for the
future.

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-03-22 15:31:47 -07:00
Yijie Ma ac7faf75ba
Fix a race on vptr for UnimplementedAsyncRequest (#32547)
It is reported in https://github.com/grpc/grpc/issues/32356 that there
is a race on vptr for `UnimplementedAsyncRequest` which would cause
crashes for multi-threaded server if clients send unimplemented RPC
request to the server.

The cause is that the server requests a call for
`UnimplementedAsyncRequest` in its base class `GenericAsyncRequest` when
the `vptr` still points to the base class's `vtable`. If the call went
in and another server thread picks up the tag before the `vptr` points
back to the derived class's `vtable`, it would call the wrong virtual
function and also this is a data race. This fix makes the request of the
call inside the derived class's constructor.

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-03-10 13:00:56 -08:00
Jan Tattermusch a27b86fb95
Switch linux RBE to ubuntu18.04, get rid of rbe_autoconfig (#32559)
First step in the modernization of our RBE stack (see
go/rbe-tech-debt-notes).

- Get rid of the deprecated rbe_autoconfig and start using
[rbe_configs_gen](https://github.com/bazelbuild/bazel-toolchains#rbe_configs_gen---cli-tool-to-generate-configs)
+ check in the generated toolchain configs.
- Switch from marketplace.gcr.io/google/rbe-ubuntu16-04 to
marketplace.gcr.io/google/rbe-ubuntu18-04 (this image is still not owned
by us, but at least it's newer and demonstrates how a switch to a newer
docker image is done).
- provide script for generating the linux RBE toolchain configs.
- cleanup RBE configuration in the bazelrc files used for remote build
2023-03-10 09:34:53 +01:00
Yash Tibrewal 97ba987132
GCP Observability: Docs on Init (#32573)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-03-09 12:27:21 -08:00
Gregory Cooke ca9e365002
Added verified_root_cert_subject pass up through cpp api (#32335)
PR #32215 added the verified root cert subject to the lower level
`tsi_peer`. This PR is a companion to that and completes the feature by
bubbling the information up to the `TsiCustomVerificationCheckRequest`
which is part of the user facing API for implementing custom
verification callbacks.
2023-03-08 13:41:26 -05:00
edmondj cd4154f21d
Fixed memory leak in Alarm's move assignment 2023-03-07 15:43:12 -08:00
AJ Heller 1910986819
[build] Fix clang warnings in Windows code (#32448)
This fixes the problems identified while building with clang-cl on
Windows, with build arguments `/std:c++14 /W4`

Passes internal checks: cl/511562057

----

We can't yet enable a clang-cl build as part of our continuous
integration tests due to a few issues:

protobuf fails an `unused-parameter` warning check in v4.21 (the current
pinned version) on Windows. The upgrade to v4.22 is evidently painful
and in progress. Without maintaining a patch against protobuf, or
disabling warnings-as-errors somehow for the protobuf code alone, we'll
need to upgrade our dependency before we can automate the clang-cl build
for Windows.

Next, our Windows CI environment does not have clang installed. There
has been some work over the past year to create custom kokoro images,
but that work has apparently stalled after trading hands a few times.
Using our current images, installing clang every time we run the job may
be our best bet (likely from precompiled binaries that we host
ourselves), but it will eat up more CI resources.

Finally, some of the default build configurations are incorrect for
clang-cl. For example `-Wall` in clang-cl translates roughly to
`-Weverything` in clang linux, whereas `-W4` in clang-cl translates more
closely to `-Wall -Wextra`. This configuration in the gRPC bazel build
is not currently platform-specific, it will need to be updated.
Similarly, `-std=c++14` is an unknown argument on Windows (should be
`/std:c++14`), and should not be in the bazelrc. This will likely need
the same platform-specific support.
2023-02-23 16:33:34 -08:00
AJ Heller 070ce3630a
[EventEngine] Revert "Revert "EventEngine::RunAt: C++ Alarm (#30024)" (#30147)" (#32461)
This reverts commit 0fc0384b5a.

Major changes: this code calls `GetDefaultEventEngine` once on Alarm
init instead of 7 times throughout.

I will run benchmarks to ensure b/237283941 is not reproduced.


<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->

---------

Co-authored-by: drfloob <drfloob@users.noreply.github.com>
2023-02-23 10:05:30 -08:00
Vignesh Babu 8fdc82f33d
[EventEngine] Update Endpoint API to return true if read/write succeeds immediately (#32370)
Requires cherrypick
2023-02-23 09:51:21 -08:00
Yash Tibrewal bf23bb2fa7
Gcp Observability Logging: Batching and Graceful close (#32436)
This PR adds batching support for GCP Observability logging. So instead
of the naive creating a new RPC to cloud logging for each logging event,
we now batch the log events to meet one of the following requirements -
* Batch size of 1000
* Batch memory consumption of 1MB
* A timeout period of 1sec after which we flush the accumulated batch
irrespective of the size.

There can also be cases where for some reason the RPCs fail or the batch
just accumulates to a very large size(100000 entries or 10MB in size).
In such cases, we just log the events with gpr_log instead of just
continuing to accumulate.

Additionally, `GcpObservabilityClose()` has been added to gracefully
shut off logging where we block till all the currently logged events are
flushed. (We might be able to gracefully shut off stats and tracing in
the future too.)

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-02-22 22:32:44 -08:00
Yash Tibrewal d93d0022cd
GCP Observability: Fix constant labels support for stats when tracing is disabled (#32451)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-02-22 14:49:55 -08:00
Mark D. Roth 50e6af14ff
Bump dev version to 1.54.0-dev (#32426) 2023-02-21 12:03:19 -08:00
AJ Heller ffe3968d0b
[EventEngine] Add advice against blocking work in callbacks (#32397)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-02-15 14:00:55 -08:00
Yousuk Seung b98f527260
Revert "Revert "Revert "Revert "server: introduce ServerMetricRecorde… (#32301)
* Revert "Revert "Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)" (#32279)" (#32293)"

This reverts commit 1f960697c5.

* Do not create CallMetricRecorder if call is null.
2023-02-06 12:00:39 -08:00
Craig Tiller 1f960697c5
Revert "Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)" (#32279)" (#32293)
This reverts commit 4475e74c6a.
2023-02-03 13:18:23 -08:00
Yousuk Seung 4475e74c6a
Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)" (#32279)
* Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)"

This reverts commit deb1e25543.

* Fix by caching call metric recording stuff in async request

PR #32106 caused msan errors in some tests while de-referencing the
server object where async calls are active after the server is
destroyed. Instead cache the ServerMetricRecorder pointer.

* copyright headers fixed

* clang fixes.
2023-02-03 11:49:32 -08:00
Yijie Ma 08555c8723
Add enforcement for cpp-style comment license in check_copyright.py (#32273)
* enforce cpp style comment copyright license in check_copyright.py

also add functionality to fix(replace) c-style comment with cpp style
comment

* fix tools/codegen/core/gen_stats_data.py

* review
2023-02-02 13:00:20 -08:00
Xuan Wang deb1e25543
Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)
This reverts commit c7f641da0d.
2023-02-02 09:58:29 -08:00
Yousuk Seung c7f641da0d
server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)
* backend metric sampling

* Comments addressed.

* More comments addressed.

* Pushing changes left behind locally.

* Removed empty lines

* Update OrcaService to use ServerMetricRecorder (no named metrics yet)

* Comments addressed.

* More comments addressed

* More comments addressed.

* Comments fixed

* Comments addressed.

* Test fixed

* make seq returned always up-to-date

* skip atomic load when not cached

* Fixed ABSL_GUARDED_BY

* Comments addressed except client_lb_end2end_test

* test updated

* Comments addressed

* BUILD fix.

* BackendMetricDataState moved to a separate header

* comments addressed

* Fixed clang and buildifier errors

* More sanity check errors fixed.

* Fixed xds tests

* Ran generate_projects.sh

* Comments addressed

* comments addressed.

* generate project

* Build fixed

* generate project

* sanity check errors fixed

* test fixed

* Backup poller period override moved to main()

* Also move cfstream override

* Clang fixes, sanitize

* generate_projects.sh

* portable print format fix

* Removed outdated comment
2023-02-01 14:54:57 -08:00
AJ Heller d14020d4e2
[EventEngine] Add invalid handle types to the public API (#32202)
* [EventEngine] Add invalid handle types to the public API

* Automated change: Fix sanity tests

* add definition for static constexpr members

* sanitize

Co-authored-by: drfloob <drfloob@users.noreply.github.com>
2023-01-25 17:31:34 -08:00
AJ Heller 6a92df7237
[fixup] EventEngine Nomenclature (#32195) 2023-01-25 09:31:46 -08:00
Craig Tiller 7b02b7c253
[stats] Add experiment to use transport reported request latency whenever available (#32168)
* [stats] Add experiment to use transport reported request latency whenever available

* redux: add a new metric

* review-feedback

* Automated change: Fix sanity tests

* fix

* fix

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-01-24 08:09:59 -08:00
Vignesh Babu 8cf04e9a54
[EventEngine] Modify iomgr to allow creation of event engine clients and client side endpoints (#31661)
* [WIP] EventEngine iomgr endpoint shims

* [WIP] EventEngine::Endpoint iomgr shims for the PosixEventEngine

* Util functions to help with posix engine listener implementation

* sanity

* update comments in posix_engine_listener_utils.h

* review comments

* iwyu

* revert prev commit

* iwyu

* update build

* update

* regenerate projects

* regenerate projects

* minor fixes

* update BUILD

* sanity

* update build

* regenerate projects

* fix unused parameter

* sanity

* update

* sanity

* regenerate_projects

* remove unused variable

* start

* update

* regenerate_projects

* sanity

* update

* fixes

* update

* regenerate_projects

* update

* fix sanity and msan failure

* more fixes

* build failure

* update

* fix

* sanity

* fixes

* update

* regenerate projects

* fix sanity

* review comments

* An EventEngine subclass to be implemented by all posix based event engines

* sanity

* comments

* update

* review comments

* re-word

* fix

* update

* review comments

* regenerate projects

* syntax fix

* add lock free event benchmark

* releasable mutex lock

* fix build isue

* update

* start

* regenerate projects

* update

* fix

* windows build

* update

* windows portability issue

* update

* update

* update

* update

* format

* update

* update

* update

* update

* update

* fix sanity

* regenerate projects

* update

* iwyu

* Fix resolved address length related bugs in tcp_socket_utils and listener_utils

* iwyu

* cleanup src/core/lib/event_engine/tcp_socket_utils.cc

* iwyu

* fix

* regenerate projects

* fix sanity

* re-write endpoint shim

* more re-write

* cleanup

* update

* review comments

* build issue

* more build issue fixes plus adding event_engine_trace

* even more build issue fixes

* iwyu

* add static_cast

* update

* remove redundant code

* update

* deduplicate

* iwyu

* Fix review comments and regenerate_projects

* sanity

* review comments

* fix include guards

Co-authored-by: AJ Heller <hork@google.com>
2023-01-20 19:47:14 -08:00
Yijie Ma c5b8fd681f
Add an additional < marker in comment blocks after the documented functions (#32114)
* Add an additional < marker in the comment block for documentation block
after the member

* review
2023-01-18 16:55:51 -08:00
Yash Tibrewal 05491fb2f6
Gcp Observabiliy : Add constant labels support for stats and tracing (#32128)
* Gcp Observabiliy : Add constant labels support for stats and tracing

* Register GCP Observability labels

* sanity

* Fix build for CI

* TEST_TAG_KEY fix

* Remove duplicitous constant label setting
2023-01-18 14:32:36 -08:00
AJ Heller 8ac043638c
[EventEngine] Add ResolvedAddress default constructor (#32142)
* [EventEngine] Add ResolvedAddress default constructor

* alternative
2023-01-18 12:22:38 -08:00
Jan Tattermusch 7fd808f6f9
Bump version to 1.53.0-dev (on master branch) (#32086)
* bump version to 1.53.0-dev

* regenerate projects
2023-01-18 09:41:33 +01:00
Yousuk Seung 6b31b8d346
Add MakeChannelArgumentOption() for void* (#32096) 2023-01-13 17:15:30 -08:00
Craig Tiller 67f364e23e
[cleanup] Eliminate usage of GRPC_ASSERT(false...); (#31757)
* crash function

* progress

* fix

* fix

* Automated change: Fix sanity tests

* fix

* fix

* fix

* fixes

* Automated change: Fix sanity tests

* fix

* Automated change: Fix sanity tests

* fix

* fix

* use cpp attr

* Automated change: Fix sanity tests

* fix

* fix

* fix

* fix

* fix

* fix

* add exclusion

* fix

* typo

* fix

* fmt

* Update tcp_socket_utils.cc

* Automated change: Fix sanity tests

* fix

* revert php changes

* Automated change: Fix sanity tests

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-01-11 08:50:32 -08:00
Yash Tibrewal 75f2a2c1c6
GCP Observability: Client attempt top latency (#32064) 2023-01-10 13:40:33 -08:00
Mark D. Roth b0d7131ab3
WRR: plumb qps in backend metric data (#31903)
* plumb qps in backend metric data

* pull in updated orca proto

* gen_upb_api

* Automated change: Fix sanity tests

Co-authored-by: markdroth <markdroth@users.noreply.github.com>
2023-01-09 10:07:58 -08:00
AJ Heller 5db6de8634
[༺ EventEngine ༻] Specify requirements for Run* immediate execution (#32028)
* [༺ EventEngine༻ ] Specify requirements for Run* immediate execution

Also adds test suite tests that may catch non-conforming implementations
(Run called ~3200 times in case it's non-deterministic).

* fix

* verbiage; longer timeout

* fix int type comparison warning

* rm illegal term

* Automated change: Fix sanity tests

* disable tests

Co-authored-by: drfloob <drfloob@users.noreply.github.com>
2023-01-05 16:26:00 -08:00
Vignesh Babu 8bd2b9af52
Fixing some bugs in posix event engine (#31785)
* update

* regenerate projects

* fix sanity

* review comments

* fix

* add lock free event benchmark

* releasable mutex lock

* fix build isue

* update

* fix sanity

* regenerate projects

* iwyu

* Fix resolved address length related bugs in tcp_socket_utils and listener_utils

* iwyu

* cleanup src/core/lib/event_engine/tcp_socket_utils.cc

* update endpoint to use SliceCast into MutableSlice

* fix sanity

* review comments

* review comments

* regenerate projects

* fix

* rm unused variable

* fix
2023-01-03 17:52:46 -08:00
Vignesh Babu 4f9ba0f49b
Expand SliceCast to support casts to MutableSlice type (#31996)
* Expand SliceCast to support casts to MutableSlice type

* update
2022-12-30 11:36:12 -08:00
Yijie Ma f99b8b5bc4
Convert c-style comments to C++-style comments (#31923)
* baseline

* fix clang-tidy

* manually revert these files

* manually fixup at eof

* revert 2 more files

* change check_deprecated_grpc++.py

* change end2end_defs.include template

* fix check_include_guards.py

* untrack tools/distrib/python/convert_cstyle_comments_to_cpp.py

not yet ready to be submitted

* fix

yapf check_include_guards.py
remove a space...

* fix version.cc.template

* fix version_info.h.template
2022-12-22 23:01:53 -08:00
Craig Tiller 5664d6ee6b
[slice] Mutable slice cast (#31959)
* [slice] Mutable slice cast

* fix
2022-12-22 12:45:25 -08:00
Craig Tiller 72c296a3c2
[event_engine] Add SliceCast (#31831)
* [event_engine] Add SliceCast

* Automated change: Fix sanity tests

* windows-fix

* comments

* comments

* comments

* build-fix

* fix

* Update port_platform.h

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2022-12-20 19:46:27 -08:00
Yash Tibrewal 3683ff924b
GCP Observability: Add server latency metric (#31900) 2022-12-16 09:32:56 -08:00
Cheng-Yu Chung 1b05576aa3
[cleanups] Clean up `GrpcLibraryInitializer` (#31894)
* Clean up `GrpcLibraryInitializer`

* Update

* Automated change: Fix sanity tests
2022-12-17 01:08:35 +08:00
Yash Tibrewal 6a97f492ff
Add method to add annotations to call tracer (#31901) 2022-12-16 02:28:59 +05:30
Yash Tibrewal 5100b33e0e
GCP Observability: Add payload metrics (#31899) 2022-12-15 11:07:09 -08:00