Commit Graph

253 Commits

Author SHA1 Message Date
Yousuk Seung c02b3e695c
xDS: Include orca named_metrics in LRS load reports (#32690)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-03-28 12:04:01 -07:00
Craig Tiller 175ccc3a90
Reland global config changes (#32661)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->

---------

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-03-27 18:55:24 -07:00
Yousuk Seung 16c03db9ac
Revert "Revert "WRR: Support EPS" (#32723)" (#32725)
This reverts commit 7bd9267f32.




<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-03-27 16:49:50 -07:00
Esun Kim 7bd9267f32
Revert "WRR: Support EPS" (#32723)
Reverts grpc/grpc#32657
2023-03-27 11:53:17 -07:00
Yousuk Seung 4429066516
WRR: Support EPS (#32657)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-03-24 16:23:51 -07:00
Jan Tattermusch 0c1797cd9f
Revert "[config] Move global config alongside core configuration" (#32659)
Reverts grpc/grpc#30788

(it breaks grpc_objc_bazel_test (see
https://github.com/grpc/grpc/pull/30788#issuecomment-1476372187) and
also seems to be breaking some other internal stuff).
2023-03-20 08:22:29 -07:00
Craig Tiller b7a83305e6
[config] Move global config alongside core configuration (#30788)
This is a big rewrite of global config.

It does a few things, all somewhat intertwined:
1. centralize the list of configuration we have to a yaml file that can
be parsed, and code generated from it
2. add an initialization and a reset stage so that config vars can be
centrally accessed very quickly without the need for caching them
3. makes the syntax more C++ like (less macros!)
4. (optionally) adds absl flags to the OSS build

This first round of changes is intended to keep the system where it is
without major changes. We pick up absl flags to match internal code and
remove one point of deviation - but importantly continue to read from
the environment variables. In doing so we don't force absl flags on our
customers - it's possible to configure grpc without the flags - but
instead allow users that do use absl flags to configure grpc using that
mechanism. Importantly this lets internal customers configure grpc the
same everywhere.

Future changes along this path will be two-fold:
1. Move documentation generation into the code generation step, so that
within the source of truth yaml file we can find all documentation and
data about a configuration knob - eliminating the chance of forgetting
to document something in all the right places.
2. Provide fuzzing over configurations. Currently most config variables
get stashed in static constants across the codebase. To fuzz over these
we'd need a way to reset those cached values between fuzzing rounds,
something that is terrifically difficult right now, but with these
changes should simply be a reset on `ConfigVars`.

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->

---------

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-03-17 16:56:45 -07:00
Yousuk Seung 3cc76171a9
Merge per-request and per-server named metrics field-wise (#32634)
We currently take named metrics recorded per-request only. Instead we
should merge field-wise.

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-03-16 13:30:04 -07:00
Yijie Ma 2ce147a131
Add a test case to verify SubchannelStreamClient retry when Health.Watch ends (#31850)
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-02-28 19:14:57 -08:00
Mark D. Roth 9dd6a98ed6
e2e tests: update regex used for connection failure messages (#32325)
* e2e tests: update regex used for connection failure messages

* attempt to tweak regex syntax to work on mac
2023-02-08 14:31:07 -08:00
Yousuk Seung b98f527260
Revert "Revert "Revert "Revert "server: introduce ServerMetricRecorde… (#32301)
* Revert "Revert "Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)" (#32279)" (#32293)"

This reverts commit 1f960697c5.

* Do not create CallMetricRecorder if call is null.
2023-02-06 12:00:39 -08:00
Craig Tiller 1f960697c5
Revert "Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)" (#32279)" (#32293)
This reverts commit 4475e74c6a.
2023-02-03 13:18:23 -08:00
Yousuk Seung 4475e74c6a
Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)" (#32279)
* Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)"

This reverts commit deb1e25543.

* Fix by caching call metric recording stuff in async request

PR #32106 caused msan errors in some tests while de-referencing the
server object where async calls are active after the server is
destroyed. Instead cache the ServerMetricRecorder pointer.

* copyright headers fixed

* clang fixes.
2023-02-03 11:49:32 -08:00
Xuan Wang deb1e25543
Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)
This reverts commit c7f641da0d.
2023-02-02 09:58:29 -08:00
Yousuk Seung c7f641da0d
server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)
* backend metric sampling

* Comments addressed.

* More comments addressed.

* Pushing changes left behind locally.

* Removed empty lines

* Update OrcaService to use ServerMetricRecorder (no named metrics yet)

* Comments addressed.

* More comments addressed

* More comments addressed.

* Comments fixed

* Comments addressed.

* Test fixed

* make seq returned always up-to-date

* skip atomic load when not cached

* Fixed ABSL_GUARDED_BY

* Comments addressed except client_lb_end2end_test

* test updated

* Comments addressed

* BUILD fix.

* BackendMetricDataState moved to a separate header

* comments addressed

* Fixed clang and buildifier errors

* More sanity check errors fixed.

* Fixed xds tests

* Ran generate_projects.sh

* Comments addressed

* comments addressed.

* generate project

* Build fixed

* generate project

* sanity check errors fixed

* test fixed

* Backup poller period override moved to main()

* Also move cfstream override

* Clang fixes, sanitize

* generate_projects.sh

* portable print format fix

* Removed outdated comment
2023-02-01 14:54:57 -08:00
Mark D. Roth 76c82265b4
WRR: implement WRR LB policy (#31904)
* WRR: port StaticStrideScheduler to OSS

* WIP

* Automated change: Fix sanity tests

* fix build

* remove unused aliases

* fix another type mismatch

* remove unnecessary include

* move benchmarks to their own file, and don't run it on windows

* Automated change: Fix sanity tests

* add OOB reporting

* generate_projects

* clang-format

* add config parser test

* clang-tidy and minimize lock contention

* add config defaults

* add oob_reporting_period config field and add basic test

* Automated change: Fix sanity tests

* fix test

* change test to use basic RR

* WIP: started exposing peer address to LB policy API

* first WRR test passing!

* small cleanup

* port RR fix to WRR

* test helper refactoring

* more test helper refactoring

* WIP: trying to fix test to have the right weights

* more WIP -- need to make pickers DualRefCounted

* fix timer ref handling and get tests working

* clang-format

* iwyu and generate_projects

* fix build

* add test for OOB reporting

* keep only READY subchannels in the picker

* add file missed in a previous commit

* fix sanity

* iwyu

* add weight expiration period

* add tests for weight update period and OOB reporting period

* Automated change: Fix sanity tests

* lower bound for timer interval

* consistently apply grpc_test_slowdown_factor()

* cache time in test

* add blackout_period tests

* avoid some unnecessary copies

* clang-format

* add field to config test

* simplify orca watcher tracking

* attempt to fix build

* iwyu

* generate_projects

* add "_experimental" suffix to policy name

* WRR: update tests to cover qps plumbing

* WIP

* more WIP

* basic WRR e2e test working

* add OOB test

* fix sanity

* ignore duplicate addresses

* Automated change: Fix sanity tests

* add new tracer to doc/environment_variables.md

* retain scheduler state across pickers

* Automated change: Fix sanity tests

* use separate mutexes for scheduler and timer

* sort addresses to avoid index churn

* remove fetch_sub for wrap around in RR case

Co-authored-by: markdroth <markdroth@users.noreply.github.com>
2023-01-20 12:14:37 -08:00
Craig Tiller 67f364e23e
[cleanup] Eliminate usage of GRPC_ASSERT(false...); (#31757)
* crash function

* progress

* fix

* fix

* Automated change: Fix sanity tests

* fix

* fix

* fix

* fixes

* Automated change: Fix sanity tests

* fix

* Automated change: Fix sanity tests

* fix

* fix

* use cpp attr

* Automated change: Fix sanity tests

* fix

* fix

* fix

* fix

* fix

* fix

* add exclusion

* fix

* typo

* fix

* fmt

* Update tcp_socket_utils.cc

* Automated change: Fix sanity tests

* fix

* revert php changes

* Automated change: Fix sanity tests

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-01-11 08:50:32 -08:00
Mark D. Roth daded8a59f
WRR: update tests to cover qps plumbing (#32068) 2023-01-11 08:14:29 -08:00
Cheng-Yu Chung e9b287997a
Remove `include/grpcpp/impl/codegen/sync.h` (#31545)
* Remove `include/grpcpp/impl/codegen/sync.h`

* Fix sanity tests
2022-11-22 13:45:53 -05:00
Esun Kim 0ba0c4bf74
Added googletest clang-tidy checks (#31610)
* Added google-googletest

* Autofix

* Fix test names

* Fix
2022-11-18 13:02:20 -08:00
Mark D. Roth ddfa85f42b
xDS: fix error message when the request does not match any route (#31593)
* improve error message when there is no matching route

* clean up ConfigSelector API

* clang-format

* fix client_lb_end2end_test

* clang-format
2022-11-09 14:29:22 -08:00
Esun Kim 530b702259
Update Clang 15 (#31518)
* clang-15

* iwyu

* run clang_format_code

* Clang-Tidy config

* Fix misc-unused-using-decls

* Fixed performance-inefficient-vector-operation

* Sanity
2022-11-04 09:08:03 -07:00
Mark D. Roth b6bdde391a
client_lb_e2e_test: add another error case to the regex (#31318) 2022-10-11 16:13:32 -07:00
Mark D. Roth 126393c9c1
client_lb_end2end_test: update failure regex to include "Socket closed" (#31242) 2022-10-05 10:09:17 -07:00
Esun Kim 3c96517fc0
Simplified GRPC_ERROR_CREATE macros (#31222)
* Simplified GRPC_ERROR_CREATE macros

* Code migration
2022-10-04 15:55:47 -07:00
Craig Tiller 9c98d67d7a
[c++14] absl::make_unique --> std::make_unique (#31159)
* [c++14] absl::make_unique --> std::make_unique

* fix

* fix
2022-09-29 06:45:17 -07:00
Craig Tiller d9ac89a441
[gprpp] absl::Notification polyfill (#31008)
* [gprpp] absl::Notification polyfill

* grpc_core::

* Automated change: Fix sanity tests

* fix

* Automated change: Fix sanity tests

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2022-09-16 09:06:03 -07:00
Craig Tiller c2ab8c99bb
Revert "Revert "[c++] Move environment functions to C++ (#30937)" (#30986)" (#30988)
This reverts commit 96264e07b8.
2022-09-14 14:19:36 -07:00
Craig Tiller 96264e07b8
Revert "[c++] Move environment functions to C++ (#30937)" (#30986)
This reverts commit 74c0d6fe3f.
2022-09-14 11:53:55 -07:00
Mark D. Roth e5aadf985e
Revert "Revert "client_channel: allow LB policy to communicate update errors to resolver (#30809)" (#30970)" (#30981)
This reverts commit 1648bc08e0.
2022-09-14 11:46:42 -07:00
Craig Tiller 74c0d6fe3f
[c++] Move environment functions to C++ (#30937)
* [gprpp] Move env to C++

* move headers/impl

* Automated change: Fix sanity tests

* fix

* fix

* Automated change: Fix sanity tests

* Update http_proxy.cc

* fix

* fix

* rename

* fix merge

* fix

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2022-09-14 10:17:15 -07:00
Richard Belleville 1648bc08e0
Revert "client_channel: allow LB policy to communicate update errors to resolver (#30809)" (#30970)
This reverts commit 9ff943b81e.
2022-09-13 15:07:50 -07:00
Mark D. Roth 9ff943b81e
client_channel: allow LB policy to communicate update errors to resolver (#30809)
* client_channel: allow LB policy to communicate update errors to resolver

* fix tests

* Automated change: Fix sanity tests

* fix build

* fix another "ignoring return value" warning

* fix use-after-move

* fix channel to invoke resolver callback when service config fails on initial resolution

* remove outdated TODO

* improve comments

* fix PollingResolver to defer re-resolution requests while waiting for result-health callback

* Automated change: Fix sanity tests

* absl::exchange -> std::exchange

* fix dns_resolver_cooldown_test

Co-authored-by: markdroth <markdroth@users.noreply.github.com>
2022-09-12 17:17:03 -07:00
Vignesh Babu 70dd34b438
Revert "Revert "Modifying iomgr tcp code to use event engine EndpointConfig instead of channel_args"" (#30509)
* Revert "Revert "Modifying iomgr tcp code to use event engine EndpointConfig instead of channel_args (#30028)" (#30495)"

This reverts commit 0b1a8c984b.

* bug fix

* bug fix

* fix sanity
2022-09-01 16:51:17 -07:00
Mark D. Roth bf9304ef17
client_channel, client_auth: rewrite disallowed status codes from the control plane (#30789)
* client_channel: rewrite illegal status codes from control plane

* rewrite illegal status codes for call creds

* move fail_lb policy out of retry_lb_fail test so it can be reused

* test resolver and LB policy status rewrites

* add test for ConfigSelector status rewriting

* attempt to add client_auth filter unit test

* fix client_auth_filter test

* cleanup test

* fix build

* fix some memory leaks

* Automated change: Fix sanity tests

* Update client_auth_filter_test.cc

* fix build

* code review comments

* clang-tidy

Co-authored-by: markdroth <markdroth@users.noreply.github.com>
Co-authored-by: Craig Tiller <ctiller@google.com>
2022-09-01 15:15:14 -07:00
Mark D. Roth 2c28b44852
subchannel: include IP address in status messages (#30784)
* subchannel: include IP address in status messages

* Automated change: Fix sanity tests

* add comment about lock annotations

* clang-format

Co-authored-by: markdroth <markdroth@users.noreply.github.com>
2022-08-30 17:01:59 -07:00
Craig Tiller 535168431a
[lb_policy] Move to core configuration system (#30483)
* [load_balancing] Move interfaces to src/core/lib/load_balancing

* Automated change: Fix sanity tests

* progress

* progress

* fix

* missed file

* Automated change: Fix sanity tests

* update tests

* Automated change: Fix sanity tests

* Automated change: Fix sanity tests

* fix

* fix

* Automated change: Fix sanity tests

* fix

* fixes

* fix

* fix

* fix

* fix

* fix

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2022-08-27 08:35:40 -07:00
Alisha Nanda b94fb2b894
Fix PickFirstTest.PendingUpdateAndSelectedSubchannelFails flake in client_lb_end2end_test (#30741)
* ConnectionAttemptInjector: fix tsan failures

* Add hold to test

* Add hold to test

* Address comments

* Address comments

* Address comments

* Fix typo

Co-authored-by: Mark D. Roth <roth@google.com>
2022-08-26 14:26:54 -07:00
Mark D. Roth 121a08f6a9
end2end tests: apply test slowdown factor in various places where it was missed (#30749)
* e2e tests: add test scaling factor to durations in channel args

* apply test scaling factor when encoding durations in xDS protos

* apply scaling factor in fixed timeout in WaitForNack()

* fix overflow

* clang-format

* adjust timeouts in fault injection tests

* add missing slowdown factor

* clang-format

* add tests for duration multiplication
2022-08-26 07:35:28 -07:00
Mark D. Roth 03b6b01043
ConnectionAttemptInjector: fix tsan failures (#30730) 2022-08-24 12:47:17 -07:00
Mark D. Roth 72e76f6a86
client_lb_e2e_test: fix flake in PickFirstTest.CheckStateBeforeStartWatch (#30698) 2022-08-22 14:42:10 -07:00
Vignesh Babu 9f9d87cfa6
Fix CBF on mac by switching to grpc_core::Mutex and grpc_core::CondVar (#30666) 2022-08-19 16:59:18 -07:00
Mark D. Roth 221a969e04
client_lb_end2end_test: fix flake caused by not doing graceful server shutdown (#30647) 2022-08-19 13:56:42 -07:00
Alisha Nanda 96af4084c9
Fix StatusCancelledWithoutStartingRecvTrailingMetadata flake in client_lb_end2end_test (#30629)
* Fix metadata flake in client_lb_end2end_test

* Address review comments

* Return bool

* Address review comment
2022-08-19 12:37:51 -07:00
Alisha Nanda faa0679c57
Fix flake in client_lb_end2end_test (#30601)
* Fix ubsan failure

* Fix build failures with istio test

* Fix other status code

* Fix flake in client_lb_end2end_test
2022-08-16 14:17:46 -07:00
Vignesh Babu 0b1a8c984b
Revert "Modifying iomgr tcp code to use event engine EndpointConfig instead of channel_args (#30028)" (#30495)
This reverts commit 2c977a4d98.
2022-08-04 11:35:27 -07:00
Vignesh Babu 2c977a4d98
Modifying iomgr tcp code to use event engine EndpointConfig instead of channel_args (#30028)
* convert tcp code to use EndpointConfig and successfully compile

* regenerate projects

* copying config options used in tcp code to create map backed endpoint configs

* fix minor build issues

* fix includes in tests

* fix some build and test issues

* modifying endpoint config definition from an abstract interface into a concrete implementation

* minor fix

* add unreachable code check

* fix some windows build issues

* fix more build issues

* adding a grpc_tcp_generic_options struct to hold data extracted from EndpointConfig

* code formatting

* simplify

* fix sanity checks

* fix windows issues

* some posix fixes

* disabling copy constructor and copy assignment operator for endpoint config

* fix sanity checks

* fix syntax error

* fix weird formatting

* fix build issue

* fix review comments

* reverting un-necessary change

* remove channel args completely from windows impl since it is unused

* remove unused headers and fix usage in endpoint_pair_windows.cc

* cleanup

* cleanup

* fix some initialization issues

* re-generate projects

* removing some files

* fix ifdef for windows

* fixing windows build issue due to improper header file include

* Automated change: Fix sanity tests

* Automated change: Fix sanity tests

* review comments

* add comment

* review comments

* rename GetPointer to GetVoidPointer

* cleanup

* Automated change: Fix sanity tests

* removing EndpointConfig::Get method

* Automated change: Fix sanity tests

* minor fix

Co-authored-by: Vignesh2208 <Vignesh2208@users.noreply.github.com>
2022-08-03 13:41:48 -07:00
Mark D. Roth f4edc883ab
service config API: use absl::Status instead of grpc_error (#30321)
* service config API: use absl::Status instead of grpc_error

* Automated change: Fix sanity tests

* add missing build deps

* attempt to work around build breakage on older compilers

* trying the work-around in more spots

* more work-arounds

* more workarounds

* Automated change: Fix sanity tests

* work around another compiler problem

* Automated change: Fix sanity tests

* Automated change: Fix sanity tests

Co-authored-by: markdroth <markdroth@users.noreply.github.com>
2022-07-26 09:29:07 -07:00
Mark D. Roth 201213c681
client channel: add tests for authority override precedence (#30172)
* client channel: add tests for authority override precedence

* add unit test

* Automated change: Fix sanity tests

Co-authored-by: markdroth <markdroth@users.noreply.github.com>
2022-07-18 16:02:59 -07:00
Craig Tiller d304712f64
[channel_args] Spread grpc_core::ChannelArgs through client channel code (#30008)
* [channel_args] Spread grpc_core::ChannelArgs through client channel code

* progress

* progress

* grpc compiles

* uniqueptr+compiles

* fix

* fix

* fix

* fix

* fix

* fix

* fixes

* fix

* fix

* fixes

* fix-lb

* fix

* fix up arg construction

* fix

* fix

* fix

* fix

* move to const& to save bytes

* fix

* fix

* progress

* review feedback

* fix

* fix

* fixes

* fixes

* Automated change: Fix sanity tests

* Automated change: Fix sanity tests

* Automated change: Fix sanity tests

* Automated change: Fix sanity tests

* fix

* fix

* Automated change: Fix sanity tests

* Automated change: Fix sanity tests

* fix

* debug-helper

* Automated change: Fix sanity tests

* fix

* Automated change: Fix sanity tests

* fix

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2022-07-07 19:19:11 -07:00