Commit Graph

6077 Commits

Author SHA1 Message Date
Craig Tiller 033d55ffd3
[arena] Fix ABA problem in pooled allocation (#32336)
The pooled allocator currently has an ABA issue in the allocation path.

This change should fix that - algorithm is described reasonably well in
the PR.
<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
2023-02-09 18:06:19 -08:00
Mark D. Roth 9dd6a98ed6
e2e tests: update regex used for connection failure messages (#32325)
* e2e tests: update regex used for connection failure messages

* attempt to tweak regex syntax to work on mac
2023-02-08 14:31:07 -08:00
Craig Tiller ae969b274b
[chttp2] Eliminate legacy huffman decoder (#32289)
* [chttp2] Eliminate legacy huffman decoder

* Automated change: Fix sanity tests

---------

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-02-07 19:53:41 -08:00
Craig Tiller 135f6e84b6
[flake] Remove inproc thread stress tests (#32283)
These tests are failing because they're running with too few threads,
however if we give them sufficient threads to catch bugs they're flaky.

Remove them and get the team some bandwidth back.
2023-02-07 08:58:41 -08:00
Yousuk Seung b98f527260
Revert "Revert "Revert "Revert "server: introduce ServerMetricRecorde… (#32301)
* Revert "Revert "Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)" (#32279)" (#32293)"

This reverts commit 1f960697c5.

* Do not create CallMetricRecorder if call is null.
2023-02-06 12:00:39 -08:00
Craig Tiller 1f960697c5
Revert "Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)" (#32279)" (#32293)
This reverts commit 4475e74c6a.
2023-02-03 13:18:23 -08:00
Yousuk Seung 4475e74c6a
Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)" (#32279)
* Revert "Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)"

This reverts commit deb1e25543.

* Fix by caching call metric recording stuff in async request

PR #32106 caused msan errors in some tests while de-referencing the
server object where async calls are active after the server is
destroyed. Instead cache the ServerMetricRecorder pointer.

* copyright headers fixed

* clang fixes.
2023-02-03 11:49:32 -08:00
Eugene Ostroukhov 72872fc29d
stateful session affinity: add e2e test for draining state (#32248) 2023-02-02 14:59:10 -08:00
Yijie Ma 08555c8723
Add enforcement for cpp-style comment license in check_copyright.py (#32273)
* enforce cpp style comment copyright license in check_copyright.py

also add functionality to fix(replace) c-style comment with cpp style
comment

* fix tools/codegen/core/gen_stats_data.py

* review
2023-02-02 13:00:20 -08:00
AJ Heller e4af983819
Reland "EventEngine::RunAfter: GrpcLb" (#32262)" (#32269)
There was a ~1% flake in grpclb end2end tests that was reproducible in opt builds, manifesting as a hang, usually in a the SingleBalancerTest.Fallback test. Through experimentation, I found that by skipping the death test in the grpclb end2end test suite, the hang was no longer reproducible in 10,000 runs. Similarly, moving this test to the end of the suite, or making it run first (as is the case in this PR) resulted in 0 failures in 3000 runs.

It's unclear to me yet why the death test causes things to be unstable in this way. It's clear from the logs that one test does affect the rest, grpc_init is done once for all tests, so all tests utilize the same EventEngine ... until the death test completes, and a new EventEngine is created for the next test.

I think this death test is sufficiently artificial that it's fine to change the test ordering itself, and ignore the wonky intermediate state that results from it.

Reproducing the flake:

```
tools/bazel --bazelrc=tools/remote_build/linux.bazelrc test \
  -c opt \
  --test_env=GRPC_TRACE=event_engine \
  --runs_per_test=5000 \
  --test_output=summary \
  test/cpp/end2end/grpclb_end2end_test@poller=epoll1 
```
2023-02-02 11:20:42 -08:00
Xuan Wang deb1e25543
Revert "server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)" (#32272)
This reverts commit c7f641da0d.
2023-02-02 09:58:29 -08:00
Craig Tiller 0893fca089
[promises] Roll forward server promise calls with fixes (#32261)
* stuff

* spam-cleanup

* Revert "Revert "[promises] Server call (#31448)" (#32260)"

This reverts commit dbfb888e1f.

* spam-cleanup

* Automated change: Fix sanity tests

* asserts-for-sanity

---------

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-02-01 16:07:46 -08:00
Yousuk Seung c7f641da0d
server: introduce ServerMetricRecorder API and move per-call reporting from a C++ interceptor to a C-core filter (#32106)
* backend metric sampling

* Comments addressed.

* More comments addressed.

* Pushing changes left behind locally.

* Removed empty lines

* Update OrcaService to use ServerMetricRecorder (no named metrics yet)

* Comments addressed.

* More comments addressed

* More comments addressed.

* Comments fixed

* Comments addressed.

* Test fixed

* make seq returned always up-to-date

* skip atomic load when not cached

* Fixed ABSL_GUARDED_BY

* Comments addressed except client_lb_end2end_test

* test updated

* Comments addressed

* BUILD fix.

* BackendMetricDataState moved to a separate header

* comments addressed

* Fixed clang and buildifier errors

* More sanity check errors fixed.

* Fixed xds tests

* Ran generate_projects.sh

* Comments addressed

* comments addressed.

* generate project

* Build fixed

* generate project

* sanity check errors fixed

* test fixed

* Backup poller period override moved to main()

* Also move cfstream override

* Clang fixes, sanitize

* generate_projects.sh

* portable print format fix

* Removed outdated comment
2023-02-01 14:54:57 -08:00
Craig Tiller 439c7518a3
[thread_stress_test] reduce thread count to a reasonable number (#32263) 2023-02-01 12:16:18 -08:00
Craig Tiller dbfb888e1f
Revert "[promises] Server call (#31448)" (#32260)
* Revert "[promises] Server call (#31448)"

This reverts commit bbeb15006a.

* clang-fmt
2023-02-01 09:18:04 -08:00
Mark D. Roth 75f4ee6e5e
xDS: fix bug when receiving an update for an aggregate cluster child (#32249) 2023-01-31 14:58:03 -08:00
Eugene Ostroukhov b168e9cc9d
stateful session affinity: add end2end tests (#32009)
Co-authored-by: Mark D. Roth <roth@google.com>
2023-01-31 10:10:36 -08:00
Craig Tiller 32be2c2179
[benchmark] Fix static initialization bug in bm_huffman_decode (#32245)
* [benchmark] Fix static initialization bug in bm_huffman_decode

* fix performance
2023-01-31 07:07:08 -08:00
Craig Tiller bbeb15006a
[promises] Server call (#31448)
* add experiment

* allow instantiation

* scratchings

* scratchings

* sniffly

* Automated change: Fix sanity tests

* fix

* fix

* fix

* Automated change: Fix sanity tests

* progress

* change pipe labels to enable server code to be written

* better api

* Automated change: Fix sanity tests

* progress

* [promises] Implementation of deadline for server-based-calls

* compression filter compiles again

* Automated change: Fix sanity tests

* fix

* server tracing fixes

* get client initial metadata

* progress

* progress

* server call surface progress

* Automated change: Fix sanity tests

* move payload

* server-progress

* recv-message-server-connchan

* logging

* fix context-gate

* recv fix@top

* Automated change: Fix sanity tests

* recv close on server

* top termination start

* [promises] Move Empty to be first class

* fixes

* fix

* flow control fix

* got to orphan!

* orphan

* call orphan

* spam cleanup

* fix

* new cancelation semantics

* progress

* large metadata fixes

* fix

* fix

* log

* better logs

* fix-chanz

* logging, necessaryness

* fix typo

* fixes

* fix

* fix

* fix-pipe

* cleanup logging

* fix

* build-fix

* fix

* Automated change: Fix sanity tests

* logging

* Automated change: Fix sanity tests

* Automated change: Fix sanity tests

* better primitive

* Revert "better primitive"

This reverts commit 119b5ee244.

* fix

* fix

* trrracing

* Automated change: Fix sanity tests

* get-trailing-metadata

* cancellation

* Automated change: Fix sanity tests

* add transform pipeline to pipe

* add transform pipeline to pipe

* interceptor lists

* new server initial md api into filters

* convert connected_channel

* convert call

* initial promise based filter conversion

* convert promise based filter

* build fixes

* compile fix

* fixes

* fix ordering

* fixes

* check-metadata

* revert later: debug code

* better debug

* fix metadata ordering with messages in promise based filter

* fix ordering problem between batch completion and promise completion

* properly handle failure on receive message path on client

* more debug, fix a repoll bug in pbf

* Automated change: Fix sanity tests

* fixes

* Automated change: Fix sanity tests

* cleanup logging

* fixes

* missing file

* fixes

* logging

* Automated change: Fix sanity tests

* fixes

* convert logging filter

* fix

* Automated change: Fix sanity tests

* fix bad server response test

* Revert "Disable logging test (#32049)"

This reverts commit 5fc92eaeae.

* fix

* Automated change: Fix sanity tests

* fix memory leaks, logging

* Automated change: Fix sanity tests

* slice refcount debugging

* asan-canaries

* leak-fix

* leak-fix

* Automated change: Fix sanity tests

* fix

* fix

* fix

* fix

* fix

* Automated change: Fix sanity tests

* fix

* remove mistaken line

* add-comment

* fix refcounting bug

* Automated change: Fix sanity tests

* rename variable

* renames

* bleh

* carry pipe close status from bottom of pipe to top to appease
recv-close-on-server

* backport cancellation

* Revert "carry pipe close status from bottom of pipe to top to appease"

This reverts commit fa33301dcd.

* fix

* Automated change: Fix sanity tests

* review-feedback

* comment-ordering

* monostate

* renames

* undo-review-feedback

* fix

* review-feedback

* review-feedback

* fix

* review-feedback

* drop debugloc constructor

* interceptor-list-rev-feedback

* interceptor-list-rev-feedback

* pipe test

* review-feedback

* undo-mistaken-change

* Automated change: Fix sanity tests

* pipe error state

* detect send/recv failures and report

* iwyu, build

* fix submodules

* fix

* warning

* cleanup

* Automated change: Fix sanity tests

* fix

* fix for windows

* fix

* null pointer fix

* iwyu

* gen projex

---------

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-01-30 21:39:11 -08:00
Mark D. Roth 05a75d2d8c
rls_e2e_test: increase RPC timeout (#32222) 2023-01-26 14:54:11 -08:00
AJ Heller 6a92df7237
[fixup] EventEngine Nomenclature (#32195) 2023-01-25 09:31:46 -08:00
AJ Heller 2e69349731
[codegen] Escape variable delimiters in raw comment printing (#32190) 2023-01-24 14:39:19 -08:00
Craig Tiller 7b02b7c253
[stats] Add experiment to use transport reported request latency whenever available (#32168)
* [stats] Add experiment to use transport reported request latency whenever available

* redux: add a new metric

* review-feedback

* Automated change: Fix sanity tests

* fix

* fix

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-01-24 08:09:59 -08:00
Yash Tibrewal beffe313e5
GCP Observability Logging: Base64 Encode metadata, message and status-details (#32184)
* GCP Observability Logging: Base64 Encode metadata, message and status-details

* Use padding for message and status-details
2023-01-23 18:57:09 -08:00
Mark D. Roth 1c4b138322
Revert "Revert "WRR: xDS LB policy support (#32067)" (#32180)" (#32182)
This reverts commit d7193a28c4.
2023-01-23 14:14:32 -08:00
Yash Tibrewal 7f7a352ccc
Fix flakiness in test/cpp/ext/filters/census/... (#32174) 2023-01-23 12:13:45 -08:00
Alisha Nanda d7193a28c4
Revert "WRR: xDS LB policy support (#32067)" (#32180)
This reverts commit 8a024963f5.
2023-01-23 09:16:11 -08:00
Mark D. Roth 8a024963f5
WRR: xDS LB policy support (#32067)
* WRR: port StaticStrideScheduler to OSS

* WIP

* Automated change: Fix sanity tests

* fix build

* remove unused aliases

* fix another type mismatch

* remove unnecessary include

* move benchmarks to their own file, and don't run it on windows

* Automated change: Fix sanity tests

* add OOB reporting

* generate_projects

* clang-format

* add config parser test

* clang-tidy and minimize lock contention

* add config defaults

* add oob_reporting_period config field and add basic test

* Automated change: Fix sanity tests

* fix test

* change test to use basic RR

* WIP: started exposing peer address to LB policy API

* first WRR test passing!

* small cleanup

* port RR fix to WRR

* test helper refactoring

* more test helper refactoring

* WIP: trying to fix test to have the right weights

* more WIP -- need to make pickers DualRefCounted

* fix timer ref handling and get tests working

* clang-format

* iwyu and generate_projects

* fix build

* add test for OOB reporting

* keep only READY subchannels in the picker

* add file missed in a previous commit

* fix sanity

* iwyu

* add weight expiration period

* add tests for weight update period and OOB reporting period

* Automated change: Fix sanity tests

* lower bound for timer interval

* consistently apply grpc_test_slowdown_factor()

* cache time in test

* add blackout_period tests

* avoid some unnecessary copies

* clang-format

* add field to config test

* simplify orca watcher tracking

* attempt to fix build

* iwyu

* generate_projects

* update xds proto dependency

* add xDS LB policy entry to registry

* add "_experimental" suffix to policy name

* update LB policy name and remove debug log

* add env var protection

* generate_projects

* gen_upb_api

* WRR: update tests to cover qps plumbing

* WIP

* Automated change: Fix sanity tests

* more WIP

* basic WRR e2e test working

* add OOB test

* add xDS WRR e2e test

* clang-format

* fix sanity

* ignore duplicate addresses

* Automated change: Fix sanity tests

* add new tracer to doc/environment_variables.md

* retain scheduler state across pickers

* Automated change: Fix sanity tests

* use separate mutexes for scheduler and timer

* sort addresses to avoid index churn

* remove fetch_sub for wrap around in RR case

Co-authored-by: markdroth <markdroth@users.noreply.github.com>
2023-01-23 08:08:52 -08:00
Yash Tibrewal aa11978541
Fix include guards for src/ and test/ directories (#32167)
* Fix include guards for src/ and test/ directories

* Sanity

* Update new files
2023-01-20 16:27:27 -08:00
Mark D. Roth 76c82265b4
WRR: implement WRR LB policy (#31904)
* WRR: port StaticStrideScheduler to OSS

* WIP

* Automated change: Fix sanity tests

* fix build

* remove unused aliases

* fix another type mismatch

* remove unnecessary include

* move benchmarks to their own file, and don't run it on windows

* Automated change: Fix sanity tests

* add OOB reporting

* generate_projects

* clang-format

* add config parser test

* clang-tidy and minimize lock contention

* add config defaults

* add oob_reporting_period config field and add basic test

* Automated change: Fix sanity tests

* fix test

* change test to use basic RR

* WIP: started exposing peer address to LB policy API

* first WRR test passing!

* small cleanup

* port RR fix to WRR

* test helper refactoring

* more test helper refactoring

* WIP: trying to fix test to have the right weights

* more WIP -- need to make pickers DualRefCounted

* fix timer ref handling and get tests working

* clang-format

* iwyu and generate_projects

* fix build

* add test for OOB reporting

* keep only READY subchannels in the picker

* add file missed in a previous commit

* fix sanity

* iwyu

* add weight expiration period

* add tests for weight update period and OOB reporting period

* Automated change: Fix sanity tests

* lower bound for timer interval

* consistently apply grpc_test_slowdown_factor()

* cache time in test

* add blackout_period tests

* avoid some unnecessary copies

* clang-format

* add field to config test

* simplify orca watcher tracking

* attempt to fix build

* iwyu

* generate_projects

* add "_experimental" suffix to policy name

* WRR: update tests to cover qps plumbing

* WIP

* more WIP

* basic WRR e2e test working

* add OOB test

* fix sanity

* ignore duplicate addresses

* Automated change: Fix sanity tests

* add new tracer to doc/environment_variables.md

* retain scheduler state across pickers

* Automated change: Fix sanity tests

* use separate mutexes for scheduler and timer

* sort addresses to avoid index churn

* remove fetch_sub for wrap around in RR case

Co-authored-by: markdroth <markdroth@users.noreply.github.com>
2023-01-20 12:14:37 -08:00
Mark D. Roth 7a96bed837
xDS: fix WeightedClusters total weight handling (#32134)
* xDS: fix WeightedClusters total weight handling

* iwyu
2023-01-19 08:01:26 -08:00
Yash Tibrewal 21618f3aef
GCP Observability: Add labels support for logging (#32130)
* GCP Observability: Add labels support for logging

* clang format

* Reviewer comments
2023-01-18 16:08:30 -08:00
Yash Tibrewal 05491fb2f6
Gcp Observabiliy : Add constant labels support for stats and tracing (#32128)
* Gcp Observabiliy : Add constant labels support for stats and tracing

* Register GCP Observability labels

* sanity

* Fix build for CI

* TEST_TAG_KEY fix

* Remove duplicitous constant label setting
2023-01-18 14:32:36 -08:00
Yash Tibrewal 25465100fd
LoggingTest: s/GUARDED_BY/ABSL_GUARDED_BY (#32144) 2023-01-18 14:24:11 -08:00
Yijie Ma f2bcdf3bbc
Add a test case for infinite past or unix epoch expiry in alarm_test.cc (#32101)
* initial commit

* review
2023-01-18 11:09:17 -08:00
Yash Tibrewal 33be15bdb6
Gcp Observability: Lazily initialize channels post-init (#32091)
* Gcp Observability: Lazily initialize channels post-init

* IWYU and fix build deps

* Run RegistryPostInit for client census filters too

* Remove unused function
2023-01-13 11:13:20 -08:00
Craig Tiller 67f364e23e
[cleanup] Eliminate usage of GRPC_ASSERT(false...); (#31757)
* crash function

* progress

* fix

* fix

* Automated change: Fix sanity tests

* fix

* fix

* fix

* fixes

* Automated change: Fix sanity tests

* fix

* Automated change: Fix sanity tests

* fix

* fix

* use cpp attr

* Automated change: Fix sanity tests

* fix

* fix

* fix

* fix

* fix

* fix

* add exclusion

* fix

* typo

* fix

* fmt

* Update tcp_socket_utils.cc

* Automated change: Fix sanity tests

* fix

* revert php changes

* Automated change: Fix sanity tests

Co-authored-by: ctiller <ctiller@users.noreply.github.com>
2023-01-11 08:50:32 -08:00
Mark D. Roth daded8a59f
WRR: update tests to cover qps plumbing (#32068) 2023-01-11 08:14:29 -08:00
Mark D. Roth e5a37d59fa
xDS: don't NACK RouteConfig with a VirtualHost containing no valid routes (#32069) 2023-01-11 08:03:55 -08:00
Yash Tibrewal d5c1dc1667
GCP Observability: Fix env var names (#32066) 2023-01-10 14:02:07 -08:00
Yash Tibrewal 5fc92eaeae
Disable logging test (#32049) 2023-01-10 10:50:56 -08:00
cut4th aa2956b05b
Change ProtoReflectionDescriptorDatabase to take grpc::ChannelInterface. (#31858)
Updates the ProtoReflectionDescriptorDatabase ctor to take a reference
to std::shared_ptr<grpc::ChannelInterface> rather than
std::shared_ptr<grpc::Channel>. This helps code that is making use of
grpc::ChannelInterface from having to perform a cast from the Base to
the Derived when creating the refelction db.
2023-01-06 18:05:38 -08:00
Yash Tibrewal ad6f6c49e0
Logging filter (#31755)
* initial

* Intermediate

* Another try

* Try multiple necesary pulls

* Filter works other than client half close

* Fixes

* Add a cancelled RPC test

* Handle trailer only responses

* Tests for disabled logging and truncated payloads

* Fix authority and peer

* Add TODOs and asserts for half-close

* Fix tests for half-close and cancel

* 2d748fcb1cf45cac62729b8346ad15e6abc79e97

* Fix sanity checks

* Strict bazel build

* Fix package

* IWYU

* Fix cmake

* Explicit cast to string

* Size casts

* Fix Arena leak and disable macos build for now

* Reviewer comments
2023-01-06 13:52:15 -08:00
Hannah Shi d8a52a5043
build multi arc ios unit test for m1 (#31930) 2023-01-05 16:27:21 -08:00
Vignesh Babu 4806da6041
Fix flakiness in xds_cluster_end2end_test: XdsTest/CdsTest.CircuitBreaking/V3 (#32027)
* Fix flakiness in xds_cluster_end2end_test: XdsTest/CdsTest.CircuitBreaking/V3

* update comment and add sleep in one more instance
2023-01-05 15:46:54 -08:00
apolcyn c952d37814
[testing] add test client for use in XDS federation tests (#32020)
[testing] add test client for use in XDS federation tests
2023-01-05 14:14:47 -08:00
Yash Tibrewal d3d4dd7325
StatsPluginEnd2EndTest: Add QueueOnceLoadBalancingPolicy to remove flakiness from test (#32019)
* StatsPluginEnd2EndTest: Add QueueOnceLoadBalancingPolicy to remove flakiness from test

* Unused arg

* Remove unnecessary DelegatingPicker

* Reviewer comments

* explicit constructor

* Remove unnecessary dependency

* Reviewer comments

* Sanity

* clang tidy
2023-01-05 12:45:52 -08:00
Cheng-Yu Chung 384982ba2b
[bugfixes] Fix vulnerability for PyYAML (#32007)
* Fix vulnerability for PyYAML

* Fix

* Revert "Fix"

This reverts commit 5522d226c4.

* Fix

* Fix format
2023-01-04 12:33:29 +08:00
Yijie Ma f99b8b5bc4
Convert c-style comments to C++-style comments (#31923)
* baseline

* fix clang-tidy

* manually revert these files

* manually fixup at eof

* revert 2 more files

* change check_deprecated_grpc++.py

* change end2end_defs.include template

* fix check_include_guards.py

* untrack tools/distrib/python/convert_cstyle_comments_to_cpp.py

not yet ready to be submitted

* fix

yapf check_include_guards.py
remove a space...

* fix version.cc.template

* fix version_info.h.template
2022-12-22 23:01:53 -08:00
Esun Kim 27bf7238f4
Added include guard to mock headers (#31909)
* Added include guard to mock headers

* Fix test
2022-12-16 14:42:41 -08:00