This PR is copied from #34276, since I did not have permissions to add commits to it. That PR has been verified to work (see the top-level description). This PR just makes the gRPC tests pass (e.g. adding includes, clang formatting).
Closes#34874
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/34874 from matthewstevenson88:pull_34276 d5fb73e5b2
PiperOrigin-RevId: 609107146
- Extract build metadata for some external dependencies from bazel
build. This is achieved by letting extract_metadata_from_bazel_xml.py
analyze some external libraries and sources. The logic is basically the
same as for internal libraries, I only needed to teach
extract_metadata_from_bazel_xml.py which external libraries it is
allowed to analyze.
* currently, the list of source files is automatically determined for
`z`, `upb`, `re2` and `gtest` dependencies (at least for the case where
we're building in "embedded" mode - e.g. mostly native extensions for
python, php, ruby etc. - cmake has the ability to replace some of these
dependencies by actual cmake dependency.)
- Eliminate the need for manually written gen_build_yaml.py for some
dependencies.
- Make the info on target dependencies in build_autogenerated.yaml more
accurate and complete. Until now, there were some depdendencies that
were allowed to show up in build_autogenerated.yaml and some that were
being skipped. This made generating the CMakeLists.txt and Makefile
quite confusing (since some dependencies are being explicitly mentioned
and some had to be assumed by the build system).
- Overhaul the Makefile
* the Makefile is currently only used internally (e.g. for ruby and PHP
builds)
* until now, the makefile wasn't really using the info about which
targets depend on what libraries, but it was effectively hardcoding the
depedendency data (by magically "knowing" what is the list of all the
stuff that e.g. "grpc" depends on).
* After the overhaul, the Makefile.template now actually looks at the
library dependencies and uses them when generating the makefile. This
gives a more correct and easier to maintain makefile.
* since csharp is no longer on the master branch, remove all mentions of
"csharp" targets in the Makefile.
Other notable changes:
- make extract_metadata_from_bazel_xml.py capable of resolving workspace
bind() rules (so that it knows the real name of the target that is
referred to as e.g. `//external:xyz`)
TODO:
- [DONE] ~~pkgconfig C++ distribtest~~
- [DONE} ~~update third_party/README to reflect changes in how some deps
get updated now.~~
Planned followups:
- cleanup naming of some targets in build metadata and buildsystem
templates: libssl vs boringssl, ares vs cares etc.
- further cleanup of Makefile
- further cleanup of CMakeLists.txt
- remote the need from manually hardcoding extra metadata for targets in
build_autogenerated.yaml. Either add logic that determines the
properties of targets automatically, or use metadata from bazel BUILD.
* [cleanup] Remove profiling timers
- nobody has used this system in years
- if we needed it, we'd probably rewrite it at this point to be something more modern
- let's remove it until that need arises
* fix
* fixes
* replace darwin checks in extconf.rb to exclude TruffleRuby
* inherit RANLIB and STRIP from RbConfig, set LDXX
* enable overriding ranlib command in top-level makefile
* ensure the -no_warning_for_no_symbols flag is only used with Apple's ranlib
* don't embed openssl & zlib on truffleruby
* add RbConfig's cppflag to CPPFLAGS when using TruffleRuby
* this ensure the paths to find the system's OpenSSL are set up correctly with TruffleRuby (includes being able to find an OpenSSL installed via Homebrew etc)
* don't statically link standard libraries on Linux with Truffleruby
* This does not work when compiling to bitcode.
* Prefer SIGTERM to SIGQUIT for graceful shutdown in examples
* Overriding SIGQUIT is suboptimal, for example on JVM where it is very
useful to dump the thread stacktraces.
* Keep the rb_tr_abi_version symbol for TruffleRuby in grpc_c.so
* Otherwise TruffleRuby cannot verify the ABI version is correct.
* See https://github.com/oracle/truffleruby/issues/2386
* Use RbConfig::CONFIG['STRIP'] instead of just `strip`
* Use a local variable for apple_toolchain for consistency
* Add a comment about -static-libgcc -static-libstdc++ and TruffleRuby
* Split comment into two for openssl/zlib
Co-authored-by: Nicolas Laurent <nicolas.laurent@oracle.com>
* Drop ABSL_RANDOM_HWAES_FLAGS compiler flags
Older CPUs that do not have SSE4.1 would crash with the Ruby native gem
due to an illegal instruction exception.
The Abseil random library isn't being used at the moment
(https://github.com/grpc/grpc/pull/26476), and there's no reason gRPC
needs to force SSE4.1 instructions on all platforms at the moment. There
are other hardware-specific issues that need to be ironed out for this
to work: https://github.com/grpc/grpc/pull/26479
When the `-msse4` compiler flag was enabled, the Abseil code started
using the `pinsrb` instruction:
```
$ elfx86exts abseil-cpp/absl/time/internal/cctz/src/time_zone_libc.o
MODE64 (ret)
CMOV (cmovne)
SSE2 (movdqa)
SSE41 (pinsrb)
SSE1 (movaps)
CPU Generation: Penryn
```
Closes https://github.com/grpc/grpc/issues/27095
* Revert "Drop ABSL_RANDOM_HWAES_FLAGS compiler flags"
This reverts commit 3b7cc74d39.
* Drop -msse4 compiler flag
Older CPUs that do not have SSE4.1 would crash with the Ruby native gem
due to an illegal instruction exception.
The Abseil random library isn't being used at the moment
(https://github.com/grpc/grpc/pull/26476), and there's no reason gRPC
needs to force SSE4.1 instructions on all platforms at the moment. There
are other hardware-specific issues that need to be ironed out for this
to work: https://github.com/grpc/grpc/pull/26479
When the `-msse4` compiler flag was enabled, the Abseil code started
using the `pinsrb` instruction:
```
$ elfx86exts abseil-cpp/absl/time/internal/cctz/src/time_zone_libc.o
MODE64 (ret)
CMOV (cmovne)
SSE2 (movdqa)
SSE41 (pinsrb)
SSE1 (movaps)
CPU Generation: Penryn
```
This was previously needed because gcc 4.8 wouldn't compile without the
`-msse4` and `-maes` flags.
However, per
97db2bf7fb
gcc 5.0+ automatically detects whether these options are enabled.
clang still needs `-maes` since including `wmmintrin.h` expects the AES
option to be enabled.
Closes https://github.com/grpc/grpc/issues/27095
* Ran `2to3` on `tools/buildgen` and the `gen_build_yaml.py` files.
* Updated the invocations of python to be explicitly Python 3.
* Changed the loader in `mako_renderer.py` to allow pickle to find the module for pickling `external_version.Version`
* Added a few open-as-binary to placate readers/writers expecting bytes.
* Hand tweaked the templates to remove `iteritems` and fiddle with encodings.
Partially addresses #24359.
Matthew Stevenson