* ci: add AWS EFA wheel build/release to official CI/CD
Adds EFA (libfabric) wheel variants to the wheel pipeline, mirroring the
existing non-cuda / cuda13 / npu variant pattern. The EFA transport memory
path is CUDA-aware (FI_HMEM_CUDA / GPUDirect under USE_CUDA=ON, FI_HMEM=system
otherwise), so two PyPI packages are produced:
mooncake-transfer-engine-efa USE_EFA=ON USE_CUDA=ON (GPU)
mooncake-transfer-engine-efa-non-cuda USE_EFA=ON USE_CUDA=OFF (CPU/DRAM)
No EFA hardware is needed to build: USE_EFA only needs libfabric headers/lib
to compile and link, which the stock ubuntu runner gets from the distro
libfabric-dev package. auditwheel already excludes libfabric/libefa from the
wheel (build_wheel.sh) so they resolve to the user system AWS EFA install at
runtime, avoiding the dual-libfabric / device-claim conflict. The distro
libfabric (1.x) is ABI-forward-compatible with the AWS EFA libfabric (2.x)
loaded at runtime; the EFA transport uses only long-stable fi_* APIs.
Changes:
- scripts/build_wheel.sh: add EFA_BUILD / EFA_NON_CUDA_BUILD variants that
rename the package to mooncake-transfer-engine-efa{,-non-cuda}.
- .github/workflows/ci_efa.yml: PR-time build validation (one python per
variant), wired into ci.yml CI Gate. Verifies libfabric is NOT bundled.
- .github/workflows/release-efa.yaml: tag-triggered full python matrix
(3.10-3.13) x {cuda, non-cuda}, publishes to GitHub Release + PyPI.
Verified end-to-end on a p5.48xlarge: both variants build, produce the
correct package names, and exclude libfabric from the wheel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(efa): document PyPI install and EFA wheel CI/CD
Now that EFA wheels are published by the release pipeline, document the
recommended path (pip install mooncake-transfer-engine-efa / -efa-non-cuda)
before the build-from-source instructions, and note the EFA_BUILD /
EFA_NON_CUDA_BUILD variant env vars plus the ci_efa.yml / release-efa.yaml
workflows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci(efa): split EFA release into per-package workflows; bump CUDA to 12.8.1
Address review feedback on PR #2565:
- Split release-efa.yaml (which built both cuda + non-cuda via a matrix.variant)
into two standalone workflows, one per PyPI package, mirroring the existing
release.yaml / release-non-cuda.yaml pattern:
* release-efa.yaml -> mooncake-transfer-engine-efa (USE_CUDA=ON)
* release-efa-non-cuda.yaml -> mooncake-transfer-engine-efa-non-cuda (USE_CUDA=OFF)
Each has its own build matrix, libfabric-exclusion check, artifact pattern, and
publish-release/PyPI step, so a package publishes from a dedicated workflow
(per-package trusted publisher / artifact pattern).
- Bump CUDA Toolkit 12.4.1 -> 12.8.1 in ci_efa.yml and release-efa.yaml to match
the version used across the rest of the repo workflows (ShangmingCai review).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: whn09 <whn09@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
af26a48c03