* fix: route EFA same-process loopback transfers through local copy
The EFA provider's SHM intra-node path performs a host memcpy into
FI_HMEM_CUDA device buffers and segfaults on the first same-host
transfer (loopback self-transfer), e.g. checkpoint-engine P2P weight
update on a single TP=8 node. See ofiwg/libfabric#12328.
Detect same-process self-loopback in EfaContext::submitPostSend (peer
NIC path equals our own nicPath(), whose server_name embeds the
per-process RPC port, so the match guarantees the peer is this very
process on this device) and satisfy the copy locally with a GPU-aware
cudaMemcpy (cudaMemcpyDefault), bypassing EFA entirely. Same-host
cross-process peers carry a different port, never match, and still go
through EFA.
The copy direction honors the slice opcode, mirroring fi_read/fi_write:
WRITE copies source_addr -> dest_addr, READ copies dest_addr ->
source_addr (the two are distinct local buffers, so it is not a
symmetric self-copy).
This mirrors how the RDMA transport already treats loopback as a
special case (rdma_endpoint.cpp self-connected QP); RDMA relies on NIC
hardware loopback and is unaffected by the libfabric SHM bug.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: add EFA GPU loopback test for FI_HMEM_CUDA same-host transfers
Add efa_gpu_loopback_test.cpp, the CUDA-device-memory counterpart of
efa_transport_test.cpp (which only covers host/numa loopback). It
reproduces the EFA SHM intra-node segfault on FI_HMEM_CUDA buffers
(ofiwg/libfabric#12328) and validates EfaContext::tryLoopbackCopy:
* GpuLoopbackWrite — same-host GPU WRITE must not crash.
* GpuLoopbackWriteThenRead — WRITE then READ with byte-accurate
verification, exercising both copy directions.
* GpuLoopbackMultiWrite — batched GPU writes through the per-slice
loopback short-circuit.
Self-skips when no EFA device or no CUDA GPU is present. Registered
under `USE_EFA AND USE_CUDA`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* efa: log fabric name and document same-host GPU loopback segfault
Add fabric_attr->name to the EFA device init log so the active fabric
(efa-rdm vs efa-direct) is visible at runtime — both share the same
provider name, FI_EP_RDM type, and <device>-rdm domain name, so the
fabric name is the only field that distinguishes them.
Revise the "Single-host loopback" doc section:
- Correct the same-host fast-path attribution: the memcpy fast path is
supplied by the SHM provider (FI_EFA_ENABLE_SHM_TRANSFER, default on),
not by FI_EFA_USE_DEVICE_RDMA. Confirmed at runtime: a default
(device-RDMA-enabled) config still reports "Opened fabric: shm".
- Add a warning that the default SHM path host-memcpy's into FI_HMEM_CUDA
destinations and segfaults on GPU buffers (ofiwg/libfabric#12328);
document the same-process tryLoopbackCopy short-circuit and the
FI_EFA_ENABLE_SHM_TRANSFER=0 workaround for cross-process GPU peers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
268622fa70