[CCF Archive] Store object type eviction policy submission #3
Loading…
Reference in New Issue
No description provided.
Delete Branch "kancel:ccf-archive-pr2746"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
本 PR 只作为比赛作品提交要求的 GitLink 归档。
由于 GitLink 的主分支落后于 GitHub 侧，本 PR 不作为上游代码评审的主要入口。具体提交与技术讨论请参考：
[Store] Add hidden state object data type[Store] adjusted lease timeout policy[Store] Add object type accounting to BatchEvict[Store] object type eviction policy本 PR 的目标是满足比赛要求中的 GitLink 托管/归档要求。
* feat(efa): idempotent warmupSegment for repeated calls Adds EfaContext::peekEndpoint() — a non-creating, normalized-key lookup — and uses it in EfaTransport::warmupSegment() to short-circuit when every (local_ctx, peer_nic) pair is already connected. Repeated warmup calls on the same segment now return immediately instead of firing a 256-thread std::async fan-out each time. Verified on p6-B300 (16 NICs × 16 peer NICs = 256 endpoints): iter 0: 7.95s (full handshake) iter 1: 0.000s (short-circuit hit) iter 2: 0.000s (short-circuit hit) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(efa): evict-on-ENOMEM + drop-on-failure for endpoint store Recover from QP exhaustion (fi_enable returns -FI_ENOMEM at the device cap, 768/device on p6-B300) and prevent dead endpoints from piling up under callers that drift endpoint keys (e.g. keys carrying a timestamp). Three places: * EfaContext::endpoint (slow path): on construct failure, evict stale endpoints once and retry. Eviction only scans when we actually ran out of room, so normal traffic pays nothing. * EfaTransport::warmupSegment: if setupConnectionsByActive fails, drop the endpoint via deleteEndpoint(normalizedKey) instead of leaving a dead fid_ep holding its QP slot. * EfaContext::submitPostSend: if submitPostSend returns non-zero and the endpoint is no longer connected, drop it for the same reason. Reproduced on B300: without the fix, injecting ":drift<iter>" into the peer_nic_path exhausts fi_enable at iter 48 (768/16 peer NICs). With the fix, 325 iterations complete with RSS bounded at ~6 GB. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(efa): SRD shared-endpoint refactor — drop per-peer fid_ep model Under SRD (FI_EP_RDM), a single fid_ep per local NIC can address any number of peers via fi_av_insert. The old model created 16 per-peer endpoints per connection (one per local NIC), consuming 16 QPs each — hitting the 768 QP cap after only 48 peers and forcing an LRU + evict-on-ENOMEM dance. This change: - EfaContext owns a single shared fid_ep + peer address vector (`peer_map_`) - EfaEndPoint becomes a thin per-peer fi_addr_t holder (no fi_endpoint, no fi_enable) — handshake is RPC + fi_av_insert only - submitPostSend delegates to EfaContext::submitSlicesOnPeer, using the peer's AV index against the shared endpoint - setPeerNicPath detaches stale AV slot on peer reconnect so drift scenarios re-handshake cleanly - Deletes EfaEndpointStore + LRU/ENOMEM eviction machinery — no longer needed - warmupSegment short-circuit updated for the shared-endpoint flow P5EN drift stress (1 target ↔ 1 initiator, customer shape 328 GB, 16 NIC, 180s target lifetime): - iter 0 (fresh peer) first_batch: 33.6 ms (was ~8.95 s → 266× faster) - iter 1–3 (drift) first_batch: 260–340 ms (was ~8.95 s → ~26× faster) - steady: ~215–220 GB/s, 0 failures across 204 M completed ops - QP growth per added peer: 0 (was 16) — no more 48-peer cliff See mooncake-transfer-engine/example/efa_srd_refactor_validation/ for raw CSV and target-teardown timings. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(efa): expose warmup_efa_segment in Python binding The C API (warmupEfaSegment) and Rust FFI (warmup_efa_segment) already pre-connect every (local_ctx, peer_nic) pair so the first submitTransfer does not stall on handshake RPC + fi_av_insert. The pybind TransferEngine did not expose this, so Python-driven integrations (vLLM / SGLang) could not benefit from the first-call latency improvement from the SRD shared- endpoint refactor (iter 0 first_batch 33.6 ms vs 8.95 s baseline). Mirror the C API shape: thin pass-through that dynamic_casts the installed "efa" transport and calls EfaTransport::warmupSegment. No-op on non-EFA builds or when the EFA transport is not installed. Tested: built with USE_EFA=ON on p5en, imported engine.TransferEngine, and confirmed `warmup_efa_segment(segment_name: str) -> int` is bound. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(efa): apply code_format.sh (clang-format-20) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(efa): drop internal drift_stress/validation dirs + stale gflags doc note - Remove mooncake-transfer-engine/example/drift_stress/ — depends on the out-of-tree customer_pattern Rust binary and isn't usable by upstream contributors. - Remove mooncake-transfer-engine/example/efa_srd_refactor_validation/ — validation artifacts for the SRD refactor; the headline numbers live in the PR description instead. - Remove the libgflags-dev note from docs/.../efa_transport.md — it is pulled in transitively by dependencies.sh and no longer needs a manual install step. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(efa): close shared endpoint before dropping peer map in teardown Calling fi_av_remove() on peer addresses after the shared fid_ep was still open — or calling it at all during context teardown when the AV itself is about to be fi_av_close()d — trips a libfabric EFA-provider assertion, causing the efa_transport_test unit tests to segfault on engine destruction. Reorder EfaContext::deconstruct() to close shared_ep_ first, then detach every peer handle via a new markDetachedForTeardown() that just resets the AV-slot field without calling fi_av_remove(). fi_av_close() below invalidates every slot in one shot, so no per-slot removal is required on the teardown path. Verified: all 5 EFATransportTest cases (InstallTransport, LoopbackWrite, WriteAndRead, MultiWrite, StressMultipleBatches) now pass on P5EN. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(efa): add coverage for warmupSegment, batch register, large xfer, re-open Five additional EFATransportTest cases, filling gaps in the existing suite (which only exercised install + 4 single-buffer write/read paths): - WarmupSegmentLoopback — covers EfaTransport::warmupSegment() and its idempotent short-circuit on the second call, plus the empty / self-name no-op path. - WarmupSegmentNotFound — warmupSegment must fail cleanly (not hang) for an unknown segment name. - RegisterMemoryBatch — registerLocalMemoryBatch + unregisterLocalMemoryBatch round-trip. - LargeTransfer — 128 MB buffer, 64 x 1 MB slices; exercises WR / CQ pacing in EfaContext::submitSlicesOnPeer well past the 16 x 64 KB MultiWrite ceiling. - RepeatedOpenSegment — openSegment() on the same peer repeatedly must keep transferring correctly; guards the setPeerNicPath detach path that target-restart drift depends on. docs/.../efa_transport.md's Unit Tests table updated to match. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): restrict CTest invocation to the efa tests Running the full ctest suite on an EFA-only host trips etcd / mooncake_master requirements in the TCP / metadata / store tests, which can read as "EFA broke the build". The EFA doc should not recommend that invocation — filter to the two EFA binaries and call out the reason in a note. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): drop redundant MC_METADATA_SERVER / MC_LOCAL_SERVER_NAME block The defaults (P2PHANDSHAKE and 127.0.0.1:12345) are what the loopback unit tests actually want, and nothing in the current test suite needs an override, so documenting these "env var exports" next to the test invocation just makes readers wonder whether they are mandatory. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): drop obsolete CUDA_VISIBLE_DEVICES tip for CPU-to-CPU runs The tip existed because libfabric's EFA provider used to dlopen libcudart at fi_getinfo time, creating a CUDA context even for CPU-only benchmarks. EfaContext::construct now sets FI_HMEM=system when the build is not GPU-enabled, which already prevents that context from being created — the CUDA_VISIBLE_DEVICES="" workaround is no longer needed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): lead with a bench command that hits line rate The previous "Initiator Node" snippet used block_size=65536 / threads=8 / buf=1GB, which reaches only ~50 GB/s on an 8x400G host and reads as "EFA is broken" to anyone copy-pasting it. Replace with block=1MB / threads=32 / buf=4GB — the same values that produce the GB/s numbers quoted in the Benchmark Results tables — and add a short sentence telling readers exactly why block_size matters. Also clarify that the MC_SLICE_SIZE no-effect note covers both CPU-to-CPU and GPU-to-GPU paths (verified: no EFA code reads globalConfig().slice_size, only rdma_transport and kunpeng_transport do). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): drop MC_SLICE_SIZE references, sync target --buffer_size EFA transport never reads globalConfig().slice_size — the only readers are rdma_transport and kunpeng_transport — so the env-var table row, the "MC_SLICE_SIZE has no effect" tuning-tip bullet, and the pre-optimization tuning-history details block were all documentation debt rather than user guidance. Drop all three. Also: the target command defaulted to --buffer_size=1GB (the flag's default), but the recommended initiator command now uses 4GB. Since the target allocates and registers exactly FLAGS_buffer_size bytes and the initiator writes into offsets [0, buffer_size), they have to match. Add --buffer_size=4294967296 to the target snippet and a short line calling out the constraint. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(efa): skip preTouchMemory for GPU buffers, clarify target flags preTouchMemory() does a CPU-side store per page, which was gated only by "chunk >= 4 GiB". That trips a segfault the moment a user registers a >=4 GiB cudaMalloc buffer — exactly what the recommended benchmark config does. Restrict the pre-touch to host (cpu:*) memory so VRAM registration stays a pure libfabric call. Also amend the benchmark docs: - call out that --use_vram / --gpu_id apply to the target too, and that GPU-to-GPU runs need --gpu_id=-1 on both sides - mention --use_vram=false as the alternative to a -DUSE_CUDA=OFF build for CPU-to-CPU runs Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): default benchmark snippet to gpu_id=-1 + threads=16 Align the benchmark quick-start with what actually reaches line rate on a p5en.48xlarge pair (≈350 GB/s write, verified). The previous snippet recommended --threads=32 and no --gpu_id override, which: - ran on one GPU (default gpu_id=0), so buffers sat on one NUMA node and half the NICs were cross-NUMA — settled at ≈56 GB/s - ran into the SRD shared-endpoint WR cap (16 × 256 = 4096 WRs vs 32 × 128 = 4096 outstanding slices, zero headroom) and triggered "timed out waiting for CQ drain" Rework the section: lead with the two knobs that matter (--gpu_id=-1 on both sides, --block_size=1048576), add --gpu_id=-1 to the target snippet for symmetry, switch the initiator to --threads=16, and explain in a note why 16 is the ceiling (so readers know where the limit comes from, not just that we picked it). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): complete the Key Parameters table The table was missing several flags that the bench snippet in the same section actually uses (--mode, --protocol, --metadata_server, --segment_id, --use_vram), and some of the flags it did list had defaults / descriptions that did not match the binary (e.g. default operation is read, not write). Sync every row to DEFINE_* in transfer_engine_bench.cpp and add explicit "what this is for on EFA" hints next to the two flags (--block_size, --gpu_id) that most determine whether the run hits line rate. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): reorder benchmark sections by peak throughput Lead with p6-b300 (752 GB/s peak), then p5en (347 GB/s), then p6-b200 (313 GB/s). Readers skim the first table to decide whether EFA transport is fast enough for them; putting the best-performing platform first avoids undersell. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): reorder benchmarks by hardware generation, not raw GB/s Previous reorder put p5en (H200) above p6-b200 (B200) because the p5en rows happen to show a higher headline number, but B200 is a newer generation than H200 and the p6-b200 numbers there are an older snapshot. Fix the ordering to reflect hardware recency: p6-b300 → p6-b200 → p5en. Same change applied to the Cross-Transport Comparison table. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): refresh p5en bench results (366 GB/s write / 304 GB/s read) Re-swept on P5EN (H200, 16×200G) with the SRD shared-endpoint build and --buffer_size=4GB per GPU. New peaks: - Write: 365.66 GB/s @ threads=16, batch=128 (up from 347 GB/s) - Read: 303.78 GB/s @ threads=16, batch=32 (up from 308 GB/s, but now with a clear optimal config) The new table preserves enough rows to show the two interesting axes — write wants big batches, read wants small ones — and the Cross-Transport Comparison is updated to 366 GB/s (~91% of the 400 GB/s line rate). B200/B300 rows are unchanged (no hardware available to re-sweep). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): mark B200/B300 bench rows stale, refresh p5en CPU, drop cross-transport table - p5en CPU-to-CPU re-swept on current SRD shared-endpoint build: peak write 213.57 GB/s @ threads=48/batch=32, peak read 212.18 GB/s @ threads=16/batch=32 (vs old 192 GB/s / 182 GB/s). Table shows a few representative points — throughput is essentially flat across the parameter grid because the run is DRAM-bound. - B200 and B300 sections tagged as predating the SRD refactor and the recent tuning work; a re-sweep is pending hardware availability. - Drop the "Cross-Transport Comparison" subsection and the "EFA vs RoCE RDMA" paragraph — both were derived numbers that quickly rot and were already redundant with the per-platform tables above them. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): add block_size + buffer_size sweep tables as evidence Two sweeps on p5en GPU-to-GPU (SRD shared-endpoint build, --gpu_id=-1) at the peak config (threads=16, batch=128), backing up the tuning claims elsewhere in this file: - block_size: 64 KB default reaches only 26% of peak. Write throughput climbs steeply through 512 KB, plateaus between 1 MB and 2 MB. 2 MB is MC_EFA_STRIPING_THRESHOLD (a different code path). 1 MB is the recommended value — within 4% of the 2 MB peak and safely below the striping boundary. - buffer_size: only needs to clear block × batch × threads (2 GB in the peak config). 2 GB vs 4 GB differs by ~3% on write, read is flat within noise. The example commands use 4 GB purely as a generous default, not because smaller fails. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): rewrite Tuning Tips around the SRD shared-endpoint model Two stale claims under Tuning Tips were written for the old per-peer endpoint code: - "Increase threads to 32-48": under the shared endpoint each NIC has a 256 WR cap, and threads × batch past 16×256 = 4096 trips the CQ-drain timeout. The sweep on p5en shows 16 threads is the ceiling, not the floor. - "Write peaks at threads=32, read at threads=16 / buffer=2GB max per GPU": both taken from the pre-refactor B300 snapshot. Current p5en peaks are both at threads=16; the 2GB/GPU limit was the VRAM preTouch segfault that is now fixed. Rewrite the bullets to describe what the SRD shared-endpoint actually constrains: block_size sweet spot, threads × batch WR cap, the write-vs-read batch-size split, and --gpu_id=-1 on both sides. Add an explicit line that buffer_size only needs to clear block × batch × threads (pointing readers at the sweep table). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(efa): remove broken per-request striping across NICs submitTransferTask had a branch that, when request.length exceeded MC_EFA_STRIPING_THRESHOLD (default 2 MB), split the request into num_nics slices and dispatched one per NIC. The intent was to speed up large single transfers by paralleling across all NICs. A sweep on p5en (SRD shared-endpoint build, 16 NICs x 200G) shows the branch is a ~20x negative optimization in every bench config: block=1 MB, threads=16, batch=128: ON 16 GB/s vs OFF 366 GB/s block=8 MB, threads=16, batch=16: ON 18 GB/s vs OFF 355 GB/s block=32 MB, threads=16, batch=4: ON 18 GB/s vs OFF 328 GB/s Even the designed-for scenario (threads=1, batch=1, single huge request) only wins by 1.2x (27 vs 23 GB/s) — far below the 16x expected from parallel dispatch, because the per-slice post_lock / peer lookup still serialize the dispatch. And that "win" is only reachable with threads=1 — any realistic caller with multiple threads can parallelize on its own without this mechanism. Since the branch silently degrades every common bench / workload configuration above the 2 MB threshold and delivers no real win even in its best case, delete it: - efa_transport.cpp: drop the LARGE TRANSFER if-branch and the kStripingThreshold read. - config.h / config.cpp: drop the efa_striping_threshold field and the MC_EFA_STRIPING_THRESHOLD env var. - docs/.../efa_transport.md: drop the Environment Variable row, the "Note on EFA slicing" striping explanation, and the "striping off" callouts in the benchmark tables and tuning tips. Verified: all 10 EFATransportTest cases pass, and the peak p5en bench run is unchanged (352.81 GB/s write @ threads=16, batch=128). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): number the sequential subsections for easier skimming Four groups of level-3 / level-4 headings in this doc read as step-by-step instructions (do A first, then B), but only two of them were numbered while the others relied on order alone. Result: readers had to count positions to tell "how far through" they were. Number every such group consistently: - Performance Benchmark: Target Node (1) / Initiator Node (2) - Usage with vLLM: Prefill Instance (1) / Decode Instance (2) - Benchmark Results: p6-b300 (1) / p6-b200 (2) / p5en (3) Reference-style sections (Key Parameters, Tuning Tips, Warmup, Technical Details, Troubleshooting) stay unnumbered — those aren't ordered and numbering them would imply a reading path that doesn't exist. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(efa): add efa_first_submit_probe for warmup / first-submit timing A standalone cross-node probe that measures, separately: - warmupSegment() latency, and - per-submit latency for the first N single-transfer submits so we can isolate the handshake / av_insert cost (warmup=OFF makes it land on submit #0) and compare pre-refactor vs post-refactor code without having to squint at 10s throughput averages. Not enabled by default ctest — only built when USE_EFA=ON. Meant for manual two-host runs alongside transfer_engine_bench. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): replace warmup numbers with a cross-branch p5en measurement The old "Eager endpoint warmup" numbers (4 s stall / 13.5 ms warmed) were from a B300 snapshot that predated the SRD shared-endpoint refactor, so they described the OLD code. Replace with a fresh cross-node measurement on p5en that compares both branches head to head using efa_first_submit_probe: - Warmup itself: 1.1 s (SRD shared endpoint) vs 17 s (per-peer fid_ep on upstream main) — ~15x faster and much more stable (SRD runs were all 1.13-1.14 s; old code swung 9-17 s across three reps). - Cold first submit with NO warmup: 26 ms (SRD) vs 99 ms (old) — ~4x faster, because the shared endpoint removes the per-peer fi_endpoint / fi_enable step that used to dominate the first send. The prior text's "first-batch stall" framing also reads as if the whole first submitTransfer always pays the full handshake cost. That's not quite how it works — the initiator round-robins across local NICs, so only a handful of pairs warm on each call, and the stall amortizes across the first several submits. Fix the wording. Add a pointer to the probe source (example/efa_first_submit_probe.cpp) so the measurement is reproducible. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): reframe "first-request latency" around two audiences The previous version led with a long explanation of warmupSegment and its internals, which made it sound like callers had to opt in to get any first-request speedup from this PR. That isn't the case: the SRD shared-endpoint refactor cuts the cold submit from 99 ms to 26 ms automatically, and vLLM / SGLang (which don't call warmupSegment) see that 4x win without any integration work on their side. The explicit warmupSegment API is an additional 15x improvement on top of that, but it matters mainly to direct Mooncake callers. Restructure the section accordingly: - Lead with a 2x2 table showing both numbers side by side. - Frame the ~4x cold-submit win as the default, no-code-change benefit everyone gets. - Frame warmupSegment as the opt-in path for callers that want sub-10 ms first-request latency. - Explicitly note that vLLM / SGLang currently don't call it. - Keep the peer handshake-daemon bottleneck note so readers understand why the warmup scales linearly. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): reference #1944 explicitly instead of "this PR" "This PR" dangles once the doc lands on main — it's fine in review context, less fine afterwards. Use the concrete issue number so readers (including future us) can follow the link. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): unwrap soft-wrapped paragraphs, let the renderer flow them Several blockquote notes, tuning-tip bullets, and the "first-request latency" section used ~70-column soft wraps, which look fine rendered but make the raw markdown look ragged — the line after 'Peer addressing resolves lazily:' used to stop before the line was actually full, which reads as a formatting bug in plain text. Let each logical paragraph / bullet be a single line and defer wrapping to the browser / sphinx renderer. No rendered change. Also update the PR body on #1944 the same way. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): fix post-refactor stale references (architecture, thread safety, comparison) Four places still described the pre-#1944 code: 1. docs/.../efa_transport.md "EFA Transport Architecture" diagram — showed fid_ep inside EfaEndpoint, which is wrong under the shared-endpoint model. fid_ep lives on EfaContext (shared_ep_); EfaEndPoint only holds the peer's fi_addr_t AV slot. Redraw. 2. docs/.../efa_transport.md "Thread Safety" — said the transport "adds per-endpoint spinlocks" (plural, per-endpoint). In the new code there's exactly one post_lock_ per EfaContext guarding the shared fid_ep. Reword. 3. docs/.../efa_transport.md "EFA vs RoCE RDMA" comparison rows — used the pre-refresh p5en GPU number (347 GB/s) and the stale B200 row (313 GB/s). Replace with the current p5en peak from the benchmark tables above (365 GB/s GPU, 213 GB/s CPU), drop the row pointing at a now-absent B200 re-sweep. 4. efa_context.cpp:589 — "Fast path: peer info pre-resolved by submitTransferTask's striping path". The striping path was removed in an earlier commit in this PR; the fast path now just catches slices whose peer_nic_path the caller already filled in. Reword without referencing the removed mechanism. No behavioral change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(efa): address pr review — drop unused params, sysconf once, drop friend-class, expand probe comment Addresses trivial review feedback on #1944: - EfaContext::construct() no longer takes num_comp_channels / port / gid_index. These three parameters were plumbed through from the original RDMA-era signature but the EFA/libfabric path never read them; the previous commit just silenced warnings with (void) casts. Remove them from the signature and the one caller in EfaTransport::installSglangTransport. - preTouchMemory() was calling sysconf(_SC_PAGESIZE) twice (once for the >0 check and once for the value). Cache the result in a local. - `friend class EfaEndPoint` in efa_context.h was unnecessary — EfaEndPoint only touches public methods on EfaContext (nicPath, localEpAddr, engine, insertPeerAddr, removePeerAddr, submitSlicesOnPeer). Drop the friend decl. - Expand the header comment on efa_first_submit_probe.cpp to say what problem it exists to measure and why transfer_engine_bench alone isn't sufficient — reviewer asked what the example is for. No behavioral change. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * perf(efa): faster hex<->binary AV address handling Reviewer pointed out two sources of overhead in the handshake path: 1. localEpAddr() built the hex string via std::ostringstream with std::hex / std::setw / std::setfill, which pulls in stream formatting for every byte. Replace with a direct table-driven encoder writing into a pre-sized std::string. 2. insertPeerAddr() decoded the hex using substr() + strtol() for every byte (a temporary std::string and a libc base-16 parse per pair). Replace with a single pass using a hexNibble() helper and a pre-sized buffer. 3. Loopback called insertPeerAddr(localEpAddr()), encoding the bytes to hex and then immediately decoding back. Expose the raw bytes via localEpAddrBytes() and add an insertPeerAddrBytes() overload; loopback now skips the round-trip entirely. Handshake path is not the dominant cost in steady-state serving, but warmupSegment() fires 256 of these on a fresh 16x16 topology and every one of them was doing the extra work. The new path also avoids the per-byte std::string allocation (strtol path). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(efa): use std::atomic for wr_depth / cq outstanding, fix O(N^2) erase Replace `volatile int` + __sync_* builtins with std::atomic<int> on the shared-endpoint pacing counters (EfaContext::wr_depth_ and EfaCq::outstanding). `volatile` never implied atomicity under the C++ memory model, and mixing it with __sync builtins was technically UB; the atomic version is both portable and more explicit about the intended memory ordering (acq_rel on the mutating ops, relaxed on the speculative load used for CAS). Also fix an O(N^2) batch consumption pattern in submitSlicesOnPeer: `slice_list.erase(begin, begin + batch_count)` shifted the tail on every iteration. Consume via a moving cursor instead, and splice retry slices in place when -FI_EAGAIN forces a partial post. Behavior is unchanged on the happy path; the retry path now preserves the original slice order at the cursor, matching the previous insert-at- front semantics. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(efa): drop stale efa_latency_bench.{py,png}, document efa_first_submit_probe The `efa_latency_bench.py` script and its `efa_latency_bench.png` output shipped with the original EFA transport PR (#1509) two years ago. No markdown doc references them, the script hard-codes old private IPs as defaults, and nothing guarantees the chart is still representative. Remove both. Expand the First-request latency section of efa_transport.md to actually document `efa_first_submit_probe`: what it measures (cold-submit + eager-warmup cost, which `transfer_engine_bench`'s 10s average hides), how to run it (target / initiator commands), the expected output layout, all flags, and the two situations where it is worth running (deciding whether your app needs `warmupSegment()`, and comparing PR branches on the same hardware). Marked it explicitly as EFA-specific — RDMA/TCP transports do not have an equivalent warmup concept, so folding this into `transfer_engine_bench` would muddy the generic benchmark for no gain. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: EC2 Default User <ec2-user@ip-172-31-8-212.us-west-1.compute.internal>When WITH_NVIDIA_PEERMEM=OFF, openRdmaDevice() validated DMA-BUF support by calling cuDeviceGet(&cuDevice, i) where i is the verbs enumeration index from ibv_get_device_list(). This assumed verbs device order matches CUDA device order, which is incorrect. On a GB300 with 10 NICs and 4 GPUs, NICs at verbs index 4-9 called cuDeviceGet with indices 4-9, which don't exist. This disabled 6 of 10 NICs with "Failed to query CUDA device", limiting RDMA throughput to 4 NICs (18 GB/s) instead of all 10 (41 GB/s). Fix: - Look up the topology matrix to find which CUDA devices list this RNIC in their preferred_hca or avail_hca, then validate DMA-BUF only for those specific devices. - Check avail_hca in addition to preferred_hca because runtime NIC selection can fall back to avail_hca after disableDevice() removes a failed preferred NIC. Without this, the fallback NIC may not have been validated for the GPU that ends up using it. - Add explicit cuInit(0) before cuDeviceGet. The old code relied on implicit CUDA driver initialization from a prior cudaGetDeviceCount call in topology discovery. Making it explicit removes a fragile ordering dependency. Validation on GB300 (single-host VRAM read, GPU 1 -> GPU 0, all 10 NICs auto-discovered): Topology discovery complete. Found 10 HCAs. RDMA device: mlx5_0 ... mlx5_9 [all 10 register, GIDs elided] Pre-fix only the 4 NICs at verbs index 0-3 would have passed validation; mlx5_4-9 would have been disabled with "Failed to query CUDA device". With the fix, the bench reaches 41 GB/s saturation (see follow-up topology commit) instead of being capped at ~18 GB/s. Co-authored-by: nishikant <nishikant.kp00@gmail.com>Say we have 3 transfer engine instances T0, T1, T2, in the following case with P2PHANDSHAKE, they will form a circular dead lock: T0.listener is handling connection request from T1: -> setupConnectionsByPassive() -> getSegmentDescByName(T1) -> exchangeMetadata(T1) -> wait for T1.listener processing T1.listener is handling connection request from T2: -> setupConnectionsByPassive() -> getSegmentDescByName(T2) -> exchangeMetadata(T2) -> wait for T2.listener processing T2.listener is handling connection request from T0: -> setupConnectionsByPassive() -> getSegmentDescByName(T0) -> exchangeMetadata(T0) -> wait for T0.listener processing T0 -> T1 -> T2 -> T0 To fix this, we can remove the calling to getSegmentDescByName completely from the connection establish process, and exchange necessary connection information through HandShakeDesc. This can also significantly simplify the connection process. Signed-off-by: Chen Jinlong <chenjinlong.cjl@alibaba-inc.com>* [Store] Expose is_local_disk_replica() to Python in ReplicaDescriptor Replica::Descriptor is a 3-way std::variant {MemoryDescriptor, DiskDescriptor, LocalDiskDescriptor} with a corresponding C++ predicate per type. The Python wrapper was missing is_local_disk_replica. Mooncake's offload pipeline (NotifyOffloadSuccess) constructs LocalDiskDescriptor exclusively. With only is_memory_replica / is_disk_replica exposed to Python, every LOCAL_DISK descriptor returned by the master to a Python caller would test False on both predicates and be misclassified (e.g. as "unknown" in tier diagnostics) regardless of whether the actual load succeeded. --------- Co-authored-by: Zhewen Li <zhewenli@inferact.ai> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>* [TENT] Refactor RDMA transport with unidirectional endpoint lifecycle This commit aligns TENT RDMA transport with the source implementation, focusing on endpoint lifecycle management, configuration consolidation, and code organization improvements. **Endpoint Lifecycle (Unidirectional)** - Rename enum Status -> EndpointState to avoid ambiguity with mooncake:⛺:Status - Define explicit state machine: EP_UNINIT -> EP_HANDSHAKING -> EP_READY -> EP_DESTROYING -> EP_DESTROYED - Remove redundant active_/inactive_time_ members, use status_ for all state judgment - Implement unidirectional lifecycle: endpoints never reset or reuse - Add resetConnection() for marking failed endpoints for destruction - Deprecate reset() to prevent accidental endpoint reuse **Two-Phase QP Destruction** - beginDestroy(): Mark endpoint as EP_DESTROYING, transition QPs to ERR state - finishDestroy(): Wait for inflight WRs to drain, then destroy QPs - Add destroy_start_time_ for timeout enforcement (30s default) - Fix deconstructUnlocked() to maintain EP_DESTROYED state (no rollback to EP_UNINIT) **Endpoint Store Cleanup** - Unify remove() and removeRef() into single remove(RdmaEndPoint*) method - Add terminal state checking in getOrInsert() - auto-remove and recreate - Fix evictOne() to call beginDestroy() before moving to waiting_list - Fix reclaim() to use finishDestroy() instead of getInflightSlices() - Add waiting_list_len_ early return check in FIFO.reclaim() **Configuration Management** - Migrate PCIe Relaxed Ordering from environment variables to config - Add backward compatibility mappings for legacy MC_* environment variables: - MC_NUM_CQ_PER_CTX, MC_NUM_COMP_CHANNELS_PER_CTX, MC_IB_PORT - MC_GID_INDEX, NCCL_IB_GID_INDEX, MC_MAX_CQE_PER_CTX - MC_MAX_EP_PER_CTX, MC_NUM_QP_PER_EP, MC_MAX_SGE, MC_MAX_WR - MC_MAX_INLINE, MC_PKEY_INDEX, MC_MTU, MC_IB_TC - MC_IB_PCI_RELAXED_ORDERING, MC_WORKERS_PER_CTX - MC_SLICE_SIZE, MC_RETRY_CNT, MC_DISABLE_GPU_DIRECT_RDMA - Add RdmaTransport::config() public accessor for config-driven decisions **Code Quality** - Update state checks from CONNECTED -> EP_READY - Simplify status checks by removing active_ dependency - Improve logging for state transitions Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Reformat code * Avoid use RdmaEndpoint::reset() * Fix code issues --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>* [Store] L2->L1 promotion-on-hit: Tier A observability + max_per_heartbeat knob Adds Prometheus metrics for the promotion-on-hit funnel and exposes the previously-hardcoded kMaxPerHeartbeat as a config knob. Metrics (master_promotion_*) ---------------------------- Funnel: - promotion_in_flight (gauge): mirror of promotion_in_flight_ - promotion_admitted_total: tasks past all gates, enqueued - promotion_completed_total: NotifyPromotionSuccess success - promotion_completed_bytes_total: bytes promoted (sum of source object_size) - promotion_failed_total: NotifyPromotionFailure accepted - promotion_expired_total: reaper Part 4 sweeps Rejection (per gate): - promotion_rejected_frequency_total: below admission threshold - promotion_rejected_watermark_total: DRAM at or above high watermark - promotion_rejected_cap_total: promotion_in_flight at queue limit Together: admitted = completed + failed + expired + in_flight. Wires a working 'promotion feature health' Grafana panel: rate(admitted) vs rate(completed) shows yield; rate(rejected_*) breaks down where work is dropping; in_flight / promotion_queue_limit shows saturation. promotion_max_per_heartbeat knob -------------------------------- The previous compile-time constant kMaxPerHeartbeat = 1 (in PromotionObjectHeartbeat) capped per-client throughput at ~6 promotions/min with the default 10s heartbeat, making promotion_queue_limit = 50000 mostly theoretical. Exposed as MasterServiceConfig::promotion_max_per_heartbeat, wired through the flag parser (FLAGS_promotion_max_per_heartbeat) and default_config.GetUInt32 path. Constructor clamps 0 -> 1 so a mistyped config doesn't silently halt promotion delivery. Startup log now includes max_per_heartbeat= alongside threshold= / queue_limit=. Default stays 1 (no behavior change on existing deployments). Operators with small objects + RDMA-rich clusters can raise it; the new MaxPerHeartbeatKnobControlsBatchSize test exercises 3. Tests ----- - MetricsFunnelTracksSuccessfulPromotion: single full lifecycle bumps admitted/completed/completed_bytes correctly and brings in_flight back to baseline. - MetricsRejectionCountersIncrementOnGateMiss: each of frequency and cap counters increments when its branch fires. - MaxPerHeartbeatKnobControlsBatchSize: knob=3, 5 admitted tasks drain across 3+2+0 heartbeats. - MaxPerHeartbeatZeroClampsToOne: pathological config clamps to 1. Suite: 32/32 pass (was 28; +4 new). No behavior change in existing tests. * [Store] L2->L1 promotion-on-hit: cover watermark gate in rejection test MetricsRejectionCountersIncrementOnGateMiss claimed coverage of all three rejection counters but only exercised frequency and cap. Add a sub-case that forces the watermark gate by configuring eviction_high_watermark_ratio = 0.0, asserting that promotion_rejected_watermark_total increments. * [Store] L2->L1 promotion-on-hit: cover RemoveAll/BatchRemove cleanup upstream #2180 introduced EraseMetadataEntry as the centralized metadata-erase helper and routed RemoveAll, BatchRemove, and RemoveByRegex through it, so promotion_tasks cleanup on those paths is already correct on main. The metric instrumentation (dec_promotion_in_flight + inc_promotion_cancelled) is wired into EraseMetadataEntry in this branch's earlier commit so every site that erases metadata bumps the funnel counters consistently. Add regression tests for the three paths so any future refactor that reintroduces a metadata.erase without going through EraseMetadataEntry will fail the suite: - RemoveAllErasesPromotionTask - BatchRemoveErasesPromotionTask (normal-completion branch) - BatchRemoveStaleHandleErasesPromotionTask * [Store] L2->L1 promotion-on-hit: bump reaper-test sleep margin to 3s Three tests configure put_start_release_timeout_sec=1 and then sleep 2s waiting for the eviction-thread reaper to expire the promotion task. The reaper schedule is `now - last_discard_time > put_start_release_timeout_sec_` (strict greater-than), so a 1s release with a 2s sleep leaves only ~1s margin between the reaper firing and the assertion. CI runs observed intermittent failures when scheduling jitter erased that margin. Bump the sleep to 3s in the three affected tests so the margin is ~2s. Configuration values unchanged. Affected: - StalePromotionReaper - RemoveDuringPromotion - AllocStartRejectsReapedTask Verified 5 consecutive clean runs of all three under -j1 build.The client-side ResolveClusterNamespace() in etcd leader coordinator, redis leader coordinator, and redis snapshot catalog store all used 'mooncake' as the fallback default, while the master-side --cluster_id flag defaults to 'mooncake_cluster' (DEFAULT_CLUSTER_ID). This inconsistency caused clients to look up a different etcd/redis key than what the master registered, making them unable to discover the master when both sides use defaults. Fix by replacing the hardcoded 'mooncake' fallback with the DEFAULT_CLUSTER_ID constant ('mooncake_cluster') so that client and master use the same namespace out of the box. Also update documentation to reflect the corrected default value.* feat(transfer-engine): extract Device API and IBGDA transport layer Move IBGDA files from mooncake-ep to mooncake-transfer-engine (git mv): - 6 headers: mooncake_ibgda/ → transport/device/ibgda/ - 1 source: mlx5gda.cpp → transport/device/mlx5gda.cpp Add new Device API layer under transport/device/: - device_transport.h: P2pTransport + RdmaTransport interfaces - device_ops.cuh: DeviceOps function pointer table (bottom IR) - comm_device.cuh, p2p_device.cuh, ibgda_device.cuh: device contexts - cuda_ops.cuh, musa_ops.cuh: platform DeviceOps implementations - ibgda_device_transport.cpp: RdmaTransport IBGDA implementation - p2p_device_transport.cpp: P2pTransport NVLink/MTLink implementation Modify transfer-engine: - transfer_engine.h/impl: add getOrCreateP2pTransport/RdmaTransport - gpu_vendor/musa.h: add MUSA API aliases - Build: link mlx5, add device/ subdirectory Minimal EP changes (include paths + build system only): - Update include paths to new transport/device/ibgda/ location - Remove mlx5gda.cpp from EP sources - Forward EP_USE_MUSA env var in BuildEpExt.cmake Root CMakeLists.txt: guard CUDAToolkit with USE_CUDA * feat(example): add device_transport_example for P2P Device API Two-rank example demonstrating the full Device API lifecycle: - Host side: P2pTransport for IPC handle exchange and peer mapping - Device side: CommCtx + mc_route_put + mc_signal for GPU-initiated P2P data transfer and notification Uses file-based IPC handle exchange (no external dependencies). Requires 2 GPUs with P2P access (NVLink or PCIe). * fix(example): handle missing TORCH_CUDA_ARCH_LIST * fix(example): convert torch arch format to CMake CUDA format * fix(example): construct CommCtx manually on host side * fix(example): add using namespace mooncake::device in kernels * fix(example): pass CommCtx by value, add barrier before cleanup - Pass CommCtx by value to kernel (CUDA copies to param space) instead of dereferencing a device pointer on host (caused segfault). - Add file-based barrier so rank 0 waits for rank 1 to finish before freeing its GDR buffer (IPC handle only valid while allocation exists). - Change default metadata_server to P2PHANDSHAKE (no etcd dependency). * fix: address review feedback — fence ordering, error checks, QP leak - musa_ops.cuh: fix fence ordering for acquire/release semantics (fence after load for acquire, before store/atomic for release) - ibgda_device_transport.cpp: check cudaMalloc return values, add num_qps >= num_ranks guard, destroy QP on rst2init failure - p2p_device_transport.cpp: add device_count > 0 guard - device_transport_example.cu: validate kDataBytes % 16 == 0 and kDataBytes <= kSignalWordOffset * style: apply clang-format to Device API files * fix: guard device transport code with USE_CUDA/USE_MUSA macros The device transport accessors (getOrCreateP2pTransport, getOrCreateRdmaTransport) and their member variables were not guarded by USE_CUDA/USE_MUSA preprocessor macros. When building with USE_CUDA=OFF (the default), the device transport source files aren't compiled but the headers and implementations still reference them, causing linker errors in CI build-flags and build jobs. * fix: set CMAKE_CUDA_STANDARD 20 and make ibgda PUBLIC - common.cmake: add CMAKE_CUDA_STANDARD 20 so nvcc compiles host code in C++20 mode, matching CMAKE_CXX_STANDARD. Fixes "starts_with is not a member of std::string" when compiling .cu files that indirectly include common.h. - transport/CMakeLists.txt: change ibgda from PRIVATE to PUBLIC so mlx5gda_* symbols are visible to downstream consumers (Go p2p store via transfer_engine). Fixes undefined reference errors for mlx5gda_destroy_qp, mlx5dv_devx_umem_reg, etc. * fix: compile mlx5gda into device_transport and link mlx5 for Go consumers The previous attempt (PUBLIC ibgda) did not work because the Go p2p-store and mooncake-store binaries link libtransfer_engine.a via hand-written cgo ldflags, which bypass CMake's target_link_libraries propagation entirely. - transport/device: compile mlx5gda.cpp directly into the device_transport OBJECT library (like every other transport module) instead of a separate ibgda STATIC lib, so mlx5gda_* symbols flow into libtransfer_engine.a and are visible to all consumers regardless of how they link. - transport: link libmlx5 (PUBLIC) since ibgda_device_transport.cpp / mlx5gda.cpp call mlx5dv_devx_* / mlx5dv_init_obj directly. - p2p-store/build.sh, mooncake-store/go/build.sh, ci.yml: add -lmlx5 to the hand-written cgo ldflags so the DevX symbols resolve. - example: set CUDA_STANDARD 20 on device_transport_example so nvcc compiles common.h (std::string::starts_with) in C++20 mode. * fix: CUDA_EXTENSIONS OFF for example, add -lm for Go consumers Follow-up to compiling mlx5gda.cpp into device_transport: - example: nvcc has no gnu++20 dialect, so CUDA_STANDARD 20 with the default CUDA_EXTENSIONS=ON fails at CMake generate ("does not know the compile flags"). Set CUDA_EXTENSIONS OFF to request plain -std=c++20. - p2p-store/build.sh, mooncake-store/go/build.sh: mlx5gda.cpp uses log2/ceil (<cmath>); now that its object lives in libtransfer_engine.a, the hand- written cgo ldflags need -lm to resolve log2@GLIBC_2.29. (ci.yml already had -lm.) * fix(ci): gate Device API GPU example off by default, link mlx5 for Rust The Docker build failed at CMake generate because device_transport_example needs the CUDA20 dialect (transfer_engine.h -> common.h uses C++20 std::string::starts_with), which the older CMake in the CI image cannot enable. CUDA_EXTENSIONS OFF did not help since the limitation is the CMake version, not the dialect flavor. Gate this manual, 2-GPU example behind a new BUILD_DEVICE_TRANSPORT_EXAMPLE option (default OFF) so the default build no longer requires CUDA20. Also link mlx5 in the mooncake-store Rust build script: the IBGDA device transport (mlx5 DevX) is now compiled into transfer_engine, so the Rust test link step needs -lmlx5 to resolve mlx5dv_devx_* symbols.* [TE] Guard stoull in EFA getMaxPteEntries against invalid input std::stoull in a static initializer throws on non-numeric input, causing std::terminate. Add try-catch to fall back to default. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Reject negative MC_EFA_MAX_PTE_ENTRIES values stoull accepts "-1" and wraps to ULLONG_MAX, passing the val > 0 check. Add minus-sign guard to treat negative input as invalid. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix clang-format violation in getMaxPteEntries Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Replace negative-sign check with range validation Per reviewer feedback: use upper-bound check instead of string inspection. stoull("-1") wraps to ULLONG_MAX which fails the range check naturally. Also catches absurdly large positive values. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Simplify getMaxPteEntries per maintainer request Remove MC_EFA_MAX_PTE_ENTRIES env var override entirely. The default 22M PTE entries is sufficient for all practical EFA workloads. Eliminates parsing complexity. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Remove stale env var comment Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>he /latest key initialization in EtcdOpLogStore::Init() was gated by enable_batch_write_, which is only true for WRITER role instances. However, the master's snapshot path creates EtcdOpLogStore with default parameters (enable_batch_write=false), so Init() always skipped the /latest key creation. This caused 'key not found' errors on first deployment when snapshot resolution tried to read /oplog/{cluster_id}/latest. Fix: Remove the enable_batch_write_ guard. The initialization uses CAS (Create-if-not-exist) semantics, so concurrent callers are safe — only the first one actually creates the key. Signed-off-by: leonzzhu <leonzzhu@tencent.com>Thread cluster_id into the SnapshotCatalogStore layer so that all S3 object paths (data, descriptor, latest.txt) are scoped per cluster: mooncake_master_snapshot/{cluster_id}/{snapshot_id}/... Changes: - Rename kSnapshotRoot to kSnapshotRootBase; add BuildSnapshotRoot(cluster_id) - All path helper functions now accept snapshot_root parameter - Add GetSnapshotRoot() pure virtual to SnapshotCatalogStore interface - EmbeddedSnapshotCatalogStore accepts cluster_id in constructor - RedisSnapshotCatalogStore computes snapshot_root_ from cluster_namespace_ - MasterService uses catalog_store_->GetSnapshotRoot() instead of hardcoded path - Backward compatible: empty cluster_id produces original path Signed-off-by: leonzzhu <leonzzhu@tencent.com>* [Store] test: add comprehensive MasterAdminServer HTTP endpoint tests Add 49 unit tests in master_admin_server_test.cpp covering all 16 HTTP endpoints served by MasterAdminServer: Always-available endpoints: - /metrics, /metrics/summary, /health, /role, /ha_status, /leader Service-dependent endpoints: - /query_key, /get_all_keys, /get_all_segments, /get_segments_detail, /query_segment, /batch_query_keys - /api/v1/drain_jobs (create/query/cancel) - /api/v1/segments/status Tests cover: all runtime states, service available/unavailable transitions, error cases (invalid params, missing keys, nonexistent resources), and end-to-end drain job lifecycle.EFA's libfabric provider returns 64-bit memory-region keys via fi_mr_key(), but BufferDesc::{lkey,rkey} and Transport::Slice::{source_lkey, dest_rkey,dest_rkeys} stored them as uint32_t, truncating the upper 32 bits on both the metadata-encode path and the fi_read/fi_write submit path. In practice EFA keys have been small monotonic values that fit in 32 bits, so this latent bug has not yet manifested, but it is incorrect. Inspired by #2535 (HPE Slingshot/cxi backend), which hit the same issue and widened the CXI path. This change applies the analogous fix to EFA, guarded by USE_EFA so RDMA verbs builds (whose ibv keys are genuinely 32-bit) are byte-for-byte unchanged. Verified on a P5EN-1 <-> P5EN-2 pair (p5.48xlarge, 32 EFA NICs) with transfer_engine_bench, A/B against unmodified main: GPU-to-GPU: baseline 362.44 GB/s vs fixed 363.38 GB/s CPU-to-CPU: baseline 35.28 GB/s vs fixed 37.70 GB/s EFA unit tests (efa_gpu_loopback_test, efa_transport_test, transport_uint_test, efa_c_api_test) all pass. No regression. Co-authored-by: whn09 <whn09@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>* ci: add AWS EFA wheel build/release to official CI/CD Adds EFA (libfabric) wheel variants to the wheel pipeline, mirroring the existing non-cuda / cuda13 / npu variant pattern. The EFA transport memory path is CUDA-aware (FI_HMEM_CUDA / GPUDirect under USE_CUDA=ON, FI_HMEM=system otherwise), so two PyPI packages are produced: mooncake-transfer-engine-efa USE_EFA=ON USE_CUDA=ON (GPU) mooncake-transfer-engine-efa-non-cuda USE_EFA=ON USE_CUDA=OFF (CPU/DRAM) No EFA hardware is needed to build: USE_EFA only needs libfabric headers/lib to compile and link, which the stock ubuntu runner gets from the distro libfabric-dev package. auditwheel already excludes libfabric/libefa from the wheel (build_wheel.sh) so they resolve to the user system AWS EFA install at runtime, avoiding the dual-libfabric / device-claim conflict. The distro libfabric (1.x) is ABI-forward-compatible with the AWS EFA libfabric (2.x) loaded at runtime; the EFA transport uses only long-stable fi_* APIs. Changes: - scripts/build_wheel.sh: add EFA_BUILD / EFA_NON_CUDA_BUILD variants that rename the package to mooncake-transfer-engine-efa{,-non-cuda}. - .github/workflows/ci_efa.yml: PR-time build validation (one python per variant), wired into ci.yml CI Gate. Verifies libfabric is NOT bundled. - .github/workflows/release-efa.yaml: tag-triggered full python matrix (3.10-3.13) x {cuda, non-cuda}, publishes to GitHub Release + PyPI. Verified end-to-end on a p5.48xlarge: both variants build, produce the correct package names, and exclude libfabric from the wheel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(efa): document PyPI install and EFA wheel CI/CD Now that EFA wheels are published by the release pipeline, document the recommended path (pip install mooncake-transfer-engine-efa / -efa-non-cuda) before the build-from-source instructions, and note the EFA_BUILD / EFA_NON_CUDA_BUILD variant env vars plus the ci_efa.yml / release-efa.yaml workflows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(efa): split EFA release into per-package workflows; bump CUDA to 12.8.1 Address review feedback on PR #2565: - Split release-efa.yaml (which built both cuda + non-cuda via a matrix.variant) into two standalone workflows, one per PyPI package, mirroring the existing release.yaml / release-non-cuda.yaml pattern: * release-efa.yaml -> mooncake-transfer-engine-efa (USE_CUDA=ON) * release-efa-non-cuda.yaml -> mooncake-transfer-engine-efa-non-cuda (USE_CUDA=OFF) Each has its own build matrix, libfabric-exclusion check, artifact pattern, and publish-release/PyPI step, so a package publishes from a dedicated workflow (per-package trusted publisher / artifact pattern). - Bump CUDA Toolkit 12.4.1 -> 12.8.1 in ci_efa.yml and release-efa.yaml to match the version used across the rest of the repo workflows (ShangmingCai review). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: whn09 <whn09@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>These three transports snapshot the output `status` BEFORE polling the hardware (cudaEventQuery / hixl GetTransferStatus). When the poll transitions the task to COMPLETED/FAILED/TIMEOUT, only the task is updated; `status` keeps the pre-poll PENDING value, so the caller sees the completion one poll cycle late. Move the `status = TransferStatus{...}` read to after the poll on every return path. Same class of bug already fixed for io_uring/gds. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Pull request closed