agent_swarm drives the core scenario: several agents share a long repo
context and differ only in a short instruction. It runs the workload with
sharing off (baseline) and on, erasing llama's own per-slot cache between
runs so the comparison isolates cross-process reuse, and reports honest
end-to-end TTFT, recomputed prefill tokens, hit rate and throughput.
smoke_e2e is a minimal two-agent cross-GPU reuse check; microbench_store
and microbench_striped measure raw and striped store bandwidth;
run_matrix sweeps a config and plot renders the figures.
scripts/env.sh keeps every toolchain, cache and runtime directory under
an out-of-tree state root so the source stays clean and nothing is
written under /. The setup_* scripts install a local Go toolchain, a
venv with mooncake-transfer-engine, a CUDA build of llama.cpp and the
GGUF models; the *_start/stop and demo scripts bring the stack up and
down, scoped to this integration's own ports and paths.
Two small, additive patches that make upstream Ollama a participant of
the KVCache Bus, verified against ollama/ollama main at 1abd56b.
0001 adds a self-contained HTTP client for the sidecar's prepare/commit
endpoints plus parsing of the options.mooncake.* request extension
(enable, namespace, read, write, block_size, transport, replica_num,
soft_pin), without touching any existing symbol.
0002 injects --slot-save-path into the bundled llama-server command line
when OLLAMA_MOONCAKE_SLOT_SAVE_PATH is set, exposing the /slots
save|restore endpoints the sidecar drives. Both are gated, so an
unconfigured Ollama behaves exactly as before. The README documents how
to apply them and the one call-site hook left as a reference, and notes
0002 as a clean upstream-PR candidate aligned with Ollama issue 14872.
The Stage-1 path serialises a sequence's KV through host memory and a
file (llama.cpp /slots save), which is several times slower than a raw
in-process copy because of the double copy described in llama.cpp issue
8915. Stage 2 targets the high-performance route.
seqstate is a cgo binding (build tag cgo_stage2) for
llama_state_seq_get/set_data_ext. With LLAMA_STATE_SEQ_FLAGS_ON_DEVICE
the KV tensors stay in device buffers, ready to be registered with the
Mooncake Transfer Engine for GPUDirect RDMA with no host copy. It is
gated behind a build tag because it links libllama and only applies to
the embedded deployment; the default HTTP path needs none of it.
omb_kvbench is an in-process microbenchmark against libllama that
compares the file-save path, the raw host export and the on-device size,
and verifies that a sequence's KV round-trips correctly (export seq 0,
import into seq 1, identical next-token argmax). It quantifies the
file-path overhead the on-device route removes.
A long-lived gRPC service that owns one warm MooncakeDistributedStore
handle and exposes it to the Go sidecar, which cannot link the Python
bindings directly. Using the official store client keeps the integration
on the supported API surface.
KV snapshots are multi-GiB, so the proxy moves them by file path: PutFile
reads a slot file and stores it, GetFile materialises an object straight
into the slot directory, and the blob is copied at most once. Objects are
striped into <=64 MiB chunks transferred in parallel with
batch_put_from / batch_get_into, because single-object RDMA degrades for
large objects while striped batches sustain full bandwidth. A pool of
pre-registered (pinned) staging buffers amortises RDMA memory
registration across operations. Eviction or lease-expiry races on read
are reported as not-found so the caller falls back to recompute rather
than failing.
A filesystem backend (no master required) backs the local baseline, and
a selftest verifies put/get round-trips and existence checks over both
backends.
Wire the pieces into the bridged daemon.
orchestrator implements the three-stage reuse flow. Lookup builds the
per-block keys and finds the longest stored prefix with a single batched
existence query (correct across nodes). Prepare additionally restores the
matched KV into a target llama.cpp slot when the arbiter approves, so the
server only prefills the uncached tail. Commit saves the slot KV and
stores it under the block key for the largest block-aligned prefix, with
skip-if-exists giving single-writer dedup when concurrent agents share a
prefix. It feeds the arbiter the full restore wall time and the observed
prefill rate so estimates self-calibrate, and tracks per-model KV
bytes/token for transfer sizing.
llamabridge drives the stock llama.cpp server over HTTP: /tokenize,
/completion (reading timings.prompt_n to count avoided prefill),
/slots save|restore|erase and /props. No llama.cpp source change is
needed for this path.
metrics exposes Prometheus instrumentation; the headline series is the
cumulative count of prompt tokens not re-prefilled thanks to reuse.
server adapts the orchestrator onto the gRPC KVCacheBus and an HTTP/JSON
gateway, and cmd/bridged is the daemon entrypoint serving gRPC (TCP and
unix socket), the JSON gateway and /metrics.
Define the two service contracts and the object-store abstraction the
orchestrator builds on.
bridge.proto is the agent-facing KVCacheBus: Lookup (read-only longest
prefix match plus arbiter decision), Prepare (Lookup plus restore of the
matched KV into a llama.cpp slot) and Commit (save the slot KV back to
the store). storeproxy.proto is the sidecar-to-proxy contract; KV blobs
are addressed by file path so a multi-GiB snapshot is copied at most once
and never streams through the Go process.
The store package exposes a small Backend interface with two
implementations: a Mooncake backend that forwards file-path Put/Get and
batched existence checks to the Python store proxy over gRPC, and a
local filesystem backend used as the cross-process baseline and a
dependency-free fallback. Generated stubs are checked in so the module
builds without protoc.
Introduce the algorithmic core of the Ollama KVCache Bus, an integration
that lets independent Ollama/llama.cpp processes share prompt KV through
the Mooncake Store.
cachekey derives a content-addressed key from a model fingerprint (model
digest, tokenizer/RoPE hashes parsed from GGUF metadata, KV dtype/layout,
context length, block size) and a forward-chained per-block hash of the
prompt tokens, where h_i = H(h_{i-1} || block_i). Chaining makes
longest-prefix matching exact: two prompts agree on h_i iff they share
every token of blocks 0..i. KV produced under different model parameters
lands in a disjoint key space and is never reused.
prefixindex maintains an in-memory radix tree over those chained hashes,
the cross-process analogue of a RadixAttention tree whose nodes reference
remote KV snapshots. It answers longest-prefix lookups in O(blocks) and
tracks per-prefix fan-out (hotness) and recency for replication and
eviction decisions.
arbiter decides restore-vs-recompute per request. A cached prefix only
helps when the store can deliver its KV faster than the GPU regenerates
it; the arbiter estimates both from rates it learns online (EWMA, seeded
on first observation) per model, and restores only when it is cheaper.
This keeps KV sharing loss-free across hardware regimes.
Includes unit tests for hash determinism, prefix consistency, model
separation, longest-prefix matching, eviction ordering and the arbiter's
adapt-to-recompute behaviour.
Immediately completing the batch on transfer failures will result in
use-after-free problems because other tasks in the batch might be still
in processing.
Signed-off-by: Chen Jinlong <chenjinlong.cjl@alibaba-inc.com>
When MC_STORE_MEMCPY is not explicitly set, auto-detect based on
installed transports: enable memcpy only when TCP is the sole
transport (no RDMA, NVLink, etc.), since TCP loopback is less
efficient than direct memcpy for same-host transfers. In RDMA
environments the default remains disabled, as RDMA is more
resource-efficient.
Add TransferEngine::isTcpOnly() API that checks whether TCP is the
only installed transport via MultiTransport::transport_map_. This
is future-proof: any new transport registered via installTransport()
is automatically accounted for without maintaining a protocol list.
TENT path returns false unconditionally since TENT already rejects
TCP loopback without MC_STORE_MEMCPY.
Signed-off-by: Tianchen Ding <dtcccc@linux.alibaba.com>
* feat(efa): auto-split large MR registrations exceeding max_mr_size
Buffers larger than the EFA device's max_mr_size are now transparently
split into chunks, each registered as a separate MR. This fixes the
silent truncation bug where only the first max_mr_size bytes were
registered, causing transfers to unregistered regions to fail at runtime.
Key changes:
- Query EFA device max_mr_size via ibverbs during init (libfabric does
not expose this) and clamp globalConfig accordingly
- Auto-split buffers > (max_mr_size - 1GB) into chunks in
registerLocalMemoryInternal, each with its own BufferDesc metadata
- Track chunk mappings for proper cleanup in unregisterLocalMemory
- Change lkey()/rkey() from exact-match to range lookup (matching
mrDesc() pattern) so key lookups work for any address within a chunk
- Replace silent truncation in EfaContext with a hard error as safety net
- Remove preTouchMemory truncation to touch the full buffer
Tested on P5EN (16 EFA NIC, max_mr_size=192GB): 191GB single-chunk
registration succeeds. Auto-split triggers correctly for larger buffers
(200GB splits into 191GB + 9GB). Total registerable size per buffer is
bounded by system pinned_vm limits (~191GB on 16-NIC P5EN).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): per-NIC partition for large buffer MR registration
Each EFA NIC can only register up to max_mr_size total. The previous
auto-split approach registered every chunk on ALL NICs, hitting the
per-NIC limit for buffers > max_mr_size. This change assigns each
chunk to a disjoint subset of NICs, enabling registration of buffers
up to max_mr_size × num_NICs (e.g. ~1.5TB on P5EN with 16 NICs).
Key changes:
- chunk_limit = max_mr_size / 2 (was max_mr_size - 1GB) for headroom
- NIC assignment: chunks distributed evenly across available NICs
- selectDevice(): skips NICs with rkey=0 (unassigned for that chunk)
- Striping path: filters by lkey!=0 to avoid unregistered NICs
- Unregister: only deregisters from assigned NICs per chunk
- ChunkRegistration struct tracks per-chunk NIC assignments
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): PTE-aware auto-split replaces conservative max_mr_size/2 threshold
The previous chunk_limit of max_mr_size/2 (~96GB on P5EN) caused unnecessary
buffer splitting even when hugepages were available. This detects the actual
backing page size via /proc/self/smaps and computes the PTE-based limit:
- 4KB pages: 22M PTEs × 4KB = 88GB (genuine hardware constraint)
- 2MB hugepages: 22M PTEs × 2MB = 44TB (effectively max_mr_size)
Verified on P5EN (H200, 16 EFA): 100GB pool with hugepages no longer splits,
restoring full 16-NIC throughput (108 GB/s vs 46 GB/s with the old threshold).
MR registration time dropped from 302s to 1.6s.
Adds MC_EFA_MAX_PTE_ENTRIES env var for override (default 22M).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add KV cache prefix transfer benchmark script
Python benchmark to measure EFA transfer performance for LLM prefix
cache hit scenarios. Tests different pool sizes (10GB-500GB) and prefix
lengths (4K-32K tokens) to evaluate per-NIC partition impact on
transfer latency and throughput.
Default KV bytes/token matches GLM-5.1 (754B MoE, MLA attention):
(kv_lora_rank=512 + qk_rope_head_dim=64) * 2 * 78 layers = 88KB/token
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(bench): add connection warmup before benchmark loop
The first prefix size's measurements were skewed by EFA connection
establishment (openSegment, endpoint creation). Add 3 small transfers
before entering the benchmark loop to warm up the connection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(bench): per-offset warmup and p50 throughput reporting
Warmup now exercises all offsets the benchmark will measure, eliminating
first-access TLB/page-fault outliers (4K token p99 dropped from 80ms to
3.5ms). Throughput is reported from p50 latency instead of avg for more
stable numbers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(bench): add --threads option for concurrent prefix transfer
Adds optional multi-threaded transfer support. Each thread transfers a
chunk of the prefix in parallel via separate transfer_sync_read calls.
Default is 1 (single transfer, same as before). Testing shows threads=2
matches single-thread throughput (~108 GB/s), while higher values add
scheduling overhead with no benefit since EFA already stripes internally.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): full NIC coverage for multi-chunk MR registration
When a buffer exceeds max_mr_size and must be split into multiple chunks,
register every chunk on ALL NICs instead of disjoint per-NIC partition,
as long as total PTE usage per NIC fits within the PTE budget.
With hugepages (2MB), 500GB buffer uses only 250K PTE/NIC (budget: 24M),
so all 16 NICs cover every address. Falls back to disjoint partition when
PTE budget is exceeded (e.g. 4KB pages with large buffers).
500GB pool throughput: 35 GB/s (disjoint, 5-6 NIC) → 108 GB/s (full, 16 NIC).
Registration time unchanged (~9.5s) due to parallel MR registration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(bench): add batch memory registration benchmark
Test script for registering multiple independent memory blocks
(e.g. multi-tenant KV cache pools). Supports both per-block
register_memory and batch_register_memory APIs. Target mode
allocates and registers N blocks; initiator mode transfers
data and measures throughput.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply clang-format to EFA transport files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): expose discoverTopology C API and Rust bindings
Add discoverTopology() to the C API and discover_topology()/install_transport()
to the Rust bindings, enabling EFA transport initialization from C/Rust callers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(efa): add C API test for discoverTopology and EFA transport
Verify that the new discoverTopology() C API correctly populates the
device list, enabling installTransport("efa") and memory registration
via the pure C interface (used by Rust/Go bindings).
All 4 tests passed on p5en.48xlarge (16 EFA NICs).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): O(log n) MR lookup and chunk registration rollback
- Replace unordered_map with std::map for mr_map_ and use upper_bound
for O(log n) range lookups in rkey/lkey/mrDesc instead of O(n) scan
- Add rollbackChunks lambda to unregister already-registered chunks on
failure in registerLocalMemoryInternal, preventing MR leaks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply clang-format to efa_c_api_test.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: rename thr_tag to thread_tag to pass typos spell check
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): round-robin multi-chunk MR and large MR registration test
When a buffer splits into more chunks than available NICs, round-robin
assign chunks across NICs with per-NIC PTE budget validation, instead
of hard-failing with "Buffer requires N chunks but only M NICs".
Add efa_single_nic_large_mr_test: tests single-NIC and all-NIC large
MR registration with hugepages. Supports --chunk_gb for multi-buffer
mode (e.g. 200×2GB).
Verified on P5EN (16 EFA NIC, 2MB hugepages):
- 1 NIC × 200×2GB (400GB): 53.7s
- 16 NICs × 200×2GB (400GB): 64.8s
- 1 NIC × 200GB single buffer: 2.7s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add cross-node transfer test for multi-buffer MR registration
Tests 200x2GB buffer registration on all NICs with actual data transfer
between two P5EN nodes. Supports target/initiator modes with single-read
and multi-block batch benchmarks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): avoid CUDA context leak when built without GPU support
libfabric 2.4's EFA provider dlopens libcudart/libcuda during
fi_getinfo/fi_domain to detect HMEM support, even though the caller
never touches GPU memory. This creates a CUDA primary context on
GPU 0 and permanently holds ~616 MiB of device memory.
When Mooncake is built with USE_CUDA=OFF (and USE_HIP=OFF), set
FI_HMEM=system before fi_getinfo and drop FI_MR_HMEM from the domain
hints so the provider skips GPU hmem initialization entirely.
* feat(efa): multi-thread initiator and wildcard location for 16-NIC coverage
Register target buffers with wildcard location "*" instead of "cpu:0"
so initiator-side remote NIC selection distributes evenly across all 16
NICs (both NUMA nodes). With "cpu:0", selectDevice only picked NUMA-0's
8 NICs, leaving NUMA-1 idle — throughput capped at ~107 GB/s instead of
~149 GB/s on P5EN (16×EFA 200Gbps).
Also convert the initiator from single-threaded to multi-threaded
(--threads flag) and print the actual P2P handshake address.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add p6-b300 bandwidth results (752 GB/s GPU, 230 GB/s CPU)
- GPU-to-GPU peak 752 GB/s write / 713 GB/s read at ~94% line rate
(16×400 Gbps = 800 GB/s theoretical)
- CPU-to-CPU peak 230 GB/s, DRAM-limited on Xeon 8559C
(NUMA-0 NICs 90 Gbps vs NUMA-1 NICs 53 Gbps)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(rust): customer_pattern sample for per-NUMA multi-MR registration
Adds a Rust binary mirroring the customer's EFA usage pattern: target
registers many MRs per NUMA (e.g. 16 NICs x N x 10GB via cpu:<numa>
locations); initiator reads/writes a specific (numa, buffer_index)
tuple, using --source-numa to pick which local NIC set is exercised.
Also makes rust/build.rs robust against non-default transfer_engine
build configurations: etcd-cpp-api is opt-in via MOONCAKE_WITH_ETCD=1,
CUDA linking is opt-in via MOONCAKE_WITH_CUDA=1 with CUDART_LIB_DIR
override, and libbase.a + libasio.so + libfabric paths are picked up
whether the repo uses the standalone or top-level CMake layout.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(rust): drop customer_pattern sample from upstream
Revert Cargo.toml [[bin]] additions and remove the bench demo source so
the public tree no longer ships it. Keep it gitignored locally so it can
be iterated on without accidental re-adds.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(efa): eager endpoint warmup for segments to remove first-batch stall
libfabric FI_EP_RDM endpoints resolve peer addresses lazily — the first
submitTransfer() to a new segment serializes fi_av_insert + handshake over
every (local_ctx, peer_nic) pair. On 16-NIC instances this is a ~4 s stall
on the first 100 × 0.5 MB batch (measured on p6-B300).
Add EfaTransport::warmupSegment(name) that pre-connects all pairs
concurrently via std::async, plus C wrapper warmupEfaSegment() (guarded by
USE_EFA) and Rust binding TransferEngine::warmup_efa_segment(). Idempotent,
safe to re-run after metadata changes. RDMA/TCP paths untouched.
Measured on p6-B300 (16 × 16 endpoints, dual-NUMA initiator):
first-batch: 4043 ms -> 13.5 ms (~300x)
steady-state: 141 GB/s -> 230 GB/s
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(efa): enable auto-split when max_mr_size is not configured
When MC_EFA_MAX_MR_SIZE is unset, max_mr was 0 and chunk_limit collapsed
to 0, bypassing the PTE-aware split entirely. Large 4KB-paged buffers
then hit the per-NIC PTE ceiling at registration time.
Fall back to pte_limit so splitting kicks in based on the PTE budget
alone when no explicit max_mr_size is provided.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* style: apply clang-format to EFA transport/tests and PEP8 to kvcache bench
Pure formatting changes to satisfy CI format hook (clang-format-20) and
address Gemini review comment on kvcache_prefix_bench.py indentation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-13-185.us-east-2.compute.internal>
Co-authored-by: whn09 <whn09@github.com>
* feat(tent): add FaultProxyTransport for fault injection testing
Introduce a decorator Transport that wraps any real Transport and
injects configurable faults (submit failures, status corruption,
artificial latency) according to a FaultPolicy. This enables
integration testing of the failover state machine without hardware.
New files:
- fault_proxy_transport.h: header-only FaultProxyTransport + FaultPolicy
- fault_proxy_test.cpp: 8 GTest cases covering unit, failover, and
policy mutation scenarios
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix(tent): fix off-by-one in ExhaustAllTransports test
The test had 3 failover attempts matching kMaxAttempts=3, so
failover_count == kMaxAttempts (not greater). Add a 4th attempt
to properly exceed the limit, matching resubmitTransferTask logic
which uses `++count > max` (strictly greater).
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor: format code style in fault proxy transport
- Adjust line breaks and indentation for consistent formatting
- No functional changes, only code style improvements
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor: use thread-safe random number generation in fault proxy transport
- Replace instance RNG with thread-local static method for thread safety
- Simplify fault injection logic and remove redundant atomic operations
Signed-off-by: staryxchen <staryxchen@tencent.com>
* feat(transport): add null check and override memory management methods in
FaultProxyTransport
- Add assert to ensure real transport is not null in constructor
- Implement override methods for addMemoryBuffer, allocateLocalMemory,
freeLocalMemory, and warmupMemory to delegate to real transport
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
PutToLocalFile and FileStorage::OffloadObjects crash with SIGSEGV when
slice.ptr points to GPU device memory, because CPU memcpy cannot access
GPU virtual addresses. The RDMA memory-replica path is unaffected.
Add synchronous Device-to-Host staging via PinnedBufferPool before data
reaches the disk-write paths:
- New gpu_staging_utils.h: shared IsDevicePointer/CopyDeviceToHost/
SetDevice helpers with cross-platform support (CUDA/HIP/MUSA/MACA/
Ascend CANN)
- New PinnedBufferPool: thread-safe pinned host memory pool with
max capacity limit (default 32) and O(1) swap-pop acquire
- PutToLocalFile: sync D2H on calling thread, PutRevoke on failure
- OffloadObjects: D2H staging before BatchOffload; on per-slice failure
the entire object is skipped to prevent partial/corrupt data
- CMakeLists: auto-detect CUDAToolkit/HIP/Ascend independently of
global USE_CUDA flag, with explicit PRIVATE compile definitions
- CI: add -lcudart to Go test CGO_LDFLAGS when CUDA is present
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: expose batch_replica_clear in Python binding
Add batch_replica_clear(keys, segment_name) to PyClient/RealClient/DummyClient.
Allows explicit deletion of replicas for lease-expired keys.
Also adds Client::getClientId() public getter.
---------
Signed-off-by: hnts03-moreh <geonwoo.choi@moreh.io>
* fix(efa): add fi_read support and endpoint eviction to EFA transport
1. EFA submitPostSend() now branches on slice->opcode to call fi_read
or fi_write, matching RDMA transport behavior. Previously only
fi_write was implemented, breaking remote-read workloads on EFA.
2. EfaEndpointStore now tracks endpoint activity and evicts stale
endpoints when approaching AV capacity. This prevents AV exhaustion
in long-running services communicating with many transient peers.
- Endpoints marked active on access, inactive on set_active(false)
- Configurable inactive timeout (default 5 min)
- evictStale() removes endpoints inactive beyond timeout
- removeDisconnected() cleans up broken connections
- Eviction triggered automatically in getOrInsert() at capacity
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* perf(efa): eliminate per-slice overhead with NIC-count-based striping
Replace fixed 64KB slicing with NIC-count-based chunking in EFA
submitTransferTask. For large transfers (>128KB), creates one slice
per active NIC instead of thousands of small slices, dramatically
reducing spinlock, atomic, and allocation overhead.
- Large transfers: stripe across all active NICs (1 chunk per NIC)
- Small transfers: single slice on one NIC, no sub-slicing
- Fallback: per-slice retry for edge cases (unregistered memory)
Example: 240MB transfer on 32 NICs now creates 32 slices vs 3840,
matching NIXL approach of one fi_write per rail per descriptor.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): add FI_MR_HMEM to mr_mode hints for GPU memory registration
EFA provider requires FI_MR_HMEM in domain mr_mode hints to support
heterogeneous (GPU) memory registration via fi_mr_reg. Without this
flag, fi_mr_reg fails with EFAULT (Bad address) when registering
CUDA device memory buffers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style(efa): apply clang-format-20 formatting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): restore lost changes from merge — endpoint eviction, batched WR, docs
Restores 5 commits that were lost during the upstream merge:
- AV entry removal on disconnect to prevent target degradation
- Endpoint reuse for same physical peer (normalizeNicPath)
- Batched WR submission in submitPostSend
- Pre-resolved remote peer info for striped transfers
- Documentation: GPU-to-GPU (313 GB/s), CPU-to-CPU (222 GB/s) benchmarks,
CPU-only build instructions, B300/P5 instance types
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): use dependencies.sh for build deps, add EFA-specific extras
Replace manual package list with dependencies.sh script reference.
Add libgflags-dev as EFA-specific extra dependency.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add tip to use CUDA_VISIBLE_DEVICES for CPU-to-CPU benchmarks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add DLAMI troubleshooting for CUDA env and Python activation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add LIBRARY_PATH for CUDA libs, remove unneeded CUDAToolkit_ROOT
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add block_size=1MB to all benchmark configurations
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): merge clone step into build dependencies section
dependencies.sh already runs git submodule update --init --recursive,
so the separate clone section was redundant. Move git clone into the
Prerequisites section and renumber build steps.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): add SGLang usage, Docker troubleshooting, reorder sections
- Move Usage with vLLM/SGLang after benchmark results (was too early)
- Add Usage with SGLang section (EFA patch, env vars, Docker example)
- Add libfabric version mismatch in Docker troubleshooting
- Reorder p5en results: GPU-to-GPU before CPU-to-CPU (consistent with B200)
- Update p5en single-instance CPU results: Write 179 GB/s, Read 185 GB/s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): use GlobalConfig max_ep_per_ctx (65536) as default, round-robin CQ assignment
- Change EfaEndpointStore and EfaContext::construct default max_endpoints
from 256 to 65536, consistent with GlobalConfig.max_ep_per_ctx
- Add round-robin CQ assignment (nextCq) for new endpoints instead of
always using cq_list_[0], distributing load across CQs
- Pass CQ outstanding counter directly to endpoint construct
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style(efa): clang-format, move striping threshold to GlobalConfig
- Run clang-format on all EFA transport files and common.h
- Move MC_EFA_STRIPING_THRESHOLD from inline std::getenv to GlobalConfig
mechanism (loaded centrally in loadGlobalConfig), per reviewer feedback
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: add 'wqs' to typos ignore list (EFA API function name)
efadv_query_qp_wqs is a libfabric EFA API symbol, not a typo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add per-transfer latency benchmark script
Automated script that measures single-transfer latency across block sizes
by SSHing into target/initiator nodes and running transfer_engine_bench
with threads=1, batch_size=1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add --threads and --batch_size args to latency bench script
Allow configurable threads/batch_size for multi-thread bandwidth scaling tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(efa): add --env flag to pass env vars to remote bench via SSH
Allows setting MC_EFA_STRIPING_THRESHOLD and other env vars on the
remote initiator without manual SSH config.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: fix typos check by adding wqs to extend-words
The extend-ignore-words key is not recognized by typos v1.30.2 used
in CI. Add wqs (EFA API function name efadv_query_qp_wqs) to the
[default.extend-words] section which is the correct format.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: revert formatting-only changes in common.h
Revert pointer/reference style changes (`&`/`*` placement) that were
introduced by clang-format but are unnecessary for this PR. Keep only
the normalizeNicPath() function addition.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(efa): clarify that MC_SLICE_SIZE does not apply to EFA transport
EFA transport uses NIC-count-based striping (since commit 84d7dc5),
not fixed-size slicing. block_size directly determines per fi_write/
fi_read size for transfers below the striping threshold. Update docs
to reflect this and correct outdated tuning advice.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(efa): address review issues - nextCq guard, error path double-free, GPU MR registration
1. nextCq(): add empty guard to prevent division-by-zero when cq_list_ is empty
2. submitTransferTask error paths: replace getSliceCache().deallocate(s) with
s->markFailed() to prevent double-free (TransferTask destructor also deallocates)
and ensure tasks complete with FAILED status instead of hanging
3. registerMemoryRegionInternal(): use fi_mr_regattr() with explicit FI_HMEM_CUDA
iface and device ordinal for GPU memory, since EFA provider fi_mr_reg()
hardcodes iface=FI_HMEM_SYSTEM
4. Make total_bytes updates atomic (__sync_fetch_and_add) for thread safety
Tested: unit tests (5/5), cross-node GPU benchmark (write 250 GB/s, read 269 GB/s)
on p5en.48xlarge.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style(efa): apply clang-format to efa_context.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-8-212.us-west-1.compute.internal>
Co-authored-by: whn09 <whn09@github.com>
* [PG][TENT]: fix hang bug
* [fix]: code format
* Always kWildcardLocation for CPU and make MooncakeBarrierWorkCuda use BackoffWaiter
* CUDAStreamPool for TENT and non-blocking enq_stream for PG.
* Temporary workaround for deprecated getLocalTopology.
* [tent]: update getMachineID to use /proc/sys/kernel/random/boot_id to
verify
* code format
---------
Co-authored-by: caozhanhao <cao2013zh@163.com>
* build(tent): introduce tent_link_group for library linking
- Add tent_link_group interface library to manage tent and related dependencies
- Update CMake targets to use tent_link_group instead of tent for linking
- Implement link group with -Wl,--start-group and -Wl,--end-group to resolve
circular dependencies
Signed-off-by: staryxchen <staryxchen@tencent.com>
* style: reformat CMakeLists files with consistent indentation and spacing
- Apply consistent indentation (2 spaces) across all modified CMakeLists files
- Normalize conditional statement formatting (if/else/endif)
- Improve line wrapping for better readability in long commands
- Standardize target_link_libraries and other CMake command formatting
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
* feat(tent): replace raw RdmaEndPoint* with weak_ptr for lifecycle safety
RdmaSlice::ep_weak_ptr was a raw pointer despite its name, creating a
dangling-pointer risk when endpoints are evicted from the EndpointStore
cache while slices are still in-flight.
Changes:
- RdmaEndPoint now inherits std::enable_shared_from_this (endpoints are
already managed as shared_ptr in FIFOEndpointStore/SIEVEEndpointStore)
- RdmaSlice::ep_weak_ptr changed from RdmaEndPoint* to
std::weak_ptr<RdmaEndPoint>
- submitSlices() assigns via shared_from_this() instead of raw this
- All dereference sites in workers.cpp now call .lock() and gracefully
handle nullptr (endpoint already destroyed) by marking slices FAILED
- Add endpoint_lifecycle_test with 7 test cases verifying weak_ptr
semantics: lock-while-alive, expire-after-release,
shared_from_this, slice access patterns, multi-slice, and reset
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* perf(tent): hoist shared_from_this() out of submitSlices loop
Move the shared_from_this() call before the loop so we create a single
shared_ptr and assign it to each slice's weak_ptr, avoiding N redundant
atomic refcount increment/decrement pairs per batch submission.
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix(tent): decrement inflight_slices when endpoint expired on timeout
When a slice times out and its endpoint weak_ptr has already expired,
the inflight_slices counter was not decremented, causing the worker to
never enter suspension and skewing load balancing decisions.
Add fetch_sub(1) in the null-endpoint timeout path to keep the counter
consistent with the actual number of in-flight slices.
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously, NUMA-aware global segment allocation was gated behind
`!ipc_socket_path_.empty()`, meaning it only activated in standalone
(Dummy+Real Client) mode. In practice, the vast majority of
deployments (mooncake_store_service, sglang/vLLM integration, Python
store.setup()) use RealClient-only mode where ipc_socket_path is
always empty, so NUMA optimization never triggered.
Remove the ipc_socket_path condition so that all RDMA deployments
benefit from NUMA-segmented memory allocation and NUMA-aware NIC
routing via selectDevice.
* feat: enhance memory registration with transport type support
- Add transport type filtering to getSupportedTransports for specific
type requests instead of always returning all transports
- Return early with InvalidArgument when no transport is available
during registerLocalMemory
- Improve same_machine detection in getTransportType to handle
LOCAL_SEGMENT_ID and empty machine_id cases
- Simplify benchmark to use MemoryOptions with default UNSPEC type,
removing unnecessary branching
Signed-off-by: staryxchen <staryxchen@tencent.com>
* Refine TENT same-machine segment lookup
* refactor: improve code formatting for same_machine condition
- Align logical operators for better readability
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
* [TransferEngine] Wire up cross-transport failover with safety limits and observability
The existing resubmitTransferTask() had complete failover logic (increment
xport_priority, resolve next transport, resubmit) but was never called.
This commit activates the failover path and adds production safeguards:
- Wire getTransferStatus() to call resubmitTransferTask() on FAILED tasks
- Add failover_count to TaskInfo with configurable max_failover_attempts (default 3)
- Add transportTypeName() helper and structured LOG(INFO) for failover events
- Add tent_transport_failover_total Prometheus counter metric
- Add unit tests for failover state machine, config loading, and limit checks
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix: prevent failover from overwriting permanent FAILED in batch status
Move the failover attempt before status aggregation so that a
successfully resubmitted task appears as PENDING to the existing
if/else-if chain. Previously the code unconditionally set
overall_status.s = PENDING after a successful resubmit, which could
overwrite a permanent FAILED from another task in the same batch.
Now the `else if (task_status.s != PENDING)` branch naturally skips
PENDING tasks, and only truly-failed tasks set overall_status to
FAILED.
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
zbtrs2