* [Store] L2->L1 promotion-on-hit: Tier A observability + max_per_heartbeat knob
Adds Prometheus metrics for the promotion-on-hit funnel and exposes the
previously-hardcoded kMaxPerHeartbeat as a config knob.
Metrics (master_promotion_*)
----------------------------
Funnel:
- promotion_in_flight (gauge): mirror of promotion_in_flight_
- promotion_admitted_total: tasks past all gates, enqueued
- promotion_completed_total: NotifyPromotionSuccess success
- promotion_completed_bytes_total: bytes promoted (sum of source object_size)
- promotion_failed_total: NotifyPromotionFailure accepted
- promotion_expired_total: reaper Part 4 sweeps
Rejection (per gate):
- promotion_rejected_frequency_total: below admission threshold
- promotion_rejected_watermark_total: DRAM at or above high watermark
- promotion_rejected_cap_total: promotion_in_flight at queue limit
Together: admitted = completed + failed + expired + in_flight. Wires a
working 'promotion feature health' Grafana panel: rate(admitted) vs
rate(completed) shows yield; rate(rejected_*) breaks down where work
is dropping; in_flight / promotion_queue_limit shows saturation.
promotion_max_per_heartbeat knob
--------------------------------
The previous compile-time constant kMaxPerHeartbeat = 1 (in
PromotionObjectHeartbeat) capped per-client throughput at ~6
promotions/min with the default 10s heartbeat, making
promotion_queue_limit = 50000 mostly theoretical.
Exposed as MasterServiceConfig::promotion_max_per_heartbeat, wired
through the flag parser (FLAGS_promotion_max_per_heartbeat) and
default_config.GetUInt32 path. Constructor clamps 0 -> 1 so a
mistyped config doesn't silently halt promotion delivery. Startup
log now includes max_per_heartbeat= alongside threshold= /
queue_limit=.
Default stays 1 (no behavior change on existing deployments).
Operators with small objects + RDMA-rich clusters can raise it; the
new MaxPerHeartbeatKnobControlsBatchSize test exercises 3.
Tests
-----
- MetricsFunnelTracksSuccessfulPromotion: single full lifecycle
bumps admitted/completed/completed_bytes correctly and brings
in_flight back to baseline.
- MetricsRejectionCountersIncrementOnGateMiss: each of frequency
and cap counters increments when its branch fires.
- MaxPerHeartbeatKnobControlsBatchSize: knob=3, 5 admitted tasks
drain across 3+2+0 heartbeats.
- MaxPerHeartbeatZeroClampsToOne: pathological config clamps to 1.
Suite: 32/32 pass (was 28; +4 new). No behavior change in existing
tests.
* [Store] L2->L1 promotion-on-hit: cover watermark gate in rejection test
MetricsRejectionCountersIncrementOnGateMiss claimed coverage of all
three rejection counters but only exercised frequency and cap. Add a
sub-case that forces the watermark gate by configuring
eviction_high_watermark_ratio = 0.0, asserting that
promotion_rejected_watermark_total increments.
* [Store] L2->L1 promotion-on-hit: cover RemoveAll/BatchRemove cleanup
upstream #2180 introduced EraseMetadataEntry as the centralized
metadata-erase helper and routed RemoveAll, BatchRemove, and
RemoveByRegex through it, so promotion_tasks cleanup on those paths
is already correct on main. The metric instrumentation
(dec_promotion_in_flight + inc_promotion_cancelled) is wired into
EraseMetadataEntry in this branch's earlier commit so every site
that erases metadata bumps the funnel counters consistently.
Add regression tests for the three paths so any future refactor
that reintroduces a metadata.erase without going through
EraseMetadataEntry will fail the suite:
- RemoveAllErasesPromotionTask
- BatchRemoveErasesPromotionTask (normal-completion branch)
- BatchRemoveStaleHandleErasesPromotionTask
* [Store] L2->L1 promotion-on-hit: bump reaper-test sleep margin to 3s
Three tests configure put_start_release_timeout_sec=1 and then sleep
2s waiting for the eviction-thread reaper to expire the promotion
task. The reaper schedule is `now - last_discard_time >
put_start_release_timeout_sec_` (strict greater-than), so a 1s
release with a 2s sleep leaves only ~1s margin between the reaper
firing and the assertion. CI runs observed intermittent failures
when scheduling jitter erased that margin.
Bump the sleep to 3s in the three affected tests so the margin is
~2s. Configuration values unchanged.
Affected:
- StalePromotionReaper
- RemoveDuringPromotion
- AllocStartRejectsReapedTask
Verified 5 consecutive clean runs of all three under -j1 build.
- mooncake-transfer-engine: add parentheses around && within ||, add
static_cast for narrowing, mark unused function [[maybe_unused]]
- mooncake-store: fix member reorder warnings, add std::ignore for
unused results, fix missing field initializers, mark unused variables
- mooncake-integration: fix sign-compare comparison, mark unused
functions [[maybe_unused]]
- All fixes are semantic-preserving (no behavior changes)
* fix(store): join dummy_client_monitor_thread on shutdown to prevent std::terminate
The dummy_client_monitor_thread_ (std::thread) was started in
start_dummy_client_monitor() but never joined or detached. On
~RealClient(), the still-joinable std::thread triggers std::terminate(),
crashing standalone mooncake_client on every clean shutdown.
Add stop_dummy_client_monitor() following the same pattern as
stop_ipc_server(): set the running flag to false, then join. The call
is placed in tearDownAll_internal() before stop_http_server() and before
the dummy_client_mutex_ lock, avoiding the early-return skip and the
lock-then-join deadlock.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* feat: add Hygon DCU/DTK and Iluvatar CoreX platform support
Add build system and runtime support for two CUDA-compatible domestic
accelerator platforms:
- Hygon DCU with DTK SDK (USE_HYGON, /opt/dtk/cuda/cuda-11/)
- Iluvatar CoreX SDK (USE_COREX, /usr/local/corex/)
Both platforms expose CUDA-compatible APIs, so the integration follows
the same pattern as existing CUDA-like platforms (MUSA, MACA): add the
new macros to all platform guard chains and register SDK paths in CMake.
---------
Co-authored-by: KarmaD7 <KarmaD7@users.noreply.github.com>
* [Store] fix: require same-process endpoint for LOCAL_MEMCPY strategy
isLocalTransfer compared only the IP of handle.transport_endpoint_ to
the local endpoint, so two processes on the same host (same IP, different
ports) were treated as LOCAL_MEMCPY-eligible. The memcpy worker then
dereferenced handle.buffer_address_, which is a virtual address only valid
in the owning process, and segfaulted inside __memcpy_avx512_unaligned_erms.
This was latent before #1936 (MC_STORE_MEMCPY defaulted to off). The
TCP-only auto-enable exposed it on multi-process workloads such as the
TorchSpec inference/trainer pipeline.
Compare the full transport endpoint instead, matching the check already
used by Client::IsReplicaOnLocalMemory. Cross-process same-host transfers
now correctly fall through to TRANSFER_ENGINE; same-process transfers
still take the memcpy fast path.
Fixes the crash reported with MC_STORE_MEMCPY auto-enabled on TCP-only hosts.
---------
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
* feat: add ObjectDataType enum and metadata propagation (Phase 1 of #1673)
Introduce a data type classification system for objects stored in
Mooncake Store, as agreed in the RFC discussion on issue #1673.
Changes:
- Add ObjectDataType enum (UNKNOWN, KVCACHE, TENSOR, WEIGHT, etc.)
in mooncake-store/include/types.h
- Extend ReplicateConfig with a data_type field (default UNKNOWN)
- Propagate data_type through PutStart into ObjectMetadata
- Serialize/deserialize data_type in snapshot metadata, with backward
compatibility for old snapshots (gracefully handles missing field)
- Expose ObjectDataType enum and data_type field in Python bindings
- Add unit tests for enum values, defaults, and PutStart propagation
Existing clients that don't set data_type will continue to work
unchanged (defaults to UNKNOWN everywhere).
---------
Co-authored-by: Yufeng He <40085740+universeplayer@users.noreply.github.com>
* fix(store): enable local memcpy for metadata local replicas
Pass the client local hostname into TransferSubmitter and use it when
detecting local transfers. This allows metadata-service descriptors, which
use the hostname as the segment identifier, to correctly select LOCAL_MEMCPY
for local reads while preserving transfer-engine endpoint matching for P2P
descriptors.
Add coverage for TCP local memcpy auto-enable behavior across P2P and
metadata modes, including remote same-host cases that should continue using
TRANSFER_ENGINE. Also verify hot-cache hits do not increment the admission
sketch when LOCAL_MEMCPY is selected.
The SpinLock::lock() inner spin loop used memory_order_relaxed for
flag.test(), which can fail to promptly observe unlock() on ARM/RISC-V.
- Use memory_order_acquire on test_and_set in both fast and slow paths
- Keep relaxed loads only inside the inner PAUSE() spin loop for performance
- This ensures proper happens-before synchronization on weakly-ordered CPUs
* support engram
Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
* add test case for engram
Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
* add docs for engram
Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(master-metrics): track local SSD storage in Master metrics
Master logs always showed "SSD Storage: 0 B / 0 B" even when local SSD
offloading was active. Three root causes fixed:
Bug 1 — allocated size never tracked for LocalDiskReplica (replica.h)
The LocalDiskReplica constructor did not call inc_allocated_file_size(),
so the numerator stayed zero regardless of how many objects were offloaded.
Symmetric fixes applied to the destructor and move-assignment operator.
Also fixed a pre-existing bug: id_ and refcnt_ were uninitialized in the
LocalDiskReplica constructor.
Bug 2 — total SSD capacity unknown to Master
Master has no direct visibility into client-side SSD configuration, so
file_total_capacity_ (the denominator) was always 0. A new dedicated RPC
ReportSsdCapacity(client_id, ssd_total_capacity_bytes) is added; clients
call it once in FileStorage::Init() after MountLocalDiskSegment succeeds.
Master stores the value per-client in LocalDiskSegment::ssd_total_capacity_bytes
and updates MasterMetricManager via inc/dec on change. Old clients that
lack this RPC simply never call it — OffloadObjectHeartbeat signature is
unchanged, ensuring backward compatibility.
Bug 3 — data race in UnmountLocalDiskSegment (segment.cpp)
ssd_total_capacity_bytes was read without holding offloading_mutex_ while
OffloadObjectHeartbeat writes it under that lock (C++ UB). Fixed by reading
inside a scoped lock block, then releasing the lock before erase() to avoid
unlocking an already-destroyed mutex.
* Update mooncake-store/src/master_service.cpp
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [Store] Add lock-free MmapArena allocator for buffer mmap path
Replace per-allocation mmap() syscalls in allocate_buffer_mmap_memory()
with a lock-free atomic bump allocator (MmapArena). Pre-allocates a
configurable pool (default 64GB) and serves allocations via CAS loop,
reducing allocation latency from ~1us (mmap syscall) to ~50ns (atomic).
Allocation lifecycle is static: all callers (ClientBufferAllocator,
global segments in RealClient::setup_internal) allocate at startup and
free at shutdown. The arena outlives all allocations, so the bump-only
(no individual free) design is correct for this usage pattern.
Feature-flagged via gflags:
--use_mmap_arena_allocator (default: true)
--mmap_arena_pool_size (default: 64GB)
Falls back to direct mmap() when arena is disabled, fails to init,
or is exhausted.
Cherry-picked from flow-ipc-poc branch (utils.cpp perf path only).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Store] Fix three correctness issues in MmapArena
1. Honor caller's alignment contract: allocate() now accepts a
per-call alignment parameter and uses max(arena default, caller
request). allocate_buffer_mmap_memory() forwards its alignment
argument to the arena. Previously, the caller's alignment was
silently ignored — the arena always used 64-byte alignment
regardless of what the caller requested.
2. Remove MAP_POPULATE from arena pool mmap: the default pool is
64GB but callers typically use only a fraction (e.g. 4GB of
segments). MAP_POPULATE would pre-fault all 64GB of pages upfront,
causing seconds of startup delay and potentially triggering OOM
on machines with less physical memory. Pages now fault on demand.
3. Make alignment_ atomic and store it BEFORE the CAS on pool_base_:
previously alignment_ was a plain size_t written AFTER the release
CAS, so the store was not in the happens-before relationship
established by the acquire-release pair on pool_base_. Now both
alignment_ and pool_size_ are stored before the CAS with the
release fence guaranteeing their visibility to readers.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [Store] Fix LOCAL_MEMCPY segfault in multi-process and GPU memory scenarios
Two bugs caused segfaults when MC_STORE_MEMCPY was enabled with
multiple processes on the same node (e.g. vLLM DP8 + mp executor):
1. isLocalTransfer() compared only IPs (stripping ports via
extractIpAddress), so all processes on the same host were
incorrectly identified as "local". Cross-process virtual
addresses are invalid → segfault. Fix: compare full ip:port
endpoint, restoring the correct behavior from before #1226.
2. workerThread() used bare std::memcpy which cannot operate on
cudaMalloc device pointers from CPU code → segfault. Fix:
detect GPU pointers via IsDevicePointer() and use cudaMemcpy
(cudaMemcpyDefault) for GPU memory, keeping std::memcpy for
the CPU-only fast path.
Add gpu_staging::CopyAuto() to gpu_staging_utils.h that uses
cudaMemcpyDefault/hipMemcpyDefault for auto-direction copy.
* Modify code format
Signed-off-by: LCAIZJ <leichao139636@163.com>
---------
Signed-off-by: LCAIZJ <leichao139636@163.com>
Co-authored-by: leichao.lc <leichao.lc@antgroup.com>
* [Store] add tensor object metadata and TP upsert APIs
Introduce explicit tensor object metadata for tensor read/write paths, add TP-aware tensor upsert wrappers, and update tests/docs for the new serialized layout.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
PutToLocalFile and FileStorage::OffloadObjects crash with SIGSEGV when
slice.ptr points to GPU device memory, because CPU memcpy cannot access
GPU virtual addresses. The RDMA memory-replica path is unaffected.
Add synchronous Device-to-Host staging via PinnedBufferPool before data
reaches the disk-write paths:
- New gpu_staging_utils.h: shared IsDevicePointer/CopyDeviceToHost/
SetDevice helpers with cross-platform support (CUDA/HIP/MUSA/MACA/
Ascend CANN)
- New PinnedBufferPool: thread-safe pinned host memory pool with
max capacity limit (default 32) and O(1) swap-pop acquire
- PutToLocalFile: sync D2H on calling thread, PutRevoke on failure
- OffloadObjects: D2H staging before BatchOffload; on per-slice failure
the entire object is skipped to prevent partial/corrupt data
- CMakeLists: auto-detect CUDAToolkit/HIP/Ascend independently of
global USE_CUDA flag, with explicit PRIVATE compile definitions
- CI: add -lcudart to Go test CGO_LDFLAGS when CUDA is present
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: expose batch_replica_clear in Python binding
Add batch_replica_clear(keys, segment_name) to PyClient/RealClient/DummyClient.
Allows explicit deletion of replicas for lease-expired keys.
Also adds Client::getClientId() public getter.
---------
Signed-off-by: hnts03-moreh <geonwoo.choi@moreh.io>
* [Store] add get_into_range and src_offset plumbing
Add single-key range read support to Mooncake Store:
- get_into_range: read [src_offset, src_offset+size) from an object
into (buffer + dst_offset)
- submitRangeRead in TransferSubmitter with src_offset support
- Get() overload and TransferReadRange in Client
- DummyClient RPC path via get_into_range_dummy_helper
- PyClient virtual interface for get_into_range
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [Store] Add OpLog abstraction layer with multi-backend support for HA
Introduce a complete OpLog infrastructure for hot-standby replication,
extracted as a standalone layer with no changes to master_service.
New components:
- OpLogStore interface + EtcdOpLogStore / LocalfsOpLogStore backends
- OpLogChangeNotifier interface + Etcd watch / Polling implementations
- OpLogStoreFactory for backend selection at runtime
- OpLogSerializer for binary serialization/deserialization of entries
- OpLogManager: in-memory buffer with global sequence-id allocation
- OpLogApplier: ordered application with gap handling
- OpLogReplicator: orchestrates notifier + applier on standby side
- HaMetricManager: HA observability metrics
Supporting additions:
- ErrorCode::OPLOG_ENTRY_NOT_FOUND
- Unit tests for all new components
- Integration tests: localfs_hot_standby_integration_test
* style(store): run code format after rebase
* fix(store): repair etcd oplog batch flush after rebase
* fix by comments
* Restore fencing test to match batch write behavior
The batching write path falls back to per-key Put on BatchCreate
transaction failure, which does not detect same-seq-different-content
conflicts. Restore the original comment from main instead of asserting
ETCD_OPERATION_ERROR.
* Move oplog files under ha/oplog/ subtree
Oplog is part of the HA runtime. Move all oplog-related headers and
sources from the top-level include/ and src/ into ha/oplog/ to align
with the existing ha/ directory structure (leadership/, snapshot/).
Also remove the unused include/ha/oplog_store.h interface that has no
references anywhere in the codebase.
* Fix oplog include paths for files added on main after rebase
master_service.cpp and snapshot_child_process_test.cpp gained
etcd_oplog_store.h includes on main while this branch was in flight.
Update them to the new ha/oplog/ path.
---------
Co-authored-by: haodedu <haodedu@tencent.com>
LZW
Yeuvoir