* [Store] Add OpLog abstraction layer with multi-backend support for HA
Introduce a complete OpLog infrastructure for hot-standby replication,
extracted as a standalone layer with no changes to master_service.
New components:
- OpLogStore interface + EtcdOpLogStore / LocalfsOpLogStore backends
- OpLogChangeNotifier interface + Etcd watch / Polling implementations
- OpLogStoreFactory for backend selection at runtime
- OpLogSerializer for binary serialization/deserialization of entries
- OpLogManager: in-memory buffer with global sequence-id allocation
- OpLogApplier: ordered application with gap handling
- OpLogReplicator: orchestrates notifier + applier on standby side
- HaMetricManager: HA observability metrics
Supporting additions:
- ErrorCode::OPLOG_ENTRY_NOT_FOUND
- Unit tests for all new components
- Integration tests: localfs_hot_standby_integration_test
* style(store): run code format after rebase
* fix(store): repair etcd oplog batch flush after rebase
* fix by comments
* Restore fencing test to match batch write behavior
The batching write path falls back to per-key Put on BatchCreate
transaction failure, which does not detect same-seq-different-content
conflicts. Restore the original comment from main instead of asserting
ETCD_OPERATION_ERROR.
* Move oplog files under ha/oplog/ subtree
Oplog is part of the HA runtime. Move all oplog-related headers and
sources from the top-level include/ and src/ into ha/oplog/ to align
with the existing ha/ directory structure (leadership/, snapshot/).
Also remove the unused include/ha/oplog_store.h interface that has no
references anywhere in the codebase.
* Fix oplog include paths for files added on main after rebase
master_service.cpp and snapshot_child_process_test.cpp gained
etcd_oplog_store.h includes on main while this branch was in flight.
Update them to the new ha/oplog/ path.
---------
Co-authored-by: haodedu <haodedu@tencent.com>
With a 16MB pool (= 1 slab, since Slab::kSize = 16MB), a startup race
in MemoryPool::allocate() can cause spurious nullptr returns: the thread
assigning the slab increments currSlabAllocSize_ before calling
addSlabAndAllocate(), causing other threads to see
allSlabsAllocated()=true and bail out immediately. Using 32MB (2 slabs)
ensures allSlabsAllocated() remains false during the first slab
assignment, allowing other threads to enter the slow path and wait.
The write_thread_pool_ has 2 threads, so concurrent PutToLocalFile calls
can complete in arbitrary order, causing the FIFO eviction queue to not
match the logical Put order. Wait for each key's DISK replica before
putting the next to guarantee key_0 is always the oldest entry.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [STORE] abstract snapshot catalog in master service
Introduce SerializerSnapshotStore as the snapshot catalog adapter for the existing serializer backend and route MasterService persist, restore, and cleanup flows through it instead of open-coding latest marker updates and snapshot directory scans.
Also add focused unit coverage for publish/get/list/delete behavior so the snapshot catalog path can evolve independently from the payload storage path.
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] distinguish missing snapshots from backend read errors
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] add Redis snapshot catalog backend
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] share Redis connection helpers across HA backends
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
Co-authored-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
* [STORE] unify Redis test helpers
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] tighten snapshot catalog state handling
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
---------
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Store] Add hard pin mechanism for eviction-protected objects
Objects created with ReplicateConfig.with_hard_pin=true are never
evicted by the eviction policy, providing guaranteed persistence
for model weights in RL and model management workloads.
Changes:
- ReplicateConfig: add with_hard_pin field (default false)
- ObjectMetadata: add hard_pinned boolean, set at creation via PutStart
- BatchEvict: skip hard-pinned objects in all eviction passes
- Serialization: persist hard_pinned in snapshots (backward compatible
with old format that lacks the field)
- Tests: verify hard-pinned objects survive eviction, coexist with
soft pin, and can still be explicitly removed
* store: split client HA/control-plane threads and suppress zero-seg heartbeats
Refactor the store client control plane so leader monitoring, storage
heartbeat, and task polling are managed separately, and route leader
switching through a single serialized SwitchLeader() path.
This changes zero-global-segment behavior so setup no longer starts the
storage heartbeat/task polling control plane when no segment is mounted,
which avoids flooding master with useless ping traffic from zero-seg
clients. The storage control plane now starts lazily from actual mount
paths.
Also add regression and smoke coverage for:
- zero-seg clients not pinging before mount
- heartbeat starting after mount
- non-HA reconnect/remount behavior
- zero-seg HA smoke and non-zero ping/fetch-task smoke
Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
* format
Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
* fix
Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
---------
Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
Add the Redis leader coordinator behind the HA backend abstraction and wire it into the store and test builds so Redis can participate as a first-class leadership backend alongside etcd.
Also make the HA/e2e test harness backend-agnostic, add Redis leadership regression tests, and fix the client test wrapper to pass HA master entries correctly so the chaos suite exercises the real HA path.
---------
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] introduce HA backend abstraction
- add HA backend types, coordinator/oplog/snapshot interfaces and etcd baseline
- migrate master supervisor and client HA path to the new coordinator layer
- remove legacy ha_helper and migrate HA tests to LeaderCoordinator
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] optimize address HA backend
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] split supervisor warmup and serve phases
Split leadership startup into an explicit warmup phase and a serve phase so the supervisor no longer stops the RPC server before async_start() is entered.
Also factor the repeated release/retry paths into small helpers to keep the leadership lifecycle readable while preserving the existing HA semantics.
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] add leadership loss reason to serve monitor
Switch the serve-phase monitor to a callback so leadership observation stays separate from the concrete shutdown action.
Thread explicit leadership loss reasons through the callback so the supervisor can distinguish renewal errors from normal leadership loss in logging and follow-up policy.
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] move leadership monitoring into HA backend
Introduce leadership monitor callback/handle in the HA abstraction, let the etcd backend own serve-phase liveness monitoring, and add etcd regression coverage.
Also add a keepalive-ready handshake in the etcd wrapper to close the startup race between keepalive registration and cancellation.
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
---------
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
Co-authored-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
* add /metrics and /metrics/summary HTTP endpoints to RealClient
* add integration tests for /metrics and /metrics/summary endpoints
* apply clang-format to metrics endpoint handlers
* add metrics data correctness test with put/get verification
Merge the metrics endpoint test and a new transfer stats verification
test into a single test case to avoid RealClient setup/teardown
resource contention that caused segfaults with 8 sequential tests.
The combined test verifies:
- /metrics and /metrics/summary return 200 before any transfers
- After put/get, Prometheus output contains write_bytes, read_bytes,
put_latency_count, and get_latency_count
- Summary output shows Put and Get sections
---------
Co-authored-by: haodedu <haodedu@tencent.com>
* [Store] enable dummy client zero-copy get_buffer via shared hot cache
- Extract ShmHelper into standalone shm_helper.h/cpp for reuse
- Convert LocalHotCache allocation from malloc to memfd (use_shm mode)
- Add IPC protocol for dummy clients to request hot cache fd from real client
- Add acquire/release RPC for hot cache (ref_count based lifecycle)
- Add acquire/release RPC for allocator-backed buffers (dummy path)
- Add batch variants for hot cache and allocator acquire RPCs
- Extend BufferHandle with view mode (non-owning, custom release callback)
- Implement DummyClient::get_buffer with hot cache fast path + allocator fallback
- Unify store_py.cpp get() to use get_buffer for both real and dummy clients
- Remove obsolete get_buffer_info virtual interface
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Store] enable dummy client get_buffer/batch_get_buffer in Python bindings
Remove use_dummy_client_ guards that blocked get_tensor, batch_get_tensor,
and batch_get_buffer for dummy clients, now that dummy supports these APIs
via shared hot cache + allocator fallback.
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Store] fix batch_get_buffer_internal to use dummy's allocator
batch_acquire_buffer_dummy was calling batch_get_buffer_internal without
passing the dummy's allocator, causing buffers to be allocated in the
real client's memory instead of the dummy's shm region. Add optional
client_buffer_allocator parameter to batch_get_buffer_internal and pass
it from batch_acquire_buffer_dummy.
Also add is_hot_cache_ptr() to DummyClient for verifying whether a
pointer falls within the hot cache shm region.
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Store] add dummy client get_buffer/batch_get_buffer unit tests
Test correctness, hot cache shm path, allocator fallback, and
performance with 500 MB per key, 7 GB batch.
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Store] allow local data into hot cache when shm mode is active
In shm mode the hot cache is shared with dummy clients via IPC, so
local data must also be cached for zero-copy access. Add IsShm()
accessor to LocalHotCache and gate the local-skip filter in
ProcessSlicesAsync on non-shm mode only.
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Store] rename hot cache env vars to MC_STORE_ prefix
Align LOCAL_HOT_CACHE_SIZE, LOCAL_HOT_BLOCK_SIZE, and
LOCAL_HOT_CACHE_USE_SHM with the project-wide MC_STORE_ naming
convention. Also use IsReplicaOnLocalMemory() for correct local
detection under P2PHANDSHAKE mode, and gate hot cache shm mode
behind MC_STORE_LOCAL_HOT_CACHE_USE_SHM=1 (default off).
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Store] real client: add bounds check, IPC auth, recv timeout, lambda fix
- Validate offset + size <= hot_cache_size_ in get_buffer/batch_get_buffer
- Authenticate client_id in handle_ipc_shm_fd_request against registered dummies
- Add SO_RCVTIMEO (5s) on IPC server client sockets to prevent blocking
- Fix lambda in batch_get_buffer to capture key directly instead of growing vector
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Store] reduce test payload sizes for CI memory constraints
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
---------
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Store] feat: CXL storage full features.
* fix(ci): resolve cxl test failure and code format
* fix(ci): add CXL protocol support and fix code format issues
* [Store] feat: CXL storage full features, reset and rm extern/pybind
* [TE] Support TCP fallback in EFA build and improve EFA documentation (#1523)
When building with USE_EFA=ON, auto_discover is disabled to prevent
RDMA transport installation (QP creation fails on EFA devices). This
means TCP transport is also not installed automatically. Add explicit
TCP transport installation for non-EFA protocols in the EFA build path.
Documentation changes:
- build.md: Add USE_EFA option and clarify USE_CUDA default/purpose
- supported-protocols.md: Add EFA as a supported protocol
- efa_transport.md: Add USE_CUDA=ON to build command, document GPU
memory requirement
Co-authored-by: whn09 <whn09@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
* [Store] Optimize BucketStorageBackend for reduced lock contention and add delete safety (#1456)
* fix(ci): resolve cxl test failure and code format
* [Store] Add Local Cache Mechanism for Mooncake Store Client (#1226)
* feat(Store): add local hot cache for client
* feat(Store): add client local hot cache log to show performance
* fix: local hot cache initialize bug
* fix(Store): Mooncake put slice is max 16MB， so make local hot cache block 16MB
* feat(Store): move local hot cache initialization to Client::Create
* feat(Store): local hot cache remove unused small block implementation
* feat(Store): add client local hot cache unit test
* fix(Store): modify client local hot cache suit with v0.3.7
* feat(Store): change local hot cache unit tes
* fix: initialize local hot cache with negative value
* feat: use in process master and metadata fro local hot cache unit test.
* feat: update local hot cache to one replica one slice version
* fix: local hot cache unit test use in process master service
* fix: code style fix
* fix: fix dirty read when client wants to read a previously hitted hot block but the hot block is modified by incoming put actions
* fix: local hot cache unit test use in process master service
* fix: code format fix
* fix: fix comment problems for
* feat: add local hot asynchronous queue size limit
* fix: local hot cache task involves the block so that there is no memcpy operation when inserting local hot cache
* fix: code check fix
* fix: update block in_use prop to reference count
---------
Co-authored-by: shichangzhang064 <zhangshichang@h-partners.com>
* fix(ci): add CXL protocol support and fix code format issues
* fix: address comments from code review
* fix(ci): resolve cxl test failure
* fix(ci): resolve ci error
---------
Co-authored-by: 王鹤男 <wanghenan09@gmail.com>
Co-authored-by: whn09 <whn09@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
Co-authored-by: Mahesh Bapatu <153306023+maheshrbapatu@users.noreply.github.com>
Co-authored-by: Shichang Zhang <77728761+Shichang-Zhang@users.noreply.github.com>
Co-authored-by: shichangzhang064 <zhangshichang@h-partners.com>
* feat(Store): add local hot cache for client
* feat(Store): add client local hot cache log to show performance
* fix: local hot cache initialize bug
* fix(Store): Mooncake put slice is max 16MB， so make local hot cache block 16MB
* feat(Store): move local hot cache initialization to Client::Create
* feat(Store): local hot cache remove unused small block implementation
* feat(Store): add client local hot cache unit test
* fix(Store): modify client local hot cache suit with v0.3.7
* feat(Store): change local hot cache unit tes
* fix: initialize local hot cache with negative value
* feat: use in process master and metadata fro local hot cache unit test.
* feat: update local hot cache to one replica one slice version
* fix: local hot cache unit test use in process master service
* fix: code style fix
* fix: fix dirty read when client wants to read a previously hitted hot block but the hot block is modified by incoming put actions
* fix: local hot cache unit test use in process master service
* fix: code format fix
* fix: fix comment problems for
* feat: add local hot asynchronous queue size limit
* fix: local hot cache task involves the block so that there is no memcpy operation when inserting local hot cache
* fix: code check fix
* fix: update block in_use prop to reference count
---------
Co-authored-by: shichangzhang064 <zhangshichang@h-partners.com>
* [Store]: add task executor feature with unit and executor test
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: add some optimizations to task executor
Refactor the task_executor into the client_service and add a
task structure on the client side. Additionally, remove the
existence check in the execute function;
only retrying should be performed if replica allocation fails.
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: get source replica from copyStart or moveStart api
* [Store]: call move or copy end if the target replica already exist to complete the replication task
* [Store]: use the max_retry_attempts in master side
* [Store]: add client integration test and set default max_retry_attempts to 10
* [Doc] update task api introduction
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store] Set copy and move as private methods
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Doc]: change the default task max_retry_attempts to 10
* [Doc]: fix some description error
* [Store]: allocate the buffer size to be a multiple of 16MB
* [Store]: validate the replica is in local and directly construct slices from replica buffer address instead of copy the data to local buffer
* [Store]: change the validate logic to directly use transfer engine endpoint or local_hostname_.
* [Store] add e2e ci test for copy and move api
* [Store] refactor the client move and copy function
* [Store] fix the e2e test
* [Store] remove unused code
* [Store] add source field when build replica copy payload in the task_manager_test
* [Store] rename back to snake case for split_into_slices function and also remove hard code for client poll count
* [Store] revert mis deleted field when resolve conflicts
* [Store] change test to validate the real behaviour
* [Store] change the default task fetch size to 16
* [Doc]: change the replica copy/move sequence diagram
* [Store] add new split_into_slice method
* [Store] change the real client to use split_to_slice with buffer handle parameters
---------
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
Co-authored-by: Vincent Gao <vincentbo@linux.alibaba.com>
* fixed odr
* fixed cmakelist format
* fixed cmakelist format
* formatted
* fix for ci
* fix for ci
* fix for ci
* fix for ci
* commit for ci
* fix for ci
* fix for ci
* fix for ci
* [Store] feat: add cxl storage for mooncake store
* Update extern/pybind11 to match main
* fix: use fake cxl device to bypass ci-test error
* Fix code formatting in segment.cpp
---------
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
* feat: Add deterministic test failure injection for partial success testing
- Add SetTestFailurePredicate() virtual method to StorageBackendInterface
- Implement test failure injection in StorageBackendAdaptor and OffsetAllocatorStorageBackend
- Refactor TestPartialSuccessBehavior helper to use predicate-based injection
- Remove repetitive key creation code (use loops)
- Add PartialSuccess tests for both Adaptor and OffsetAllocator backends
- Update OutOfSpace test to handle partial success semantics (value==0)
This enables deterministic testing of partial success behavior without
relying on approximate space exhaustion, making tests more reliable.
* Addressed review comments
* [Store]: initialize the basic task data structure
* [Store]: implement the task manager
* [Store]: change the mutex to custom mutext
* [Store]: implement copy and move, query task api
* [Store]: change the task manager lock to be shared
* [Store]: change the task to struct when submit
* [Store]: add fetch tasks api
* [Store]: add metrics for query and fetch task api
* [Store]: add update task status api
* [Store]: rename the updateTask api and add pending and processing task limit
* [Store]: expose the task manager paramters to master config
* [Store]: remove client id from client service
* [Store]: change the copy and move api definition to createCopyTask and createMoveTask.
* [Store]: add pending and processing task timeout support
* [Store]: add exposure api to real_client_main and also rename method to snake_case style
* [Store]: directly cast to int64_t
* [Store]: remove unused code and fix the api description
* [Store]: fix metrics issue for mark_task_to_complete
* [Store] Decouple master from transfer_engine dependencies
Fixes#992
Change transfer_engine linkage from PUBLIC to PRIVATE in mooncake_store library to prevent propagating its dependencies (e.g., Ascend, CUDA) to targets that don't need it. The master service only handles metadata management and doesn't perform data transfers, so it shouldn't require transfer engine dependencies.
Changes:
- Set transfer_engine as PRIVATE dependency in mooncake_store
- Explicitly link transfer_engine for mooncake_client binary
- Add transfer_engine to all test targets that use client functionality
- Add transfer_engine to e2e test targets
- corrected target_compile_options from mooncake_master to mooncake_client
Benefits:
- Reduces master binary size
- Eliminates unnecessary Ascend/CUDA dependencies for master
* Update mooncake-store/src/CMakeLists.txt
---------
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
lujh