* [Store] Add lock-free MmapArena allocator for buffer mmap path
Replace per-allocation mmap() syscalls in allocate_buffer_mmap_memory()
with a lock-free atomic bump allocator (MmapArena). Pre-allocates a
configurable pool (default 64GB) and serves allocations via CAS loop,
reducing allocation latency from ~1us (mmap syscall) to ~50ns (atomic).
Allocation lifecycle is static: all callers (ClientBufferAllocator,
global segments in RealClient::setup_internal) allocate at startup and
free at shutdown. The arena outlives all allocations, so the bump-only
(no individual free) design is correct for this usage pattern.
Feature-flagged via gflags:
--use_mmap_arena_allocator (default: true)
--mmap_arena_pool_size (default: 64GB)
Falls back to direct mmap() when arena is disabled, fails to init,
or is exhausted.
Cherry-picked from flow-ipc-poc branch (utils.cpp perf path only).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Store] Fix three correctness issues in MmapArena
1. Honor caller's alignment contract: allocate() now accepts a
per-call alignment parameter and uses max(arena default, caller
request). allocate_buffer_mmap_memory() forwards its alignment
argument to the arena. Previously, the caller's alignment was
silently ignored — the arena always used 64-byte alignment
regardless of what the caller requested.
2. Remove MAP_POPULATE from arena pool mmap: the default pool is
64GB but callers typically use only a fraction (e.g. 4GB of
segments). MAP_POPULATE would pre-fault all 64GB of pages upfront,
causing seconds of startup delay and potentially triggering OOM
on machines with less physical memory. Pages now fault on demand.
3. Make alignment_ atomic and store it BEFORE the CAS on pool_base_:
previously alignment_ was a plain size_t written AFTER the release
CAS, so the store was not in the happens-before relationship
established by the acquire-release pair on pool_base_. Now both
alignment_ and pool_size_ are stored before the CAS with the
release fence guaranteeing their visibility to readers.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [Store] Add hard pin mechanism for eviction-protected objects
Objects created with ReplicateConfig.with_hard_pin=true are never
evicted by the eviction policy, providing guaranteed persistence
for model weights in RL and model management workloads.
Changes:
- ReplicateConfig: add with_hard_pin field (default false)
- ObjectMetadata: add hard_pinned boolean, set at creation via PutStart
- BatchEvict: skip hard-pinned objects in all eviction passes
- Serialization: persist hard_pinned in snapshots (backward compatible
with old format that lacks the field)
- Tests: verify hard-pinned objects survive eviction, coexist with
soft pin, and can still be explicitly removed
* [Store]: add task executor feature with unit and executor test
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: add some optimizations to task executor
Refactor the task_executor into the client_service and add a
task structure on the client side. Additionally, remove the
existence check in the execute function;
only retrying should be performed if replica allocation fails.
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: get source replica from copyStart or moveStart api
* [Store]: call move or copy end if the target replica already exist to complete the replication task
* [Store]: use the max_retry_attempts in master side
* [Store]: add client integration test and set default max_retry_attempts to 10
* [Doc] update task api introduction
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store] Set copy and move as private methods
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Doc]: change the default task max_retry_attempts to 10
* [Doc]: fix some description error
* [Store]: allocate the buffer size to be a multiple of 16MB
* [Store]: validate the replica is in local and directly construct slices from replica buffer address instead of copy the data to local buffer
* [Store]: change the validate logic to directly use transfer engine endpoint or local_hostname_.
* [Store] add e2e ci test for copy and move api
* [Store] refactor the client move and copy function
* [Store] fix the e2e test
* [Store] remove unused code
* [Store] add source field when build replica copy payload in the task_manager_test
* [Store] rename back to snake case for split_into_slices function and also remove hard code for client poll count
* [Store] revert mis deleted field when resolve conflicts
* [Store] change test to validate the real behaviour
* [Store] change the default task fetch size to 16
* [Doc]: change the replica copy/move sequence diagram
* [Store] add new split_into_slice method
* [Store] change the real client to use split_to_slice with buffer handle parameters
---------
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
Co-authored-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store] add HugePage support
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* doc: add huge page env introduction
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
---------
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
* [Store] feat: Add BatchReplicaClear API for manual cache cleanup
Implement the `BatchReplicaClear` API to allow clients to manually clear
cached object replicas. This feature enables explicit storage management
by deleting replicas for specified keys. The implementation is end-to-end,
from the client to the master service.
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store] feat: Fix BatchReplicaClear monitor issues
Fixed issues with monitoring metrics not being updated and function naming.
Fix the issue of unclear descriptions in the file.
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
---------
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store] feat: Add BatchQueryIp API for querying multiple client IPs
Add BatchQueryIp API with client test and master server test
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Doc] Add API documents for BatchQueryIp
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
---------
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: Add independent deployment implementation for Client
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Doc]: mooncake-store: add introduce for client standalone mode
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [CI]: Add dummy client test
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Store]: Change dummy client setup into a new func
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [Store]: Add more log for dummy client
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
---------
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* feat(store): add client heartbeat support for non ha mode
This change extends client functionality to handle master reconnection and segment remounting in non-HA mode, adds corresponding test infrastructure, and improves utility functions for HTTP operations and port allocation.
Signed-off-by: xiaguan <751080330@qq.com>
* refactor(tests): reorganize test entries in CMakeLists.txt
* update docs
---------
Signed-off-by: xiaguan <751080330@qq.com>
Co-authored-by: Ke Yang <oneday117@qq.com>
Dmitry Barsukoff