- mooncake-transfer-engine: add parentheses around && within ||, add
static_cast for narrowing, mark unused function [[maybe_unused]]
- mooncake-store: fix member reorder warnings, add std::ignore for
unused results, fix missing field initializers, mark unused variables
- mooncake-integration: fix sign-compare comparison, mark unused
functions [[maybe_unused]]
- All fixes are semantic-preserving (no behavior changes)
* [Store] Add hard pin mechanism for eviction-protected objects
Objects created with ReplicateConfig.with_hard_pin=true are never
evicted by the eviction policy, providing guaranteed persistence
for model weights in RL and model management workloads.
Changes:
- ReplicateConfig: add with_hard_pin field (default false)
- ObjectMetadata: add hard_pinned boolean, set at creation via PutStart
- BatchEvict: skip hard-pinned objects in all eviction passes
- Serialization: persist hard_pinned in snapshots (backward compatible
with old format that lacks the field)
- Tests: verify hard-pinned objects survive eviction, coexist with
soft pin, and can still be explicitly removed
* [Store]: initialize the basic task data structure
* [Store]: implement the task manager
* [Store]: change the mutex to custom mutext
* [Store]: implement copy and move, query task api
* [Store]: change the task manager lock to be shared
* [Store]: change the task to struct when submit
* [Store]: add fetch tasks api
* [Store]: add metrics for query and fetch task api
* [Store]: add update task status api
* [Store]: rename the updateTask api and add pending and processing task limit
* [Store]: expose the task manager paramters to master config
* [Store]: remove client id from client service
* [Store]: change the copy and move api definition to createCopyTask and createMoveTask.
* [Store]: add pending and processing task timeout support
* [Store]: add exposure api to real_client_main and also rename method to snake_case style
* [Store]: directly cast to int64_t
* [Store]: remove unused code and fix the api description
* [Store]: fix metrics issue for mark_task_to_complete
* [Store] feat: Add BatchReplicaClear API for manual cache cleanup
Implement the `BatchReplicaClear` API to allow clients to manually clear
cached object replicas. This feature enables explicit storage management
by deleting replicas for specified keys. The implementation is end-to-end,
from the client to the master service.
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store] feat: Fix BatchReplicaClear monitor issues
Fixed issues with monitoring metrics not being updated and function naming.
Fix the issue of unclear descriptions in the file.
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
---------
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store] feat: Add BatchQueryIp API for querying multiple client IPs
Add BatchQueryIp API with client test and master server test
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Doc] Add API documents for BatchQueryIp
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
---------
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* Refactor(store): Remove BufStatus and segment_name for AllocatedBuffer
- Removed BufStatus enum and related functionality from AllocatedBuffer.
- Updated AllocatedBuffer constructor to eliminate segment_name parameter.
- Enhanced getSegmentName method to retrieve segment name from the allocator.
- Adjusted logging and tests to reflect changes in buffer status handling.
* fix fmt
* update comment
* feat(allocator): Add transport endpoint support for segment transfers
* refactor(Client): Replace segment_name with transport_endpoint in buffer descriptors and tests
* refactor(test): switch to in-proc master and remove external service management
* style: format code and adjust transfer task timeout
* feat(Client::MountSegment): set te_endpoint conditionally based on metadata_connstring for P2P mode
* Allow configuring default KV lease TTL via environment variable
Read DEFAULT_KV_LEASE_TTL environment variable at server startup and
parse as uint64_t value. This enables CI tests to override default
configuration without code changes.
Signed-off-by: xiaguan <751080330@qq.com>
* style: format CI config and add [[nodiscard]] to methods
Signed-off-by: xiaguan <751080330@qq.com>
* refactor(tests): remove local_hostname parameter and improve test helpers
The commit removes the unused local_hostname parameter from TransferSubmitter constructor, updates the associated initialization logic, and simplifies test code by introducing helper functions for segment creation. The changes improve code maintainability while preserving functionality.
* refactor(TransferSubmitter): Remove unused local_hostname_ member and validation check
* docs(store): update setup examples for p2p
---------
Signed-off-by: xiaguan <751080330@qq.com>
* refactor: introduce expected pattern for error handling in master service
- Replace ErrorCode return types with tl::expected<T, ErrorCode> pattern
- Improve error handling clarity by separating success values from error codes
- Update MasterService methods to return expected<void, ErrorCode> or expected<T, ErrorCode>
- Modify RPC service interfaces to support expected pattern
- Update all related tests to handle new expected return types
- Add necessary includes for ylt/util/expected.hpp
This change makes error handling more explicit and type-safe:
- Success cases can be accessed via .value()
- Error cases can be accessed via .error()
- Eliminates ambiguity between success and error states
Future work:
- Extend expected pattern to RPC response types
- Enhance error code system for more comprehensive error handling
* Refactor error handling to use expected<T,E> pattern instead of ErrorCode
- Updated MasterService methods to replace ylt::expected with tl::expected for better error handling.
- Modified BatchGetReplicaList, BatchPutStart, BatchPutEnd, and other methods to return tl::expected types.
- Enhanced ClientIntegrationTest to handle expected results from Put, Get, Remove, and other operations using tl::expected.
- Adjusted error handling in clientctl and master_metrics_test to utilize the new expected type.
- Improved overall error reporting in tests to provide clearer feedback on operation failures.
* fix test compile
* refactor: update client implementation and remove master.proto
- Enhanced client.h and client.cpp with new functionality
- Removed obsolete master.proto file
- Updated master_client.cpp and transfer_task.cpp
- Improved integration and stress tests
* fix: update Python integration to work with new batch API
- Replace BatchObjectInfo with vector<vector<Replica::Descriptor>>
- Update BatchPut to handle new return type vector<tl::expected<void, ErrorCode>>
- Fix BatchQuery API usage to work with new expected pattern
- Convert unordered_map to vector format for BatchPut parameter compatibility
* refine master log and metric
* fix(ci): should alloc first
* merge main
* fix tests
* enable client ssd offload and storage persistence
* add storage_root_path in all tests setup() initialization and add related description in doc
* clean up headers and improve code readability - Added consistent Doxygen-style comments to all header files - Removed redundant code and outdated comments - Optimized function execution logic in LocalFile
* Revert "add storage_root_path in all tests setup() initialization and add related description in doc"
This reverts commit 159442d44e.
revert old high-level api test and doc modification
* Restore the high-level API to its original state and modify it to introduce the storage path through environment variables.
* add local_file_test and thread_pool_test
* feat(client_ssd_offload): implement async writes and fix locking bugs
- Refactor write operations to use thread pool for async file I/O
- Fix potential double-unlock bug by adding atomic is_locked_ flag
- Add corrupted file cleanup on write failure:
- Auto-delete files with failed writes in destructor
- Prevent subsequent reads of corrupted data
* add support for remove , remove_all , isexist interface etc.
* feat(kvcache): implement cluster isolation with session IDs
* Remove precompilation parameters to simplify build configuration
* Add session ID mechanism for cluster isolation:
- Master node now generates unique session IDs on initialization
- All persistent operations are scoped under session-specific subdirectories
* edit two parameters client get, add persisitence path in client rather than store_py.cpp
* add support for batch api conflict , refactor replica.descriptor to support file and memory type
* add test branch
* add ci ssd
* change python test
* add log for fail
* change querykey return value type
* fix bug
* fix bug
* add sleep for removefile
* fix sleep
* edit ci.yml and fix delete before write problem
* add comment for storage_backend
* spell check
* fix name problem and decrease errorcode for file
* add pytest for ssd offload
* edit test
* edit test
* fix test
* fix bug
* fix test
* Modify the thread pool value capture to reference capture to fix the issue of significant performance degradation when writing files with put.
* add async getfrom file in batchget transfertask. delete file_storage_backend
* add support for HA in cluster_id subdirectory, change session_id to fsdir
* add persistence in batchput
* add disk allocate for get_into py interface
* fix bug
* temp
* fix bug in submit fileread task for std:move(slices)
* edit querykey to return optional<descriptor>, add interface batchquerykey for storagebackend
* fix confict in batchget, add batchget/batchput test
* fix bug
* fix bug
* comment batch test
* fix conflict and add batch_get_into file test
* fix test bug
* fix test bug
* fix conflict
Major changes include: Client failover. Refactor SegmentManager. On the client side, limit the segment name to be equal to the localhost name. Add clientctl for manual e2e tests.
* [Store] New feature: lease and LRU eviction
* Slightly adjust the ordering of the unit test cases
* Fix failed client_integration_test due to new the lease feature
* Modify test_distributed_object_store.py: adjust unit test behaviour accordingly
* bugfix: don't evict objects whose put are still not complete
* Refactor the codes according to review comments
* Add eviction related metrics in master;
Improve a unit test case for eviction.
* Fix a typo
* Refactor some codes;
Fix a unit test;
* Update docs: add lease and eviction policy
* make eviction_ratio configurable;
improve eviction algorithm to better achieve evicted_count / object_count = eviction_ration, especially when there are not so many objects.
* Fix a typo
* [Store] Improve: Add RemoveAll rpc for remove all keys
* Add UT for RemoveAll and refactor code
* Add 3 more UT to address suggestion.
* Change to return removed_count rather than ErrorCode
* [Refactor] [MooncakeStore] Migrate Master Service from gRPC to coro_rpc
* fix typos and install yalantinglibs
* fix master service test
* [Refactor] rename cache_allocator to mooncake_store;
* Update DistributedObjectStore
* Set minimum logging severity to WARN
* Refactor MasterClient to use send_request for thread safety
* seems a bug
* fix bug
Mooncake Store is a high-performance **distributed key-value (KV) cache storage engine** designed specifically for LLM inference scenarios.
Unlike traditional caching systems such as Redis or Memcached, Mooncake Store is positioned as **a distributed KV cache rather than a generic caching system**. The key difference is that in the former, the key is derived from the value through hashing, so value is immutable after inserting (although the key/value pair may be garbage collected).
Mooncake Store provides low-level object storage and management capabilities, while specific caching strategies (e.g., eviction policies) are left to upper-layer frameworks (like vLLM) or users for implementation, offering higher flexibility and customizability.
Key features of Mooncake Store include:
- **Object-level storage operations**: Mooncake Store provides simple and easy-to-use object-level APIs, including `Put`, `Get`, and `Remove` operations.
- **Multi-replica support**: Mooncake Store supports storing multiple data replicas for the same object, effectively alleviating hotspots in access pressure.
- **Eventual consistency**: Mooncake Store ensures that `Get` operations read complete and correct data, but does not guarantee the latest written data. This eventual consistency model ensures high performance while simplifying system design.
- **High bandwidth utilization**: Mooncake Store supports striping and parallel I/O transfer of large objects, fully utilizing multi-NIC aggregated bandwidth for high-speed data reads and writes.
- **Dynamic resource scaling**: Mooncake Store supports dynamically adding and removing nodes to flexibly handle changes in system load, achieving elastic resource management (to be refined in future versions).
---------
Co-authored-by: Feng Ren <alogfans@gmail.com>
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com>
Co-authored-by: susun <751080330@qq.com>
Co-authored-by: liusy58 <liusy58@linux.alibaba.com>
LZW