Commit Graph

20 Commits

Author SHA1 Message Date
CAICAII 9b50b354b3
[Store] Robustify ConfigDict size parsing (#2206) 2026-06-02 10:33:08 +08:00
LZW be93707047
[Store] feat: add graceful segment unmount APIs (#2065) 2026-05-16 18:21:22 +08:00
ゆり 377dcba3fb
[Store] Add comprehensive test suite for Python binding error handling (#2097)
Add 20+ C++ tests and 40+ Python tests covering error handling,
corner cases, and previously untested API endpoints for the
mooncake-store Python binding layer.

C++ tests (pybind_client_test.cpp):
- batchIsExist with mixed existing/missing keys and before setup
- getSize for existing keys and before setup
- removeByRegex with matching, non-matching, and pre-setup cases
- removeAll on empty store
- batchRemove with valid, non-existent, and pre-setup keys
- put_parts and put_batch then batch_get_buffer roundtrip
- health_check after setup and before initialization
- get_hostname verification
- Double tearDownAll idempotency
- Empty batch operations
- Mount non-existent file, unmount invalid segment IDs

Python tests (test_mooncake_store_service_api.py):
- /api/put: success, missing key/value, empty key, store failure
- /api/get: success, not found, empty bytes, store exception
- /api/exist: true, false, store exception
- /api/remove: success, failure, store exception
- /api/remove_all: success, zero keys, failure, store exception
- /api/reconfigure: decode/prefill success, missing params, invalid mode,
  remount, unmount failure rollback
- /api/mount: negative/float/string/missing size
- /api/unmount: missing segment_ids, empty list, success
- /api/mount_shm: store failure, defaults, invalid names
- /api/unmount_shm: string coercion, empty list
- _shm_name_to_path: 11 unit tests for path validation edge cases

Closes #654

Co-authored-by: yurekami <yurekami@users.noreply.github.com>
2026-05-15 14:52:01 +08:00
Cruz Zhao 4326adbd51
[engram] support engram (#1483)
* support engram

Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>

* add test case for engram

Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>

* add docs for engram

Signed-off-by: Cruz Zhao <CruzZhao@linux.alibaba.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-08 19:52:35 +08:00
LZW bd7ed21017
[Store] feat(store): add allocate-backed segment mount and unmount APIs (#2006) 2026-05-08 16:02:29 +08:00
fatSheep 7d0978b9ca
[Store] Add Upsert API for in-place object updates (#1662)
Co-authored-by: fatSheep <tzh2005t@gmail.com>
2026-04-03 10:24:37 +08:00
dongb0 3897e1b53e
[Store]add unit test for exception handling of RealClient (#1546) 2026-02-22 16:02:01 +07:00
tang c755916ebc
[Store][Feature]: add task manager component (#1326)
* [Store]: initialize the basic task data structure

* [Store]: implement the task manager

* [Store]: change the mutex to custom mutext

* [Store]: implement copy and move, query task api

* [Store]: change the task manager lock to be shared

* [Store]: change the task to struct when submit

* [Store]: add fetch tasks api

* [Store]: add metrics for query and fetch task api

* [Store]: add update task status api

* [Store]: rename the updateTask api and add pending and processing task limit

* [Store]: expose the task manager paramters to master config

* [Store]: remove client id from client service

* [Store]: change the copy and move api definition to createCopyTask and createMoveTask.

* [Store]: add pending and processing task timeout support

* [Store]: add exposure api to real_client_main and also rename method to snake_case style

* [Store]: directly cast to int64_t

* [Store]: remove unused code and fix the api description

* [Store]: fix metrics issue for mark_task_to_complete
2026-01-07 11:27:18 +08:00
Copilot e59749a156
[Store] Fix compilation warnings for missing field initializers (#1232)
* Initial plan

* Fix compilation warnings for visibility and missing initializers

Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>

* Complete fix for compilation warnings

Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>

* Remove build artifacts and update .gitignore

Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>

* Use pragma GCC diagnostic to suppress visibility warnings instead of attributes

Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>

* Narrow scope of pragma diagnostic directives for lambda

Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>

* fix

* Revert all changes in mooncake-transfer-engine directory

Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: stmatengss <11641725+stmatengss@users.noreply.github.com>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
2025-12-18 16:27:40 +08:00
yejj 1840577d68
[Store] add pybind for get_replica_desc (#1121)
* add pybind for get_replica

* rename pybinb funcs name & update api doc

* fix spell problem
2025-12-04 15:45:52 +08:00
EkiRui 7517db4585
[Store] feat: Add standalone deployment implementation for Client (#1084)
* [Store]: Add independent deployment implementation for Client

Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>

* [Doc]: mooncake-store: add introduce for client standalone mode

Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>

* [CI]: Add dummy client test

Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>

* [Store]: Change dummy client setup into a new func

Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>

* [Store]: Add more log for dummy client

Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>

---------

Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
2025-11-25 23:57:30 +08:00
ykwd ea1ffe7a40
[CI] Fix a CI BUG in PyClientTest:TestSetupExistTransferEngine (#1016) 2025-11-04 17:08:02 +08:00
ykwd f3ed886cc8
[Store] Fix CI bugs & Improve log output & Refactor TE Initialization (#1006) 2025-11-04 14:13:57 +08:00
Chao Lei 2d3d02a0a8
[Store] Add APIs: batch_put_from_multi_buffers and batch_get_into_multi_buffers (#929)
Co-authored-by: youxiao <youxiao@huawei.com>
2025-10-16 11:17:23 +08:00
JinYan Su ca09e509cf
fix(store): Fix integer overflow in get_into/batch_get_into for values > 4GB (#920)
* Fix integer overflow in get_into and batch_get_into for values > 4GB

Problem:
- get_into() and batch_get_into() returned 'int' (32-bit signed)
- This caused integer overflow when handling values larger than ~2GB
- Values > 4GB would wrap around and return incorrect results

Solution:
- Changed return type from 'int' to 'int64_t' for both methods
- Updated batch_get_into to return std::vector<int64_t>
- Updated test assertions to use int64_t

Impact:
- Can now handle values up to ~9.2 exabytes (2^63 - 1)
- Backward compatible for values < 2GB
- Internal implementation already used int64_t, so this aligns public API
- Python bindings automatically handle int64_t -> Python int conversion

Files changed:
- mooncake-store/include/pybind_client.h
- mooncake-store/src/pybind_client.cpp
- mooncake-store/tests/pybind_client_test.cpp

* docs: Update Doxygen comment for batch_get_into return type

Update the documentation comment to accurately reflect that batch_get_into
returns a vector of 64-bit integers, not just 'integers'. This clarifies
that the return type can handle values larger than 2GB/4GB.
2025-10-14 11:26:02 +08:00
ykwd c5cce40ee7
[Store] Check If Get Operation Completed Within Lease (#778) 2025-09-24 11:43:23 +08:00
JinYan Su af4fb4d5d4
feat(store): support transfer engine p2phandshake (#852)
* feat(allocator): Add transport endpoint support for segment transfers

* refactor(Client): Replace segment_name with transport_endpoint in buffer descriptors and tests

* refactor(test): switch to in-proc master and remove external service management

* style: format code and adjust transfer task timeout

* feat(Client::MountSegment): set te_endpoint conditionally based on metadata_connstring for P2P mode

* Allow configuring default KV lease TTL via environment variable

Read DEFAULT_KV_LEASE_TTL environment variable at server startup and
parse as uint64_t value. This enables CI tests to override default
configuration without code changes.

Signed-off-by: xiaguan <751080330@qq.com>

* style: format CI config and add [[nodiscard]] to methods

Signed-off-by: xiaguan <751080330@qq.com>

* refactor(tests): remove local_hostname parameter and improve test helpers

The commit removes the unused local_hostname parameter from TransferSubmitter constructor, updates the associated initialization logic, and simplifies test code by introducing helper functions for segment creation. The changes improve code maintainability while preserving functionality.

* refactor(TransferSubmitter): Remove unused local_hostname_ member and validation check

* docs(store): update setup examples for p2p

---------

Signed-off-by: xiaguan <751080330@qq.com>
2025-09-22 16:30:15 +08:00
JinYan Su 785e939523
refactor(store): use dedicated thread for signal handling (#840)
* refactor(store): use dedicated thread for signal handling and atomic cleanup

* refactor(store): switch PyClient to shared_ptr and improve resource management

* refactor(ResourceTracker): use leaked heap object to avoid static destruction order issues

* test: increase unittest verbosity to show running tests

* try fix ci

* fix setup after close

* fix comments
2025-09-16 14:41:19 +08:00
JinYan Su d8f1ee24b2
[Store] feat(store): add duplicate rpc_meta key check and CI integration (#818)
* feat(store): add duplicate rpc_meta key check and CI integration

* fix(KVBootstrapServer): disallow duplicate rpc_meta keys in PUT requests

* revert ci changes

* fix ci
2025-09-10 12:10:32 +08:00
JinYan Su d2fa93dfb2
refactor(store): move python bidning to `pybind_client` (#723)
* refactor(store): replace SliceBuffer with BufferHandle in Python bindings

* refactor(store_py): introduce PyClient and wrapper class for Python bindings

* fix

Signed-off-by: Jinyang Su <751080330@qq.com>

---------

Signed-off-by: Jinyang Su <751080330@qq.com>
2025-08-11 14:09:42 +08:00