The write_thread_pool_ has 2 threads, so concurrent PutToLocalFile calls
can complete in arbitrary order, causing the FIFO eviction queue to not
match the logical Put order. Wait for each key's DISK replica before
putting the next to guarantee key_0 is always the oldest entry.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [Bugfix] Fix Pimpl pattern compilation error in S3SnapshotObjectStore
Move destructor definition from header (= default) to source file,
where the Impl class is fully defined. This fixes the incomplete type
error when std::unique_ptr<Impl> tries to instantiate its destructor.
Signed-off-by: timzhang0727
* [Build] Improve Go download reliability and smart GOPROXY configuration
1. Go download: Add mirror fallback for restricted network environments.
Try go.dev first, then golang.google.cn, then mirrors.aliyun.com.
Each mirror has a 30s timeout and 2 retries before falling back.
2. GOPROXY: Automatically detect restricted network environments based
on whether the Go download fell back to a CN mirror. Only set
GOPROXY (goproxy.cn, goproxy.io, direct) when CN mirror was used
and GOPROXY is not already configured by the user. This avoids
unnecessarily routing traffic through CN proxies in environments
with normal international network access (e.g. GitHub CI).
Signed-off-by: timzhang0727
---------
Signed-off-by: timzhang0727
Co-authored-by: Tianxiang Zhang <timskyzhang@tencent.com>
* [Store] Add C API for Mooncake Store
Introduce an extern "C" API layer (store_c.h / store_c.cpp) over the
existing C++ RealClient, enabling any FFI-capable language to consume
Mooncake Store without linking against C++ symbols.
Covers: lifecycle (create/destroy/setup), put/get (including zero-copy
and batch variants), existence/size checks, remove operations, buffer
registration, and health checks.
* Address review: add try-catch guards, c_str_or helper, idiomatic vectors
- Wrap all extern "C" functions in try-catch to prevent C++ exceptions
from crossing the FFI boundary (undefined behavior)
- Add c_str_or() helper to deduplicate null-check fallback pattern
- Use reserve/emplace_back and range constructors for batch vectors
* fix C API code formatting
* Address Copilot review: fix destroy leak, input validation, null key rejection
- destroy(): move delete handle outside try-catch to prevent leak if
tearDownAll() throws
- init_all(): default protocol to "tcp" instead of empty string
- put_from(), get_into(): reject NULL buffer when size > 0
- batch_put_from(), batch_get_into(): validate per-element keys (non-null)
and buffers (non-null when size > 0) before calling into RealClient
- batch_is_exist(): reject null keys instead of silently converting to ""
* [Build] Manage yalantinglibs as extern submodule
Move yalantinglibs under extern as a git submodule so dependency setup uses the repo-managed source instead of downloading an archive into thirdparties.
* Apply suggestions from code review
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Changes:
- Add job dependencies in ci.yml (build jobs depend on spell-check/clang-format)
- Convert ci_ascend.yml and integration-test.yml to workflow_run trigger
- Add paths-filter to avoid running on docs-only PRs
- Fix context variables in integration-test.yml for workflow_run
- Add paths filtering to ci_cu13.yml
This PR fixes three issues in the local TENT Python path
that block stable integration:
- make the built Python module importable as tent
- load TransferEngine(<config_path>) as a config file via loadFile(...)
- fix RDMA status string construction in rdma_transport.cpp
Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
* [Store] Fix SSD offload failure in Metadata Server mode (#1729)
Use Client::IsReplicaOnLocalMemory() in FileStorage::BatchQuerySegmentSlices
instead of directly comparing transport_endpoint_ with GetTransportEndpoint().
The direct comparison only works in P2P_HANDSHAKE mode where both sides use
the RDMA NIC IP. In Metadata Server mode, transport_endpoint_ is set to
local_hostname_ while GetTransportEndpoint() returns the RDMA IP, causing
a mismatch in multi-NIC environments and INVALID_KEY errors.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: correct TP zero-copy put semantics
* fix: apply clang-format to store binding
* fix: default batch TP-from results to errors
* fix: return success for batch TP put-from
Ensure batch_put_tensor_with_tp_impl reports zero on fully successful shard writes so TP zero-copy Python tests don't fail with false INVALID_PARAMS results.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* tests: align TP zero-copy benchmark with get API signature
Keep the TP zero-copy benchmark focused on put-from semantics by removing an unsupported split_dim argument from batch_get_tensor_with_tp_into.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* trigger ci
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
* fix(tent): address build issues and enable CI coverage
* fix(ci): disable tent in standalone te build
* fix(tent): honor use_cuda in cmake
* fix(tent): decode http metadata test query keys
* revert(ci): restore workflow to main
* [TE] Fix simultaneous open handshake in RdmaEndpoint
* Keep the same logic for ERDMA.
* apply gemini-code-assist's suggestion.
* Fix endpoint reinitialization.
* Add disconnect and waiting with back-off.
* Add eRDMA Endpoint Re-establishment Test
* Include the test in cmake
* Address reviewer comments
* Better log message.
During Elastic EP rank recovery, the recovered rank must re-run expensive initialization (especially CUDA graph capture).
Previously, this required the recovered and healthy ranks to enter the same communication phase, which paused healthy-rank inference and increased recovery disruption.
This change enables **asynchronous recovered-rank initialization** in Mooncake PG: recovered ranks can initialize in isolation first, then join the live process group after local recovery work is finished.
* [STORE] abstract snapshot catalog in master service
Introduce SerializerSnapshotStore as the snapshot catalog adapter for the existing serializer backend and route MasterService persist, restore, and cleanup flows through it instead of open-coding latest marker updates and snapshot directory scans.
Also add focused unit coverage for publish/get/list/delete behavior so the snapshot catalog path can evolve independently from the payload storage path.
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] distinguish missing snapshots from backend read errors
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] add Redis snapshot catalog backend
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] share Redis connection helpers across HA backends
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
Co-authored-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
* [STORE] unify Redis test helpers
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* [STORE] tighten snapshot catalog state handling
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
---------
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
* fix: preserve explicit tent metadata config
* Fix tent config override for string rpc port
* Validate tent rpc port overrides and test startup path
* Format tent config override test
* [Store] Add hard pin mechanism for eviction-protected objects
Objects created with ReplicateConfig.with_hard_pin=true are never
evicted by the eviction policy, providing guaranteed persistence
for model weights in RL and model management workloads.
Changes:
- ReplicateConfig: add with_hard_pin field (default false)
- ObjectMetadata: add hard_pinned boolean, set at creation via PutStart
- BatchEvict: skip hard-pinned objects in all eviction passes
- Serialization: persist hard_pinned in snapshots (backward compatible
with old format that lacks the field)
- Tests: verify hard-pinned objects survive eviction, coexist with
soft pin, and can still be explicitly removed
Introduce a first-pass MACA integration with a dedicated USE_MACA switch, transfer-engine linkage, and a MACA vendor compatibility header so Mooncake can build and run basic store setup/put/get on MACA environments.
* store: split client HA/control-plane threads and suppress zero-seg heartbeats
Refactor the store client control plane so leader monitoring, storage
heartbeat, and task polling are managed separately, and route leader
switching through a single serialized SwitchLeader() path.
This changes zero-global-segment behavior so setup no longer starts the
storage heartbeat/task polling control plane when no segment is mounted,
which avoids flooding master with useless ping traffic from zero-seg
clients. The storage control plane now starts lazily from actual mount
paths.
Also add regression and smoke coverage for:
- zero-seg clients not pinging before mount
- heartbeat starting after mount
- non-HA reconnect/remount behavior
- zero-seg HA smoke and non-zero ping/fetch-task smoke
Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
* format
Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
* fix
Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
---------
Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
Add the Redis leader coordinator behind the HA backend abstraction and wire it into the store and test builds so Redis can participate as a first-class leadership backend alongside etcd.
Also make the HA/e2e test harness backend-agnostic, add Redis leadership regression tests, and fix the client test wrapper to pass HA master entries correctly so the chaos suite exercises the real HA path.
---------
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
Some IDEs (like CLion) uses CMakeLists.txt for smart context. This PR adds an optional flag `EP_USE_IDE` that IDE users can enable. It is switched off by default, so that it will not affect production systems.
copilot-swe-agent[bot]
QiuRuiqi