The pull_request_target trigger introduced in PR #1989 was incorrectly
preserved during the merge of PR #1992 due to conflict resolution.
Remove it along with the fork-routing conditions in ascend-test and
integration-test, since the mirror URL is now hardcoded in ci_ascend.yml
and vars/secrets access is no longer needed.
Signed-off-by: staryxchen <staryxchen@tencent.com>
Fork PRs trigger pull_request events where GitHub Actions withholds
vars and secrets. This causes vars.ASCEND_GITHUB_MIRROR_URLS to resolve
as empty on the self-hosted Ascend runner, blocking mirror-based checkout.
Add pull_request_target as a complementary trigger and route both
ascend-test and integration-test by fork origin: non-fork PRs continue
on pull_request, fork PRs are handled by pull_request_target where vars
and secrets are available. Duplicate runs are avoided by skipping the
opposite event for each case.
Signed-off-by: staryxchen <staryxchen@tencent.com>
PutToLocalFile and FileStorage::OffloadObjects crash with SIGSEGV when
slice.ptr points to GPU device memory, because CPU memcpy cannot access
GPU virtual addresses. The RDMA memory-replica path is unaffected.
Add synchronous Device-to-Host staging via PinnedBufferPool before data
reaches the disk-write paths:
- New gpu_staging_utils.h: shared IsDevicePointer/CopyDeviceToHost/
SetDevice helpers with cross-platform support (CUDA/HIP/MUSA/MACA/
Ascend CANN)
- New PinnedBufferPool: thread-safe pinned host memory pool with
max capacity limit (default 32) and O(1) swap-pop acquire
- PutToLocalFile: sync D2H on calling thread, PutRevoke on failure
- OffloadObjects: D2H staging before BatchOffload; on per-slice failure
the entire object is skipped to prevent partial/corrupt data
- CMakeLists: auto-detect CUDAToolkit/HIP/Ascend independently of
global USE_CUDA flag, with explicit PRIVATE compile definitions
- CI: add -lcudart to Go test CGO_LDFLAGS when CUDA is present
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Changes:
- Add job dependencies in ci.yml (build jobs depend on spell-check/clang-format)
- Convert ci_ascend.yml and integration-test.yml to workflow_run trigger
- Add paths-filter to avoid running on docs-only PRs
- Fix context variables in integration-test.yml for workflow_run
- Add paths filtering to ci_cu13.yml
Some IDEs (like CLion) uses CMakeLists.txt for smart context. This PR adds an optional flag `EP_USE_IDE` that IDE users can enable. It is switched off by default, so that it will not affect production systems.
* build: add memory-aware compile/link parallelism
Auto-detect available memory and CPU cores at configure time to calculate
safe parallel job limits. With Ninja generator, creates separate job pools
for compilation (~1.5GB/job) and linking (~4GB/job) so high-core machines
can compile fast without OOM during linking.
Changes:
- New mooncake-common/limit_jobs.cmake module
- Include from common.cmake for all build modes
- Switch Dockerfile and CI workflows to Ninja
- User can override via -DPARALLEL_COMPILE_JOBS / -DPARALLEL_LINK_JOBS
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix(ci): remove sudo from Ascend CI install step
The Ascend CI job runs inside a Docker container as root, where sudo
is not available. This caused exit code 127 after a successful build.
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* [CI] add CU13_BUILD environment variable to CI workflow
* Disable etcd, redis, and examples in CI workflow
* Apply suggestion from @stmatengss
* Update ci_cu13.yml
* docs: update pull request template with module checklist and refined change types
- Add "Module" section with checkboxes for various project components
- Simplify "Type of Change" section by removing nested structure
- Add "Refactor" as a new change type option
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [Store] feat: CXL storage full features.
* fix(ci): resolve cxl test failure and code format
* fix(ci): add CXL protocol support and fix code format issues
* [Store] feat: CXL storage full features, reset and rm extern/pybind
* [TE] Support TCP fallback in EFA build and improve EFA documentation (#1523)
When building with USE_EFA=ON, auto_discover is disabled to prevent
RDMA transport installation (QP creation fails on EFA devices). This
means TCP transport is also not installed automatically. Add explicit
TCP transport installation for non-EFA protocols in the EFA build path.
Documentation changes:
- build.md: Add USE_EFA option and clarify USE_CUDA default/purpose
- supported-protocols.md: Add EFA as a supported protocol
- efa_transport.md: Add USE_CUDA=ON to build command, document GPU
memory requirement
Co-authored-by: whn09 <whn09@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
* [Store] Optimize BucketStorageBackend for reduced lock contention and add delete safety (#1456)
* fix(ci): resolve cxl test failure and code format
* [Store] Add Local Cache Mechanism for Mooncake Store Client (#1226)
* feat(Store): add local hot cache for client
* feat(Store): add client local hot cache log to show performance
* fix: local hot cache initialize bug
* fix(Store): Mooncake put slice is max 16MB， so make local hot cache block 16MB
* feat(Store): move local hot cache initialization to Client::Create
* feat(Store): local hot cache remove unused small block implementation
* feat(Store): add client local hot cache unit test
* fix(Store): modify client local hot cache suit with v0.3.7
* feat(Store): change local hot cache unit tes
* fix: initialize local hot cache with negative value
* feat: use in process master and metadata fro local hot cache unit test.
* feat: update local hot cache to one replica one slice version
* fix: local hot cache unit test use in process master service
* fix: code style fix
* fix: fix dirty read when client wants to read a previously hitted hot block but the hot block is modified by incoming put actions
* fix: local hot cache unit test use in process master service
* fix: code format fix
* fix: fix comment problems for
* feat: add local hot asynchronous queue size limit
* fix: local hot cache task involves the block so that there is no memcpy operation when inserting local hot cache
* fix: code check fix
* fix: update block in_use prop to reference count
---------
Co-authored-by: shichangzhang064 <zhangshichang@h-partners.com>
* fix(ci): add CXL protocol support and fix code format issues
* fix: address comments from code review
* fix(ci): resolve cxl test failure
* fix(ci): resolve ci error
---------
Co-authored-by: 王鹤男 <wanghenan09@gmail.com>
Co-authored-by: whn09 <whn09@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com>
Co-authored-by: Mahesh Bapatu <153306023+maheshrbapatu@users.noreply.github.com>
Co-authored-by: Shichang Zhang <77728761+Shichang-Zhang@users.noreply.github.com>
Co-authored-by: shichangzhang064 <zhangshichang@h-partners.com>
* [Store]: add task executor feature with unit and executor test
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: add some optimizations to task executor
Refactor the task_executor into the client_service and add a
task structure on the client side. Additionally, remove the
existence check in the execute function;
only retrying should be performed if replica allocation fails.
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: get source replica from copyStart or moveStart api
* [Store]: call move or copy end if the target replica already exist to complete the replication task
* [Store]: use the max_retry_attempts in master side
* [Store]: add client integration test and set default max_retry_attempts to 10
* [Doc] update task api introduction
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store] Set copy and move as private methods
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Doc]: change the default task max_retry_attempts to 10
* [Doc]: fix some description error
* [Store]: allocate the buffer size to be a multiple of 16MB
* [Store]: validate the replica is in local and directly construct slices from replica buffer address instead of copy the data to local buffer
* [Store]: change the validate logic to directly use transfer engine endpoint or local_hostname_.
* [Store] add e2e ci test for copy and move api
* [Store] refactor the client move and copy function
* [Store] fix the e2e test
* [Store] remove unused code
* [Store] add source field when build replica copy payload in the task_manager_test
* [Store] rename back to snake case for split_into_slices function and also remove hard code for client poll count
* [Store] revert mis deleted field when resolve conflicts
* [Store] change test to validate the real behaviour
* [Store] change the default task fetch size to 16
* [Doc]: change the replica copy/move sequence diagram
* [Store] add new split_into_slice method
* [Store] change the real client to use split_to_slice with buffer handle parameters
---------
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
Co-authored-by: Vincent Gao <vincentbo@linux.alibaba.com>
* feat: use setuptools_scm for more elegant version and support MOONCAKE_LOCAL_VERSION env
* feat: more compatible
* feat: add fetch-tags for build ci
* feat: manually add Fetch git tags actions
* feat: use fetch-depth: 0 and filter: tree:0
---------
Co-authored-by: huangweixiao <huangweixiao@moonshot.cn>
fatSheep