* [Store]: add task executor feature with unit and executor test
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: add some optimizations to task executor
Refactor the task_executor into the client_service and add a
task structure on the client side. Additionally, remove the
existence check in the execute function;
only retrying should be performed if replica allocation fails.
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store]: get source replica from copyStart or moveStart api
* [Store]: call move or copy end if the target replica already exist to complete the replication task
* [Store]: use the max_retry_attempts in master side
* [Store]: add client integration test and set default max_retry_attempts to 10
* [Doc] update task api introduction
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Store] Set copy and move as private methods
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
* [Doc]: change the default task max_retry_attempts to 10
* [Doc]: fix some description error
* [Store]: allocate the buffer size to be a multiple of 16MB
* [Store]: validate the replica is in local and directly construct slices from replica buffer address instead of copy the data to local buffer
* [Store]: change the validate logic to directly use transfer engine endpoint or local_hostname_.
* [Store] add e2e ci test for copy and move api
* [Store] refactor the client move and copy function
* [Store] fix the e2e test
* [Store] remove unused code
* [Store] add source field when build replica copy payload in the task_manager_test
* [Store] rename back to snake case for split_into_slices function and also remove hard code for client poll count
* [Store] revert mis deleted field when resolve conflicts
* [Store] change test to validate the real behaviour
* [Store] change the default task fetch size to 16
* [Doc]: change the replica copy/move sequence diagram
* [Store] add new split_into_slice method
* [Store] change the real client to use split_to_slice with buffer handle parameters
---------
Signed-off-by: Vincent Gao <vincentbo@linux.alibaba.com>
Co-authored-by: Vincent Gao <vincentbo@linux.alibaba.com>
* feat(metrics): add task completion latency tracking and detailed metrics reporting
- Add histogram metric for tracking transfer task completion latency with
configurable buckets
- Record task start time in submitTransfer and calculate latency upon completion
- Enhance metrics reporting thread to include latency distribution statistics
- Add skip_metrics parameter to getBatchTransferStatus to avoid double counting
- Update task structure to include start_time field for latency calculation
Signed-off-by: staryxchen <staryxchen@tencent.com>
* feat: add metrics support for submitTransferWithNotify
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(metrics): improve metrics recording logic with early returns
- Replace nested if statements with early returns using goto for cleaner flow
- Add boundary checks for task_id and validate start_time before recording
- Only record metrics for COMPLETED status and reset start_time to prevent
duplicates
Signed-off-by: staryxchen <staryxchen@tencent.com>
* refactor(metrics): simplify bucket count tracking logic
- Remove conditional checks for vector bounds by ensuring prev_bucket_counts_
size matches bucket_counts
- Consolidate previous snapshot update into a single assignment
Signed-off-by: staryxchen <staryxchen@tencent.com>
* fix(metrics): conditionally enable metrics collection
- Add metrics_enabled_ flag check before collecting metrics
- Wrap metrics collection logic with flag condition to avoid unnecessary
operations
- Ensure metrics are only recorded when explicitly enabled
Signed-off-by: staryxchen <staryxchen@tencent.com>
* docs: clarify Transfer Engine metrics limitation with TENT
- Add note about MC_TE_METRIC not being supported when using Transfer Engine
TENT in deployment guide
- Add same note to Python API reference for MC_TE_METRIC environment variable
Signed-off-by: staryxchen <staryxchen@tencent.com>
---------
Signed-off-by: staryxchen <staryxchen@tencent.com>
* feat(store): disable auto discovery by default, require rdma_devices for RDMA
* feat(client): support optional auto-discovery with default for RDMA
* update comment
* update doc
* update hello world
* feat(allocator): Add transport endpoint support for segment transfers
* refactor(Client): Replace segment_name with transport_endpoint in buffer descriptors and tests
* refactor(test): switch to in-proc master and remove external service management
* style: format code and adjust transfer task timeout
* feat(Client::MountSegment): set te_endpoint conditionally based on metadata_connstring for P2P mode
* Allow configuring default KV lease TTL via environment variable
Read DEFAULT_KV_LEASE_TTL environment variable at server startup and
parse as uint64_t value. This enables CI tests to override default
configuration without code changes.
Signed-off-by: xiaguan <751080330@qq.com>
* style: format CI config and add [[nodiscard]] to methods
Signed-off-by: xiaguan <751080330@qq.com>
* refactor(tests): remove local_hostname parameter and improve test helpers
The commit removes the unused local_hostname parameter from TransferSubmitter constructor, updates the associated initialization logic, and simplifies test code by introducing helper functions for segment creation. The changes improve code maintainability while preserving functionality.
* refactor(TransferSubmitter): Remove unused local_hostname_ member and validation check
* docs(store): update setup examples for p2p
---------
Signed-off-by: xiaguan <751080330@qq.com>
tang