Mooncake/mooncake-store/include
skygragon aad5111781
[Store] Support degraded startup when master is unavailable (#1930)
* [Store] Support degraded startup when master is unavailable

Problem:
P2P client startup fails when master is unavailable, blocking the
entire application. This is problematic for scenarios where the
client should be able to start independently and wait for master
to become available.

Solution:
1. Refactor P2PClientService::Init to support degraded startup:
   - Try ConnectToMaster/RegisterClient, allow failure
   - Set HA state to FULL or DEGRADED based on connection result
   - Skip MountSegment in degraded mode
   - Heartbeat thread will recover when master becomes available

2. Optimize initialization order:
   - ConnectToMaster/RegisterClient first
   - StartHeartbeat before InitTransferEngine (faster heartbeat)
   - InitTransferEngine and InitStorage after heartbeat starts

3. Add HARecoveryManager::SetState() for setting initial state

4. Make connection_interrupted_ atomic for thread-safe access

Changes:
- ha_recovery_manager.h: Add SetState() method
- client_service.h/cpp: Support degraded startup in heartbeat logic
- p2p_client_service.cpp: Refactor Init flow with degraded support

Co-Authored-By: Claude (antchat/GLM-5) <noreply@anthropic.com>

* [Store] Refactor HA recovery manager and client service

1. Remove unused master_server_entry parameter from ReconnectToMaster.
   In non-HA mode, current_master_address is already initialized to
   master_server_entry, making the parameter redundant.

2. Replace SetState() with constructor initial_state parameter.
   SetState() bypasses the state machine transition logic, which breaks
   encapsulation. Initial state should only be set during construction.

3. Add WaitForReady() to encapsulate initialization wait logic.
   Extracts the ready_for_recovery_ and data_manager_ check into a
   separate method for better readability in RecoveryPipelineMain.

Co-Authored-By: Claude (antchat/GLM-5) <noreply@anthropic.com>

* [Store] Add TODO comment for race window in segment mount logic

Add documentation explaining the race condition between IsDegraded()
check and MountSegment() call, and outline future improvement plan
to decouple storage layer initialization from master interaction.

Co-Authored-By: Claude (antchat/GLM-5) <noreply@anthropic.com>

---------

Co-authored-by: wl177541 <wl177541@antgroup.com>
Co-authored-by: Claude (antchat/GLM-5) <noreply@anthropic.com>
2026-04-22 20:56:56 +08:00
..
cachelib_memory_allocator [Store]Refactor hierarchy MasterService and heartbeat mechanism 2026-03-04 14:42:59 +08:00
hf3fs code format & enable code format checking in ci (#677) 2025-08-02 15:50:25 +08:00
offset_allocator fix(store): add mutex locks for thread-safe metrics retrieval (#804) 2025-09-04 16:54:11 +08:00
tiered_cache [Store ]Implement Master HA recovery based on client (#1876) 2026-04-13 19:33:23 +08:00
utils refactor: introduce expected pattern for error handling in master service (#562) 2025-07-03 14:54:06 +08:00
allocation_strategy.h [Store] pub_tensor for multiple replica (#1148) 2025-12-08 22:38:36 +08:00
allocator.h [Store]Refactor hierarchy MasterService and heartbeat mechanism 2026-03-04 14:42:59 +08:00
async_memcpy_executor.h [Store] Parallelize Local and Remote Read/Write in Batch Scenarios (#1921) 2026-04-20 11:47:43 +08:00
async_metadata_notifier.h [Store ]Implement Master HA recovery based on client (#1876) 2026-04-13 19:33:23 +08:00
centralized_client_manager.h [Store] Implement the client service and add adaptation to the P2P architecture for the interface on the client side (#1631) 2026-03-17 16:17:35 +08:00
centralized_client_meta.h [Store]Refactor hierarchy MasterService and heartbeat mechanism 2026-03-04 14:42:59 +08:00
centralized_client_service.h [Store] Add HTTP metrics endpoint for Client (#1934) 2026-04-21 17:35:20 +08:00
centralized_master_client.h [Store]Refactor hierarchy MasterService and heartbeat mechanism 2026-03-04 14:42:59 +08:00
centralized_master_service.h [Store]Refactor hierarchy MasterService and heartbeat mechanism 2026-03-04 14:42:59 +08:00
centralized_rpc_service.h [Store]Refactor hierarchy MasterService and heartbeat mechanism 2026-03-04 14:42:59 +08:00
centralized_segment_manager.h [Store]Refactor hierarchy MasterService and heartbeat mechanism 2026-03-04 14:42:59 +08:00
client_buffer.hpp [Store] Add HTTP metrics endpoint for Client (#1934) 2026-04-21 17:35:20 +08:00
client_config_builder.h [Store] Add HTTP metrics endpoint for Client (#1934) 2026-04-21 17:35:20 +08:00
client_manager.h [Store ]Implement Master HA recovery based on client (#1876) 2026-04-13 19:33:23 +08:00
client_meta.h [Store]Refactor hierarchy MasterService and heartbeat mechanism 2026-03-04 14:42:59 +08:00
client_metric.h [Store] feat: Implement a FileStorage component to manage the lifecycle of key-value data (#1031) 2025-12-09 15:00:57 +08:00
client_rpc_service.h [Store] Implement Lock-Free P2P Route Cache (#1793) 2026-04-02 14:46:37 +08:00
client_rpc_types.h [Store] Implement the client service and add adaptation to the P2P architecture for the interface on the client side (#1631) 2026-03-17 16:17:35 +08:00
client_service.h [Store] Support degraded startup when master is unavailable (#1930) 2026-04-22 20:56:56 +08:00
config_helper.h [Store] Refine Complicated Constructor Parameters (#748) 2025-08-18 19:31:45 +08:00
data_manager.h [Store] Parallelize Local and Remote Read/Write in Batch Scenarios (#1921) 2026-04-20 11:47:43 +08:00
dummy_client.h [Store] Add orchestration script for matrix sweeps of stress_workload_test and Optimize client read path of P2P Structure (#1779) 2026-04-01 17:13:26 +08:00
etcd_helper.h Skip etcd go package compilation in default. 2025-06-19 07:25:17 +00:00
eviction_strategy.h code format & enable code format checking in ci (#677) 2025-08-02 15:50:25 +08:00
file_interface.h code format & enable code format checking in ci (#677) 2025-08-02 15:50:25 +08:00
file_storage.h [Store] Implement the client service and add adaptation to the P2P architecture for the interface on the client side (#1631) 2026-03-17 16:17:35 +08:00
ha_helper.h Feature: support custom key prefix for issue 957 (#958) 2025-11-03 10:35:07 +08:00
ha_recovery_manager.h [Store] Support degraded startup when master is unavailable (#1930) 2026-04-22 20:56:56 +08:00
heartbeat_type.h [Store] Add ut for P2P-Mooncake-Store (#1715) 2026-03-23 15:08:13 +08:00
http_metadata_server.h [Store] add c++ http metadata server in mooncake master (#766) 2025-08-25 13:56:37 +08:00
hybrid_metric.h [Store] Add support for static labels (host IP/cluster name) in client metrics (#1081) 2025-11-29 14:09:35 +08:00
master_client.h [Store] Parallelize Local and Remote Read/Write in Batch Scenarios (#1921) 2026-04-20 11:47:43 +08:00
master_config.h [Store] Add ut for P2P-Mooncake-Store (#1715) 2026-03-23 15:08:13 +08:00
master_metric_manager.h [Store] Optimize batch_put with BatchGetWriteRoute RPC (#1947) 2026-04-21 22:28:25 +08:00
master_service.h [Store] Add ut for P2P-Mooncake-Store (#1715) 2026-03-23 15:08:13 +08:00
mutex.h [Store] Implement Lock-Free P2P Route Cache (#1793) 2026-04-02 14:46:37 +08:00
p2p_client_manager.h [Store ]Implement Master HA recovery based on client (#1876) 2026-04-13 19:33:23 +08:00
p2p_client_meta.h [Store ]Implement Master HA recovery based on client (#1876) 2026-04-13 19:33:23 +08:00
p2p_client_service.h [Store] Optimize batch_put with BatchGetWriteRoute RPC (#1947) 2026-04-21 22:28:25 +08:00
p2p_master_client.h [Store] Optimize batch_put with BatchGetWriteRoute RPC (#1947) 2026-04-21 22:28:25 +08:00
p2p_master_service.h [Store] Optimize batch_put with BatchGetWriteRoute RPC (#1947) 2026-04-21 22:28:25 +08:00
p2p_rpc_service.h [Store] Optimize batch_put with BatchGetWriteRoute RPC (#1947) 2026-04-21 22:28:25 +08:00
p2p_rpc_types.h [Store] Optimize batch_put with BatchGetWriteRoute RPC (#1947) 2026-04-21 22:28:25 +08:00
p2p_segment_manager.h [Store]Refactor hierarchy MasterService and heartbeat mechanism 2026-03-04 14:42:59 +08:00
peer_client.h [Store] Implement Lock-Free P2P Route Cache (#1793) 2026-04-02 14:46:37 +08:00
pyclient.h [Store] Add orchestration script for matrix sweeps of stress_workload_test and Optimize client read path of P2P Structure (#1779) 2026-04-01 17:13:26 +08:00
real_client.h [Store] Add orchestration script for matrix sweeps of stress_workload_test and Optimize client read path of P2P Structure (#1779) 2026-04-01 17:13:26 +08:00
replica.h [Store] Implement the client service and add adaptation to the P2P architecture for the interface on the client side (#1631) 2026-03-17 16:17:35 +08:00
route_cache.h [Store] Implement Lock-Free P2P Route Cache (#1793) 2026-04-02 14:46:37 +08:00
rpc_helper.h refactor: introduce expected pattern for error handling in master service (#562) 2025-07-03 14:54:06 +08:00
rpc_service.h [Store] Add ut for P2P-Mooncake-Store (#1715) 2026-03-23 15:08:13 +08:00
rpc_types.h [Store] Add ut for P2P-Mooncake-Store (#1715) 2026-03-23 15:08:13 +08:00
segment_manager.h [Store] Add ut for P2P-Mooncake-Store (#1715) 2026-03-23 15:08:13 +08:00
serializer.h [Store] Serialize/Deserialize Offset Allocator (#760) 2025-08-21 11:35:54 +08:00
storage_backend.h [Store] Make tiered scheduler incremental and harden SSD tier space management (#1675) 2026-03-23 14:01:24 +08:00
task_handle.h [Store] Parallelize Local and Remote Read/Write in Batch Scenarios (#1921) 2026-04-20 11:47:43 +08:00
thread_pool.h [Store] Implement the client service and add adaptation to the P2P architecture for the interface on the client side (#1631) 2026-03-17 16:17:35 +08:00
transfer_task.h [Store] One Replica Has One Slice (#1032) 2025-11-10 16:35:08 +08:00
types.h [Store] Add HTTP metrics endpoint for Client (#1934) 2026-04-21 17:35:20 +08:00
utils.h [Store] feat: Implement a unified storage interface to simplify integration and extension (#1185) 2025-12-17 22:33:05 +08:00
version.h.in [Store] add version checking between client and server (#1061) 2025-11-24 20:26:57 +08:00