zbtrs2
/
Mooncake
forked from mooncake-track/Mooncake
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[Doc] update doc for *Regex interface (#776) 

* [Doc] update doc for *Regex interface

Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>

* Fix

Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>

* fix

Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>

---------

Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
This commit is contained in:
Xuchun Shang 2025-08-25 11:05:24 +08:00 committed by GitHub
parent 494fe8a43a
commit 2833d8be8b
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
8 changed files with 278 additions and 19 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

78
doc/en/mooncake-store-preview.md
View File

@ -107,6 +107,23 @@ tl::expected<void, ErrorCode> Remove(const ObjectKey& key);
Used to delete the object corresponding to the specified key. This interface marks all data replicas associated with the key in the storage engine as deleted, without needing to communicate with the corresponding storage node (Client).
### QueryByRegex
```C++
tl::expected<std::unordered_map<std::string, std::vector<Replica::Descriptor>>, ErrorCode>
QueryByRegex(const std::string& str);
```
Used to query the replica information for all objects whose keys match the given regular expression. This is useful for batch operations or for retrieving a group of related objects. The operation is performed on the Master and returns a map of keys to their replica lists.
### RemoveByRegex
```C++
tl::expected<long, ErrorCode> RemoveByRegex(const ObjectKey& str);
```
Used to delete all objects from the store whose keys match the specified regular expression. This provides a powerful way to perform bulk deletions. The command returns the number of objects that were successfully removed.
### Master Service
The cluster's available resources are viewed as a large resource pool, managed centrally by a Master process for space allocation and guiding data replication
@ -150,6 +167,9 @@ service MasterService {
// Get the list of replicas for an object
rpc GetReplicaList(GetReplicaListRequest) returns (GetReplicaListResponse);
// Get replica lists for objects matching a regex
rpc GetReplicaListByRegex(GetReplicaListByRegexRequest) returns (GetReplicaListByRegexResponse);
// Start Put operation, allocate storage space
rpc PutStart(PutStartRequest) returns (PutStartResponse);
@ -159,6 +179,9 @@ service MasterService {
// Delete all replicas of an object
rpc Remove(RemoveRequest) returns (RemoveResponse);
// Remove objects matching a regex
rpc RemoveByRegex(RemoveByRegexRequest) returns (RemoveByRegexResponse);
// Storage node (Client) registers a storage segment
rpc MountSegment(MountSegmentRequest) returns (MountSegmentResponse);
@ -184,7 +207,28 @@ message GetReplicaListResponse {
- **Response**: `GetReplicaListResponse` containing the status code status_code and the list of replica information `replica_list`.
- **Description**: Used to retrieve information about all available replicas for a specified key. The Client can select an appropriate replica for reading based on this information.
2. PutStart
2. GetReplicaListByRegex
```protobuf
message GetReplicaListByRegexRequest {
required string key_regex = 1;
};
message ObjectReplicaList {
repeated ReplicaInfo replica_list = 1;
};
message GetReplicaListByRegexResponse {
required int32 status_code = 1;
map<string, ObjectReplicaList> object_map = 2; // Matched objects and their replica information.
};
```
- **Request**: GetReplicaListByRegexRequest, which contains the regular expression key_regex to be matched.
- **Response**: GetReplicaListByRegexResponse, which contains a status_code and an object_map. The keys of this map are the successfully matched object keys, and the values are the lists of replica information for each key.
- **Description**: Used to query for all keys and their replica information that match the specified regular expression. This interface facilitates bulk queries and management.
3. PutStart
```protobuf
message PutStartRequest {
@ -204,7 +248,7 @@ message PutStartResponse {
- **Response**: `PutStartResponse` containing the status code status_code and the allocated replica information replica_list.
- **Description**: Before writing an object, the Client must call PutStart to request storage space from the Master Service. The Master Service allocates space based on the config and returns the allocation results (`replica_list`) to the Client. The Client then writes data to the storage nodes where the allocated replicas are located. The need for both start and end steps ensures that other Clients do not read partially written values, preventing dirty reads.
3. PutEnd
4. PutEnd
```protobuf
message PutEndRequest {
@ -220,7 +264,7 @@ message PutEndResponse {
- **Response**: `PutEndResponse` containing the status code status_code.
- **Description**: After the Client completes data writing, it calls `PutEnd` to notify the Master Service. The Master Service updates the object's metadata, marking the replica status as `COMPLETE`, indicating that the object is readable.
4. Remove
5. Remove
```protobuf
message RemoveRequest {
@ -236,7 +280,24 @@ message RemoveResponse {
- **Response**: `RemoveResponse` containing the status code `status_code`.
- **Description**: Used to delete the object and all its replicas corresponding to the specified key. The Master Service marks all replicas of the corresponding object as deleted.
5. MountSegment
6. RemoveByRegex
```protobuf
message RemoveByRegexRequest {
required string key_regex = 1;
};
message RemoveByRegexResponse {
required int32 status_code = 1;
optional int64 removed_count = 2; // The number of objects removed.
};
```
- **Request**: RemoveByRegexRequest, which contains the regular expression key_regex to be matched.
- **Response**: RemoveByRegexResponse, which contains a status_code and the number of objects that were removed, removed_count.
- **Description**: Used to delete all objects and their corresponding replicas for keys that match the specified regular expression. Similar to the Remove interface, this is a metadata operation where the Master Service marks the status of all matched object replicas as removed.
7. MountSegment
```protobuf
message MountSegmentRequest {
@ -252,7 +313,7 @@ message MountSegmentResponse {
The storage node (Client) allocates a segment of memory and, after calling `TransferEngine::registerLocalMemory` to complete local mounting, calls this interface to mount the allocated continuous address space to the Master Service for allocation.
6. UnmountSegment
8. UnmountSegment
```protobuf
message UnmountSegmentRequest {
@ -309,17 +370,20 @@ Before writing an object, the Client calls PutStart to request storage space all
```C++
ErrorCode GetReplicaList(const std::string& key,
std::vector<ReplicaInfo>& replica_list);
tl::expected<std::unordered_map<std::string, std::vector<Replica::Descriptor>>, ErrorCode>
GetReplicaListByRegex(const std::string& str);
```
The Client requests the Master Service to retrieve the replica list for a specified key, allowing the Client to select an appropriate replica for reading based on this information.
The Client requests the Master Service to retrieve the replica list for a specified key or for all object keys matching a specified regular expression, allowing the Client to select an appropriate replica for reading based on this information.
- Remove
```C++
tl::expected<void, ErrorCode> Remove(const std::string& key);
tl::expected<long, ErrorCode> RemoveByRegex(const std::string& str);
```
The Client requests the Master Service to delete all replicas corresponding to the specified key.
The Client requests the Master Service to delete all replicas corresponding to the specified key or for all object keys that match the specified regular expression.
### Buffer Allocator

73
doc/zh/mooncake-store-preview.md
View File

@ -113,6 +113,23 @@ tl::expected<void, ErrorCode> Remove(const ObjectKey& key);
用于删除指定 key 对应的对象。该接口标记存储引擎中与 key 关联的所有数据副本已被删除，不需要与对应存储节点(Client)通信。
### RemoveByRegex
```C++
tl::expected<long, ErrorCode> RemoveByRegex(const ObjectKey& str);
```
用于删除与正则表达式匹配的所有 key 对应的对象。其余能力类似 Remove。
### QueryByRegex
```C++
tl::expected<std::unordered_map<std::string, std::vector<Replica::Descriptor>>, ErrorCode>
QueryByRegex(const std::string& str);
```
用于查询与正则表达式匹配的所有 key 对应的对象。
### Master Service
将集群中所有可用的资源看做一个巨大的资源池，由一个中心化的 Master 进程进行空间分配，并指导实现数据复制（**注意 Master Service 不接管任何的数据流，只是提供对应的元数据信息**）。
@ -154,6 +171,9 @@ service MasterService {
// 获取对象的副本列表
rpc GetReplicaList(GetReplicaListRequest) returns (GetReplicaListResponse);
// 获取与正则表达式匹配的对性的副本列表
rpc GetReplicaListByRegex(GetReplicaListByRegexRequest) returns (GetReplicaListByRegexResponse);
// 开始 Put 操作，分配存储空间
rpc PutStart(PutStartRequest) returns (PutStartResponse);
@ -163,6 +183,9 @@ service MasterService {
// 删除对象的所有副本
rpc Remove(RemoveRequest) returns (RemoveResponse);
// 删除与正则表达式匹配的对性的所有副本
rpc RemoveByRegex(RemoveByRegexRequest) returns (RemoveByRegexResponse);
// 存储节点(Client)注册存储段
rpc MountSegment(MountSegmentRequest) returns (MountSegmentResponse);
@ -189,7 +212,29 @@ message GetReplicaListResponse {
说明: 用于获取指定 key 的所有可用副本的信息。Client 可以根据这些信息选择合适的副本进行读取。
2. PutStart
2. GetReplicaListByRegex
```protobuf
message GetReplicaListByRegexRequest {
required string key_regex = 1;
};
message ObjectReplicaList {
repeated ReplicaInfo replica_list = 1;
};
message GetReplicaListByRegexResponse {
required int32 status_code = 1;
map<string, ObjectReplicaList> object_map = 2; // 匹配到的对象及其副本信息
};
```
* 请求: GetReplicaListByRegexRequest，包含需要匹配的正则表达式 key_regex。
* 响应: GetReplicaListByRegexResponse，包含状态码 status_code 和一个 object_map，该 map 的键是匹配成功的对象 key，值是该 key 对应的副本信息列表。
说明: 用于查询与指定正则表达式匹配的所有 key 及其副本信息。该接口方便进行批量查询和管理。
3. PutStart
```protobuf
message PutStartRequest {
@ -210,7 +255,7 @@ message PutStartResponse {
说明: Client 在写入对象前，需要先调用 PutStart 向 `Master Service` 申请存储空间。`Master Service` 会根据 config 分配空间，并将分配结果（replica_list）返回给 Client。Client 随后将数据写入到分配副本所在的存储节点。 之所以需要 start 和 end 两步，是为确保其他Client不会读到正在写的值，进而造成脏读。
3. PutEnd
4. PutEnd
```protobuf
message PutEndRequest {
@ -227,7 +272,7 @@ message PutEndResponse {
Client 完成数据写入后，调用 PutEnd 通知 `Master Service`。`Master Service` 将更新对象的元数据信息，将副本状态标记为 COMPLETE，表示该对象可以被读取。
4. Remove
5. Remove
```protobuf
message RemoveRequest {
@ -244,7 +289,25 @@ message RemoveResponse {
用于删除指定 key 对应的对象及其所有副本。Master Service 将对应对象的所有副本状态标记为删除。
5. MountSegment
6. RemoveByRegex
```protobuf
message RemoveByRegexRequest {
required string key_regex = 1;
};
message RemoveByRegexResponse {
required int32 status_code = 1;
optional int64 removed_count = 2; // 被删除的对象数量
};
```
* 请求: RemoveByRegexRequest，包含需要匹配的正则表达式 key_regex。
* 响应: RemoveByRegexResponse，包含状态码 status_code 和被删除对象的数量 removed_count。
说明: 用于删除与指定正则表达式匹配的所有对象及其全部副本。与 Remove 接口类似，这是一个元数据操作，Master Service 将所有匹配对象的副本状态标记为删除。
7. MountSegment
```protobuf
message MountSegmentRequest {
@ -260,7 +323,7 @@ message MountSegmentResponse {
存储节点(Client)自己分配一段内存，然后在调用`TransferEngine::registerLoalMemory` 完成本地挂载后，调用该接口，将分配好的一段连续的地址空间挂载到`Master Service`用于分配。
6. UnmountSegment
8. UnmountSegment
```protobuf
message UnmountSegmentRequest {

78
docs/source/design/mooncake-store-preview.md
View File

@ -107,6 +107,23 @@ tl::expected<void, ErrorCode> Remove(const ObjectKey& key);
Used to delete the object corresponding to the specified key. This interface marks all data replicas associated with the key in the storage engine as deleted, without needing to communicate with the corresponding storage node (Client).
### QueryByRegex
```C++
tl::expected<std::unordered_map<std::string, std::vector<Replica::Descriptor>>, ErrorCode>
QueryByRegex(const std::string& str);
```
Used to query the replica information for all objects whose keys match the given regular expression. This is useful for batch operations or for retrieving a group of related objects. The operation is performed on the Master and returns a map of keys to their replica lists.
### RemoveByRegex
```C++
tl::expected<long, ErrorCode> RemoveByRegex(const ObjectKey& str);
```
Used to delete all objects from the store whose keys match the specified regular expression. This provides a powerful way to perform bulk deletions. The command returns the number of objects that were successfully removed.
### Master Service
The cluster's available resources are viewed as a large resource pool, managed centrally by a Master process for space allocation and guiding data replication
@ -150,6 +167,9 @@ service MasterService {
// Get the list of replicas for an object
rpc GetReplicaList(GetReplicaListRequest) returns (GetReplicaListResponse);
// Get replica lists for objects matching a regex
rpc GetReplicaListByRegex(GetReplicaListByRegexRequest) returns (GetReplicaListByRegexResponse);
// Start Put operation, allocate storage space
rpc PutStart(PutStartRequest) returns (PutStartResponse);
@ -159,6 +179,9 @@ service MasterService {
// Delete all replicas of an object
rpc Remove(RemoveRequest) returns (RemoveResponse);
// Remove objects matching a regex
rpc RemoveByRegex(RemoveByRegexRequest) returns (RemoveByRegexResponse);
// Storage node (Client) registers a storage segment
rpc MountSegment(MountSegmentRequest) returns (MountSegmentResponse);
@ -184,7 +207,28 @@ message GetReplicaListResponse {
- **Response**: `GetReplicaListResponse` containing the status code status_code and the list of replica information `replica_list`.
- **Description**: Used to retrieve information about all available replicas for a specified key. The Client can select an appropriate replica for reading based on this information.
2. PutStart
2. GetReplicaListByRegex
```protobuf
message GetReplicaListByRegexRequest {
required string key_regex = 1;
};
message ObjectReplicaList {
repeated ReplicaInfo replica_list = 1;
};
message GetReplicaListByRegexResponse {
required int32 status_code = 1;
map<string, ObjectReplicaList> object_map = 2; // Matched objects and their replica information.
};
```
- **Request**: GetReplicaListByRegexRequest, which contains the regular expression key_regex to be matched.
- **Response**: GetReplicaListByRegexResponse, which contains a status_code and an object_map. The keys of this map are the successfully matched object keys, and the values are the lists of replica information for each key.
- **Description**: Used to query for all keys and their replica information that match the specified regular expression. This interface facilitates bulk queries and management.
3. PutStart
```protobuf
message PutStartRequest {
@ -204,7 +248,7 @@ message PutStartResponse {
- **Response**: `PutStartResponse` containing the status code status_code and the allocated replica information replica_list.
- **Description**: Before writing an object, the Client must call PutStart to request storage space from the Master Service. The Master Service allocates space based on the config and returns the allocation results (`replica_list`) to the Client. The Client then writes data to the storage nodes where the allocated replicas are located. The need for both start and end steps ensures that other Clients do not read partially written values, preventing dirty reads.
3. PutEnd
4. PutEnd
```protobuf
message PutEndRequest {
@ -220,7 +264,7 @@ message PutEndResponse {
- **Response**: `PutEndResponse` containing the status code status_code.
- **Description**: After the Client completes data writing, it calls `PutEnd` to notify the Master Service. The Master Service updates the object's metadata, marking the replica status as `COMPLETE`, indicating that the object is readable.
4. Remove
5. Remove
```protobuf
message RemoveRequest {
@ -236,7 +280,24 @@ message RemoveResponse {
- **Response**: `RemoveResponse` containing the status code `status_code`.
- **Description**: Used to delete the object and all its replicas corresponding to the specified key. The Master Service marks all replicas of the corresponding object as deleted.
5. MountSegment
6. RemoveByRegex
```protobuf
message RemoveByRegexRequest {
required string key_regex = 1;
};
message RemoveByRegexResponse {
required int32 status_code = 1;
optional int64 removed_count = 2; // The number of objects removed.
};
```
- **Request**: RemoveByRegexRequest, which contains the regular expression key_regex to be matched.
- **Response**: RemoveByRegexResponse, which contains a status_code and the number of objects that were removed, removed_count.
- **Description**: Used to delete all objects and their corresponding replicas for keys that match the specified regular expression. Similar to the Remove interface, this is a metadata operation where the Master Service marks the status of all matched object replicas as removed.
7. MountSegment
```protobuf
message MountSegmentRequest {
@ -252,7 +313,7 @@ message MountSegmentResponse {
The storage node (Client) allocates a segment of memory and, after calling `TransferEngine::registerLocalMemory` to complete local mounting, calls this interface to mount the allocated continuous address space to the Master Service for allocation.
6. UnmountSegment
8. UnmountSegment
```protobuf
message UnmountSegmentRequest {
@ -309,17 +370,20 @@ Before writing an object, the Client calls PutStart to request storage space all
```C++
ErrorCode GetReplicaList(const std::string& key,
std::vector<ReplicaInfo>& replica_list);
tl::expected<std::unordered_map<std::string, std::vector<Replica::Descriptor>>, ErrorCode>
GetReplicaListByRegex(const std::string& str);
```
The Client requests the Master Service to retrieve the replica list for a specified key, allowing the Client to select an appropriate replica for reading based on this information.
The Client requests the Master Service to retrieve the replica list for a specified key or for all object keys matching a specified regular expression, allowing the Client to select an appropriate replica for reading based on this information.
- Remove
```C++
tl::expected<void, ErrorCode> Remove(const std::string& key);
tl::expected<long, ErrorCode> RemoveByRegex(const std::string& str);
```
The Client requests the Master Service to delete all replicas corresponding to the specified key.
The Client requests the Master Service to delete all replicas corresponding to the specified key or for all object keys that match the specified regular expression.
### Buffer Allocator

23
docs/source/mooncake-store-api/python-binding.md
View File

@ -585,6 +585,29 @@ if result == 0:
---
#### remove_by_regex()
Remove objects from the storage system whose keys match a regular expression.
```python
def remove_by_regex(self, regex: str) -> int
```
**Parameters:**
- `regex` (str): The regular expression to match against object keys.
**Returns:**
- `int`: The number of objects removed, or a negative value on error.
**Example:**
```python
# Remove all keys starting with "user_session_"
count = store.remove_by_regex("^user_session_.*")
if count >= 0:
print(f"Removed {count} objects")
```
---
#### remove_all()
Remove all objects from the storage system.

12
mooncake-store/include/client.h
View File

@ -74,6 +74,12 @@ class Client {
tl::expected<std::vector<Replica::Descriptor>, ErrorCode> Query(
const std::string& object_key);
/**
* @brief Queries replica lists for object keys that match a regex pattern.
* @param str The regular expression string to match against object keys.
* @return An expected object containing a map from object keys to their
* replica descriptors on success, or an ErrorCode on failure.
*/
tl::expected<
std::unordered_map<std::string, std::vector<Replica::Descriptor>>,
ErrorCode>
@ -141,6 +147,12 @@ class Client {
*/
tl::expected<void, ErrorCode> Remove(const ObjectKey& key);
/**
* @brief Removes objects from the store whose keys match a regex pattern.
* @param str The regular expression string to match against object keys.
* @return An expected object containing the number of removed objects on
* success, or an ErrorCode on failure.
*/
tl::expected<long, ErrorCode> RemoveByRegex(const ObjectKey& str);
/**

13
mooncake-store/include/master_client.h
View File

@ -56,6 +56,13 @@ class MasterClient {
[[nodiscard]] tl::expected<std::vector<Replica::Descriptor>, ErrorCode>
GetReplicaList(const std::string& object_key);
/**
* @brief Retrieves replica lists for object keys that match a regex
* pattern.
* @param str The regular expression string to match against object keys.
* @return An expected object containing a map from object keys to their
* replica descriptors on success, or an ErrorCode on failure.
*/
[[nodiscard]] tl::expected<
std::unordered_map<std::string, std::vector<Replica::Descriptor>>,
ErrorCode>
@ -139,6 +146,12 @@ class MasterClient {
*/
[[nodiscard]] tl::expected<void, ErrorCode> Remove(const std::string& key);
/**
* @brief Removes objects from the master whose keys match a regex pattern.
* @param str The regular expression string to match against object keys.
* @return An expected object containing the number of removed objects on
* success, or an ErrorCode on failure.
*/
[[nodiscard]] tl::expected<long, ErrorCode> RemoveByRegex(
const std::string& str);

13
mooncake-store/include/master_service.h
View File

@ -107,6 +107,13 @@ class MasterService {
auto QuerySegments(const std::string& segment)
-> tl::expected<std::pair<size_t, size_t>, ErrorCode>;
/**
* @brief Retrieves replica lists for object keys that match a regex
* pattern.
* @param str The regular expression string to match against object keys.
* @return An expected object containing a map from object keys to their
* replica descriptors on success, or an ErrorCode on failure.
*/
auto GetReplicaListByRegex(const std::string& regex_pattern)
-> tl::expected<
std::unordered_map<std::string, std::vector<Replica::Descriptor>>,
@ -182,6 +189,12 @@ class MasterService {
*/
auto Remove(const std::string& key) -> tl::expected<void, ErrorCode>;
/**
* @brief Removes objects from the master whose keys match a regex pattern.
* @param str The regular expression string to match against object keys.
* @return An expected object containing the number of removed objects on
* success, or an ErrorCode on failure.
*/
auto RemoveByRegex(const std::string& str) -> tl::expected<long, ErrorCode>;
/**

7
mooncake-store/include/storage_backend.h
View File

@ -133,6 +133,13 @@ class StorageBackend {
*/
void RemoveFile(const std::string& path);
/**
* @brief Removes objects from the storage backend whose keys match a regex
* pattern.
* @param regex The regular expression string to match against object keys.
* @return An expected object containing the number of removed objects on
* success, or an ErrorCode on failure.
*/
void RemoveByRegex(const std::string& key);
/**