[Doc] improve Transfer Engine C++ API Reference (#1467)
* [Doc] improve Transfer Engine C++ API Reference * Update docs/source/design/transfer-engine/cpp-api.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update docs/source/design/transfer-engine/cpp-api.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update docs/source/design/transfer-engine/index.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update docs/source/design/transfer-engine/cpp-api.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
fatSheep
GitHub
parent
53f4f9ccb4
commit
9b242111aa
|
|
@ -0,0 +1,556 @@
|
|||
# Transfer Engine C++ API Reference
|
||||
|
||||
## Overview
|
||||
This page summarizes the C++ APIs in `mooncake-transfer-engine/include/transfer_engine.h`.
|
||||
It follows the same narrative style as the design doc, while covering the full set of public C++ APIs.
|
||||
|
||||
For conceptual background, see [Transfer Engine](index.md).
|
||||
|
||||
**Core APIs vs Advanced APIs**
|
||||
- Core APIs form the minimal path to move data: initialize the engine, register memory, open segments, submit transfers, and query status.
|
||||
- Advanced APIs are optional; they help with transport control, notifications, metadata/cache maintenance, topology discovery, and debugging.
|
||||
|
||||
## Core APIs
|
||||
|
||||
### Core Usage Path (C++)
|
||||
Most integrations follow a short, repeatable path: initialize the engine, register local memory, open a target segment, submit a batch transfer, poll status, and finally free resources (batch, segment, memory). A minimal example is shown below.
|
||||
|
||||
```cpp
|
||||
#include "transfer_engine.h"
|
||||
|
||||
using mooncake::TransferEngine;
|
||||
using mooncake::TransferRequest;
|
||||
using mooncake::TransferStatus;
|
||||
|
||||
TransferEngine engine(true);
|
||||
engine.init("etcd://127.0.0.1:2379", "node0");
|
||||
|
||||
void *local_addr = /* local buffer address */;
|
||||
size_t length = /* bytes to transfer */;
|
||||
uint64_t remote_addr = /* remote address or file offset */;
|
||||
|
||||
engine.registerLocalMemory(local_addr, length, "cpu:0", true);
|
||||
|
||||
auto segment = engine.openSegment("node1");
|
||||
auto batch = engine.allocateBatchID(1);
|
||||
|
||||
TransferRequest req{};
|
||||
req.opcode = TransferRequest::WRITE;
|
||||
req.source = local_addr;
|
||||
req.target_id = segment;
|
||||
req.target_offset = remote_addr;
|
||||
req.length = length;
|
||||
|
||||
engine.submitTransfer(batch, {req});
|
||||
|
||||
TransferStatus status{};
|
||||
engine.getTransferStatus(batch, 0, status);
|
||||
|
||||
engine.freeBatchID(batch);
|
||||
engine.closeSegment(segment);
|
||||
engine.unregisterLocalMemory(local_addr);
|
||||
```
|
||||
|
||||
### Constructors
|
||||
|
||||
```cpp
|
||||
TransferEngine(bool auto_discover = false);
|
||||
TransferEngine(bool auto_discover, const std::vector<std::string>& filter);
|
||||
```
|
||||
|
||||
Constructs a Transfer Engine instance. `auto_discover` enables topology discovery; `filter` constrains discovery or device selection with a whitelist.
|
||||
|
||||
### Data Transfer
|
||||
|
||||
#### TransferEngine::TransferRequest
|
||||
|
||||
The core API provided by Mooncake Transfer Engine is submitting a group of asynchronous `TransferRequest` tasks through the `submitTransfer` interface, and querying their status through the `getTransferStatus` interface. Each `TransferRequest` specifies reading or writing a continuous data space of `length` starting from the local starting address `source`, to the position starting at `target_offset` in the segment corresponding to `target_id`.
|
||||
|
||||
The `TransferRequest` structure is defined as follows:
|
||||
|
||||
```cpp
|
||||
struct TransferRequest
|
||||
{
|
||||
enum OpCode { READ, WRITE };
|
||||
OpCode opcode;
|
||||
void *source;
|
||||
SegmentID target_id; // The ID of the target segment, which may correspond to local or remote DRAM/VRAM/NVMeof, with the specific routing logic hidden
|
||||
uint64_t target_offset;
|
||||
size_t length;
|
||||
int advise_retry_cnt = 0;
|
||||
};
|
||||
```
|
||||
|
||||
- `opcode` takes the values `READ` or `WRITE`. `READ` indicates that data is copied from the target address indicated by `<target_id, target_offset>` to the local starting address `source`; `WRITE` indicates that data is copied from `source` to the address indicated by `<target_id, target_offset>`.
|
||||
- `source` represents the DRAM/VRAM buffer managed by the current `TransferEngine`, which must have been registered in advance by the `registerLocalMemory` interface.
|
||||
- `target_id` represents the segment ID of the transfer target. The segment ID is obtained using the `openSegment` interface. Segments are divided into the following types:
|
||||
- RAM space type, covering DRAM/VRAM. As mentioned earlier, there is only one segment under the same process (or `TransferEngine` instance), which contains various types of Buffers (DRAM/VRAM). In this case, the segment name passed to the `openSegment` interface is equivalent to the server hostname. `target_offset` is the virtual address of the target server.
|
||||
- NVMeOF space type, where each file corresponds to a segment. In this case, the segment name passed to the `openSegment` interface is equivalent to the unique identifier of the file. `target_offset` is the offset of the target file.
|
||||
- `length` represents the amount of data transferred. TransferEngine may further split this into multiple read/write requests internally.
|
||||
|
||||
#### TransferEngine::allocateBatchID
|
||||
|
||||
```cpp
|
||||
BatchID allocateBatchID(size_t batch_size);
|
||||
```
|
||||
|
||||
Allocates a `BatchID`. A maximum of `batch_size` `TransferRequest`s can be submitted under the same `BatchID`.
|
||||
|
||||
- `batch_size`: The maximum number of `TransferRequest`s that can be submitted under the same `BatchID`;
|
||||
- Return value: If successful, returns a `BatchID`; on failure, returns an invalid handle (for example `INVALID_BATCH_ID`).
|
||||
|
||||
#### TransferEngine::submitTransfer
|
||||
|
||||
```cpp
|
||||
Status submitTransfer(BatchID batch_id,
|
||||
const std::vector<TransferRequest> &entries);
|
||||
```
|
||||
|
||||
Submits new `TransferRequest` tasks to `batch_id`. The task is asynchronously submitted to the background thread pool. The total number of `entries` accumulated under the same `batch_id` should not exceed the `batch_size` defined at creation.
|
||||
|
||||
- `batch_id`: The `BatchID` it belongs to;
|
||||
- `entries`: Array of `TransferRequest`;
|
||||
- Return value: If successful, returns an OK status; otherwise, returns a non-OK status.
|
||||
|
||||
#### TransferEngine::getTransferStatus
|
||||
|
||||
```cpp
|
||||
enum TransferStatusEnum
|
||||
{
|
||||
WAITING, // In the transfer phase
|
||||
PENDING, // Not supported
|
||||
INVALID, // Invalid parameters
|
||||
CANCELED, // Not supported
|
||||
COMPLETED, // Transfer completed
|
||||
TIMEOUT, // Not supported
|
||||
FAILED // Transfer failed even after retries
|
||||
};
|
||||
struct TransferStatus {
|
||||
TransferStatusEnum s;
|
||||
size_t transferred_bytes; // How much data has been successfully transferred (not necessarily an accurate value, but it is a lower bound)
|
||||
};
|
||||
Status getTransferStatus(BatchID batch_id, size_t task_id, TransferStatus &status)
|
||||
```
|
||||
|
||||
Obtains the running status of the `TransferRequest` with `task_id` in `batch_id`.
|
||||
|
||||
- `batch_id`: The `BatchID` it belongs to;
|
||||
- `task_id`: The sequence number of the `TransferRequest` to query;
|
||||
- `status`: Output Transfer status;
|
||||
- Return value: If successful, returns an OK status; otherwise, returns a non-OK status.
|
||||
|
||||
#### TransferEngine::getBatchTransferStatus
|
||||
|
||||
```cpp
|
||||
Status getBatchTransferStatus(BatchID batch_id, TransferStatus& status);
|
||||
```
|
||||
|
||||
Obtains the aggregated status of the batch and the total transferred bytes.
|
||||
|
||||
- `batch_id`: The `BatchID` it belongs to;
|
||||
- `status`: Output Transfer status;
|
||||
- Return value: If successful, returns an OK status; otherwise, returns a non-OK status.
|
||||
|
||||
#### TransferEngine::freeBatchID
|
||||
|
||||
```cpp
|
||||
Status freeBatchID(BatchID batch_id);
|
||||
```
|
||||
|
||||
Recycles `BatchID`, and subsequent operations on `submitTransfer` and `getTransferStatus` are undefined. If there are still `TransferRequest`s pending completion in the `BatchID`, the operation is refused.
|
||||
|
||||
- `batch_id`: The `BatchID` it belongs to;
|
||||
- Return value: If successful, returns an OK status; otherwise, returns a non-OK status.
|
||||
|
||||
### Space Registration
|
||||
|
||||
For the RDMA transfer process, the source pointer `TransferRequest::source` must be registered in advance as an RDMA readable/writable Memory Region space, that is, included as part of the RAM Segment of the current process. Therefore, the following functions are needed:
|
||||
|
||||
#### TransferEngine::registerLocalMemory
|
||||
|
||||
```cpp
|
||||
int registerLocalMemory(void *addr,
|
||||
size_t length,
|
||||
const std::string& location = kWildcardLocation,
|
||||
bool remote_accessible = true,
|
||||
bool update_metadata = true);
|
||||
```
|
||||
|
||||
Registers a space starting at address `addr` with a length of `length` on the local DRAM/VRAM.
|
||||
|
||||
- `addr`: The starting address of the registration space;
|
||||
- `length`: The length of the registration space;
|
||||
- `location`: The `device` corresponding to this memory segment, such as `cuda:0` indicating the GPU device, `cpu:0` indicating the CPU socket, by matching with the network card priority order table (see `installTransport`), the preferred network card is identified. You can also use `*`, Transfer Engine will try to automatically recognize the `device` corresponding to `addr`, if it fails to recognize the device, it will print a `WARNING` level log and use all network cards, no preferred network cards.
|
||||
- `remote_accessible`: Indicates whether this memory can be accessed by remote nodes.
|
||||
- `update_metadata`: Whether to publish the registration to the metadata service.
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
|
||||
#### TransferEngine::unregisterLocalMemory
|
||||
|
||||
```cpp
|
||||
int unregisterLocalMemory(void *addr, bool update_metadata = true);
|
||||
```
|
||||
|
||||
Unregisters the region.
|
||||
|
||||
- `addr`: The starting address of the registration space;
|
||||
- `update_metadata`: Whether to publish the unregistration to the metadata service.
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
|
||||
#### TransferEngine::registerLocalMemoryBatch
|
||||
|
||||
```cpp
|
||||
int registerLocalMemoryBatch(const std::vector<BufferEntry>& buffer_list,
|
||||
const std::string& location);
|
||||
```
|
||||
|
||||
Registers multiple buffers in one call to reduce registration overhead.
|
||||
|
||||
- `buffer_list`: A list of `{addr, length}` entries.
|
||||
- `location`: The `device` corresponding to these buffers.
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
|
||||
#### TransferEngine::unregisterLocalMemoryBatch
|
||||
|
||||
```cpp
|
||||
int unregisterLocalMemoryBatch(const std::vector<void*>& addr_list);
|
||||
```
|
||||
|
||||
Unregisters multiple buffers in one call.
|
||||
|
||||
- `addr_list`: A list of buffer start addresses.
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
|
||||
### Segment Management and Metadata Format
|
||||
|
||||
TransferEngine provides the `openSegment` function, which obtains a `SegmentHandle` for subsequent `Transport` transfers.
|
||||
```cpp
|
||||
SegmentHandle openSegment(const std::string& segment_name);
|
||||
```
|
||||
|
||||
- `segment_name`: The unique identifier of the segment. For RAM Segment, this needs to be consistent with the `server_name` filled in by the peer process when initializing the TransferEngine object.
|
||||
- Return value: If successful, returns the corresponding `SegmentHandle`; otherwise, returns an invalid handle.
|
||||
|
||||
```cpp
|
||||
int closeSegment(SegmentHandle segment_id);
|
||||
```
|
||||
|
||||
- `segment_id`: The unique identifier of the segment.
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
|
||||
#### TransferEngine::removeLocalSegment
|
||||
|
||||
```cpp
|
||||
int removeLocalSegment(const std::string& segment_name);
|
||||
```
|
||||
|
||||
Removes local segment metadata entries.
|
||||
|
||||
- `segment_name`: The local segment name to remove.
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
|
||||
#### TransferEngine::CheckSegmentStatus
|
||||
|
||||
```cpp
|
||||
Status CheckSegmentStatus(SegmentID sid);
|
||||
```
|
||||
|
||||
Checks whether a segment is reachable and valid.
|
||||
|
||||
- `sid`: The segment identifier.
|
||||
- Return value: If successful, returns an OK status; otherwise, returns a non-OK status.
|
||||
|
||||
<details>
|
||||
<summary><strong>Metadata Format</strong></summary>
|
||||
|
||||
```
|
||||
// Used to find the communicable address and exposed rpc port based on server_name.
|
||||
// Created: when calling TransferEngine::init().
|
||||
// Deleted: when TransferEngine is destructed.
|
||||
Key = mooncake/rpc_meta/[server_name]
|
||||
Value = {
|
||||
'ip_or_host_name': 'node01'
|
||||
'rpc_port': 12345
|
||||
}
|
||||
|
||||
// For segments, the key naming method of mooncake/[proto]/[segment_name] is used, and the segment name can use the Server Name.
|
||||
// A segment corresponds to a machine, and a buffer corresponds to different segments of memory or different files or different disks on the machine. Different buffers of the same segment are in the same fault domain.
|
||||
|
||||
// RAM Segment, used by RDMA Transport to obtain transfer information.
|
||||
// Created: command line tool register.py, at this time buffers are empty, only fill in the information that can be known in advance.
|
||||
// Modified: TransferEngine at runtime through register / unregister to add or delete Buffer.
|
||||
Key = mooncake/ram/[segment_name]
|
||||
Value = {
|
||||
'server_name': server_name,
|
||||
'protocol': rdma,
|
||||
'devices': [
|
||||
{ 'name': 'mlx5_2', 'lid': 17, 'gid': 'fe:00:...' },
|
||||
{ 'name': 'mlx5_3', 'lid': 22, 'gid': 'fe:00:...' }
|
||||
],
|
||||
'priority_matrix': {
|
||||
"cpu:0": [["mlx5_2"], ["mlx5_3"]],
|
||||
"cpu:1": [["mlx5_3"], ["mlx5_2"]],
|
||||
"cuda:0": [["mlx5_2"], ["mlx5_3"]],
|
||||
},
|
||||
'buffers': [
|
||||
{
|
||||
'name': 'cpu:0',
|
||||
'addr': 0x7fa16bdf5000,
|
||||
'length': 1073741824,
|
||||
'rkey': [1fe000, 1fdf00, ...], // The length is the same as the number of elements in the 'devices' field
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
// Created: command line tool register.py, determine the file path that can be mounted.
|
||||
// Modified: command line tool mount.py, add a mapping of the machine mounting the file to the file path on the mounting machine to the buffers.local_path_map.
|
||||
Key = mooncake/nvmeof/[segment_name]
|
||||
Value = {
|
||||
'server_name': server_name,
|
||||
'protocol': nvmeof,
|
||||
'buffers':[{
|
||||
'length': 1073741824,
|
||||
'file_path': "/mnt/nvme0" // The file path on this machine
|
||||
'local_path_map': {
|
||||
"node01": "/mnt/transfer_engine/node01/nvme0", // The machine mounting the file -> The file path on the mounting machine
|
||||
...
|
||||
},
|
||||
},
|
||||
{
|
||||
'length': 1073741824,
|
||||
'file_path': "/mnt/nvme1",
|
||||
'local_path_map': {
|
||||
"node02": "/mnt/transfer_engine/node02/nvme1",
|
||||
...
|
||||
},
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
</details>
|
||||
|
||||
### HTTP Metadata Server
|
||||
|
||||
The HTTP server should implement three following RESTful APIs, while the metadata server configured to `http://host:port/metadata` as an example:
|
||||
|
||||
1. `GET /metadata?key=$KEY`: Get the metadata corresponding to `$KEY`.
|
||||
2. `PUT /metadata?key=$KEY`: Update the metadata corresponding to `$KEY` to the value of the request body.
|
||||
3. `DELETE /metadata?key=$KEY`: Delete the metadata corresponding to `$KEY`.
|
||||
|
||||
For specific implementation, refer to the demo service implemented in Golang at [mooncake-transfer-engine/example/http-metadata-server](../../../mooncake-transfer-engine/example/http-metadata-server).
|
||||
|
||||
### Initialization
|
||||
|
||||
TransferEngine needs to be initialized by calling the `init` method before further actions:
|
||||
```cpp
|
||||
TransferEngine();
|
||||
|
||||
int init(const std::string &metadata_conn_string,
|
||||
const std::string &local_server_name);
|
||||
```
|
||||
There is also an extended form to override RPC binding:
|
||||
```cpp
|
||||
int init(const std::string &metadata_conn_string,
|
||||
const std::string &local_server_name,
|
||||
const std::string &ip_or_host_name,
|
||||
uint64_t rpc_port);
|
||||
```
|
||||
- `metadata_conn_string`: Connecting string of metadata storage servers, i.e., the IP address/hostname of `etcd`/`redis` or the URI of the http service.
|
||||
The general form is `[proto]://[hostname:port]`. For example, the following metadata server addresses are legal:
|
||||
- Using `etcd` as a metadata storage service: `"10.0.0.1:2379"` or `"etcd://10.0.0.1:2379"`.
|
||||
- Using `redis` as a metadata storage service: `"redis://10.0.0.1:6379"`
|
||||
- Using `http` as a metadata storage service: `"http://10.0.0.1:8080/metadata"`
|
||||
|
||||
- `local_server_name`: The local server name, ensuring uniqueness within the cluster. It also serves as the name of the RAM Segment that other nodes refer to the current instance (i.e., Segment Name).
|
||||
- `ip_or_host_name`: Optional explicit bind address or hostname for the RPC service.
|
||||
- `rpc_port`: Optional explicit RPC port.
|
||||
|
||||
```cpp
|
||||
~TransferEngine();
|
||||
```
|
||||
|
||||
Reclaims all allocated resources and also deletes the global metadata server information.
|
||||
|
||||
#### TransferEngine::freeEngine
|
||||
|
||||
```cpp
|
||||
int freeEngine();
|
||||
```
|
||||
|
||||
Releases resources early without waiting for destruction.
|
||||
|
||||
## Advanced APIs
|
||||
These APIs are optional; use them when you need manual transport control, coordination notifications, metadata/cache refresh, topology discovery, or debugging. They are not required for the basic data path.
|
||||
|
||||
### Multi-Transport Management
|
||||
|
||||
The `TransferEngine` class internally manages multiple backend `Transport` classes.
|
||||
And it will discover the topology between CPU/CUDA and RDMA devices automatically
|
||||
(more device types are working in progress, feedbacks are welcome when the automatic discovery mechanism is not accurate),
|
||||
and it will install `Transport` automatically based on the topology.
|
||||
|
||||
#### TransferEngine::installTransport
|
||||
|
||||
```cpp
|
||||
Transport* installTransport(const std::string& proto, void** args);
|
||||
```
|
||||
|
||||
Installs a transport backend explicitly.
|
||||
|
||||
- `proto`: Transport protocol name, such as `rdma`, `tcp`, or `nvmeof`.
|
||||
- `args`: Transport-specific arguments.
|
||||
> Note: In TENT, `installTransport` is not exposed (removed from the public API, including compatibility surfaces). Transport selection is internal to TENT.
|
||||
|
||||
#### TransferEngine::uninstallTransport
|
||||
|
||||
```cpp
|
||||
int uninstallTransport(const std::string& proto);
|
||||
```
|
||||
|
||||
Uninstalls a transport backend.
|
||||
|
||||
- `proto`: Transport protocol name.
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
|
||||
#### TransferEngine::getTransport
|
||||
|
||||
```cpp
|
||||
Transport* getTransport(const std::string& proto);
|
||||
```
|
||||
|
||||
Returns a transport instance by protocol name, mainly for advanced inspection or debugging.
|
||||
|
||||
### Notifications
|
||||
|
||||
TransferEngine can send and receive lightweight notifications across segments to coordinate data movement.
|
||||
|
||||
#### TransferEngine::submitTransferWithNotify
|
||||
|
||||
```cpp
|
||||
Status submitTransferWithNotify(BatchID batch_id,
|
||||
const std::vector<TransferRequest>& entries,
|
||||
TransferMetadata::NotifyDesc notify_msg);
|
||||
```
|
||||
|
||||
Submits a batch transfer and delivers a notification upon completion.
|
||||
|
||||
- `notify_msg`: A `{name, msg}` payload delivered to the receiver.
|
||||
- Typical use: signal that a transferred buffer or KV-cache slice is ready to consume on the receiver side.
|
||||
|
||||
#### TransferEngine::getNotifies
|
||||
|
||||
```cpp
|
||||
int getNotifies(std::vector<TransferMetadata::NotifyDesc>& notifies);
|
||||
```
|
||||
|
||||
Gets pending notifications from peers.
|
||||
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
- Typical use: receiver-side polling loop to trigger follow-up actions (e.g., attaching the transferred buffer to a scheduler or cache).
|
||||
|
||||
#### TransferEngine::sendNotifyByID
|
||||
|
||||
```cpp
|
||||
int sendNotifyByID(SegmentID target_id, TransferMetadata::NotifyDesc notify_msg);
|
||||
```
|
||||
|
||||
Sends a notification to a specific segment by ID.
|
||||
- Typical use: control-plane message such as "ready", "invalidate", or "retry" tied to a known segment handle.
|
||||
|
||||
#### TransferEngine::sendNotifyByName
|
||||
|
||||
```cpp
|
||||
int sendNotifyByName(std::string remote_agent, TransferMetadata::NotifyDesc notify_msg);
|
||||
```
|
||||
|
||||
Sends a notification to a remote agent by name.
|
||||
- Typical use: best-effort coordination when you only know the peer's name (segment name), not the handle.
|
||||
|
||||
### Segment Cache and Metadata Access
|
||||
|
||||
#### TransferEngine::syncSegmentCache
|
||||
|
||||
```cpp
|
||||
int syncSegmentCache(const std::string& segment_name = "");
|
||||
```
|
||||
|
||||
Synchronizes local segment cache from the metadata service.
|
||||
|
||||
- `segment_name`: If empty, refreshes all segments; otherwise, refreshes the specified segment.
|
||||
- Typical use: when peers dynamically register/unregister buffers and you need to refresh before opening or submitting transfers.
|
||||
|
||||
#### TransferEngine::getMetadata
|
||||
|
||||
```cpp
|
||||
std::shared_ptr<TransferMetadata> getMetadata();
|
||||
```
|
||||
|
||||
Returns the metadata subsystem instance for advanced workflows.
|
||||
- Typical use: advanced integration or debugging of metadata content beyond the standard APIs.
|
||||
|
||||
### Topology and Discovery
|
||||
|
||||
#### TransferEngine::setAutoDiscover
|
||||
|
||||
```cpp
|
||||
void setAutoDiscover(bool auto_discover);
|
||||
```
|
||||
|
||||
Enables or disables topology discovery.
|
||||
- Typical use: controlled environments or debugging where auto discovery is not desired.
|
||||
|
||||
#### TransferEngine::setWhitelistFilters
|
||||
|
||||
```cpp
|
||||
void setWhitelistFilters(std::vector<std::string>&& filters);
|
||||
```
|
||||
|
||||
Sets the device whitelist used during topology discovery.
|
||||
- Typical use: restrict transfers to a subset of NICs or GPUs for performance isolation or testing.
|
||||
|
||||
#### TransferEngine::numContexts
|
||||
|
||||
```cpp
|
||||
int numContexts() const;
|
||||
```
|
||||
|
||||
Returns the number of active transport contexts.
|
||||
- Typical use: introspection and resource monitoring.
|
||||
|
||||
#### TransferEngine::getLocalTopology
|
||||
|
||||
```cpp
|
||||
std::shared_ptr<Topology> getLocalTopology();
|
||||
```
|
||||
|
||||
Returns the local topology model used for path selection.
|
||||
- Typical use: diagnose path selection or export topology to a scheduler.
|
||||
|
||||
### Introspection and Safety
|
||||
|
||||
#### TransferEngine::getLocalIpAndPort
|
||||
|
||||
```cpp
|
||||
std::string getLocalIpAndPort();
|
||||
```
|
||||
|
||||
Returns the resolved local RPC address.
|
||||
- Typical use: publish the resolved address to external components or logs.
|
||||
|
||||
#### TransferEngine::getRpcPort
|
||||
|
||||
```cpp
|
||||
int getRpcPort();
|
||||
```
|
||||
|
||||
Returns the active RPC port.
|
||||
- Typical use: verify port binding when dynamic ports are used.
|
||||
|
||||
#### TransferEngine::checkOverlap
|
||||
|
||||
```cpp
|
||||
bool checkOverlap(void* addr, uint64_t length);
|
||||
```
|
||||
|
||||
Checks whether a given address range overlaps with existing registered buffers.
|
||||
- Typical use: validate registration plans before calling `registerLocalMemory` in complex buffer managers.
|
||||
|
|
@ -157,94 +157,10 @@ The following video shows a normal run as described above, with the Target on th
|
|||
## Transfer Engine C/C++ API
|
||||
Transfer Engine provides interfaces through the `TransferEngine` class (located in `mooncake-transfer-engine/include/transfer_engine.h`), where the specific data transfer functions for different backends are implemented by the `Transport` class, currently supporting `TcpTransport`, `RdmaTransport`, `NVMeoFTransport`, `NvlinkTransport` (for NVIDIA GPUs), `IntraNodeNvlinkTransport` (for NVIDIA GPUs), and `HipTransport` (for AMD GPUs).
|
||||
|
||||
For a complete C++ API reference, see [Transfer Engine C++ API Reference](cpp-api.md).
|
||||
|
||||
### Data Transfer
|
||||
|
||||
#### TransferEngine::TransferRequest
|
||||
|
||||
The core API provided by Mooncake Transfer Engine is submitting a group of asynchronous `TransferRequest` tasks through the `submitTransfer` interface, and querying their status through the `getTransferStatus` interface. Each `TransferRequest` specifies reading or writing a continuous data space of `length` starting from the local starting address `source`, to the position starting at `target_offset` in the segment corresponding to `target_id`.
|
||||
|
||||
The `TransferRequest` structure is defined as follows:
|
||||
|
||||
```cpp
|
||||
using SegmentID = int32_t;
|
||||
struct TransferRequest
|
||||
{
|
||||
enum OpCode { READ, WRITE };
|
||||
OpCode opcode;
|
||||
void *source;
|
||||
SegmentID target_id; // The ID of the target segment, which may correspond to local or remote DRAM/VRAM/NVMeof, with the specific routing logic hidden
|
||||
size_t target_offset;
|
||||
size_t length;
|
||||
};
|
||||
```
|
||||
|
||||
- `opcode` takes the values `READ` or `WRITE`. `READ` indicates that data is copied from the target address indicated by `<target_id, target_offset>` to the local starting address `source`; `WRITE` indicates that data is copied from `source` to the address indicated by `<target_id, target_offset>`.
|
||||
- `source` represents the DRAM/VRAM buffer managed by the current `TransferEngine`, which must have been registered in advance by the `registerLocalMemory` interface.
|
||||
- `target_id` represents the segment ID of the transfer target. The segment ID is obtained using the `openSegment` interface. Segments are divided into the following types:
|
||||
- RAM space type, covering DRAM/VRAM. As mentioned earlier, there is only one segment under the same process (or `TransferEngine` instance), which contains various types of Buffers (DRAM/VRAM). In this case, the segment name passed to the `openSegment` interface is equivalent to the server hostname. `target_offset` is the virtual address of the target server.
|
||||
- NVMeOF space type, where each file corresponds to a segment. In this case, the segment name passed to the `openSegment` interface is equivalent to the unique identifier of the file. `target_offset` is the offset of the target file.
|
||||
- `length` represents the amount of data transferred. TransferEngine may further split this into multiple read/write requests internally.
|
||||
|
||||
#### TransferEngine::allocateBatchID
|
||||
|
||||
```cpp
|
||||
BatchID allocateBatchID(size_t batch_size);
|
||||
```
|
||||
|
||||
Allocates a `BatchID`. A maximum of `batch_size` `TransferRequest`s can be submitted under the same `BatchID`.
|
||||
|
||||
- `batch_size`: The maximum number of `TransferRequest`s that can be submitted under the same `BatchID`;
|
||||
- Return value: If successful, returns `BatchID` (non-negative); otherwise, returns a negative value.
|
||||
|
||||
#### TransferEngine::submitTransfer
|
||||
|
||||
```cpp
|
||||
int submitTransfer(BatchID batch_id, const std::vector<TransferRequest> &entries);
|
||||
```
|
||||
|
||||
Submits new `TransferRequest` tasks to `batch_id`. The task is asynchronously submitted to the background thread pool. The total number of `entries` accumulated under the same `batch_id` should not exceed the `batch_size` defined at creation.
|
||||
|
||||
- `batch_id`: The `BatchID` it belongs to;
|
||||
- `entries`: Array of `TransferRequest`;
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
|
||||
#### TransferEngine::getTransferStatus
|
||||
|
||||
```cpp
|
||||
enum TaskStatus
|
||||
{
|
||||
WAITING, // In the transfer phase
|
||||
PENDING, // Not supported
|
||||
INVALID, // Ilvalid parameters
|
||||
CANCELED, // Not supported
|
||||
COMPLETED, // Transfer completed
|
||||
TIMEOUT, // Not supported
|
||||
FAILED // Transfer failed even after retries
|
||||
};
|
||||
struct TransferStatus {
|
||||
TaskStatus s;
|
||||
size_t transferred; // How much data has been successfully transferred (not necessarily an accurate value, but it is a lower bound)
|
||||
};
|
||||
int getTransferStatus(BatchID batch_id, size_t task_id, TransferStatus &status)
|
||||
```
|
||||
|
||||
Obtains the running status of the `TransferRequest` with `task_id` in `batch_id`.
|
||||
|
||||
- `batch_id`: The `BatchID` it belongs to;
|
||||
- `task_id`: The sequence number of the `TransferRequest` to query;
|
||||
- `status`: Output Transfer status;
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
|
||||
#### TransferEngine::freeBatchID
|
||||
|
||||
```cpp
|
||||
int freeBatchID(BatchID batch_id);
|
||||
```
|
||||
|
||||
Recycles `BatchID`, and subsequent operations on `submitTransfer` and `getTransferStatus` are undefined. If there are still `TransferRequest`s pending completion in the `BatchID`, the operation is refused.
|
||||
|
||||
- `batch_id`: The `BatchID` it belongs to;
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
Transfer Engine provides batch-based read/write transfers between segments (DRAM/VRAM/NVMeof). A typical flow is: register local memory, open a target segment, submit a batch, and poll status. Detailed function signatures and usage are documented in the C++ API reference.
|
||||
|
||||
### Multi-Transport Management
|
||||
|
||||
|
|
@ -253,51 +169,8 @@ And it will discover the toplogy between CPU/CUDA and RDMA devices automatically
|
|||
(more device types are working in progress, feedbacks are welcome when the automatic discovery mechanism is not accurate),
|
||||
and it will install `Transport` automatically based on the topology.
|
||||
|
||||
### Space Registration
|
||||
|
||||
For the RDMA transfer process, the source pointer `TransferRequest::source` must be registered in advance as an RDMA readable/writable Memory Region space, that is, included as part of the RAM Segment of the current process. Therefore, the following functions are needed:
|
||||
|
||||
#### TransferEngine::registerLocalMemory
|
||||
|
||||
```cpp
|
||||
int registerLocalMemory(void *addr, size_t size, string location, bool remote_accessible);
|
||||
```
|
||||
|
||||
Registers a space starting at address `addr` with a length of `size` on the local DRAM/VRAM.
|
||||
|
||||
- `addr`: The starting address of the registration space;
|
||||
- `size`: The length of the registration space;
|
||||
- `location`: The `device` corresponding to this memory segment, such as `cuda:0` indicating the GPU device, `cpu:0` indicating the CPU socket, by matching with the network card priority order table (see `installTransport`), the preferred network card is identified. You can also use `*`, Transfer Engine will try to automatically recognize the `device` corresponding to `addr`, if it fails to recognize the device, it will print a `WARNING` level log and use all network cards, no preferred network cards.
|
||||
- `remote_accessible`: Indicates whether this memory can be accessed by remote nodes.
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
|
||||
#### TransferEngine::unregisterLocalMemory
|
||||
|
||||
```cpp
|
||||
int unregisterLocalMemory(void *addr);
|
||||
```
|
||||
|
||||
Unregisters the region.
|
||||
|
||||
- `addr`: The starting address of the registration space;
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
|
||||
### Segment Management and Metadata Format
|
||||
|
||||
TransferEngine provides the `openSegment` function, which obtains a `SegmentHandle` for subsequent `Transport` transfers.
|
||||
```cpp
|
||||
SegmentHandle openSegment(const std::string& segment_name);
|
||||
```
|
||||
|
||||
- `segment_name`: The unique identifier of the segment. For RAM Segment, this needs to be consistent with the `server_name` filled in by the peer process when initializing the TransferEngine object.
|
||||
- Return value: If successful, returns the corresponding `SegmentHandle`; otherwise, returns a negative value.
|
||||
|
||||
```cpp
|
||||
int closeSegment(SegmentHandle segment_id);
|
||||
```
|
||||
|
||||
- `segment_id`: The unique identifier of the segment.
|
||||
- Return value: If successful, returns 0; otherwise, returns a negative value.
|
||||
Segment metadata is stored in the metadata service. The following format is provided for reference.
|
||||
|
||||
<details>
|
||||
<summary><strong>Metadata Format</strong></summary>
|
||||
|
|
@ -379,29 +252,6 @@ The HTTP server should implement three following RESTful APIs, while the metadat
|
|||
|
||||
For specific implementation, refer to the demo service implemented in Golang at [mooncake-transfer-engine/example/http-metadata-server](../../../mooncake-transfer-engine/example/http-metadata-server).
|
||||
|
||||
### Initialization
|
||||
|
||||
TransferEngine needs to initializing by calling the `init` method before further actions:
|
||||
```cpp
|
||||
TransferEngine();
|
||||
|
||||
int init(const std::string &metadata_conn_string,
|
||||
const std::string &local_server_name);
|
||||
```
|
||||
- `metadata_conn_string`: Connecting string of metadata storage servers, i.e., the IP address/hostname of `etcd`/`redis` or the URI of the http service.
|
||||
The general form is `[proto]://[hostname:port]`. For example, the following metadata server addresses are legal:
|
||||
- Using `etcd` as a metadata storage service: `“10.0.0.1:2379”` or `“etcd://10.0.0.1:2379”`.
|
||||
- Using `redis` as a metadata storage service: `“redis://10.0.0.1:6379”`
|
||||
- Using `http` as a metadata storage service: `“http://10.0.0.1:8080/metadata”`
|
||||
|
||||
- `local_server_name`: The local server name, ensuring uniqueness within the cluster. It also serves as the name of the RAM Segment that other nodes refer to the current instance (i.e., Segment Name).
|
||||
|
||||
```cpp
|
||||
~TransferEngine();
|
||||
```
|
||||
|
||||
Reclaims all allocated resources and also deletes the global meta data server information.
|
||||
|
||||
## Using Transfer Engine to Your Projects
|
||||
|
||||
### Using C/C++ Interface
|
||||
|
|
@ -453,6 +303,14 @@ For advanced users, TransferEngine provides the following advanced runtime optio
|
|||
- `MC_PATH_ROUNDROBIN` Use round-robin mode in the RDMA path selection. This may be beneficial for transferring large bulks.
|
||||
- `MC_ENDPOINT_STORE_TYPE` Choose FIFO Endpoint Store (`FIFO`) or Sieve Endpoint Store (`SIEVE`), default is `SIEVE`.
|
||||
|
||||
## C++ API Reference
|
||||
|
||||
::::{toctree}
|
||||
:maxdepth: 1
|
||||
|
||||
cpp-api
|
||||
::::
|
||||
|
||||
## Ascend Transport Component
|
||||
|
||||
:::{toctree}
|
||||
|
|
|
|||
Loading…
Reference in New Issue