dynamo/lib/runtime

Dynamo Runtime

A Datacenter Scale Distributed Inference Serving Framework

Rust implementation of the Dynamo runtime system, enabling distributed computing capabilities for machine learning workloads.

Prerequisites

Install Rust and Cargo using rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Build

cargo build
cargo test

Start Dependencies

Docker Compose

The simplest way to deploy the pre-requisite services is using docker-compose, defined in deploy/docker-compose.yml.

# At the root of the repository:
docker compose -f deploy/docker-compose.yml up -d

This will deploy a NATS.io server and an etcd server used to communicate between and discover components at runtime.

Local (alternate)

To deploy the pre-requisite services locally instead of using docker-compose above, you can manually launch each:

• NATS.io server with Jetstream
  • example: nats-server -js --trace
• etcd server
  • follow instructions in etcd installation to start an etcd-server locally

Run Examples

When developing or running examples, any process or user that shared your core-services (etcd and nats.io) will be operating within your distributed runtime.

The current examples use a hard-coded namespace. We will address the namespace collisions later.

Most examples require etcd for service discovery. nats.io is required for KV-aware routing with event tracking; for approximate mode (--no-kv-events), NATS is optional.

Rust hello_world

With two terminals open, in one window:

cd examples/hello_world
cargo run --bin server

In the second terminal, execute:

cd examples/hello_world
cargo run --bin client

which should yield some output similar to:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.25s
     Running `target/debug/client`
Annotated { data: Some("h"), id: None, event: None, comment: None }
Annotated { data: Some("e"), id: None, event: None, comment: None }
Annotated { data: Some("l"), id: None, event: None, comment: None }
Annotated { data: Some("l"), id: None, event: None, comment: None }
Annotated { data: Some("o"), id: None, event: None, comment: None }
Annotated { data: Some(" "), id: None, event: None, comment: None }
Annotated { data: Some("w"), id: None, event: None, comment: None }
Annotated { data: Some("o"), id: None, event: None, comment: None }
Annotated { data: Some("r"), id: None, event: None, comment: None }
Annotated { data: Some("l"), id: None, event: None, comment: None }
Annotated { data: Some("d"), id: None, event: None, comment: None }

Python

See the README.md for details

The Python and Rust hello_world client and server examples are interchangeable, so you can start the Python server.py and talk to it from the Rust client.