3.3 KiB

Raw Blame History

title
Multi-Node

Multi-node Examples

This guide covers deploying vLLM across multiple nodes using Dynamo's distributed capabilities.

Prerequisites

Multi-node deployments require:

Multiple nodes with GPU resources
Network connectivity between nodes (faster the better)
Firewall rules allowing NATS/ETCD communication

Infrastructure Setup

Step 1: Start NATS/ETCD on Head Node

Start the required services on your head node. These endpoints must be accessible from all worker nodes:

# On head node (node-1)
docker compose -f deploy/docker-compose.yml up -d

Default ports:

NATS: 4222
ETCD: 2379

Step 2: Configure Environment Variables

Set the head node IP address and service endpoints. Set this on all nodes for easy copy-paste:

# Set this on ALL nodes - replace with your actual head node IP
export HEAD_NODE_IP="<your-head-node-ip>"

# Service endpoints (set on all nodes)
export NATS_SERVER="nats://${HEAD_NODE_IP}:4222"
export ETCD_ENDPOINTS="${HEAD_NODE_IP}:2379"

Deployment Patterns

Multi-node Aggregated Serving

Deploy vLLM workers across multiple nodes for horizontal scaling:

Node 1 (Head Node): Run ingress and first worker

# Start ingress
python -m dynamo.frontend --router-mode kv

# Start vLLM worker
python -m dynamo.vllm \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 8 \
  --enforce-eager

Node 2: Run additional worker

# Start vLLM worker
python -m dynamo.vllm \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 8 \
  --enforce-eager

Multi-node Disaggregated Serving

Deploy prefill and decode workers on separate nodes for optimized resource utilization:

Node 1: Run ingress and decode worker

# Start ingress
python -m dynamo.frontend --router-mode kv &

# Start decode worker
python -m dynamo.vllm \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 8 \
  --enforce-eager \
  --disaggregation-mode decode

Node 2: Run prefill worker

# Start prefill worker
python -m dynamo.vllm \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 8 \
  --enforce-eager \
  --disaggregation-mode prefill

Multi-node Tensor/Pipeline Parallelism

When the total parallelism (TP × PP) exceeds the number of GPUs on a single node, you need multiple nodes to host a single model instance. One node runs the full dynamo.vllm process (head) while additional nodes run in --headless mode, spawning only vLLM workers.

See examples/backends/vllm/launch/multi_node_tp.sh for a ready-to-use launch script that supports both head and worker roles via --head / --worker flags. The model, TP size, and node count are configurable via MODEL, TENSOR_PARALLEL_SIZE, and NNODES environment variables.

For details on the flags used for multi-node distributed execution (--master-addr, --master-port, --nnodes, --node-rank), see the vLLM multiprocessing docs.

3.3 KiB Raw Blame History Unescape Escape