# Dynamo review

> Dynamo is an open-source, datacenter-scale distributed inference serving framework that coordinates SGLang, TensorRT-LLM, and vLLM across multiple GPUs and nodes.

Devtoolwatch Board, the week of Oct 25–31, 2026. Canonical page: https://index.devtoolwatch.com/tools/ai-dynamo-dynamo/

## TL;DR

Dynamo is an orchestration layer that sits above inference engines such as SGLang, TensorRT-LLM, and vLLM and turns them into a coordinated multi-node inference system. It is for teams serving LLMs across multiple GPUs or nodes. It combines disaggregated prefill/decode, KV-aware routing, multi-tier KV caching, and SLA-driven autoscaling.

## Dynamo at a glance

- **Category:** [Inference and training](https://index.devtoolwatch.com/categories/inference-and-training/)
- **Maintained by:** Dynamo (organization, [@ai-dynamo](https://github.com/ai-dynamo) on GitHub)
- **Language:** Rust, Python, Go
- **Repository:** [github.com/ai-dynamo/dynamo](https://github.com/ai-dynamo/dynamo)
- **Website:** [docs.nvidia.com/dynamo/latest](https://docs.nvidia.com/dynamo/latest)
- **PyPI package:** [ai-dynamo](https://pypi.org/project/ai-dynamo/) 1.5.1
- **Latest release:** [v1.6.0-muse-spark-open-dev.1](https://github.com/ai-dynamo/dynamo/releases/tag/v1.6.0-muse-spark-open-dev.1) on GitHub, Oct 20, 2026
- **First published:** Mar 3, 2025

## What Dynamo does

- Dynamo separates prefill and decode into GPU pools that scale independently.
- Dynamo routes requests based on worker load and KV cache overlap to avoid redundant prefill computation.
- Dynamo's KV Block Manager offloads KV cache across GPU, CPU, SSD, and remote storage, including S3 and Azure blob.
- Dynamo's Planner is an SLA-driven autoscaler that profiles workloads and sizes GPU pools to meet latency targets.
- Dynamo's ModelExpress streams model weights GPU-to-GPU via NIXL/NVLink for faster cold-starts of new replicas.
- Dynamo provides fault tolerance through canary health checks and in-flight request migration.
- Dynamo exposes an OpenAI-compatible API and supports LLM, multimodal, and video generation workloads.

### How it works

Dynamo runs a frontend that serves HTTP and a router that sends requests to engine workers running vLLM, SGLang, or TensorRT-LLM. You can run it locally, on Slurm, or on Kubernetes. On Kubernetes, traffic goes through the Dynamo Frontend or through a Gateway API Inference Extension gateway. The core is built in Rust, with Python for extensibility.

## How to install Dynamo

**With the vLLM container**

```
docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1
```

**With uv (vLLM backend)**

```
uv pip install --prerelease=allow "ai-dynamo[vllm]"
```

**With uv (SGLang backend)**

```
uv pip install --prerelease=allow "ai-dynamo[sglang]"
```

### Requirements

- Python 3.10 or later for the ai-dynamo PyPI package.
- NVIDIA or AMD GPUs, or Intel XPUs.
- An inference engine backend: vLLM, SGLang, or TensorRT-LLM.
- TensorRT-LLM installs require pip with --extra-index-url https://pypi.nvidia.com.

### First steps

1. Start the frontend: python3 -m dynamo.frontend --http-port 8000 --discovery-backend file
2. Start a worker: python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --discovery-backend file
3. Once the worker is ready, send a chat completion request to localhost:8000/v1/chat/completions.
4. For production clusters, install the Dynamo Platform on Kubernetes and deploy with a DynamoGraphDeploymentRequest manifest.

## Who Dynamo is for

- Teams serving LLMs across multiple GPUs or nodes that need to coordinate them.
- Deployments that need to scale prefill and decode independently.
- Operators who want autoscaling tied to latency SLAs and total cost of ownership.
- Kubernetes platforms that want KV-aware routing inside the standard Gateway API.

### Limitations

- Dynamo is likely unnecessary for a single model on a single GPU. The README says the inference engine alone is probably sufficient there.
- Dynamo's KVBM support for SGLang is still in progress. It is supported on TensorRT-LLM and vLLM.
- Dynamo's zero-config deploy (DGDR) is in beta.
- Dynamo's model-specific dev snapshot releases are experimental and not recommended for production.

## Usage data (the week of Oct 25–31, 2026)

- Board rank: #195 overall, #19 in Inference and training, momentum score 45
- Downloads that week: 20.7k (+60% vs 4-week average)
- New GitHub stars per week (last 12 weeks): Aug 9: 67; Aug 16: 63; Aug 23: 82; Aug 30: 74; Sep 6: 70; Sep 13: 81; Sep 20: 42; Sep 27: 52; Oct 4: 45; Oct 11: 70; Oct 18: 60; Oct 25: 38

## Dynamo alternatives

| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | [dsh-our-free-model](https://index.devtoolwatch.com/tools/ebony-vinyl-dsh-our-free-model/) | 54.3k | – |
| #2 | [magpie](https://index.devtoolwatch.com/tools/yetone-magpie/) | 26.9k | – |
| #3 | [Strata](https://index.devtoolwatch.com/tools/niko1221-strata/) | 37.1k | – |
| #4 | [edge0](https://index.devtoolwatch.com/tools/edge0-ai-edge0/) | 8.9k | – |
| #5 | [OmniRoute](https://index.devtoolwatch.com/tools/diegosouzapw-omniroute/) | 83.2k | 106k |
| #6 | [CLIProxyAPI](https://index.devtoolwatch.com/tools/router-for-me-cliproxyapi/) | 57.1k | – |

## Dynamo FAQ

### Is Dynamo free and open source?

Dynamo's code is on GitHub (ai-dynamo/dynamo), but the repository doesn't declare a standard open-source license. Check its license file before using it commercially.

### How do I install Dynamo?

Run `docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1`. The ai-dynamo package needs Python 3.10 or newer. The install section above has the other options from the project's README.

### Is Dynamo still maintained?

Yes. The ai-dynamo/dynamo repository was last updated on Oct 21, 2026, and the latest release, v1.6.0-muse-spark-open-dev.1 on GitHub, came out on Oct 20, 2026.

### How popular is Dynamo?

Dynamo has 8.4k GitHub stars as of Nov 1, 2026. It gained 38 new stars in the week of Oct 25–31, 2026, against a 12-week average of 62 a week. Its main package had 20.7k downloads that week, +60% against its own 4-week average. That made it #19 among LLM inference and training tools on the Devtoolwatch Board.

### What are the best alternatives to Dynamo?

Other LLM inference and training tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: dsh-our-free-model (#1), magpie (#2), Strata (#3), edge0 (#4), OmniRoute (#5). The alternatives table above has their numbers.

### Does Dynamo replace vLLM or SGLang?

No. Dynamo does not replace SGLang, TensorRT-LLM, or vLLM. Dynamo is the orchestration layer above those engines and turns them into a coordinated multi-node inference system.

### When should I use Dynamo?

Use Dynamo when you serve LLMs across multiple GPUs or nodes, or when you need KV-aware routing, disaggregated prefill and decode, SLA-based autoscaling, or fast cold-starts. For a single model on a single GPU, the inference engine alone is probably sufficient.

### Where can Dynamo run?

Dynamo runs on Kubernetes, on Slurm, or locally. It supports NVIDIA and AMD GPUs and Intel XPUs. The README recommends Kubernetes for production multi-node clusters.

### Does Dynamo expose an OpenAI-compatible API?

Yes. Both Kubernetes routing topologies, Dynamo-native Frontend routing and Gateway API routing with GAIE, expose an OpenAI-compatible API.

### Can Dynamo handle KV cache larger than GPU memory?

Yes. Dynamo's KV Block Manager offloads KV cache from GPU to CPU, SSD, and remote storage, which extends effective context length beyond GPU memory.

## Sources

1. [ai-dynamo/dynamo on GitHub](https://github.com/ai-dynamo/dynamo)
2. [Dynamo README](https://github.com/ai-dynamo/dynamo/blob/main/README.md)
3. [ai-dynamo on PyPI](https://pypi.org/project/ai-dynamo/)
4. [Dynamo website](https://docs.nvidia.com/dynamo/latest)
5. [Release v1.6.0-muse-spark-open-dev.1 (Oct 20, 2026)](https://github.com/ai-dynamo/dynamo/releases/tag/v1.6.0-muse-spark-open-dev.1)
6. [Release v1.6.0-beam-1.0-501b-dev.1 (Oct 19, 2026)](https://github.com/ai-dynamo/dynamo/releases/tag/v1.6.0-beam-1.0-501b-dev.1)
7. [Release v1.6.0-nemotron-3.5-super-vl-dev.1 (Oct 15, 2026)](https://github.com/ai-dynamo/dynamo/releases/tag/v1.6.0-nemotron-3.5-super-vl-dev.1)
8. [Hacker News: CUDA 11.0](https://news.ycombinator.com/item?id=23772245)
9. [Hacker News: CUDA Toolkit Release Notes](https://news.ycombinator.com/item?id=21618774)
10. [Hacker News: Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework](https://news.ycombinator.com/item?id=43404858)
11. [Devtoolwatch Board, the week of Oct 25–31, 2026](https://index.devtoolwatch.com/)