Dynamo review

Q: How do I install Dynamo?

Run docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1. The ai-dynamo package needs Python 3.10 or newer. The install section above has the other options from the project's README.

Dynamo is an open-source, datacenter-scale distributed inference serving framework that coordinates SGLang, TensorRT-LLM, and vLLM across multiple GPUs and nodes.

Updated Nov 1, 2026. Review written Oct 22, 2026 from the project's own sources; usage data refreshed every Monday.

TL;DR

Dynamo is an orchestration layer that sits above inference engines such as SGLang, TensorRT-LLM, and vLLM and turns them into a coordinated multi-node inference system. It is for teams serving LLMs across multiple GPUs or nodes. It combines disaggregated prefill/decode, KV-aware routing, multi-tier KV caching, and SLA-driven autoscaling.

8.4kGitHub stars
+38new stars, week of Oct 25
20.7kdownloads a week, +60% vs 4-week avg
#19in Inference and training this week

Dynamo at a glance

Category: Inference and training
Maintained by: Dynamo (organization, @ai-dynamo on GitHub)
Language: Rust, Python, Go
Repository: github.com/ai-dynamo/dynamo
Website: docs.nvidia.com/dynamo/latest
PyPI package: ai-dynamo 1.5.1
Latest release: v1.6.0-muse-spark-open-dev.1 on GitHub, Oct 20, 2026
First published: Mar 3, 2025

What Dynamo does

Dynamo separates prefill and decode into GPU pools that scale independently.
Dynamo routes requests based on worker load and KV cache overlap to avoid redundant prefill computation.
Dynamo's KV Block Manager offloads KV cache across GPU, CPU, SSD, and remote storage, including S3 and Azure blob.
Dynamo's Planner is an SLA-driven autoscaler that profiles workloads and sizes GPU pools to meet latency targets.
Dynamo's ModelExpress streams model weights GPU-to-GPU via NIXL/NVLink for faster cold-starts of new replicas.
Dynamo provides fault tolerance through canary health checks and in-flight request migration.
Dynamo exposes an OpenAI-compatible API and supports LLM, multimodal, and video generation workloads.

How it works

Dynamo runs a frontend that serves HTTP and a router that sends requests to engine workers running vLLM, SGLang, or TensorRT-LLM. You can run it locally, on Slurm, or on Kubernetes. On Kubernetes, traffic goes through the Dynamo Frontend or through a Gateway API Inference Extension gateway. The core is built in Rust, with Python for extensibility.

How to install and start using Dynamo

With the vLLM container

docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1

With uv (vLLM backend)

uv pip install --prerelease=allow "ai-dynamo[vllm]"

With uv (SGLang backend)

uv pip install --prerelease=allow "ai-dynamo[sglang]"

Requirements

Python 3.10 or later for the ai-dynamo PyPI package.
NVIDIA or AMD GPUs, or Intel XPUs.
An inference engine backend: vLLM, SGLang, or TensorRT-LLM.
TensorRT-LLM installs require pip with --extra-index-url https://pypi.nvidia.com.

First steps

Start the frontend: python3 -m dynamo.frontend --http-port 8000 --discovery-backend file
Start a worker: python3 -m dynamo.vllm --model Qwen/Qwen3-0.6B --discovery-backend file
Once the worker is ready, send a chat completion request to localhost:8000/v1/chat/completions.
For production clusters, install the Dynamo Platform on Kubernetes and deploy with a DynamoGraphDeploymentRequest manifest.

Who Dynamo is for

Teams serving LLMs across multiple GPUs or nodes that need to coordinate them.
Deployments that need to scale prefill and decode independently.
Operators who want autoscaling tied to latency SLAs and total cost of ownership.
Kubernetes platforms that want KV-aware routing inside the standard Gateway API.

Limitations to know

Dynamo is likely unnecessary for a single model on a single GPU. The README says the inference engine alone is probably sufficient there.
Dynamo's KVBM support for SGLang is still in progress. It is supported on TensorRT-LLM and vLLM.
Dynamo's zero-config deploy (DGDR) is in beta.
Dynamo's model-specific dev snapshot releases are experimental and not recommended for production.

Is Dynamo growing? Usage data

In the week of Oct 25–31, 2026, Dynamo ranked #195 on the Devtoolwatch Board and #19 in Inference and training, with a momentum score of 45 out of 100. Its main package had 20.7k downloads that week, +60% against its own 4-week average.

New GitHub stars per week for the last 12 weeks, from GitHub's stargazer history (today's stargazers by the week they starred).

Show the weekly numbers

New GitHub stars per week for Dynamo
Week of	New stars
Aug 9, 2026	67
Aug 16, 2026	63
Aug 23, 2026	82
Aug 30, 2026	74
Sep 6, 2026	70
Sep 13, 2026	81
Sep 20, 2026	42
Sep 27, 2026	52
Oct 4, 2026	45
Oct 11, 2026	70
Oct 18, 2026	60
Oct 25, 2026	38

On Hacker News

CUDA 11.0, 182 points and 65 comments (Jul 8, 2020)
CUDA Toolkit Release Notes, 160 points and 119 comments (Nov 24, 2019)
Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving Framework, 150 points and 39 comments (Mar 18, 2025)
Tile IR Specification, 19 points and 3 comments (Dec 4, 2025)
Nvidia uses raytracing to model 5G network design, 6 points and 0 comments (Jul 4, 2024)

The Board ranks tools by four signals each week: new stars, new stars for the repo's size, added downloads and download growth beyond the market. How the ranking works.

Dynamo alternatives

Other LLM inference and training tools on the Board in the week of Oct 25–31, 2026, by momentum:

Inference and training ranked by momentum, the week of Oct 25–31, 2026
Rank	Tool	Stars	Downloads/wk
#1	dsh-our-free-modeldsh-our-free-model is a dsh plugin that provides DeepSeek V4.1 Flash, Kimi K3 and other models without login, sign-up or API key.	54.3k	–
#2	magpiemagpie is a local gateway and app that lets coding agents like Claude Code, Codex and Gemini CLI use any provider's model.	26.9k	–
#3	StrataStrata is a local inference engine and app that runs large open models like Qwen3.8-Flash-Next on consumer hardware with…	37.1k	–
#4	edge0edge0 is an open-source streaming inference framework that runs large sparse mixture-of-experts models on consumer hardware by streaming…	8.9k	–
#5	OmniRouteOmniRoute is an open-source AI gateway that routes requests across many model providers through one OpenAI-compatible endpoint, with…	83.2k	106k
#6	CLIProxyAPICLIProxyAPI is a Go proxy server that exposes CLI coding accounts like Codex, Claude Code and Gemini CLI through OpenAI, Gemini and…	57.1k	–

All LLM inference and training tools, ranked

Recent Dynamo releases

v1.6.0-muse-spark-open-dev.1 (pre-release), Oct 20, 2026
v1.6.0-beam-1.0-501b-dev.1 (pre-release), Oct 19, 2026
v1.6.0-nemotron-3.5-super-vl-dev.1 (pre-release), Oct 15, 2026
v1.5.0-motif-3-dev.2 (pre-release), Oct 14, 2026
v1.6.0-glm-5.3-flash-dev.1 (pre-release), Oct 14, 2026

Dynamo FAQ

Is Dynamo free and open source?

Dynamo's code is on GitHub (ai-dynamo/dynamo), but the repository doesn't declare a standard open-source license. Check its license file before using it commercially.

How do I install Dynamo?

Run docker run --gpus all --network host --rm -it nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.5.1. The ai-dynamo package needs Python 3.10 or newer. The install section above has the other options from the project's README.

Is Dynamo still maintained?

Yes. The ai-dynamo/dynamo repository was last updated on Oct 21, 2026, and the latest release, v1.6.0-muse-spark-open-dev.1 on GitHub, came out on Oct 20, 2026.

How popular is Dynamo?

Dynamo has 8.4k GitHub stars as of Nov 1, 2026. It gained 38 new stars in the week of Oct 25–31, 2026, against a 12-week average of 62 a week. Its main package had 20.7k downloads that week, +60% against its own 4-week average. That made it #19 among LLM inference and training tools on the Devtoolwatch Board.

What are the best alternatives to Dynamo?

Other LLM inference and training tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: dsh-our-free-model (#1), magpie (#2), Strata (#3), edge0 (#4), OmniRoute (#5). The alternatives table above has their numbers.

Does Dynamo replace vLLM or SGLang?

No. Dynamo does not replace SGLang, TensorRT-LLM, or vLLM. Dynamo is the orchestration layer above those engines and turns them into a coordinated multi-node inference system.

When should I use Dynamo?

Use Dynamo when you serve LLMs across multiple GPUs or nodes, or when you need KV-aware routing, disaggregated prefill and decode, SLA-based autoscaling, or fast cold-starts. For a single model on a single GPU, the inference engine alone is probably sufficient.

Where can Dynamo run?

Dynamo runs on Kubernetes, on Slurm, or locally. It supports NVIDIA and AMD GPUs and Intel XPUs. The README recommends Kubernetes for production multi-node clusters.

Does Dynamo expose an OpenAI-compatible API?

Yes. Both Kubernetes routing topologies, Dynamo-native Frontend routing and Gateway API routing with GAIE, expose an OpenAI-compatible API.

Can Dynamo handle KV cache larger than GPU memory?

Yes. Dynamo's KV Block Manager offloads KV cache from GPU to CPU, SSD, and remote storage, which extends effective context length beyond GPU memory.

Sources

Dynamo README badge

Maintainers can show the tool's weekly Board rank in their README:

[![Devtoolwatch](https://index.devtoolwatch.com/badge/ai-dynamo-dynamo.svg)](https://index.devtoolwatch.com/tools/ai-dynamo-dynamo/)

How this page is made: Devtoolwatch writes each review from the project's README, releases, package metadata and website, with an AI model, then checks every claim against the source it cites and drops what the sources don't support. The numbers come from Devtoolwatch's daily GitHub snapshots, GitHub's stargazer history, and the npm and PyPI download APIs. Spotted an error? Email [email protected].