Devtoolwatch BoardGet it every Monday

TensorRT-LLM review

TensorRT-LLM is an NVIDIA library with a Python API for defining large language models and running optimized inference on NVIDIA GPUs.

Updated Nov 1, 2026. Review written Oct 25, 2026 from the project's own sources; usage data refreshed every Monday.

TL;DR

TensorRT-LLM is NVIDIA's open-source library for optimizing LLM and visual generation inference on NVIDIA GPUs. It is for teams deploying models on NVIDIA hardware who want specialized kernels, an efficient runtime, and a Python framework they can customize and extend. TensorRT-LLM also includes components for building Python and C++ runtimes.

TensorRT-LLM at a glance

Category
Inference and training
Maintained by
NVIDIA Corporation (organization, @NVIDIA on GitHub)
Language
Python, C++, Cuda
Repository
github.com/NVIDIA/TensorRT-LLM
Website
nvidia.github.io/TensorRT-LLM
PyPI package
tensorrt-llm 1.2.1
Latest release
v1.4.0rc1 on GitHub, Oct 22, 2026
First published
Aug 16, 2023

What TensorRT-LLM does

  • TensorRT-LLM provides a Python API for defining large language models and running inference on NVIDIA GPUs.
  • TensorRT-LLM uses specialized kernels for common operations to optimize inference for LLMs and visual generation models.
  • TensorRT-LLM supports diffusion models for visual generation, which the documentation labels as beta.
  • TensorRT-LLM supports speculative decoding, guided decoding, multiple LoRA adapters, sparse attention, and KV cache compression and offloading.
  • TensorRT-LLM includes components for building Python and C++ runtimes that orchestrate inference execution.
  • TensorRT-LLM ships CLI tools for serving, benchmarking, and evaluation: trtllm-serve, trtllm-bench, and trtllm-eval.

How it works

TensorRT-LLM is used through its Python LLM API, or through the trtllm-serve CLI for online serving. Its documentation includes clients for OpenAI-style chat, completion, and responses requests. The docs also cover Prometheus metrics, a Dynamo Kubernetes example, Slurm runs with the PyTorch backend, and an LMCache KV cache connector. Container images and builds from source are documented.

How to install and start using TensorRT-LLM

From PyPI

pip install tensorrt-llm

Requirements

  • NVIDIA GPUs
  • CUDA

First steps

  1. Follow the Quick Start Guide and Installation Guide in the TensorRT-LLM documentation.
  2. Try the LLM API examples, starting with generating text, then asynchronous and streaming generation.
  3. Serve a model with trtllm-serve and query it with the OpenAI Chat Client example.
  4. Check the Model Recipes for deployment guides for specific models on Blackwell and Hopper hardware.

Who TensorRT-LLM is for

  • Large mixture-of-experts deployments that need expert parallelism or disaggregated serving
  • Developers who want to customize and extend an inference stack in Python
  • Edge deployments on Jetson AGX Orin through the v0.12.0-jetson branch

Limitations to know

  • TensorRT-LLM release notes list known issues in disaggregated serving, multi-GPU, and multi-node setups.
  • TensorRT-LLM release candidates include breaking API changes, such as removing the AutoDeploy integration.
  • TensorRT-LLM labels visual generation support as beta.

Is TensorRT-LLM growing? Usage data

In the week of Oct 25–31, 2026, TensorRT-LLM ranked #355 on the Devtoolwatch Board and #43 in Inference and training, with a momentum score of 20 out of 100. Its main package had 1.3k downloads that week, −42% against its own 4-week average.

New GitHub stars per week for TensorRT-LLM12 weeks from Aug 9, 2026: 58, 67, 53, 66, 57, 70, 53, 46, 37, 62, 102, 50 new stars, an average of 60 a week.Aug 9week of Oct 25
New GitHub stars per week for the last 12 weeks, from GitHub's stargazer history (today's stargazers by the week they starred).
Show the weekly numbers
New GitHub stars per week for TensorRT-LLM
Week ofNew stars
Aug 9, 202658
Aug 16, 202667
Aug 23, 202653
Aug 30, 202666
Sep 6, 202657
Sep 13, 202670
Sep 20, 202653
Sep 27, 202646
Oct 4, 202637
Oct 11, 202662
Oct 18, 2026102
Oct 25, 202650

On Hacker News

The Board ranks tools by four signals each week: new stars, new stars for the repo's size, added downloads and download growth beyond the market. How the ranking works.

TensorRT-LLM alternatives

Other LLM inference and training tools on the Board in the week of Oct 25–31, 2026, by momentum:

Inference and training ranked by momentum, the week of Oct 25–31, 2026
RankToolStarsDownloads/wk
#1dsh-our-free-modeldsh-our-free-model is a dsh plugin that provides DeepSeek V4.1 Flash, Kimi K3 and other models without login, sign-up or API key.54.3k–
#2magpiemagpie is a local gateway and app that lets coding agents like Claude Code, Codex and Gemini CLI use any provider's model.26.9k–
#3StrataStrata is a local inference engine and app that runs large open models like Qwen3.8-Flash-Next on consumer hardware with…37.1k–
#4edge0edge0 is an open-source streaming inference framework that runs large sparse mixture-of-experts models on consumer hardware by streaming…8.9k–
#5OmniRouteOmniRoute is an open-source AI gateway that routes requests across many model providers through one OpenAI-compatible endpoint, with…83.2k106k
#6CLIProxyAPICLIProxyAPI is a Go proxy server that exposes CLI coding accounts like Codex, Claude Code and Gemini CLI through OpenAI, Gemini and…57.1k–

All LLM inference and training tools, ranked

Recent TensorRT-LLM releases

TensorRT-LLM FAQ

Is TensorRT-LLM free and open source?

TensorRT-LLM's code is on GitHub (NVIDIA/TensorRT-LLM), but the repository doesn't declare a standard open-source license. Check its license file before using it commercially.

How do I install TensorRT-LLM?

Run pip install tensorrt-llm. The tensorrt-llm package needs Python 3.10 or newer, below 4.

Is TensorRT-LLM still maintained?

Yes. The NVIDIA/TensorRT-LLM repository was last updated on Oct 25, 2026, and the latest release, v1.4.0rc1 on GitHub, came out on Oct 22, 2026.

How popular is TensorRT-LLM?

TensorRT-LLM has 15k GitHub stars as of Nov 1, 2026. It gained 50 new stars in the week of Oct 25–31, 2026, against a 12-week average of 60 a week. Its main package had 1.3k downloads that week, −42% against its own 4-week average. That made it #43 among LLM inference and training tools on the Devtoolwatch Board.

TensorRT-LLM vs vLLM: how do they compare?

TensorRT-LLM and vLLM are both tracked by Devtoolwatch as LLM inference and training tools. TensorRT-LLM has 15k GitHub stars and vLLM has 96k. In the week of Oct 25–31, 2026, TensorRT-LLM's main package had 1.3k downloads and vLLM's had 677.2k. vLLM is an open-source Python library and engine for high-throughput, memory-efficient inference and serving of large language models. vLLM review.

TensorRT-LLM vs SGLang: how do they compare?

TensorRT-LLM and SGLang are both tracked by Devtoolwatch as LLM inference and training tools. TensorRT-LLM has 15k GitHub stars and SGLang has 38.3k. In the week of Oct 25–31, 2026, TensorRT-LLM's main package had 1.3k downloads and SGLang's had 336.7k. SGLang is an open-source inference and serving framework for large language, vision-language, and diffusion models, with OpenAI-compatible endpoints. SGLang review.

What are the best alternatives to TensorRT-LLM?

Other LLM inference and training tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: dsh-our-free-model (#1), magpie (#2), Strata (#3), edge0 (#4), OmniRoute (#5). The alternatives table above has their numbers.

What hardware does TensorRT-LLM run on?

TensorRT-LLM runs inference on NVIDIA GPUs. Its documentation includes deployment guides for Blackwell and Hopper hardware. Jetson AGX Orin has initial support in JetPack 6.1 through the v0.12.0-jetson branch.

Can TensorRT-LLM serve an OpenAI-compatible API?

TensorRT-LLM provides the trtllm-serve CLI for online serving. Its documentation includes OpenAI chat, completion, and responses client examples, plus curl clients and Prometheus metrics.

Does TensorRT-LLM support speculative decoding?

TensorRT-LLM supports speculative decoding. Its API reference lists configs for Eagle, Eagle3, MTP, N-gram, and draft-target decoding.

Does TensorRT-LLM support image or video generation?

TensorRT-LLM supports diffusion models for visual generation. The documentation marks this support as beta and includes a VisualGen API walkthrough.

Is TensorRT-LLM open source?

TensorRT-LLM is fully open source, and its development has moved to GitHub, according to the project README.

Sources

  1. NVIDIA/TensorRT-LLM on GitHub
  2. TensorRT-LLM README
  3. tensorrt-llm on PyPI
  4. TensorRT-LLM website
  5. Release v1.4.0rc1 (Oct 22, 2026)
  6. Release v1.4.0rc0 (Oct 14, 2026)
  7. Release v1.3.0rc29 (Sep 29, 2026)
  8. Hacker News: Nvidia Warp: Python framework for high-performance simulation and graphics code
  9. Hacker News: Nvidia Releases CUDA Python
  10. Hacker News: What we have learned at OpenShell applying formal methods to control AI agents
  11. Devtoolwatch Board, the week of Oct 25–31, 2026

TensorRT-LLM README badge

Maintainers can show the tool's weekly Board rank in their README:

Devtoolwatch badge for TensorRT-LLM

[![Devtoolwatch](https://index.devtoolwatch.com/badge/tensorrt-llm.svg)](https://index.devtoolwatch.com/tools/tensorrt-llm/)

How this page is made: Devtoolwatch writes each review from the project's README, releases, package metadata and website, with an AI model, then checks every claim against the source it cites and drops what the sources don't support. The numbers come from Devtoolwatch's daily GitHub snapshots, GitHub's stargazer history, and the npm and PyPI download APIs. Spotted an error? Email [email protected].