Devtoolwatch BoardFlashInfer review
FlashInfer is a GPU kernel library and kernel generator for LLM inference, covering attention, GEMM, MoE, sampling and communication operations.
TL;DR
FlashInfer is a library and kernel generator for LLM inference on NVIDIA GPUs from Turing through Blackwell. It is for developers building or tuning LLM serving engines. FlashInfer provides unified APIs for attention, GEMM and MoE with multiple backends, including FlashAttention-2/3, cuDNN, CUTLASS and TensorRT-LLM, and selects a backend automatically.
- 6.8kGitHub stars
- +57new stars, week of Oct 25
- 938.6kdownloads a week, +18% vs 4-week avg
- #15in Inference and training this week
FlashInfer at a glance
- Category
- Inference and training
- Maintained by
- FlashInfer (organization, @flashinfer-ai on GitHub)
- License
- Apache 2.0
- Language
- Cuda, Python, C++
- Repository
- github.com/flashinfer-ai/flashinfer
- Website
- flashinfer.ai
- PyPI package
- flashinfer-python 0.7.2
- Latest release
- v0.7.2 on GitHub, Oct 15, 2026
- First published
- Jul 22, 2023
What FlashInfer does
- FlashInfer provides attention kernels for decode, prefill and append, with paged and ragged KV-cache for dynamic batch serving.
- FlashInfer supports MLA attention, cascade attention for shared prefixes, block-sparse attention and fused prefill+decode POD-Attention.
- FlashInfer offers BF16, FP8 and FP4 GEMM, plus grouped GEMM for LoRA and multi-expert routing.
- FlashInfer includes fused MoE kernels with DeepSeek-V3, Llama-4 and top-k routing, and FP8 or FP4 expert weights.
- FlashInfer implements sorting-free Top-K, Top-P and Min-P sampling and chain speculative sampling.
- FlashInfer provides custom AllReduce, multi-node NVLink support and NVSHMEM integration for distributed inference.
- FlashInfer kernels are compatible with CUDAGraph and torch.compile for low-latency serving.
How it works
FlashInfer is a Python package used from PyTorch on CUDA tensors. The core package compiles or downloads kernels on first use. Optional packages supply pre-compiled kernel binaries for faster startup and offline use. A flashinfer CLI shows configuration, manages modules and caches, and exports compile commands. Serving engines such as SGLang, vLLM, TensorRT-LLM and TGI use FlashInfer for inference.
How to install and start using FlashInfer
With pip
pip install flashinfer-pythonFor Blackwell CuTe DSL kernels (CUDA 13)
pip install flashinfer-python[cu13]Verify installation
flashinfer show-configRequirements
- Python 3.10 or later, below 4.0
- NVIDIA GPU with compute capability SM 7.5 (Turing) or later
- CUDA 12.9, 13.0, or 13.4 (PyTorch nightly)
- PyTorch, used for tensors in the basic usage example
First steps
- Install FlashInfer with pip install flashinfer-python.
- Optionally run flashinfer install-cubin-wheel and flashinfer install-jit-cache-wheel to pre-compile most kernels for faster initialization and offline use.
- Run flashinfer show-config to verify the installation and view configuration.
- Call flashinfer.single_decode_with_kv_cache(q, k, v) on CUDA tensors to run single decode attention.
Who FlashInfer is for
- Developers building LLM serving engines who need attention, GEMM and MoE kernels
- Teams serving MoE or MLA models such as DeepSeek on Hopper or Blackwell GPUs
- Engineers using FP8 or FP4 quantization for inference
- Researchers creating custom attention variants through JIT
Limitations to know
- FlashInfer does not support every feature on every compute capability, according to its GPU support notes.
- FlashInfer targets NVIDIA GPUs and CUDA. The sources list no other hardware.
- The core FlashInfer package compiles or downloads kernels on first use unless the optional pre-compiled packages are installed.
- FlashInfer runtime CI covers CUDA 12.9 and 13.0. CUDA 13.4 wheels use a preview toolkit.
Is FlashInfer growing? Usage data
In the week of Oct 25–31, 2026, FlashInfer ranked #136 on the Devtoolwatch Board and #15 in Inference and training, with a momentum score of 56 out of 100. Its main package had 938.6k downloads that week, +18% against its own 4-week average.
Show the weekly numbers
| Week of | New stars |
|---|---|
| Aug 9, 2026 | 42 |
| Aug 16, 2026 | 48 |
| Aug 23, 2026 | 75 |
| Aug 30, 2026 | 59 |
| Sep 6, 2026 | 50 |
| Sep 13, 2026 | 70 |
| Sep 20, 2026 | 55 |
| Sep 27, 2026 | 34 |
| Oct 4, 2026 | 48 |
| Oct 11, 2026 | 93 |
| Oct 18, 2026 | 100 |
| Oct 25, 2026 | 57 |
The Board ranks tools by four signals each week: new stars, new stars for the repo's size, added downloads and download growth beyond the market. How the ranking works.
FlashInfer alternatives
Other LLM inference and training tools on the Board in the week of Oct 25–31, 2026, by momentum:
| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | dsh-our-free-modeldsh-our-free-model is a dsh plugin that provides DeepSeek V4.1 Flash, Kimi K3 and other models without login, sign-up or API key. | 54.3k | – |
| #2 | magpiemagpie is a local gateway and app that lets coding agents like Claude Code, Codex and Gemini CLI use any provider's model. | 26.9k | – |
| #3 | StrataStrata is a local inference engine and app that runs large open models like Qwen3.8-Flash-Next on consumer hardware with… | 37.1k | – |
| #4 | edge0edge0 is an open-source streaming inference framework that runs large sparse mixture-of-experts models on consumer hardware by streaming… | 8.9k | – |
| #5 | OmniRouteOmniRoute is an open-source AI gateway that routes requests across many model providers through one OpenAI-compatible endpoint, with… | 83.2k | 106k |
| #6 | CLIProxyAPICLIProxyAPI is a Go proxy server that exposes CLI coding accounts like Codex, Claude Code and Gemini CLI through OpenAI, Gemini and… | 57.1k | – |
Recent FlashInfer releases
FlashInfer FAQ
Is FlashInfer free and open source?
Yes. FlashInfer's source code is on GitHub (flashinfer-ai/flashinfer) under the Apache 2.0 license, so you can use it for free.
How do I install FlashInfer?
Run pip install flashinfer-python. The flashinfer-python package needs Python 3.10 or newer, below 4.0. The install section above has the other options from the project's README.
Is FlashInfer still maintained?
Yes. The flashinfer-ai/flashinfer repository was last updated on Oct 22, 2026, and the latest release, v0.7.2 on GitHub, came out on Oct 15, 2026.
How popular is FlashInfer?
FlashInfer has 6.8k GitHub stars as of Nov 1, 2026. It gained 57 new stars in the week of Oct 25–31, 2026, against a 12-week average of 61 a week. Its main package had 938.6k downloads that week, +18% against its own 4-week average. That made it #15 among LLM inference and training tools on the Devtoolwatch Board.
FlashInfer vs vLLM: how do they compare?
FlashInfer and vLLM are both tracked by Devtoolwatch as LLM inference and training tools. FlashInfer has 6.8k GitHub stars and vLLM has 96k. In the week of Oct 25–31, 2026, FlashInfer's main package had 938.6k downloads and vLLM's had 677.2k. vLLM is an open-source Python library and engine for high-throughput, memory-efficient inference and serving of large language models. vLLM review.
What are the best alternatives to FlashInfer?
Other LLM inference and training tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: dsh-our-free-model (#1), magpie (#2), Strata (#3), edge0 (#4), OmniRoute (#5). The alternatives table above has their numbers.
Does vLLM use FlashInfer?
Yes. The FlashInfer README lists vLLM among the inference engines it powers, along with SGLang, TensorRT-LLM, TGI, MLC-LLM, LightLLM, lorax and ScaleLLM.
What GPUs does FlashInfer support?
FlashInfer supports NVIDIA GPUs from Turing (SM 7.5) through Blackwell. The README lists T4, A100, L4, H100, H200, B200, B300, Jetson Thor, RTX 50 series and DGX Spark as examples. Not all features work on all compute capabilities.
How does FlashInfer relate to FlashAttention?
FlashInfer is inspired by FlashAttention. It can use FlashAttention-2/3 as one of several backends, alongside cuDNN, CUTLASS and TensorRT-LLM. FlashInfer selects a backend automatically for your hardware and workload.
Can FlashInfer run offline without compiling kernels?
Yes. Install the optional flashinfer-cubin and flashinfer-jit-cache packages with flashinfer install-cubin-wheel and flashinfer install-jit-cache-wheel. With these installed, most kernels are pre-compiled, which gives faster initialization and allows offline use.
How do I debug FlashInfer API calls?
Set the FLASHINFER_LOGLEVEL environment variable to turn on API logging. Levels are 0 (off), 1, 3 and 5. Set FLASHINFER_LOGDEST to stdout, stderr or a file path to choose where logs go.
Sources
FlashInfer README badge
Maintainers can show the tool's weekly Board rank in their README:
[](https://index.devtoolwatch.com/tools/flashinfer-ai-flashinfer/)How this page is made: Devtoolwatch writes each review from the project's README, releases, package metadata and website, with an AI model, then checks every claim against the source it cites and drops what the sources don't support. The numbers come from Devtoolwatch's daily GitHub snapshots, GitHub's stargazer history, and the npm and PyPI download APIs. Spotted an error? Email [email protected].