# FlashInfer review

> FlashInfer is a GPU kernel library and kernel generator for LLM inference, covering attention, GEMM, MoE, sampling and communication operations.

Devtoolwatch Board, the week of Oct 25–31, 2026. Canonical page: https://index.devtoolwatch.com/tools/flashinfer-ai-flashinfer/

## TL;DR

FlashInfer is a library and kernel generator for LLM inference on NVIDIA GPUs from Turing through Blackwell. It is for developers building or tuning LLM serving engines. FlashInfer provides unified APIs for attention, GEMM and MoE with multiple backends, including FlashAttention-2/3, cuDNN, CUTLASS and TensorRT-LLM, and selects a backend automatically.

## FlashInfer at a glance

- **Category:** [Inference and training](https://index.devtoolwatch.com/categories/inference-and-training/)
- **Maintained by:** FlashInfer (organization, [@flashinfer-ai](https://github.com/flashinfer-ai) on GitHub)
- **License:** Apache 2.0
- **Language:** Cuda, Python, C++
- **Repository:** [github.com/flashinfer-ai/flashinfer](https://github.com/flashinfer-ai/flashinfer)
- **Website:** [flashinfer.ai](https://flashinfer.ai)
- **PyPI package:** [flashinfer-python](https://pypi.org/project/flashinfer-python/) 0.7.2
- **Latest release:** [v0.7.2](https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.7.2) on GitHub, Oct 15, 2026
- **First published:** Jul 22, 2023

## What FlashInfer does

- FlashInfer provides attention kernels for decode, prefill and append, with paged and ragged KV-cache for dynamic batch serving.
- FlashInfer supports MLA attention, cascade attention for shared prefixes, block-sparse attention and fused prefill+decode POD-Attention.
- FlashInfer offers BF16, FP8 and FP4 GEMM, plus grouped GEMM for LoRA and multi-expert routing.
- FlashInfer includes fused MoE kernels with DeepSeek-V3, Llama-4 and top-k routing, and FP8 or FP4 expert weights.
- FlashInfer implements sorting-free Top-K, Top-P and Min-P sampling and chain speculative sampling.
- FlashInfer provides custom AllReduce, multi-node NVLink support and NVSHMEM integration for distributed inference.
- FlashInfer kernels are compatible with CUDAGraph and torch.compile for low-latency serving.

### How it works

FlashInfer is a Python package used from PyTorch on CUDA tensors. The core package compiles or downloads kernels on first use. Optional packages supply pre-compiled kernel binaries for faster startup and offline use. A flashinfer CLI shows configuration, manages modules and caches, and exports compile commands. Serving engines such as SGLang, vLLM, TensorRT-LLM and TGI use FlashInfer for inference.

## How to install FlashInfer

**With pip**

```
pip install flashinfer-python
```

**For Blackwell CuTe DSL kernels (CUDA 13)**

```
pip install flashinfer-python[cu13]
```

**Verify installation**

```
flashinfer show-config
```

### Requirements

- Python 3.10 or later, below 4.0
- NVIDIA GPU with compute capability SM 7.5 (Turing) or later
- CUDA 12.9, 13.0, or 13.4 (PyTorch nightly)
- PyTorch, used for tensors in the basic usage example

### First steps

1. Install FlashInfer with pip install flashinfer-python.
2. Optionally run flashinfer install-cubin-wheel and flashinfer install-jit-cache-wheel to pre-compile most kernels for faster initialization and offline use.
3. Run flashinfer show-config to verify the installation and view configuration.
4. Call flashinfer.single_decode_with_kv_cache(q, k, v) on CUDA tensors to run single decode attention.

## Who FlashInfer is for

- Developers building LLM serving engines who need attention, GEMM and MoE kernels
- Teams serving MoE or MLA models such as DeepSeek on Hopper or Blackwell GPUs
- Engineers using FP8 or FP4 quantization for inference
- Researchers creating custom attention variants through JIT

### Limitations

- FlashInfer does not support every feature on every compute capability, according to its GPU support notes.
- FlashInfer targets NVIDIA GPUs and CUDA. The sources list no other hardware.
- The core FlashInfer package compiles or downloads kernels on first use unless the optional pre-compiled packages are installed.
- FlashInfer runtime CI covers CUDA 12.9 and 13.0. CUDA 13.4 wheels use a preview toolkit.

## Usage data (the week of Oct 25–31, 2026)

- Board rank: #136 overall, #15 in Inference and training, momentum score 56
- Downloads that week: 938.6k (+18% vs 4-week average)
- New GitHub stars per week (last 12 weeks): Aug 9: 42; Aug 16: 48; Aug 23: 75; Aug 30: 59; Sep 6: 50; Sep 13: 70; Sep 20: 55; Sep 27: 34; Oct 4: 48; Oct 11: 93; Oct 18: 100; Oct 25: 57

## FlashInfer alternatives

| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | [dsh-our-free-model](https://index.devtoolwatch.com/tools/ebony-vinyl-dsh-our-free-model/) | 54.3k | – |
| #2 | [magpie](https://index.devtoolwatch.com/tools/yetone-magpie/) | 26.9k | – |
| #3 | [Strata](https://index.devtoolwatch.com/tools/niko1221-strata/) | 37.1k | – |
| #4 | [edge0](https://index.devtoolwatch.com/tools/edge0-ai-edge0/) | 8.9k | – |
| #5 | [OmniRoute](https://index.devtoolwatch.com/tools/diegosouzapw-omniroute/) | 83.2k | 106k |
| #6 | [CLIProxyAPI](https://index.devtoolwatch.com/tools/router-for-me-cliproxyapi/) | 57.1k | – |

## FlashInfer FAQ

### Is FlashInfer free and open source?

Yes. FlashInfer's source code is on GitHub (flashinfer-ai/flashinfer) under the Apache 2.0 license, so you can use it for free.

### How do I install FlashInfer?

Run `pip install flashinfer-python`. The flashinfer-python package needs Python 3.10 or newer, below 4.0. The install section above has the other options from the project's README.

### Is FlashInfer still maintained?

Yes. The flashinfer-ai/flashinfer repository was last updated on Oct 22, 2026, and the latest release, v0.7.2 on GitHub, came out on Oct 15, 2026.

### How popular is FlashInfer?

FlashInfer has 6.8k GitHub stars as of Nov 1, 2026. It gained 57 new stars in the week of Oct 25–31, 2026, against a 12-week average of 61 a week. Its main package had 938.6k downloads that week, +18% against its own 4-week average. That made it #15 among LLM inference and training tools on the Devtoolwatch Board.

### FlashInfer vs vLLM: how do they compare?

FlashInfer and vLLM are both tracked by Devtoolwatch as LLM inference and training tools. FlashInfer has 6.8k GitHub stars and vLLM has 96k. In the week of Oct 25–31, 2026, FlashInfer's main package had 938.6k downloads and vLLM's had 677.2k. vLLM is an open-source Python library and engine for high-throughput, memory-efficient inference and serving of large language models. [vLLM review](https://index.devtoolwatch.com/tools/vllm/)

### What are the best alternatives to FlashInfer?

Other LLM inference and training tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: dsh-our-free-model (#1), magpie (#2), Strata (#3), edge0 (#4), OmniRoute (#5). The alternatives table above has their numbers.

### Does vLLM use FlashInfer?

Yes. The FlashInfer README lists vLLM among the inference engines it powers, along with SGLang, TensorRT-LLM, TGI, MLC-LLM, LightLLM, lorax and ScaleLLM.

### What GPUs does FlashInfer support?

FlashInfer supports NVIDIA GPUs from Turing (SM 7.5) through Blackwell. The README lists T4, A100, L4, H100, H200, B200, B300, Jetson Thor, RTX 50 series and DGX Spark as examples. Not all features work on all compute capabilities.

### How does FlashInfer relate to FlashAttention?

FlashInfer is inspired by FlashAttention. It can use FlashAttention-2/3 as one of several backends, alongside cuDNN, CUTLASS and TensorRT-LLM. FlashInfer selects a backend automatically for your hardware and workload.

### Can FlashInfer run offline without compiling kernels?

Yes. Install the optional flashinfer-cubin and flashinfer-jit-cache packages with flashinfer install-cubin-wheel and flashinfer install-jit-cache-wheel. With these installed, most kernels are pre-compiled, which gives faster initialization and allows offline use.

### How do I debug FlashInfer API calls?

Set the FLASHINFER_LOGLEVEL environment variable to turn on API logging. Levels are 0 (off), 1, 3 and 5. Set FLASHINFER_LOGDEST to stdout, stderr or a file path to choose where logs go.

## Sources

1. [flashinfer-ai/flashinfer on GitHub](https://github.com/flashinfer-ai/flashinfer)
2. [FlashInfer README](https://github.com/flashinfer-ai/flashinfer/blob/main/README.md)
3. [flashinfer-python on PyPI](https://pypi.org/project/flashinfer-python/)
4. [FlashInfer website](https://flashinfer.ai)
5. [Release v0.7.2rc5 (Oct 20, 2026)](https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.7.2rc5)
6. [Release v0.7.2 (Oct 15, 2026)](https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.7.2)
7. [Release v0.7.2rc3 (Oct 13, 2026)](https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.7.2rc3)
8. [Devtoolwatch Board, the week of Oct 25–31, 2026](https://index.devtoolwatch.com/)