# Inspect AI review

> Inspect AI is an open-source Python framework from the UK AI Security Institute for writing and running large language model evaluations.

Devtoolwatch Board, the week of Oct 25–31, 2026. Canonical page: https://index.devtoolwatch.com/tools/inspect-ai/

## TL;DR

Inspect AI is a Python framework for large language model evaluations, created by the UK AI Security Institute. Inspect AI suits teams that build or run evals of coding, agentic, reasoning and knowledge tasks. Inspect AI includes over 200 pre-built evaluations, sandboxed tool execution, and support for running external agents such as Claude Code.

## Inspect AI at a glance

- **Category:** [Evals and observability](https://index.devtoolwatch.com/categories/evals-and-observability/)
- **Maintained by:** Department of Business, Energy and Industrial Strategy (organization, [@UKGovernmentBEIS](https://github.com/UKGovernmentBEIS) on GitHub)
- **License:** MIT
- **Language:** Python, Shell, Dockerfile
- **Repository:** [github.com/UKGovernmentBEIS/inspect_ai](https://github.com/UKGovernmentBEIS/inspect_ai)
- **Website:** [inspect.aisi.org.uk](https://inspect.aisi.org.uk/)
- **PyPI package:** [inspect-ai](https://pypi.org/project/inspect-ai/) 0.3.283
- **Latest release:** [0.3.283](https://github.com/UKGovernmentBEIS/inspect_ai/releases/tag/0.3.283) on GitHub, Oct 21, 2026
- **First published:** Nov 14, 2023

## What Inspect AI does

- Inspect AI provides composable datasets, agents, tools and scorers for writing reusable evaluations.
- Inspect AI ships over 200 pre-built evaluations ready to run on any model.
- Inspect AI supports custom and MCP tools, plus built-in bash, python, text editing, web search, web browsing and computer tools.
- Inspect AI can evaluate built-in agents, multi-agent setups, and external agents like Claude Code, Codex CLI and Gemini CLI.
- Inspect AI runs untrusted model code in sandboxes such as Docker, Kubernetes, Modal, Proxmox and Vagrant.
- Inspect AI includes model graded evaluations, multi-turn dialog, and prompt engineering components.
- Inspect AI offers a web-based Inspect View tool for monitoring and visualizing evaluations.

### How it works

Inspect AI installs as a Python package and runs evaluations from the command line with inspect eval, passing a task file and a model such as openai/gpt-4o. Inspect AI connects to model providers like OpenAI, Anthropic, Google and Hugging Face through their Python packages and API keys. Extensions in other Python packages can add model APIs, sandboxes, approvers, hooks and scoring techniques.

## How to install Inspect AI

**With pip**

```
pip install inspect-ai
```

### Requirements

- Python 3.11 or later.
- Access to a model, typically a provider Python package and an API key in the environment.

### First steps

1. Install Inspect AI from PyPI, and in VS Code optionally add the Inspect VS Code Extension.
2. Install a model provider package, for example with pip install openai.
3. Set the provider API key in your environment, such as OPENAI_API_KEY.
4. Run an evaluation, for example inspect eval simpleqa.py --model openai/gpt-4o.

## Who Inspect AI is for

- Researchers building custom evaluations of frontier AI models.
- Teams running established benchmarks against models without writing new evals.
- Developers evaluating coding agents or agentic tasks in sandboxed environments.

### Limitations

- Inspect AI requires access to a model, usually with a provider API key.
- Inspect AI releases often include breaking changes, such as new dataset checkpoint field handling in 0.3.283.
- Inspect AI dropped support for Python versions below 3.11 in release 0.3.281.

## Usage data (the week of Oct 25–31, 2026)

- Board rank: #301 overall, #16 in Evals and observability, momentum score 31
- Downloads that week: 727.6k (−13% vs 4-week average)
- New GitHub stars per week (last 12 weeks): Aug 9: 53; Aug 16: 51; Aug 23: 57; Aug 30: 50; Sep 6: 46; Sep 13: 56; Sep 20: 52; Sep 27: 69; Oct 4: 46; Oct 11: 57; Oct 18: 57; Oct 25: 68

## Inspect AI alternatives

| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | [SkillSpector](https://index.devtoolwatch.com/tools/nvidia-skillspector/) | 23.7k | – |
| #2 | [Judgeval](https://index.devtoolwatch.com/tools/judgmentlabs-judgeval/) | 1.1k | 80.2k |
| #3 | [A.I.G (AI-Infra-Guard)](https://index.devtoolwatch.com/tools/tencent-ai-infra-guard/) | 7.2k | – |
| #4 | [Langfuse](https://index.devtoolwatch.com/tools/langfuse/) | 36.8k | 9.8M |
| #5 | [Opik](https://index.devtoolwatch.com/tools/opik/) | 23.1k | 709.1k |
| #6 | [promptfoo](https://index.devtoolwatch.com/tools/promptfoo/) | 26.7k | 1.2M |

## Inspect AI FAQ

### Is Inspect AI free and open source?

Yes. Inspect AI's source code is on GitHub (UKGovernmentBEIS/inspect_ai) under the MIT license, so you can use it for free.

### How do I install Inspect AI?

Run `pip install inspect-ai`. The inspect-ai package needs Python 3.11 or newer.

### Is Inspect AI still maintained?

Yes. The UKGovernmentBEIS/inspect_ai repository was last updated on Oct 21, 2026, and the latest release, 0.3.283 on GitHub, came out on Oct 21, 2026.

### How popular is Inspect AI?

Inspect AI has 3.1k GitHub stars as of Nov 1, 2026. It gained 68 new stars in the week of Oct 25–31, 2026, against a 12-week average of 55 a week. Its main package had 727.6k downloads that week, −13% against its own 4-week average. That made it #16 among LLM evals and observability tools on the Devtoolwatch Board.

### Inspect AI vs DeepEval: how do they compare?

Inspect AI and DeepEval are both tracked by Devtoolwatch as LLM evals and observability tools. Inspect AI has 3.1k GitHub stars and DeepEval has 19.2k. In the week of Oct 25–31, 2026, Inspect AI's main package had 727.6k downloads and DeepEval's had 769.5k. DeepEval is an open-source Python framework for unit testing and evaluating LLM apps, agents, chatbots, voice agents, and RAG pipelines. [DeepEval review](https://index.devtoolwatch.com/tools/deepeval/)

### Inspect AI vs Langfuse: how do they compare?

Inspect AI and Langfuse are both tracked by Devtoolwatch as LLM evals and observability tools. Inspect AI has 3.1k GitHub stars and Langfuse has 36.8k. In the week of Oct 25–31, 2026, Inspect AI's main package had 727.6k downloads and Langfuse's had 9.8M. Langfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. [Langfuse review](https://index.devtoolwatch.com/tools/langfuse/)

### What are the best alternatives to Inspect AI?

Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), Judgeval (#2), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5). The alternatives table above has their numbers.

### What is Inspect AI?

Inspect AI is an open-source framework for large language model evaluations, developed by the UK AI Security Institute and Meridian Labs. Inspect AI measures coding, agentic tasks, reasoning, knowledge, behavior and multi-modal understanding.

### Does Inspect AI have a VS Code extension?

Yes. Inspect AI has a VS Code Extension that assists with authoring and debugging evaluations. The extension is not required but is highly recommended.

### Can Inspect AI evaluate external coding agents?

Yes. Inspect AI can run arbitrary external agents like Claude Code, Codex CLI and Gemini CLI, alongside its built-in and multi-agent primitives.

### How does Inspect AI handle untrusted model code?

Inspect AI uses a sandboxing system that runs untrusted model code in Docker, Kubernetes, Modal, Proxmox, Vagrant and other systems through an extension API.

### Are there docs for coding agents?

Yes. Inspect AI publishes a docs index at llms.txt, the user guide as Markdown at llms-guide.txt, and llms-full.txt with the API and CLI reference.

## Sources

1. [UKGovernmentBEIS/inspect_ai on GitHub](https://github.com/UKGovernmentBEIS/inspect_ai)
2. [Inspect AI README](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/README.md)
3. [inspect-ai on PyPI](https://pypi.org/project/inspect-ai/)
4. [Inspect AI website](https://inspect.aisi.org.uk/)
5. [Release 0.3.283 (Oct 21, 2026)](https://github.com/UKGovernmentBEIS/inspect_ai/releases/tag/0.3.283)
6. [Release 0.3.282 (Oct 19, 2026)](https://github.com/UKGovernmentBEIS/inspect_ai/releases/tag/0.3.282)
7. [Release 0.3.281 (Oct 15, 2026)](https://github.com/UKGovernmentBEIS/inspect_ai/releases/tag/0.3.281)
8. [Hacker News: Inspect: An open-source framework for large language model evaluations](https://news.ycombinator.com/item?id=49902367)
9. [Devtoolwatch Board, the week of Oct 25–31, 2026](https://index.devtoolwatch.com/)