Devtoolwatch BoardInspect AI review
Inspect AI is an open-source Python framework from the UK AI Security Institute for writing and running large language model evaluations.
TL;DR
Inspect AI is a Python framework for large language model evaluations, created by the UK AI Security Institute. Inspect AI suits teams that build or run evals of coding, agentic, reasoning and knowledge tasks. Inspect AI includes over 200 pre-built evaluations, sandboxed tool execution, and support for running external agents such as Claude Code.
- 3.1kGitHub stars
- +68new stars, week of Oct 25
- 727.6kdownloads a week, −13% vs 4-week avg
- #16in Evals and observability this week
Inspect AI at a glance
- Category
- Evals and observability
- Maintained by
- Department of Business, Energy and Industrial Strategy (organization, @UKGovernmentBEIS on GitHub)
- License
- MIT
- Language
- Python, Shell, Dockerfile
- Repository
- github.com/UKGovernmentBEIS/inspect_ai
- Website
- inspect.aisi.org.uk
- PyPI package
- inspect-ai 0.3.283
- Latest release
- 0.3.283 on GitHub, Oct 21, 2026
- First published
- Nov 14, 2023
What Inspect AI does
- Inspect AI provides composable datasets, agents, tools and scorers for writing reusable evaluations.
- Inspect AI ships over 200 pre-built evaluations ready to run on any model.
- Inspect AI supports custom and MCP tools, plus built-in bash, python, text editing, web search, web browsing and computer tools.
- Inspect AI can evaluate built-in agents, multi-agent setups, and external agents like Claude Code, Codex CLI and Gemini CLI.
- Inspect AI runs untrusted model code in sandboxes such as Docker, Kubernetes, Modal, Proxmox and Vagrant.
- Inspect AI includes model graded evaluations, multi-turn dialog, and prompt engineering components.
- Inspect AI offers a web-based Inspect View tool for monitoring and visualizing evaluations.
How it works
Inspect AI installs as a Python package and runs evaluations from the command line with inspect eval, passing a task file and a model such as openai/gpt-4o. Inspect AI connects to model providers like OpenAI, Anthropic, Google and Hugging Face through their Python packages and API keys. Extensions in other Python packages can add model APIs, sandboxes, approvers, hooks and scoring techniques.
How to install and start using Inspect AI
With pip
pip install inspect-aiRequirements
- Python 3.11 or later.
- Access to a model, typically a provider Python package and an API key in the environment.
First steps
- Install Inspect AI from PyPI, and in VS Code optionally add the Inspect VS Code Extension.
- Install a model provider package, for example with pip install openai.
- Set the provider API key in your environment, such as OPENAI_API_KEY.
- Run an evaluation, for example inspect eval simpleqa.py --model openai/gpt-4o.
Who Inspect AI is for
- Researchers building custom evaluations of frontier AI models.
- Teams running established benchmarks against models without writing new evals.
- Developers evaluating coding agents or agentic tasks in sandboxed environments.
Limitations to know
- Inspect AI requires access to a model, usually with a provider API key.
- Inspect AI releases often include breaking changes, such as new dataset checkpoint field handling in 0.3.283.
- Inspect AI dropped support for Python versions below 3.11 in release 0.3.281.
Is Inspect AI growing? Usage data
In the week of Oct 25–31, 2026, Inspect AI ranked #301 on the Devtoolwatch Board and #16 in Evals and observability, with a momentum score of 31 out of 100. Its main package had 727.6k downloads that week, −13% against its own 4-week average.
Show the weekly numbers
| Week of | New stars |
|---|---|
| Aug 9, 2026 | 53 |
| Aug 16, 2026 | 51 |
| Aug 23, 2026 | 57 |
| Aug 30, 2026 | 50 |
| Sep 6, 2026 | 46 |
| Sep 13, 2026 | 56 |
| Sep 20, 2026 | 52 |
| Sep 27, 2026 | 69 |
| Oct 4, 2026 | 46 |
| Oct 11, 2026 | 57 |
| Oct 18, 2026 | 57 |
| Oct 25, 2026 | 68 |
On Hacker News
- Inspect: An open-source framework for large language model evaluations, 5 points and 0 comments (Sep 29, 2026)
The Board ranks tools by four signals each week: new stars, new stars for the repo's size, added downloads and download growth beyond the market. How the ranking works.
Inspect AI alternatives
Other LLM evals and observability tools on the Board in the week of Oct 25–31, 2026, by momentum:
| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | SkillSpectorSkillSpector is NVIDIA's security scanner that checks AI agent skills for vulnerabilities, malicious patterns, and supply-chain risks… | 23.7k | – |
| #2 | JudgevalJudgeval is an open-source Python SDK for tracing and evaluating LLM-powered agents with prompt-based judges and online monitoring. | 1.1k | 80.2k |
| #3 | A.I.G (AI-Infra-Guard)A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak… | 7.2k | – |
| #4 | LangfuseLangfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. | 36.8k | 9.8M |
| #5 | OpikOpik is an open-source platform from Comet for tracing, evaluating, and monitoring LLM applications, RAG systems, and AI agents. | 23.1k | 709.1k |
| #6 | promptfoopromptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, including prompts, agents, and RAG pipelines. | 26.7k | 1.2M |
Recent Inspect AI releases
Inspect AI FAQ
Is Inspect AI free and open source?
Yes. Inspect AI's source code is on GitHub (UKGovernmentBEIS/inspect_ai) under the MIT license, so you can use it for free.
How do I install Inspect AI?
Run pip install inspect-ai. The inspect-ai package needs Python 3.11 or newer.
Is Inspect AI still maintained?
Yes. The UKGovernmentBEIS/inspect_ai repository was last updated on Oct 21, 2026, and the latest release, 0.3.283 on GitHub, came out on Oct 21, 2026.
How popular is Inspect AI?
Inspect AI has 3.1k GitHub stars as of Nov 1, 2026. It gained 68 new stars in the week of Oct 25–31, 2026, against a 12-week average of 55 a week. Its main package had 727.6k downloads that week, −13% against its own 4-week average. That made it #16 among LLM evals and observability tools on the Devtoolwatch Board.
Inspect AI vs DeepEval: how do they compare?
Inspect AI and DeepEval are both tracked by Devtoolwatch as LLM evals and observability tools. Inspect AI has 3.1k GitHub stars and DeepEval has 19.2k. In the week of Oct 25–31, 2026, Inspect AI's main package had 727.6k downloads and DeepEval's had 769.5k. DeepEval is an open-source Python framework for unit testing and evaluating LLM apps, agents, chatbots, voice agents, and RAG pipelines. DeepEval review.
Inspect AI vs Langfuse: how do they compare?
Inspect AI and Langfuse are both tracked by Devtoolwatch as LLM evals and observability tools. Inspect AI has 3.1k GitHub stars and Langfuse has 36.8k. In the week of Oct 25–31, 2026, Inspect AI's main package had 727.6k downloads and Langfuse's had 9.8M. Langfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. Langfuse review.
What are the best alternatives to Inspect AI?
Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), Judgeval (#2), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5). The alternatives table above has their numbers.
What is Inspect AI?
Inspect AI is an open-source framework for large language model evaluations, developed by the UK AI Security Institute and Meridian Labs. Inspect AI measures coding, agentic tasks, reasoning, knowledge, behavior and multi-modal understanding.
Does Inspect AI have a VS Code extension?
Yes. Inspect AI has a VS Code Extension that assists with authoring and debugging evaluations. The extension is not required but is highly recommended.
Can Inspect AI evaluate external coding agents?
Yes. Inspect AI can run arbitrary external agents like Claude Code, Codex CLI and Gemini CLI, alongside its built-in and multi-agent primitives.
How does Inspect AI handle untrusted model code?
Inspect AI uses a sandboxing system that runs untrusted model code in Docker, Kubernetes, Modal, Proxmox, Vagrant and other systems through an extension API.
Are there docs for coding agents?
Yes. Inspect AI publishes a docs index at llms.txt, the user guide as Markdown at llms-guide.txt, and llms-full.txt with the API and CLI reference.
Sources
- UKGovernmentBEIS/inspect_ai on GitHub
- Inspect AI README
- inspect-ai on PyPI
- Inspect AI website
- Release 0.3.283 (Oct 21, 2026)
- Release 0.3.282 (Oct 19, 2026)
- Release 0.3.281 (Oct 15, 2026)
- Hacker News: Inspect: An open-source framework for large language model evaluations
- Devtoolwatch Board, the week of Oct 25–31, 2026
Inspect AI README badge
Maintainers can show the tool's weekly Board rank in their README:
[](https://index.devtoolwatch.com/tools/inspect-ai/)How this page is made: Devtoolwatch writes each review from the project's README, releases, package metadata and website, with an AI model, then checks every claim against the source it cites and drops what the sources don't support. The numbers come from Devtoolwatch's daily GitHub snapshots, GitHub's stargazer history, and the npm and PyPI download APIs. Spotted an error? Email [email protected].