# promptfoo review

> promptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, including prompts, agents, and RAG pipelines.

Devtoolwatch Board, the week of Oct 25–31, 2026. Canonical page: https://index.devtoolwatch.com/tools/promptfoo/

## TL;DR

promptfoo is a command-line tool and library for testing prompts, models, agents, and RAG apps, and for red-team vulnerability scanning. It is for developers shipping LLM apps who want repeatable checks in CI/CD. Configs are declarative, and evals run locally, so prompts stay on your machine.

## promptfoo at a glance

- **Category:** [Evals and observability](https://index.devtoolwatch.com/categories/evals-and-observability/)
- **Maintained by:** promptfoo (organization, [@promptfoo](https://github.com/promptfoo) on GitHub)
- **License:** MIT
- **Language:** TypeScript, CSS, JavaScript
- **Repository:** [github.com/promptfoo/promptfoo](https://github.com/promptfoo/promptfoo)
- **Website:** [promptfoo.dev](https://promptfoo.dev)
- **npm package:** [promptfoo](https://www.npmjs.com/package/promptfoo) 0.125.1
- **Latest release:** [0.125.1](https://github.com/promptfoo/promptfoo/releases/tag/0.125.1) on GitHub, Oct 25, 2026
- **First published:** Apr 28, 2023

## What promptfoo does

- promptfoo runs automated evaluations of prompts and models against test cases you define.
- promptfoo red-teams LLM apps for prompt injections, jailbreaks, PII leaks, business rule violations, and insecure tool use in agents.
- promptfoo compares models side by side across providers such as OpenAI, Anthropic, Azure, Bedrock, and Ollama.
- promptfoo runs automated checks in CI/CD pipelines.
- promptfoo reviews pull requests for LLM-related security and compliance issues through code scanning.
- promptfoo generates security vulnerability reports and lets you share results with your team.

### How it works

promptfoo is a TypeScript CLI and Node.js package. You describe prompts, providers, and tests in declarative config files, then run evals with `promptfoo eval` and view the results with `promptfoo view`. It connects to LLM providers through their APIs, usually with an API key set as an environment variable. It works with any LLM API or programming language, and it runs evals locally.

## How to install promptfoo

**With npm**

```
npm install -g promptfoo
```

**With Homebrew**

```
brew install promptfoo
```

**With pip**

```
pip install promptfoo
```

### Requirements

- Node.js 22.22.0 or newer for the npm package.
- An API key for most LLM providers, set as an environment variable.

### First steps

1. Create a sample project with `promptfoo init --example getting-started`.
2. Set your provider API key as an environment variable, for example `export OPENAI_API_KEY=sk-abc123`.
3. Change into the `getting-started` folder and run `promptfoo eval`.
4. Open the results with `promptfoo view`.

## Who promptfoo is for

- Developers who want repeatable tests for prompts, agents, and RAG pipelines.
- Teams that want to scan LLM apps for security vulnerabilities before shipping.
- Teams that want to add LLM checks to CI/CD pipelines and pull request reviews.
- Developers choosing between models from several providers using their own test cases.

### Limitations

- Some integrations, including the Claude Agent, Codex, WatsonX, Langfuse, and Slack SDKs, are opt-in installs as of recent releases.
- The hosted ChatKit provider was removed in version 0.124.0.
- The npm package requires a recent Node.js version (22.22.0 or newer).

## Usage data (the week of Oct 25–31, 2026)

- Board rank: #83 overall, #6 in Evals and observability, momentum score 67
- Downloads that week: 1.2M (+16% vs 4-week average)
- New GitHub stars per week (last 12 weeks): Aug 9: 199; Aug 16: 219; Aug 23: 199; Aug 30: 189; Sep 6: 215; Sep 13: 248; Sep 20: 201; Sep 27: 220; Oct 4: 217; Oct 11: 239; Oct 18: 314; Oct 25: 306

## promptfoo alternatives

| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | [SkillSpector](https://index.devtoolwatch.com/tools/nvidia-skillspector/) | 23.7k | – |
| #2 | [Judgeval](https://index.devtoolwatch.com/tools/judgmentlabs-judgeval/) | 1.1k | 80.2k |
| #3 | [A.I.G (AI-Infra-Guard)](https://index.devtoolwatch.com/tools/tencent-ai-infra-guard/) | 7.2k | – |
| #4 | [Langfuse](https://index.devtoolwatch.com/tools/langfuse/) | 36.8k | 9.8M |
| #5 | [Opik](https://index.devtoolwatch.com/tools/opik/) | 23.1k | 709.1k |
| #7 | [garak](https://index.devtoolwatch.com/tools/garak/) | 9.8k | 12.9k |

## promptfoo FAQ

### Is promptfoo free and open source?

Yes. promptfoo's source code is on GitHub (promptfoo/promptfoo) under the MIT license, so you can use it for free. Its website also links to a pricing page (https://promptfoo.dev/pricing/) for any paid plans.

### How do I install promptfoo?

Run `npm install -g promptfoo`. It needs Node.js >=22.22.0. The install section above has the other options from the project's README.

### Is promptfoo still maintained?

Yes. The promptfoo/promptfoo repository was last updated on Oct 25, 2026, and the latest release, 0.125.1 on GitHub, came out on Oct 25, 2026.

### How popular is promptfoo?

promptfoo has 26.7k GitHub stars as of Nov 1, 2026. It gained 306 new stars in the week of Oct 25–31, 2026, against a 12-week average of 231 a week. Its main package had about 1.2M downloads that week, +16% against its own 4-week average. That made it #6 among LLM evals and observability tools on the Devtoolwatch Board.

### promptfoo vs DeepEval: how do they compare?

promptfoo and DeepEval are both tracked by Devtoolwatch as LLM evals and observability tools. promptfoo has 26.7k GitHub stars and DeepEval has 19.2k. In the week of Oct 25–31, 2026, promptfoo's main package had 1.2M downloads and DeepEval's had 769.5k. DeepEval is an open-source Python framework for unit testing and evaluating LLM apps, agents, chatbots, voice agents, and RAG pipelines. [DeepEval review](https://index.devtoolwatch.com/tools/deepeval/)

### promptfoo vs Langfuse: how do they compare?

promptfoo and Langfuse are both tracked by Devtoolwatch as LLM evals and observability tools. promptfoo has 26.7k GitHub stars and Langfuse has 36.8k. In the week of Oct 25–31, 2026, promptfoo's main package had 1.2M downloads and Langfuse's had 9.8M. Langfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. [Langfuse review](https://index.devtoolwatch.com/tools/langfuse/)

### What are the best alternatives to promptfoo?

Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), Judgeval (#2), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5). The alternatives table above has their numbers.

### What is promptfoo used for?

promptfoo is used to evaluate and red-team LLM-based apps. Developers use it to test prompts and models, compare providers side by side, scan for vulnerabilities such as prompt injection and data leaks, and automate these checks in CI/CD.

### How do you run promptfoo?

promptfoo runs from the command line. Run `promptfoo init --example getting-started`, set your provider API key, then run `promptfoo eval` and `promptfoo view` inside the project folder. You can also run any command without installing by using `npx promptfoo@latest`.

### How does promptfoo work?

promptfoo reads declarative config files that list prompts, providers, and tests. It sends the tests to the configured LLM APIs, scores the outputs, and shows the results in the CLI or a web viewer. For red teaming, it generates attacks tailored to your target application.

### Is promptfoo joining OpenAI?

Yes. The promptfoo README states that Promptfoo is now part of OpenAI. It also states that Promptfoo remains open source and MIT licensed.

### Is promptfoo safe for private prompts?

promptfoo runs LLM evals locally, so your prompts never leave your machine, according to the README. Prompts are still sent to whichever LLM provider APIs you configure.

### Does promptfoo work with Python?

Yes. promptfoo is available through `pip install promptfoo`. The README says it works with any LLM API or programming language. It also ships as a Node.js package.

## Sources

1. [promptfoo/promptfoo on GitHub](https://github.com/promptfoo/promptfoo)
2. [promptfoo README](https://github.com/promptfoo/promptfoo/blob/main/README.md)
3. [promptfoo on npm](https://www.npmjs.com/package/promptfoo)
4. [promptfoo website](https://promptfoo.dev)
5. [Release 0.125.1 (Oct 25, 2026)](https://github.com/promptfoo/promptfoo/releases/tag/0.125.1)
6. [Release 0.125.0 (Oct 23, 2026)](https://github.com/promptfoo/promptfoo/releases/tag/0.125.0)
7. [Release code-scan-action-0.2.2 (Oct 8, 2026)](https://github.com/promptfoo/promptfoo/releases/tag/code-scan-action-0.2.2)
8. [Hacker News: Questions censored by DeepSeek](https://news.ycombinator.com/item?id=42858552)
9. [Hacker News: Promptfoo Is Joining OpenAI](https://news.ycombinator.com/item?id=47312346)
10. [Hacker News: Show HN: Automated red teaming for your LLM app](https://news.ycombinator.com/item?id=40671686)
11. [Devtoolwatch Board, the week of Oct 25–31, 2026](https://index.devtoolwatch.com/)