DeepEval review

Q: How do I install DeepEval?

Run pip install -U deepeval. The deepeval package needs Python 3.9 or newer, below 4.0. The install section above has the other options from the project's README.

DeepEval is an open-source Python framework for unit testing and evaluating LLM apps, agents, chatbots, voice agents, and RAG pipelines.

Updated Nov 1, 2026. Review written Oct 22, 2026 from the project's own sources; usage data refreshed every Monday.

TL;DR

DeepEval is an open-source LLM evaluation framework that works like Pytest but is specialized for unit testing LLM apps. DeepEval is for developers building AI agents, multi-turn chatbots, voice agents, or RAG pipelines. DeepEval ships ready-to-use metrics across these areas, using LLM-as-a-judge, statistical methods, or NLP models that run locally.

19.2kGitHub stars
+140new stars, week of Oct 25
769.5kdownloads a week, +4% vs 4-week avg
#9in Evals and observability this week

DeepEval at a glance

Category: Evals and observability
Maintained by: Confident AI (organization, @confident-ai on GitHub)
License: Apache 2.0
Language: Python, TypeScript, JavaScript
Repository: github.com/confident-ai/deepeval
Website: deepeval.com
PyPI package: deepeval 4.3.1
Latest release: setup-v0.1.0 on GitHub, Oct 16, 2026
First published: Aug 10, 2023

What DeepEval does

DeepEval evaluates LLM apps end-to-end as black boxes, plus full agent trajectories and individual steps like LLM calls, tool use, and retrieval.
DeepEval provides metrics for agents, multi-turn chat, voice agents, RAG, MCP usage, and multimodal image tasks.
DeepEval supports custom criteria through G-Eval, DAG graph-based metrics, JevEval, and your own custom metrics.
DeepEval generates single-turn and multi-turn synthetic datasets for evaluation.
DeepEval benchmarks LLMs on MMLU, HellaSwag, DROP, BIG-Bench Hard, TruthfulQA, HumanEval, and GSM8K.
DeepEval can optimize prompts automatically based on evaluation results.
DeepEval traces each step of an agent and shows scores in the terminal during test runs.

How it works

DeepEval runs as Pytest-native tests in CI/CD or as Python scripts. You write test cases with metrics and run them with the deepeval CLI. Judge metrics can use any LLM, such as OpenAI through an API key or a custom model. DeepEval integrates with LangChain, LangGraph, CrewAI, LlamaIndex, Pydantic AI, OpenAI Agents, Anthropic, Google ADK, Mastra, and others. Results can optionally go to the Confident AI platform.

How to install and start using DeepEval

With pip

pip install -U deepeval

Requirements

Python 3.9 or later, below 4.0.
An OpenAI API key for the default judge model, or a configured custom model.
A Confident AI account is optional but recommended for cloud reports.

First steps

Optionally run deepeval login to create a free account for shareable cloud test reports.
Create a test file, such as test_ai_agent.py, with an LLMTestCase and a metric like GEval.
Set OPENAI_API_KEY as an environment variable, or configure a custom evaluation model.
Run the test with deepeval test run test_ai_agent.py.

Who DeepEval is for

Developers who want Pytest-style unit tests for LLM apps in CI/CD.
Teams evaluating AI agents, including tool calls, plans, and step efficiency.
Builders of RAG pipelines who need faithfulness and retrieval context metrics.
Teams building multi-turn chatbots or voice agents.

Limitations to know

DeepEval's LLM-as-a-judge metrics need a judge model, OpenAI by default, which requires an API key.
DeepEval tests known cases before shipping; production trace monitoring is handled by the separate Confident AI platform.
When logged in to the platform, DeepEval automatically logs all test cases to the cloud.

Is DeepEval growing? Usage data

In the week of Oct 25–31, 2026, DeepEval ranked #177 on the Devtoolwatch Board and #9 in Evals and observability, with a momentum score of 47 out of 100. Its main package had 769.5k downloads that week, +4% against its own 4-week average.

New GitHub stars per week for the last 12 weeks, from GitHub's stargazer history (today's stargazers by the week they starred).

Show the weekly numbers

New GitHub stars per week for DeepEval
Week of	New stars
Aug 9, 2026	129
Aug 16, 2026	194
Aug 23, 2026	169
Aug 30, 2026	166
Sep 6, 2026	120
Sep 13, 2026	106
Sep 20, 2026	120
Sep 27, 2026	156
Oct 4, 2026	136
Oct 11, 2026	200
Oct 18, 2026	167
Oct 25, 2026	140

On Hacker News

Show HN: I implemented evals metrics for LLMs that runs locally on your machine, 22 points and 3 comments (Dec 11, 2023)
Show HN: DeepEval – Evaluation and Unit Testing for LLMs, 18 points and 8 comments (Sep 25, 2023)
Show HN: DeepEval – Unit Testing for LLMs (Open Science), 6 points and 0 comments (Oct 6, 2023)
Show HN: Vibe code your agents without vibe coding your agent, 6 points and 0 comments (May 8, 2026)

The Board ranks tools by four signals each week: new stars, new stars for the repo's size, added downloads and download growth beyond the market. How the ranking works.

DeepEval alternatives

Other LLM evals and observability tools on the Board in the week of Oct 25–31, 2026, by momentum:

Evals and observability ranked by momentum, the week of Oct 25–31, 2026
Rank	Tool	Stars	Downloads/wk
#1	SkillSpectorSkillSpector is NVIDIA's security scanner that checks AI agent skills for vulnerabilities, malicious patterns, and supply-chain risks…	23.7k	–
#2	JudgevalJudgeval is an open-source Python SDK for tracing and evaluating LLM-powered agents with prompt-based judges and online monitoring.	1.1k	80.2k
#3	A.I.G (AI-Infra-Guard)A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak…	7.2k	–
#4	LangfuseLangfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents.	36.8k	9.8M
#5	OpikOpik is an open-source platform from Comet for tracing, evaluating, and monitoring LLM applications, RAG systems, and AI agents.	23.1k	709.1k
#6	promptfoopromptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, including prompts, agents, and RAG pipelines.	26.7k	1.2M

All LLM evals and observability tools, ranked

Recent DeepEval releases

setup-v0.1.0, Oct 16, 2026
python-v4.2.4, Sep 22, 2026
typescript-v0.9.13 (pre-release), Aug 24, 2026
python-v4.2.0, Aug 24, 2026
typescript-v0.9.11, Aug 21, 2026

DeepEval FAQ

Is DeepEval free and open source?

Yes. DeepEval's source code is on GitHub (confident-ai/deepeval) under the Apache 2.0 license, so you can use it for free.

How do I install DeepEval?

Run pip install -U deepeval. The deepeval package needs Python 3.9 or newer, below 4.0. The install section above has the other options from the project's README.

Is DeepEval still maintained?

Yes. The confident-ai/deepeval repository was last updated on Oct 19, 2026, and the latest release, setup-v0.1.0 on GitHub, came out on Oct 16, 2026.

How popular is DeepEval?

DeepEval has 19.2k GitHub stars as of Nov 1, 2026. It gained 140 new stars in the week of Oct 25–31, 2026, against a 12-week average of 150 a week. Its main package had 769.5k downloads that week, +4% against its own 4-week average. That made it #9 among LLM evals and observability tools on the Devtoolwatch Board.

DeepEval vs Ragas: how do they compare?

DeepEval and Ragas are both tracked by Devtoolwatch as LLM evals and observability tools. DeepEval has 19.2k GitHub stars and Ragas has 16.2k. In the week of Oct 25–31, 2026, DeepEval's main package had 769.5k downloads and Ragas's had 273k. Ragas is a Python evaluation framework for RAG and LLM applications, with LLM-based and traditional metrics plus test data generation. Ragas review.

DeepEval vs Langfuse: how do they compare?

DeepEval and Langfuse are both tracked by Devtoolwatch as LLM evals and observability tools. DeepEval has 19.2k GitHub stars and Langfuse has 36.8k. In the week of Oct 25–31, 2026, DeepEval's main package had 769.5k downloads and Langfuse's had 9.8M. Langfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents. Langfuse review.

What are the best alternatives to DeepEval?

Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), Judgeval (#2), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5). The alternatives table above has their numbers.

Is DeepEval open source?

Yes. DeepEval describes itself as an open-source LLM evaluation framework, and the repository is licensed under Apache-2.0.

Does DeepEval use LLM-as-a-judge?

Yes. Many DeepEval metrics, such as G-Eval and DAG, use LLM-as-a-judge. DeepEval also uses statistical methods, NLP models that run locally, and Jev, a System One model that returns calibrated probabilities instead of generated text.

How does DeepEval work?

DeepEval works like Pytest for LLM apps. You define test cases with inputs, actual outputs, and expected outputs, attach metrics with thresholds, and run them with deepeval test run. Each metric returns a score with an explanation.

What is DeepEval used for?

DeepEval is used to test AI agents, chatbots, voice agents, and RAG pipelines. Teams use it to choose models, prompts, and architecture, prevent prompt drift, and switch providers such as OpenAI to Claude with confidence.

Can DeepEval use a model other than OpenAI?

Yes. DeepEval metrics can be powered by any LLM you choose. The README notes that you can evaluate using your own custom model instead of OpenAI.

Is DeepEval a library?

Yes. DeepEval is a Python package installed from PyPI as deepeval, with a CLI for running tests. Releases also exist for a TypeScript version.

Sources

DeepEval README badge

Maintainers can show the tool's weekly Board rank in their README:

[![Devtoolwatch](https://index.devtoolwatch.com/badge/deepeval.svg)](https://index.devtoolwatch.com/tools/deepeval/)

How this page is made: Devtoolwatch writes each review from the project's README, releases, package metadata and website, with an AI model, then checks every claim against the source it cites and drops what the sources don't support. The numbers come from Devtoolwatch's daily GitHub snapshots, GitHub's stargazer history, and the npm and PyPI download APIs. Spotted an error? Email [email protected].