garak review

Q: How do I install garak?

Run python -m pip install -U garak. The garak package needs Python 3.11 or newer. The install section above has the other options from the project's README.

garak is a command-line LLM vulnerability scanner from NVIDIA that probes models for jailbreaks, prompt injection, data leakage, hallucination and toxicity.

Updated Nov 1, 2026. Review written Oct 22, 2026 from the project's own sources; usage data refreshed every Monday.

TL;DR

garak is a Python command-line tool that checks whether an LLM or dialog system can be made to fail in unwanted ways. It is for developers and security teams red-teaming models. The README compares it to nmap or Metasploit, but for LLMs. It combines static, dynamic and adaptive probes and connects to many model providers.

9.8kGitHub stars
+123new stars, week of Oct 25
12.9kdownloads a week, +16% vs 4-week avg
#7in Evals and observability this week

garak at a glance

Category: Evals and observability
Maintained by: NVIDIA Corporation (organization, @NVIDIA on GitHub)
License: Apache 2.0
Language: Python, HTML, TypeScript
Repository: github.com/NVIDIA/garak
Website: discord.gg/uVch4puUCs
PyPI package: garak 0.17.1
Latest release: v0.17.1 on GitHub, Oct 16, 2026
First published: May 10, 2023

What garak does

garak probes LLMs for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks and other weaknesses.
garak includes probe families such as dan, encoding, gcg, leakreplay, malwaregen, packagehallucination, xss and promptinject.
garak scores each probe with detectors and marks responses showing undesirable behavior as FAIL, with a failure rate.
garak writes a JSONL report per run plus a garak.log file, and ships a basic analysis script.
garak reports a hit@k attack success rate metric per probe/detector pair, added in v0.17.1.
garak provides an EU AI Act mapping that tags and groups probe results by risk category, added in v0.17.0.
garak includes a multi-turn GOAT probe, an agent breaker probe and a system prompt extraction probe as of v0.15.0.

How it works

garak runs from the command line as garak <options>. You choose a target with --target_type and --target_name. By default garak runs all known probes with each probe's recommended detectors, or you can narrow this with --spec. Generators cover Hugging Face, OpenAI, Replicate, Cohere, Groq, ggml, NIM, AWS Bedrock, LiteLLM and any REST endpoint. API keys are set through environment variables.

How to install and start using garak

With pip

python -m pip install -U garak

Development version from GitHub

python -m pip install -U git+https://github.com/NVIDIA/garak.git@main

Requirements

Python 3.11 or later.
An API key environment variable for hosted generators, such as OPENAI_API_KEY, REPLICATE_API_TOKEN, COHERE_API_KEY or BEDROCK_API_KEY.
garak is developed on Linux and macOS.

First steps

List the available probes with garak --list_probes.
Set the API key environment variable your provider needs, such as OPENAI_API_KEY for OpenAI models.
Run a probe against a model: python3 -m garak --target_type huggingface --target_name gpt2 --spec probes.dan.Dan_11_0
Review the FAIL rows in the terminal, then check the JSONL report and garak.log for details.

Who garak is for

Developers red-teaming an LLM application for jailbreaks and prompt injection before release.
Security teams that want an nmap-style scanner for LLMs and dialog systems.
Teams testing custom models behind REST endpoints, NIM or AWS Bedrock.
Teams mapping LLM risk findings to EU AI Act categories.

Limitations to know

garak is command-line only, with no graphical interface described in the sources.
garak runs all probes by default, and each prompt generates 10 outputs by default, so full scans can be long and costly on paid APIs.
garak's REST generator needs a short YAML config describing the endpoint.
garak's atkgen probe is a prototype and is mostly stateless.

Is garak growing? Usage data

In the week of Oct 25–31, 2026, garak ranked #144 on the Devtoolwatch Board and #7 in Evals and observability, with a momentum score of 54 out of 100. Its main package had 12.9k downloads that week, +16% against its own 4-week average.

New GitHub stars per week for the last 12 weeks, from GitHub's stargazer history (today's stargazers by the week they starred).

Show the weekly numbers

New GitHub stars per week for garak
Week of	New stars
Aug 9, 2026	82
Aug 16, 2026	82
Aug 23, 2026	180
Aug 30, 2026	49
Sep 6, 2026	106
Sep 13, 2026	87
Sep 20, 2026	66
Sep 27, 2026	61
Oct 4, 2026	102
Oct 11, 2026	97
Oct 18, 2026	98
Oct 25, 2026	123

On Hacker News

Garak, LLM Vulnerability Scanner, 211 points and 62 comments (Nov 17, 2024)
Rally – A Community of Rejected YC Founders, 23 points and 6 comments (Dec 11, 2020)
Show HN: Show HN Chat, 9 points and 3 comments (Feb 17, 2016)
Show HN: An algo based approach to beating the market, 5 points and 1 comments (Apr 22, 2019)

The Board ranks tools by four signals each week: new stars, new stars for the repo's size, added downloads and download growth beyond the market. How the ranking works.

garak alternatives

Other LLM evals and observability tools on the Board in the week of Oct 25–31, 2026, by momentum:

Evals and observability ranked by momentum, the week of Oct 25–31, 2026
Rank	Tool	Stars	Downloads/wk
#1	SkillSpectorSkillSpector is NVIDIA's security scanner that checks AI agent skills for vulnerabilities, malicious patterns, and supply-chain risks…	23.7k	–
#2	JudgevalJudgeval is an open-source Python SDK for tracing and evaluating LLM-powered agents with prompt-based judges and online monitoring.	1.1k	80.2k
#3	A.I.G (AI-Infra-Guard)A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak…	7.2k	–
#4	LangfuseLangfuse is an open source LLM engineering platform for tracing, evaluating, prompt management, and debugging LLM applications and agents.	36.8k	9.8M
#5	OpikOpik is an open-source platform from Comet for tracing, evaluating, and monitoring LLM applications, RAG systems, and AI agents.	23.1k	709.1k
#6	promptfoopromptfoo is a CLI and library for evaluating and red-teaming LLM-based apps, including prompts, agents, and RAG pipelines.	26.7k	1.2M

All LLM evals and observability tools, ranked

Recent garak releases

v0.17.1, Oct 16, 2026
v0.17.0, Sep 9, 2026
v0.16.0, Aug 4, 2026
v0.15.1, Jun 5, 2026
v0.15.0, May 1, 2026

garak FAQ

Is garak free and open source?

Yes. garak's source code is on GitHub (NVIDIA/garak) under the Apache 2.0 license, so you can use it for free.

How do I install garak?

Run python -m pip install -U garak. The garak package needs Python 3.11 or newer. The install section above has the other options from the project's README.

Is garak still maintained?

Yes. The NVIDIA/garak repository was last updated on Oct 21, 2026, and the latest release, v0.17.1 on GitHub, came out on Oct 16, 2026.

How popular is garak?

garak has 9.8k GitHub stars as of Nov 1, 2026. It gained 123 new stars in the week of Oct 25–31, 2026, against a 12-week average of 94 a week. Its main package had 12.9k downloads that week, +16% against its own 4-week average. That made it #7 among LLM evals and observability tools on the Devtoolwatch Board.

What are the best alternatives to garak?

Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), Judgeval (#2), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5). The alternatives table above has their numbers.

Is garak free?

Yes. The README describes garak as a free tool, and the repository is licensed under Apache-2.0.

What does garak test for?

garak tests whether an LLM can be made to fail. It probes for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks and other weaknesses, using static, dynamic and adaptive probes.

Which models can garak scan?

garak supports Hugging Face models, Replicate, OpenAI, Cohere, Groq, AWS Bedrock, LiteLLM, NIM, gguf models through llama.cpp, and other models reachable through REST.

How do I run only certain garak probes?

Use --spec. For example, --spec probes.promptinject runs one probe family, and --spec probes.lmrc.SlurUsage runs a single plugin.

How do I read garak results?

garak prints one row per detector for each probe. Undesirable responses are marked FAIL with a failure rate. Figures like 840/840 show total generations and how many behaved OK. Full details go to a JSONL report.

Sources

garak README badge

Maintainers can show the tool's weekly Board rank in their README:

[![Devtoolwatch](https://index.devtoolwatch.com/badge/garak.svg)](https://index.devtoolwatch.com/tools/garak/)

How this page is made: Devtoolwatch writes each review from the project's README, releases, package metadata and website, with an AI model, then checks every claim against the source it cites and drops what the sources don't support. The numbers come from Devtoolwatch's daily GitHub snapshots, GitHub's stargazer history, and the npm and PyPI download APIs. Spotted an error? Email [email protected].