# garak review

> garak is a command-line LLM vulnerability scanner from NVIDIA that probes models for jailbreaks, prompt injection, data leakage, hallucination and toxicity.

Devtoolwatch Board, the week of Oct 25–31, 2026. Canonical page: https://index.devtoolwatch.com/tools/garak/

## TL;DR

garak is a Python command-line tool that checks whether an LLM or dialog system can be made to fail in unwanted ways. It is for developers and security teams red-teaming models. The README compares it to nmap or Metasploit, but for LLMs. It combines static, dynamic and adaptive probes and connects to many model providers.

## garak at a glance

- **Category:** [Evals and observability](https://index.devtoolwatch.com/categories/evals-and-observability/)
- **Maintained by:** NVIDIA Corporation (organization, [@NVIDIA](https://github.com/NVIDIA) on GitHub)
- **License:** Apache 2.0
- **Language:** Python, HTML, TypeScript
- **Repository:** [github.com/NVIDIA/garak](https://github.com/NVIDIA/garak)
- **Website:** [discord.gg/uVch4puUCs](https://discord.gg/uVch4puUCs)
- **PyPI package:** [garak](https://pypi.org/project/garak/) 0.17.1
- **Latest release:** [v0.17.1](https://github.com/NVIDIA/garak/releases/tag/v0.17.1) on GitHub, Oct 16, 2026
- **First published:** May 10, 2023

## What garak does

- garak probes LLMs for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks and other weaknesses.
- garak includes probe families such as dan, encoding, gcg, leakreplay, malwaregen, packagehallucination, xss and promptinject.
- garak scores each probe with detectors and marks responses showing undesirable behavior as FAIL, with a failure rate.
- garak writes a JSONL report per run plus a garak.log file, and ships a basic analysis script.
- garak reports a hit@k attack success rate metric per probe/detector pair, added in v0.17.1.
- garak provides an EU AI Act mapping that tags and groups probe results by risk category, added in v0.17.0.
- garak includes a multi-turn GOAT probe, an agent breaker probe and a system prompt extraction probe as of v0.15.0.

### How it works

garak runs from the command line as garak <options>. You choose a target with --target_type and --target_name. By default garak runs all known probes with each probe's recommended detectors, or you can narrow this with --spec. Generators cover Hugging Face, OpenAI, Replicate, Cohere, Groq, ggml, NIM, AWS Bedrock, LiteLLM and any REST endpoint. API keys are set through environment variables.

## How to install garak

**With pip**

```
python -m pip install -U garak
```

**Development version from GitHub**

```
python -m pip install -U git+https://github.com/NVIDIA/garak.git@main
```

### Requirements

- Python 3.11 or later.
- An API key environment variable for hosted generators, such as OPENAI_API_KEY, REPLICATE_API_TOKEN, COHERE_API_KEY or BEDROCK_API_KEY.
- garak is developed on Linux and macOS.

### First steps

1. List the available probes with garak --list_probes.
2. Set the API key environment variable your provider needs, such as OPENAI_API_KEY for OpenAI models.
3. Run a probe against a model: python3 -m garak --target_type huggingface --target_name gpt2 --spec probes.dan.Dan_11_0
4. Review the FAIL rows in the terminal, then check the JSONL report and garak.log for details.

## Who garak is for

- Developers red-teaming an LLM application for jailbreaks and prompt injection before release.
- Security teams that want an nmap-style scanner for LLMs and dialog systems.
- Teams testing custom models behind REST endpoints, NIM or AWS Bedrock.
- Teams mapping LLM risk findings to EU AI Act categories.

### Limitations

- garak is command-line only, with no graphical interface described in the sources.
- garak runs all probes by default, and each prompt generates 10 outputs by default, so full scans can be long and costly on paid APIs.
- garak's REST generator needs a short YAML config describing the endpoint.
- garak's atkgen probe is a prototype and is mostly stateless.

## Usage data (the week of Oct 25–31, 2026)

- Board rank: #144 overall, #7 in Evals and observability, momentum score 54
- Downloads that week: 12.9k (+16% vs 4-week average)
- New GitHub stars per week (last 12 weeks): Aug 9: 82; Aug 16: 82; Aug 23: 180; Aug 30: 49; Sep 6: 106; Sep 13: 87; Sep 20: 66; Sep 27: 61; Oct 4: 102; Oct 11: 97; Oct 18: 98; Oct 25: 123

## garak alternatives

| Rank | Tool | Stars | Downloads/wk |
|---|---|---|---|
| #1 | [SkillSpector](https://index.devtoolwatch.com/tools/nvidia-skillspector/) | 23.7k | – |
| #2 | [Judgeval](https://index.devtoolwatch.com/tools/judgmentlabs-judgeval/) | 1.1k | 80.2k |
| #3 | [A.I.G (AI-Infra-Guard)](https://index.devtoolwatch.com/tools/tencent-ai-infra-guard/) | 7.2k | – |
| #4 | [Langfuse](https://index.devtoolwatch.com/tools/langfuse/) | 36.8k | 9.8M |
| #5 | [Opik](https://index.devtoolwatch.com/tools/opik/) | 23.1k | 709.1k |
| #6 | [promptfoo](https://index.devtoolwatch.com/tools/promptfoo/) | 26.7k | 1.2M |

## garak FAQ

### Is garak free and open source?

Yes. garak's source code is on GitHub (NVIDIA/garak) under the Apache 2.0 license, so you can use it for free.

### How do I install garak?

Run `python -m pip install -U garak`. The garak package needs Python 3.11 or newer. The install section above has the other options from the project's README.

### Is garak still maintained?

Yes. The NVIDIA/garak repository was last updated on Oct 21, 2026, and the latest release, v0.17.1 on GitHub, came out on Oct 16, 2026.

### How popular is garak?

garak has 9.8k GitHub stars as of Nov 1, 2026. It gained 123 new stars in the week of Oct 25–31, 2026, against a 12-week average of 94 a week. Its main package had 12.9k downloads that week, +16% against its own 4-week average. That made it #7 among LLM evals and observability tools on the Devtoolwatch Board.

### What are the best alternatives to garak?

Other LLM evals and observability tools on the Devtoolwatch Board in the week of Oct 25–31, 2026, by momentum: SkillSpector (#1), Judgeval (#2), A.I.G (AI-Infra-Guard) (#3), Langfuse (#4), Opik (#5). The alternatives table above has their numbers.

### Is garak free?

Yes. The README describes garak as a free tool, and the repository is licensed under Apache-2.0.

### What does garak test for?

garak tests whether an LLM can be made to fail. It probes for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks and other weaknesses, using static, dynamic and adaptive probes.

### Which models can garak scan?

garak supports Hugging Face models, Replicate, OpenAI, Cohere, Groq, AWS Bedrock, LiteLLM, NIM, gguf models through llama.cpp, and other models reachable through REST.

### How do I run only certain garak probes?

Use --spec. For example, --spec probes.promptinject runs one probe family, and --spec probes.lmrc.SlurUsage runs a single plugin.

### How do I read garak results?

garak prints one row per detector for each probe. Undesirable responses are marked FAIL with a failure rate. Figures like 840/840 show total generations and how many behaved OK. Full details go to a JSONL report.

## Sources

1. [NVIDIA/garak on GitHub](https://github.com/NVIDIA/garak)
2. [garak README](https://github.com/NVIDIA/garak/blob/main/README.md)
3. [garak on PyPI](https://pypi.org/project/garak/)
4. [garak website](https://discord.gg/uVch4puUCs)
5. [Release v0.17.1 (Oct 16, 2026)](https://github.com/NVIDIA/garak/releases/tag/v0.17.1)
6. [Release v0.17.0 (Sep 9, 2026)](https://github.com/NVIDIA/garak/releases/tag/v0.17.0)
7. [Release v0.16.0 (Aug 4, 2026)](https://github.com/NVIDIA/garak/releases/tag/v0.16.0)
8. [Hacker News: Garak, LLM Vulnerability Scanner](https://news.ycombinator.com/item?id=42163591)
9. [Hacker News: Rally – A Community of Rejected YC Founders](https://news.ycombinator.com/item?id=25391208)
10. [Hacker News: Show HN: Show HN Chat](https://news.ycombinator.com/item?id=11114986)
11. [Devtoolwatch Board, the week of Oct 25–31, 2026](https://index.devtoolwatch.com/)