Practical overview
When it fits
Use Promptfoo when an LLM application needs repeatable evaluation, provider comparison, regression testing or documented red-team scanning during development.
Who it is for
- Developers building LLM applications
- Teams testing prompts, agents or RAG systems
- Teams comparing LLM providers
- Development teams adding AI checks to CI/CD
Documented uses
- Evaluate prompts and model outputs with automated tests
- Compare models side by side
- Regression-test RAG applications
- Scan AI applications for vulnerabilities
- Automate evaluation and security checks in CI/CD
Requirements & limits
Promptfoo is documented as open source under the MIT license. It is available as a CLI and library, with npm, Homebrew, pip and npx installation paths documented. Most LLM providers require an API key.
- It supports command-line workflows and CI/CD integration.
- It can compare outputs across providers including OpenAI, Anthropic, Google Gemini and DeepSeek.
- Its documented scope includes both evaluation and security testing.
- The project is implemented in TypeScript and is available as a library as well as a CLI.
- Evaluations are described as running locally, while provider access may still require API keys.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Run a getting-started evaluation
Initialize Promptfoo and run the documented example evaluation.
Before you start
- Promptfoo installed through npm, Homebrew, pip or npx.
- An API key for the LLM provider used by the example.
- A shell environment and the getting-started example.
Steps
Install Promptfoo and initialize the getting-started example.
Set the relevant provider API key as an environment variable.
Change into the getting-started example directory.
Run the evaluation command.
Open the results viewer.
Expected outcome
The example evaluation runs and its results can be viewed through Promptfoo.
Guide limitations
- Most LLM providers require an API key.
- This recipe covers the documented getting-started evaluation, not a complete application-specific test plan.
Compare options for Retrieval-augmented applications
These projects document a common task and share a project role. This is a source-based comparison, not an execution benchmark or a guarantee of interchangeability.
| Project | Documented purpose | Platform | Access and requirements |
|---|---|---|---|
| Promptfoo | Evaluate prompts and model outputs with automated tests | CLI, Cloud, Self-hosted | Promptfoo is documented as open source under the MIT license. It is available as a CLI and library, with npm, Homebrew, pip and npx installation paths documented. Most LLM providers require an API key. |
| Sentrev | Compare retrieval performance across embedding models on a document collection. | CLI | The project is an open-source Python package under the MIT license. Its README documents installation with pip or from source. |
| Diragnosis | Generate synthetic evaluation questions from supplied documents. | Docker, Linux, macOS, Self-hosted | The repository identifies the software as open source under the MIT license. The README documents a local Docker application and a pip-installable Python package. |
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 30 September 2026
- Original website title
- promptfoo/promptfoo · Official homepage title · retained in its source language