What are Giskard Checks?

Giskard Checks is a lightweight Python library for testing and evaluating non-deterministic applications such as LLM-based systems.

An LLM answers differently every time, so an exact-string assertion fails on the next run. You write two things instead. A scenario is one test case: a message, or a short conversation, to send to your agent. A check is a rule the reply has to satisfy, either plain Python or a rule written in English and graded by an LLM, called a judge.

Take the example used through these docs, a support agent for a retail bank: a Python check confirms the reply quotes the reference of the disputed transaction, and a judge decides whether the reply refused to give investment advice. Tests run under pytest and report pass or fail. See the glossary for the failure types these checks look for, such as hallucination and prompt injection.

A judge is an LLM, so it is sometimes wrong in both directions. Read the failing verdicts before you act on them, and treat a passing suite as “these scenarios did not break the agent”, not as proof that it is safe.

Key features

• Checks that need no LLM call: string and regex matching, comparisons, JSON validity, semantic similarity, and Rego policies, so deterministic rules stay fast and free
• LLM judges: LLMJudge for any rule you write in English, plus ready-made Groundedness, AnswerRelevance, Contradiction, Toxicity, and Conformity
• Single-turn and multi-turn scenarios: send one message, or drive a whole conversation with UserSimulator to see whether the agent keeps track of what was said earlier
• Composition: combine checks with AllOf, AnyOf, and Not instead of writing a new check per combination
• Your own Python as a check: wrap any function with FnCheck, and use WithSpy to assert which internal calls the agent made
• Async-first, pytest-native: await a whole suite in one call, or run the same scenarios as pytest tests in CI
• Results you can store and compare: every result is a frozen Pydantic model that serializes to JSON, or to JUnit XML for your CI report

Looking for tests you don’t have to write? Giskard Scan generates an adversarial suite from a plain-language description of your agent, and returns it as an ordinary Suite you run with everything below.

Use a coding agent?

Install the Giskard Agent Skills and let your coding agent (Claude Code, Cursor, and more) design red-team suites or RAG evaluation suites directly from a plain-English description of your agent.

Start here

To see the whole thing working first, the Quickstart runs one end-to-end example. Otherwise, work through these in order.

1. Install & Configure (~5 min): get the library installed and an LLM provider configured. Giskard v3 requires Python 3.12 or newer. Judged checks call your own provider with your own key, so they cost tokens. After this you can import giskard.checks and run judged checks.
2. Your First Test (~10 min): write a scenario that passes. No API key or LLM needed. After this you can write a test case and read its result.
3. Your First LLM Call (~15 min): call a real model and evaluate the response with LLMJudge. After this you can grade an answer against a rule written in English.
4. Test Suites (~20 min): group scenarios into a suite you run with a single await. After this you can rerun the same set of tests after every change to your prompt or model.
5. CI/CD Integration (~20 min): run that suite automatically on every pull request, so a change that breaks the agent’s behavior fails the build.

Browse the docs

Installation Install the library and configure your LLM provider

Quickstart Your first scenario in under 5 minutes

Agent Skills Drop-in skills that turn Claude Code and other coding agents into Giskard specialists

Tutorials Step-by-step learning arc from your first test to reusable test suites

How-to Guides Task-oriented guides for pytest, custom checks, CI/CD, spy, and more

API Reference Complete API documentation for all checks, scenarios, and utilities

Explanation Core concepts, async design, JSONPath, and when to use which check

Use Cases Worked examples for RAG systems, agents, chatbots, and content moderation

Use cases

Giskard Checks is designed for:

• RAG evaluation: check that an answer is supported by the retrieved documents and actually answers the question asked
• Agent testing: run a multi-step workflow and assert on the trace, including which tools the agent called
• Chatbot testing: hold a multi-turn conversation with the agent and check it stays on topic and in character
• Content moderation: test refusals and unsafe outputs before you ship, with Toxicity and policy checks
• Regression testing: rerun a saved suite after a prompt or model change and see exactly which scenarios changed