Datasets for Evaluating AI Agents

A dataset is a collection of scenarios used to evaluate your agents. We allow manual scenario creation for fine-grained control, but since generative AI agents can encounter an infinite number of scenarios, automated scenario generation is often necessary, especially when you don’t have any scenarios to import.

In this section, we will walk you through how to create scenarios and datasets using the Hub interface. In general, we cover four different ways to create datasets:

Create manual scenarios Design your own scenarios using full control over the scenario creation process and explore them in the playground.

Import scenarios Import existing scenario datasets from a JSON, JSONL or CSV file, obtained from another tool, like Giskard Open Source.

Generate prompt preset scenarios Create targeted, business-specific scenarios using prompt presets. Test your agents with specific personas and business rules without editing your agent's core functionality.

Generate knowledge base scenarios Detect business failures, by generating synthetic scenarios to detect business failures, like hallucinations or denial to answer questions, using document-based queries and knowledge bases.

Dataset creation workflow

graph LR
    A[Create Dataset] --> B{Source}
    B --> C([<a href="/hub/ui/datasets/manual" target="_self">Create Manually</a>])
    B --> D([<a href="/hub/ui/datasets/import" target="_self">Import Existing</a>])
    B --> E([<a href="/hub/ui/datasets/knowledge-base" target="_self">Knowledge Base Scenarios</a>])
    B --> F([<a href="/hub/ui/datasets/prompt-preset" target="_self">Prompt Preset Scenarios</a>])
    B --> G([<a href="/hub/ui/scan" target="_self">From Scan</a>])
    C --> H[<a href="/hub/ui/annotate" target="_self">Review Scenarios</a>]
    D --> H
    E --> H
    F --> H
    G --> H