Annotate and Review Scenarios

The annotation workflow in Giskard Hub enables you to continuously improve your scenarios and evaluation metrics through an iterative, collaborative process.

Each scenario is composed of one or more interactions (a chat conversation for chat agents, or an input/output pair for structured agents) and its associated checks (e.g., an expected answer, rules that the agent must respect, etc.).

The annotation workflow follows a task-oriented approach built around two personas: the business expert, who reviews evaluation results, and the product owner, who refines scenarios and checks. Work is coordinated by distributing tasks between them.

1. Distribute tasks - Organize your review work by creating and assigning tasks to team members
2. Review test results - The business expert workflow for reviewing evaluation results and understanding failures
3. Modify scenarios - The product owner workflow for refining scenarios and validation rules

This section guides you through the complete task-oriented workflow from task distribution to scenario refinement.

Getting started

Understand metrics, failure categories and tags Understand metrics, failure categories and tags to review test results.

Distribute tasks to organize your review work Create and manage tasks to coordinate team reviews. Assign work for scan results, evaluation runs, and scenarios to ensure quality and collaboration.

Review test results Review evaluation results and understand test failures. Follow the business workflow to analyze check results, understand reasons, and take appropriate actions.

Modify scenarios Refine scenarios and validation rules. Follow the product owner workflow to draft/undraft scenarios, enable/disable checks, and structure your dataset.

Workflow overview

The annotation workflow involves two personas with distinct workflows:

Business Persona (Review Workflow):

• Reviews test results from evaluation runs or tasks
• Understands check results and failure reasons
• Reviews conversation flow and metadata
• Takes action: closes tasks if results are acceptable, or assigns modification work

Product Owner Persona (Modification Workflow):

• Modifies scenarios based on review feedback
• Drafts/undrafts scenarios
• Enables/disables checks
• Modifies check requirements
• Validates checks and structures scenarios

Next steps

Now that you understand the task-oriented annotation workflow, explore the specific workflows:

• Start with task distribution - Learn how to create and manage tasks to organize your review work task management
• Review test results - Follow the business workflow to review evaluation results review test results
• Modify scenarios - Follow the product owner workflow to refine scenarios and checks modify scenarios

Note

Getting started with annotation workflows

If you’re new to Giskard Hub, we recommend starting with:

1. Run an evaluation or run a scan and review its results to identify scenarios that need attention
2. Create tasks to organize the review work
3. Review test results following the business workflow
4. Modify scenarios as needed following the product owner workflow

For more information, see Create evaluations and Scan.