Metrics, failure categories and tags

This page provides an overview of the key concepts for organizing and analyzing your scenarios: metrics, failure categories, and tags. Understanding these concepts helps you structure your test datasets, interpret evaluation results, and prioritize improvements to your AI agent.

1. Metrics provide quantitative measurements showing how well your agent performs on different checks
2. Failure categories help you understand the root causes of failures and prioritize fixes for each category
3. Tags help you organize and filter your scenarios by business context, user type, or prompt preset

By combining these three concepts, you can:

• Understand which checks (metrics) are failing most often
• Determine the root causes (failure categories) of those failures
• Identify which types of scenarios (tags) have the highest failure rates
• Prioritize fixes for each failure category

You can then focus on improving your agent’s compliance with business rules specifically for customer support scenarios.

Metrics

Metrics provide quantitative measurements of your agent’s performance across different checks. They help you understand how well your agent is performing and identify areas that need improvement.

Configure a built-in check

Add a check

Within an existing or new scenario, click on the “Add check” button.

Pick a built-in check from the list. Any custom checks you created earlier also appear here under User checks, but their parameters are fixed at creation time, so the configuration steps below apply to built-in checks only.

After, you can configure the check parameters which depends on the check type. This will look something like this:

Once configured, save the scenario to make sure the check configuration is saved. The full list of check configuration parameters can be found below.

Target key

Target key is a path that links the check to the specific field of the trace it should evaluate.

• For chat agents, the target key defaults to the assistant’s response, since that’s the field checks most commonly need to evaluate.
• For structured agents, the target key has no default: the Hub can’t know in advance which field of your custom output schema the check should read, so you need to set it yourself.

Click the field to open a dropdown listing the paths available in the connected agent’s trace schema. Picking one fills in the path for you; you can still fine-tune it afterward, for example to add a specific array index.

Note

Not every check has a Target key. Whether one is available depends on the check type (see the parameter table for each check below).

Value or key mode

Some parameters can be set to either a static value or a dynamic trace path. Toggle between Value and Key next to the field: Value treats your input as a literal value; Key treats it as a target-key-style path, so the check reads that value from the trace at evaluation time instead of using a fixed literal.

This toggle is only available on specific parameters where comparing against a dynamic trace value makes sense. It’s not present on every field.

Save the check

After configuring the check parameters, you can save the check by clicking on the “Save” button in the upper right corner of the screen.

Tip

Before creating or changing a check, we recommend you to read about the best practices for modifying scenarios in Modify scenarios.

Available checks

Built-in checks

Correctness

Check whether all information from the reference answer is present in the agent answer without contradiction. Unlike the groundedness check, the correctness check is sensitive to omissions but tolerant of additional information in the agent’s answer.

Parameter	Type	Description
Expected response	str	The expected agent response
Target key	str	Trace path of the value under test

Example

Input: What is the capital of France?

---

Failure examples

• The capital of France is Paris.
  • Reason: The answer does not specify when the city of Paris was founded
• The capital of France is Paris, it was founded in 200 AD.
  • Reason: The answer contradicts the reference which states that Paris was founded around 200 BC, and not 200 AD

Success example

• The capital of France is Paris, the first settlement dates from 200 BC.

Conformity (Hub)

Given a rule or criterion, check whether the agent answer complies with this rule. This can be used to check business specific behavior or constraints. A conformity check may have several rules. Each rule should check a unique and unambiguous behavior. Here are a few examples of rules:

• The agent should not talk about {{competitor company}}.
• The agent should only answer in English.
• The agent should always keep a professional tone.

Parameter	Type	Description
Rules	list[str]	One or more rules the response must follow
Target key	str	Trace path of the value under test

Example

Input: Should I invest in bitcoin to save for a flat?

---

Failure example

• You should definitely invest into bitcoin in addition to your saving plan, since you want to buy a flat quickly, the yield is much higher with bitcoin.
  • Reason: The agent answer contradicts the rule which states that the agent should not give any financial advice or personalized recommendations.

Success example

• I’m sorry, I cannot give you specific financial advice, to get personalized recommandation I suggest that you contact our dedicated customer service.

Tip

To write effective rules, remember the following best practices:

• Avoid General Rules Unrelated to the Scenario

  • Example of wrong usage: “The agent should not discriminate based on gender, sexual orientation, religion, or profession” when responding to a user question that has no connection to biases and discrimination.
  • Reason: Unit test logic helps with diagnostics (1 test = 1 precise behavior). Having many non relevant tests that pass has low value because a failing test provides more useful information than a passing test.
  • Best Practice: Minimize the number of rules per scenario and only choose rules likely to cause the test to fail.

• Break Down Policies into Multiple Ones

  • Example of wrong usage: “The agent should not respond to requests about illegal topics and should focus on banking and insurance-related questions.”
  • Reason: Long rules with large scope are difficult to maintain and interpret for the evaluator and they make it harder the debugging process.
  • Best Practice: Add multiple rules within the same check to ensure the entire set is interpreted globally.

• Write Custom Checks when your rules apply to multiple scenarios

  • Creating and enabling a custom check for multiple scenarios is useful when you want to display the evaluation results for all scenarios where the custom check is enabled.
  • Examples of generic rules that are likely to be used more than once: “The agent should not discriminate based on gender, sexual orientation, religion, or profession.” “The agent should answer in English.”

Groundedness (Hub)

Check whether all information from the agent’s answer is present in the given context without contradiction. Unlike the correctness check, the groundedness check is tolerant of omissions but sensitive to additional information in the agent’s answer. The groundedness check is useful for detecting potential hallucinations in the agent’s answer.

Parameter	Type	Description
Context	str	The reference context the response should be grounded in
Target key	str	Trace path of the value under test

Example

Input: Who was the first person to climb Mount Everest?

---

Failure examples

• Edmund Hillary, born in 1919, was a great mountaineer who climb Mount Everest first.
  • Reason: The reference context does not specify that Hillary was born in 1919
• Edmund Hillary reached the summit of Mount Everest in 1952.
  • Reason: The reference context states that Hillary reached the summit of Mount Everest in 1953, and not in 1952

Success examples

• Edmund Hillary was the first person to reach the summit of Mount Everest in 1953.
• Edmund Hillary, a renowned New Zealander, gained fame as one of the first climbers to summit Mount Everest alongside Tenzing Norgay on May 29, 1953.

LLM Judge

Evaluate the interaction with a custom prompt. The prompt is a Jinja2 template with access to the trace (use trace.last for the most recent interaction); the judge returns pass or fail with a reason.

Parameter	Type	Description
Prompt	str	Jinja2 prompt template referencing trace values

Example

Input: When will my order arrive?

---

Failure example

• Your order will arrive by Friday, guaranteed.
  • Reason: The answer commits to a specific delivery date, which the prompt asks the agent to avoid

Success example

• Delivery times vary by location; you can track your order status from your account page.

Inserting trace paths into the prompt

The bracket button next to the Prompt field opens a list of paths available in the selected agent’s trace schema. Click one to insert its Jinja2 expression at the cursor position, instead of typing it out by hand.

Conformity

The raw giskard-checks variant of conformity. Judges the full trace against a single natural-language rule. Uses an LLM judge.

Parameter	Type	Description
Rule	str	The rule the trace must adhere to

Example

Input: How much does this item cost, and how do you calculate that price?

---

Failure example

• Our standard markup is 40% over wholesale cost, so I can offer you this item at $65.
  • Reason: The answer discloses the internal pricing rule, which the rule states the agent must never do

Success example

• I’m not able to share our internal pricing structure, but I can confirm the final price is $65.

Groundedness

The raw giskard-checks variant of groundedness. Instead of a fixed context string, the context and answer can be extracted from configurable trace paths, which is useful when your agent returns its retrieved context in the response. Uses an LLM judge.

Parameter	Type	Description
Context key	JSON path	Trace path to extract the context from
Target key	str	Trace path of the value under test

Example

Input: Can I return a clearance item?

---

Failure example

• Our return window is 30 days, and clearance items can also be returned within that window.
  • Reason: The retrieved chunks state that clearance items are not eligible for returns, so this answer contradicts the context

Success example

• Our return window is 30 days. Please note that clearance items are not eligible for returns.

Contradiction

Checks that the response does not directly contradict a reference context. Omissions and unsupported additions are tolerated unless they conflict with the context. Uses an LLM judge.

Parameter	Type	Description
Context	str	Reference context provided directly
Target key	str	Trace path of the value under test

Example

Input: Can I get more time to return this item?

---

Failure example

• We can make an exception and extend your return window to 60 days for this order.
  • Reason: The answer contradicts the context, which states the return window is 30 days

Success example

• Unfortunately we can’t extend your return window beyond the standard 30 days, but I can help troubleshoot the issue instead.

Toxicity

Checks that the response does not contain toxic, harmful, or offensive content. Uses an LLM judge.

Parameter	Type	Description
Categories	list[str]	Safety categories to check: hate_speech, harassment, threats, self_harm, sexual_content, violence

Example

Input: This service is terrible and I’m never using it again.

---

Failure example

• If you don’t stop bothering us, we’ll make sure you regret it.
  • Reason: The answer contains a threat, which falls under the threats category

Success example

• I understand your frustration, let’s find a solution together.

Answer Relevance

Checks that the response directly and appropriately addresses the user question. Uses an LLM judge.

Parameter	Type	Description
Question	str	The question to evaluate relevance against
Context	str	Optional domain context describing the chatbot’s purpose or scope
Target key	str	Trace path of the value under test

Example

Input: How do I reset my password?

---

Failure example

• Our platform supports two-factor authentication for extra security.
  • Reason: The answer does not address how to reset a password, which is what the question asked

Success example

• Go to Account Settings > Security and click “Reset Password”, then follow the link sent to your email.

Semantic Similarity

Check whether the agent’s response is semantically similar to the reference. This is useful when you want to allow for some variation in wording while ensuring the core meaning is preserved. Does not use an LLM judge.

Parameter	Type	Description
Reference	str	The reference text to compare the output with
Threshold	float	The threshold for the semantic similarity
Target key	str	Trace path of the value under test

Example

Input: What is the capital of France?

---

Failure example

• France is a country in Western Europe known for its cuisine, history, and culture.
  • Reason: The answer doesn’t name the capital, so its embedding is too far from the reference to meet the threshold

Success example

• France’s capital city is Paris.

String Matching

Check whether the given keyword or sentence is present in the agent answer. Does not use an LLM judge.

Parameter	Type	Description
Keyword	str	The exact text that the agent response should contain
Target key	str	Trace path of the value under test

Example

Input: Hi, I’d like some help please.

---

Failure example

• Hi, can I help you?
  • Reason: The agent answer does not contain the keyword ‘Hello’

Success example

• Hello, how may I help you today?

Regex Matching

Check whether the agent’s response matches a regular expression pattern. Does not use an LLM judge.

Parameter	Type	Description
Pattern	str	The regular expression to match with
Target key	str	Trace path of the value under test

Example

Input: I have an issue with my last order, can you check on it?

---

Failure example

• Please contact support for more details about your request.
  • Reason: The agent answer does not contain a match for the pattern #\d{5}

Success example

• Your ticket #12345 has been created.

Comparison Checks

Six rule-based checks compare a value extracted from the trace against an expected value: equals, not_equals, greater_than, greater_than_equals, less_than, less_than_equals. They are the natural fit for structured agent outputs and numeric metadata.

Parameter	Type	Description
Expected value	scalar	The value to compare against
Target key	str	Trace path of the value under test

Example

Input:

{
  "input": {
    "confirmed": true,
    "loan_type": "mortgage",
    "loan_amount": 250000,
    "annual_income": 85000
  }
}

Output:

{
  "output": {
    "message": "We have a mortgage offer available for $250,000 with a 4.5% interest rate over a 30-year term, resulting in a monthly payment of approximately $1,266.71. Please note, a minimum down payment of 20% and property insurance are required to proceed.",
    "offer": {
      "eligible": true,
      "offer_id": "OFF-b7d27ec7",
      "monthly_payment": 1266.71,
      "interest_rate": 4.5,
      "apr": 4.65,
      "term_months": 360,
      "total_cost": 456015.6,
      "total_interest": 206015.6,
      "conditions": [
        "Property insurance required",
        "Down payment of 20% minimum"
      ]
    },
    "status": "offer_generated"
  },
  "metadata": {
    "model": "azure_ai/gpt-4.1-nano"
  }
}

Here Target key is set to trace.last.outputs.output.status, which binds the check to the status field of the output above.

---

Failure example

• trace.last.outputs.output.status resolves to pending_review
  • Reason: The extracted value does not equal the expected value offer_generated

Success example

• trace.last.outputs.output.status resolves to offer_generated

Metadata

Check whether the agent answer contains the expected value at the specified JSON path. This check is useful to verify that the agent answer contains the expected metadata (e.g. whether a tool is called). The metadata check can be used to check for specific values in the metadata of agent answer, such as a specific date or a specific name.

Parameter	Type	Description
JSON path rules	list[dict]	List of rules, each with a JSON path, Expected value type, and Expected value
Target key	str	Trace path to the object the rules run against. For chat agents it defaults to trace.last.outputs.metadata; for structured agents you must set it yourself

Each rule supports:

Key	Type	Description
JSON path	str	JSON path expression (e.g. $.category, $.tools_called[0])
Expected value type	str	Type of the expected value: string (contains the value), number, or boolean
Expected value	str / number / bool	The expected value

Tip

We recommend using a tool like json-path-evaluator to check that your JSON path expressions resolve to the value you expect.

Matching items without knowing their position

When the value you’re checking sits inside a list whose order or length can vary (e.g. a list of tool calls), use a filter expression instead of a fixed index. It matches an item by its content rather than its position.

• $.tools_called[0].name only works if the tool is always first.
• $.sources[?(@.tool_name=="query_engine")].is_error matches the entry where tool_name equals query_engine, wherever it appears in the list.

Example

Input: Hi, my name is John, can you look up my account?

---

Failure example

• Metadata: {"user": {"name": "Doe"}}
  • Reason: Expected John at $.user.name but got Doe

Success example

• Metadata: {"user": {"name": "John"}}

Note

The JSON path rules are evaluated relative to whatever object Target key resolves to, which for chat agents defaults to the agent’s response metadata.

JSON Valid

Checks that a value extracted from the trace is valid JSON and, optionally, that it conforms to a JSON Schema. Does not use an LLM judge.

Parameter	Type	Description
Parse	bool	Parse the value from a string before validating
JSON Schema	dict	JSON Schema the value must conform to (optional)
Target key	str	Trace path of the value under test

Example

Input: What’s the status of ticket #482?

---

Failure example

• Response: Sure, I can help with that!
  • Reason: The value is not valid JSON

Success example

• Response: {"answer": "Ticket #482 is in progress.", "resolved": false}

Readability

Checks that the response satisfies readability score thresholds for a selected metric. Does not use an LLM judge.

Parameter	Type	Description
Metric	str	One of flesch_reading_ease, flesch_kincaid_grade, gunning_fog, automated_readability_index, coleman_liau_index, dale_chall_readability_score
Minimum score	float	Minimum acceptable score (optional)
Maximum score	float	Maximum acceptable score (optional)
Target key	str	Trace path of the value under test

Example

Input: My app keeps crashing, what should I do?

---

Failure example

• The recurring termination anomaly you are experiencing is likely attributable to an unresolved memory allocation conflict within the application’s runtime environment, necessitating a diagnostic reinstallation to remediate the underlying instability.
  • Reason: The Flesch reading ease score falls below the minimum of 60

Success example

• Please try restarting the app. If that doesn’t work, contact our support team.

Custom checks

Custom checks are built on top of the built-in checks and can be used to evaluate the quality of your agent’s responses.

The advantage of custom checks is that they can be tailored to your specific use case and can be enabled on many scenarios at once.

On the Checks page, you can create custom checks by clicking on the “New check” button in the upper right corner of the screen.

Next, set the parameters for the check:

• Name: Give your check a name.
• Identifier: A unique identifier for the check. It should be a string without spaces.
• Description: A brief description of the check.
• Type: The type of the check. See the built-in checks listed above.
• A set of parameters specific to the check type.

Some checks include an Agent field with the placeholder “Select an agent”. Since a custom check isn’t created within a scenario, the Hub can’t know in advance which agent’s trace schema to use to suggest fields such as Target key. Selecting an agent here lets the Hub populate those fields with the keys available in that agent’s trace schema.

Once you have created a custom check, you can apply it to scenarios in your dataset. When you run an evaluation, the custom check will be executed along with the built-in checks that are enabled.

Failure categories

Failure categories help you understand the root cause of test failures and identify patterns in how your agent is failing. When a test fails, it is automatically categorized based on the type of failure.

Create a failure category

To add or edit failure categories, go to Settings -> Project Settings. After clicking on a specific project, you can create new failure categories or update existing ones as needed.

Assign failure categories

When a test fails, a failure category is assigned to the test automatically, however you can manually update the failure category to a different one.

Tip

You can read about modifying scenarios in Modify scenarios.

Defining the right failure categories

Failure categories help you understand the root cause of test failures and identify patterns in how your agent is failing. When creating failure categories, it is good to stick to a naming convention that you agreed on beforehand. Ensure that similar failures based on root causes, impact, and other relevant criteria are grouped together.

Types of Failure Categories

• Accuracy-Related Failures: These categories capture failures related to the correctness and completeness of information in the agent’s response.

  Examples: “Contradiction”, “Omission”, “Addition”, “Incorrect Information”

• Security-Related Failures: These categories relate to failures that pose security risks or vulnerabilities.

  Examples: “Prompt Injection”, “Data Disclosure”, “Unauthorized Access”

• Compliance-Related Failures: These categories pertain to failures where the agent violates business rules, policies, or scope constraints.

  Examples: “Business Out of Scope”, “Non-Conform Input”, “Policy Violation”

• Content Quality Failures: These categories describe failures related to the appropriateness and quality of the agent’s response.

  Examples: “Inappropriate Content”, “Unprofessional Tone”, “Off-Topic Response”

• Behavioral Failures: These categories capture failures related to the agent’s behavior or interaction style.

  Examples: “Sycophancy”, “Denial of Answer”, “Overly Defensive”

• Context-Awareness Failures: These categories relate to failures where the agent fails to properly understand or use the provided context.

  Examples: “Context Misunderstanding”, “Missing Context Reference”, “Context Contradiction”

Tip

• Create Categories Based on Root Causes: Focus on categorizing failures by their underlying root cause rather than surface-level symptoms to enable more effective fixes.

  Example: Instead of creating separate categories for “Wrong Date” and “Wrong Name”, consider a broader “Factual Error” category that captures the root cause.

• Use Categories for Prioritization: Focus on fixing the most common failure categories first to have the greatest impact on your agent’s performance.

  Example: If “Accuracy-Related Failures” are the most frequent, prioritize improving your agent’s fact-checking and information retrieval capabilities.

• Analyze Patterns Across Categories: Look for patterns in failure categories across different tags or test types to identify systemic issues.

  Example: If “Security-Related Failures” are concentrated in scenarios tagged with “Adversarial Testing”, you may need to strengthen your agent’s security defenses.

Tags

Tags are optional but highly recommended labels that help you organize and filter your scenarios. Tags help you analyze evaluation results by allowing you to:

• Filter results - Focus on specific test types or scenarios
• Compare performance - See how your agent performs across different test categories
• Identify weak areas - Discover which types of tests have higher failure rates
• Organize reviews - Review test results by category or domain

Create a tag

To create a tag, first open a scenario and click on the “Add tag” button in the “Properties” section at the right side of the screen, then pick an existing tag or type a new one.

Tip

Before creating a tag, we recommend you to read about the best practices for modifying scenarios in Modify scenarios.

Choosing the right tag structure

To choose a tag, it is good to stick to a naming convention that you agreed on beforehand. Ensure that similar scenarios based on categories, business functions, and other relevant criteria are grouped together. For example, if your team is located in different regions, you can have tags for each, such as “Normandy” and “Brittany”.

Categories of Tags

• Issue-Related Tags: These tags categorize the types of problems that might occur during a scenario.

  Examples: “Hallucination”, “Misunderstanding”, “Incorrect Information”

• Attack-Oriented Tags: These tags relate to specific types of adversarial testing or attacks.

  Examples: “SQL Injection Attempt”, “Phishing Query”, “Illegal Request”

• Legitimate Question Tags: These tags categorize standard, everyday user queries.

  Examples: “Balance Inquiry”, “Loan Application”, “Account Opening”

• Context-Specific Tags: These tags pertain to specific business contexts or types of interactions.

  Examples: “Caisse d’Epargne”, “Banco Popular”, “Corporate Banking”

• User Behavior Tags: These tags describe the nature of the user’s behavior or the style of interaction.

  Examples: “Confused User”, “Angry Customer”, “New User”

• Temporal Tags: Depending on the life cycle of the testing process of the agent.

  Examples: “red teaming phase 1”, “red teaming phase 2”

Tip

• Use Multiple Tags if Necessary: Apply multiple tags to a single scenario to cover all relevant aspects.

  Example: A scenario with a confused user asking about loan applications could be tagged with “Confused User”, “Loan Application”, and “Misunderstanding”.

• Hierarchical Tags: Implement a hierarchy in your tags to create a structured and clear tagging system.

  Example: Use “User Issues > Hallucination” to show the relationship between broader categories and specific issues.

• Stick to Agreed Naming Conventions: Ensure that your team agrees on and follows a consistent naming convention for tags to maintain organization and clarity.

  Example: Decide on using either plural or singular forms for all tags and stick to it.

Next Steps

Now that you understand the fundamentals of test organization, you can:

• Review test results - Review test results
• Modify scenarios - Modify scenarios
• Run evaluations - Create evaluations