Custom trace types

Use a custom trace type when you want shared helpers or computed views over the full interaction history (for example, a turn count), or a custom Rich rendering for notebooks and terminals. This guide uses a small LLMTrace subclass for chat-style [user] / [assistant] formatting.

Giskard’s Trace is not OpenTelemetry: it is the immutable conversation history passed to checks and interaction callables.

Define LLMTrace

Subclass Trace and pass trace_type=LLMTrace on Scenario. The scenario runner starts from an empty trace instance of that type and appends interactions as usual.

• Use a private helper (here _conversation_markdown) to build a transcript string.
• Override __rich_console__ so Rich-based output (see below) can render Markdown. Giskard does not use a _repr_prompt_ hook; Rich uses __rich_console__ / __rich__ on the trace when building reports.
• Add @computed_field properties for values you want to reuse in custom checks.

from rich.console import Console, ConsoleOptions, RenderResult
from rich.markdown import Markdown

from giskard.checks import Trace
from pydantic import computed_field


class LLMTrace(Trace[str, str]):
    """Chat-oriented trace with a Markdown transcript for Rich."""

    @computed_field
    @property
    def turn_count(self) -> int:
        return len(self.interactions)

    def _conversation_markdown(self) -> str:
        if not self.interactions:
            return "**No interactions yet**"
        return "\n\n".join(
            f"[user]: {interaction.inputs}\n\n[assistant]: {interaction.outputs}"
            for interaction in self.interactions
        )

    def __rich_console__(
        self, console: Console, options: ConsoleOptions
    ) -> RenderResult:
        yield Markdown(self._conversation_markdown())

Scenario trace_type and annotations

Pass trace_type=LLMTrace so execution uses your class. Optional annotations={...} on the scenario is copied onto the initial trace (shared metadata such as tenant or experiment id) and appears on trace.annotations for checks and callables.

import asyncio

from giskard.checks import Scenario, FnCheck


async def main():
    return await (
        Scenario(
            "llm_trace_demo",
            trace_type=LLMTrace,
            annotations={"tenant": "acme", "env": "ci"},
        )
        .interact(
            inputs="Hello",
            outputs="Hi! How can I help?",
        )
        .interact(
            inputs="What is 2+2?",
            outputs="2 + 2 equals 4.",
        )
        .check(
            FnCheck(
                fn=lambda trace: trace.annotations.get("tenant") == "acme",
                name="tenant_annotation",
                success_message="Tenant present",
                failure_message="Missing tenant",
            )
        )
        .check(
            FnCheck(
                fn=lambda trace: isinstance(trace, LLMTrace) and trace.turn_count >= 2,
                name="min_two_turns",
                success_message="At least two turns",
                failure_message="Expected two turns",
            )
        )
        .run()
    )


result = asyncio.run(main())
result.print_report()

Output

──────────────────────────────────────────────────── ✅ PASSED ────────────────────────────────────────────────────
tenant_annotation       PASS    
min_two_turns   PASS    
────────────────────────────────────────────────────── Trace ──────────────────────────────────────────────────────
[assistant]: Hi! How can I help?                                                                                   

user: What is 2+2?                                                                                                 

[assistant]: 2 + 2 equals 4.                                                                                       
────────────────────────────────────────────────── 1 step in 0ms ──────────────────────────────────────────────────

ScenarioResult.final_trace and Rich

ScenarioResult validates final_trace as the base Trace type. After a run, type(result.final_trace) is therefore the generic Trace, even when you passed trace_type=LLMTrace. During execution, checks still receive your subclass — the FnCheck above uses isinstance(trace, LLMTrace) and trace.turn_count.

To print a Rich transcript (Markdown [user] / [assistant] blocks), rebuild an LLMTrace from the final interactions and annotations, then pass it to Console.print or use it anywhere Rich renders objects.

print_report() uses Rich on result.final_trace; that object uses the default per-interaction trace layout unless you reconstruct your subclass as below.

def as_llm_trace(trace: Trace[str, str]) -> LLMTrace:
    """Rebuild LLMTrace for Rich / repr after `Scenario.run()`."""
    return LLMTrace(
        interactions=list(trace.interactions),
        annotations=dict(trace.annotations),
    )


print("type(result.final_trace):", type(result.final_trace))
display_trace = as_llm_trace(result.final_trace)
print("display_trace.turn_count:", display_trace.turn_count)

console = Console(width=88)
console.print(display_trace)
print()
result.print_report()

Output

type(result.final_trace): <class '__main__.LLMTrace'>
display_trace.turn_count: 2

[assistant]: Hi! How can I help?                                                        

user: What is 2+2?                                                                      

[assistant]: 2 + 2 equals 4.

──────────────────────────────────────────────────── ✅ PASSED ────────────────────────────────────────────────────
tenant_annotation       PASS    
min_two_turns   PASS    
────────────────────────────────────────────────────── Trace ──────────────────────────────────────────────────────
[assistant]: Hi! How can I help?                                                                                   

user: What is 2+2?                                                                                                 

[assistant]: 2 + 2 equals 4.                                                                                       
────────────────────────────────────────────────── 1 step in 0ms ──────────────────────────────────────────────────

Custom check typed with LLMTrace

Subclass Check and annotate run(self, trace: LLMTrace) so type checkers and readers know which trace type you expect.

from giskard.checks import Check, CheckResult


@Check.register("min_turns")
class MinTurnsCheck(Check):
    """Fail if the conversation has fewer than `minimum` turns."""

    minimum: int = 1

    async def run(self, trace: LLMTrace) -> CheckResult:
        if trace.turn_count >= self.minimum:
            return CheckResult.success(
                message=f"{trace.turn_count} turn(s), minimum {self.minimum}",
            )
        return CheckResult.failure(
            message=f"Only {trace.turn_count} turn(s), need at least {self.minimum}",
        )


r2 = asyncio.run(
    Scenario("typed_check", trace_type=LLMTrace)
    .interact(inputs="Hi", outputs="Hello.")
    .check(MinTurnsCheck(minimum=1))
    .run()
)
assert r2.passed

JSONPath and resolve

Built-in checks still use paths like trace.last.outputs. Custom fields on a subclass are available in Python (for example trace.turn_count); expose data through trace.annotations or interaction metadata if you need it in JSONPath strings. See JSONPath in checks.