Generators

Input generators for creating dynamic test data and simulating user interactions.

`UserSimulator`

Generate multi-turn user messages with an LLM.

Module: giskard.checks.generators.user

persona str Required

Predefined persona name (e.g., "frustrated_customer") or a custom persona description.

max_steps int Default: 3

Maximum number of conversation turns to generate. Must be ≥ 0; 0 generates nothing.

max_retries int Default: 2

Retries per turn when generation fails. Must be ≥ 0, so each turn is attempted up to max_retries + 1 times before the generator raises.

context str | None Default: None

Optional context to customize the persona’s behavior.

generator BaseGenerator Default: get_default_generator()

LLM generator used to simulate the user’s messages. Defaults to the framework’s default generator.

Write the persona as a goal the simulated user is pursuing:

from giskard.checks.generators.user import UserSimulator
from giskard.agents.generators import Generator

user_sim = UserSimulator(
    persona="""
    You were charged twice for the same card payment.
    Describe the two charges without using the word "dispute".
    Ask how long you will wait for the money.
    """,
    max_steps=3,
    generator=Generator(model="openai/gpt-5"),
)

Using in scenarios

from giskard.checks import Scenario, FnCheck


def bank_support_agent(inputs: str) -> str:
    """Stand-in for the agent under test. Replace with your own call."""
    if "charged twice" in inputs.lower() or "duplicate" in inputs.lower():
        return "I can help with a duplicate payment. What were the date and amount?"
    return "Could you share the payment date and amount?"


test_scenario = (
    Scenario("duplicate_charge_conversation")
    .interact(inputs=user_sim, outputs=bank_support_agent)
    .check(
        FnCheck(
            fn=lambda trace: trace.last is not None
            and "payment" in trace.last.outputs.lower(),
            name="asks_about_the_payment",
        )
    )
)

result = await test_scenario.run()

The LLM can generate different messages on each run. Use fixed inputs for repeatable tests.

Goal-oriented testing

Generation stops when the LLM reports that the goal is reached, or after max_steps turns:

result = await test_scenario.run()

print(f"Turns generated: {len(result.final_trace.interactions)}")
print(f"Last reply: {result.final_trace.last.outputs}")

`LLMGenerator`

Generate inputs from an inline prompt or prompt template.

Module: giskard.checks.generators.base

prompt str | None

Inline prompt string. Jinja2 rendering applies only when as_template=True.

prompt_path str | None

Template reference, written as namespace::path/to/template.j2. giskard-scan registers the giskard.scan namespace and ships the attack templates there (for example "giskard.scan::scenarios/llm01_injection.j2"); giskard-checks registers giskard.checks, which holds the bundled judge and generator prompts. Register your own directory with giskard.agents.add_prompts_path(path, "my_namespace").

max_steps int Default: 3

Maximum conversation turns to generate. Must be ≥ 0; 0 generates nothing.

max_retries int Default: 2

Retries per turn when generation fails. Must be ≥ 0, so each turn is attempted up to max_retries + 1 times before the generator raises.

as_template bool Default: False

When True, render prompt as a Jinja2 template with trace context.

generator BaseGenerator Default: get_default_generator()

LLM generator used to produce inputs.

Exactly one of prompt or prompt_path must be provided.

import giskard.scan  # noqa: F401  — registers the "giskard.scan" prompt namespace
from giskard.checks import LLMGenerator

gen = LLMGenerator(prompt="Ask a concise question about an archive.")
gen = LLMGenerator(prompt_path="giskard.scan::scenarios/llm01_injection.j2")

Use UserSimulator for a multi-turn persona. Use LLMGenerator when you need a custom prompt.

`LLMGeneratorOutput`

Module: giskard.checks.generators.base

Internal output for each UserSimulator or LLMGenerator turn. The loop yields message and stops when goal_reached is true or message is empty.

LLMGeneratorOutput

goal_reached bool

Whether the goal has been reached and no further messages are needed.

schema_issue str | None Default: None

Set instead of message when the input schema cannot produce a user message. Triggers a retry, up to max_retries.

message T | None Default: None

The message to send. None when goal_reached is true. Cannot be set together with schema_issue.

`InputGenerator`

Base class for custom input generators.

Module: giskard.checks.core.input_generator

InputGenerator takes one type parameter, the trace type: InputGenerator[Trace]. Passing two raises TypeError at class definition.

.__call__() → AsyncGenerator[InputType, TraceType]

Yield input values one at a time. Receives the current trace via send() between yields, so the generator can adapt to prior interactions.

trace TraceType Required

The initial trace passed in when the generator is first called.

input_type type | None Default: None

The type the caller wants each yielded input to have. Accept it in your signature even if you ignore it; None means plain str.

Sequential generator

from collections.abc import AsyncGenerator

from giskard.checks import Scenario
from giskard.checks.core import Trace
from giskard.checks.core.input_generator import InputGenerator


def target(inputs: str) -> str:
    return f"Received: {inputs}"


class SequentialInputGenerator(InputGenerator[Trace]):
    inputs_list: list[str]

    async def __call__(
        self, trace: Trace, input_type: type | None = None
    ) -> AsyncGenerator[str, Trace]:
        for value in self.inputs_list:
            trace = yield value


gen = SequentialInputGenerator(
    inputs_list=[
        "Find item 42",
        "Search the archive",
        "Show its metadata",
    ]
)

scenario = Scenario("generated_inputs").interact(
    inputs=gen, outputs=target
)

Context-aware generator

from collections.abc import AsyncGenerator

from giskard.checks.core import Trace
from giskard.checks.core.input_generator import InputGenerator


class ContextAwareGenerator(InputGenerator[Trace]):
    strategy: str = "follow_up"
    max_steps: int = 5

    async def __call__(
        self, trace: Trace, input_type: type | None = None
    ) -> AsyncGenerator[str, Trace]:
        # First message: no prior interactions yet.
        trace = yield "Find item 42."

        for _ in range(self.max_steps - 1):
            if not trace.last:
                return

            last_output = trace.last.outputs.lower()
            if self.strategy == "follow_up":
                if "?" in last_output:
                    trace = yield "Search the archive."
                    continue
                if "found" in last_output:
                    trace = yield "Show its metadata."
                    continue
            trace = yield "Thanks, that helps."

Use a custom generator when Python can determine the next message. Otherwise use UserSimulator.

Usage patterns

Compare personas

Run the same target with several personas:

from giskard.checks import Scenario
from giskard.checks.generators.user import UserSimulator

personas = [
    {
        "name": "impatient",
        "persona": "Ask for a one-line answer.",
    },
    {
        "name": "novice",
        "persona": "Ask for plain-language explanations.",
    },
    {
        "name": "pushy",
        "persona": "Ask a follow-up question after each answer.",
    },
]

results = {}
for persona in personas:
    sim = UserSimulator(persona=persona["persona"], max_steps=3)
    scenario = Scenario(persona["name"]).interact(
        inputs=sim, outputs=target
    )
    results[persona["name"]] = await scenario.run()

for name, result in results.items():
    print(f"{name}: {result.status}")

Each run can generate different messages. Compare results as examples, not a fixed measurement.

Generators

UserSimulator