Simulate Users

Use UserSimulator to drive multi-turn tests with LLM-generated user inputs.

1. Configure a generator

To get started, you need to provide the LLM that will power the simulator. UserSimulator uses a generator to produce each user turn, so the same model you use for your checks can also drive realistic user behavior.

UserSimulator uses an LLM to generate realistic user messages. Set a default generator once, or pass one inline.

def support_agent(message: str) -> str:
    """Stub support agent for demonstration."""
    return "I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?"

from giskard.checks import set_default_generator
from giskard.agents.generators import Generator

set_default_generator(Generator(model="openai/gpt-5.4-nano"))

2. Create a UserSimulator with a persona

With the generator configured, we can now define who the simulated user is. The persona field acts as a system prompt for the simulator — it describes the user’s role, goal, and stopping condition. The more specific you are, the more deterministic and useful the generated conversation will be.

from giskard.checks.generators.user import UserSimulator

customer = UserSimulator(
    persona="""
    You are a customer trying to track a delayed order.
    - Start by asking about order #98765
    - Provide your name (Alex) when asked
    - Accept any resolution the support agent offers
    - Stop when the agent confirms a solution
    """,
    max_steps=8,
)

max_steps limits how many turns the simulator will generate before stopping.

3. Use the simulator as inputs in .interact()

Now we’ll wire the simulator into the scenario. Passing the UserSimulator as inputs tells the scenario to call it on each turn rather than using a fixed string — the scenario handles the loop automatically up to max_steps.

Pass the UserSimulator instance as the inputs argument. The scenario will call it repeatedly to generate each user turn.

from giskard.checks import Scenario, FnCheck

scenario = (
    Scenario("order_tracking")
    .interact(
        inputs=customer,
        outputs=lambda inputs: support_agent(inputs),
    )
    .check(
        FnCheck(fn=
            lambda trace: any(
                word in trace.last.outputs.lower()
                for word in ["resolved", "refund", "replacement", "shipped"]
            ),
            name="resolution_offered",
        )
    )
)

4. Run the scenario and inspect the trace

With the scenario built, run it and iterate over the trace to see the full conversation the simulator generated. This is especially useful when debugging a failing check — you can see exactly what the simulated user said at each step.

import asyncio
result = asyncio.run(scenario.run())

# Print every turn
for turn in result.final_trace.interactions:
    print(f"User:  {turn.inputs}")
    print(f"Agent: {turn.outputs}")
    print()

Output

User: Hi, I’m trying to track a delayed order. Can you check on order #98765 for me and tell me the current status and expected delivery date? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Thanks—my name is Alex. Can you also confirm the tracking number and whether it’s expected to arrive tomorrow as planned? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Okay—can you confirm the latest tracking update (what the carrier scanned last) and share the tracking number? Also, if it’s not arriving tomorrow, what’s the next estimated delivery date? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Got it. Since you’ve confirmed it’s arriving tomorrow, can you please share the carrier’s latest scan (date/time and location) and the tracking number so I can verify it? If it misses tomorrow, what’s the next estimated delivery date? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Thanks—before I let this go, can you please provide the tracking number and the most recent carrier scan details (date/time and location) for order #98765? And just confirm again that it’s expected to arrive tomorrow. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Actually—could you please provide the tracking number and the latest carrier scan details (date/time and location) for order #98765, and confirm one more time that it’s expected to arrive tomorrow? If you can’t access that info, let me know the next best step. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Hi—sorry, but you haven’t given me the tracking number or the latest carrier scan details yet. Please either (1) provide the tracking number plus the most recent scan (date/time and location) for order #98765, or (2) if you can’t access it, tell me the next best step and what I should expect next. Also, please confirm one final time that it’s expected to arrive tomorrow. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Hi—Alex here. I still haven’t received the tracking number or the latest carrier scan details (date/time and location). Please provide those for order #98765, or tell me the next best step to get them, and confirm it’s expected to arrive tomorrow. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

5. Check goal_reached from simulator metadata

After the scenario finishes, the simulator writes a LLMGeneratorOutput into the last interaction’s metadata. This tells you whether the user’s stated goal was achieved, a stronger signal than just checking whether the scenario passed its checks, because it reflects the simulator’s own evaluation of the conversation outcome.

from giskard.checks.generators.base import LLMGeneratorOutput

last = result.final_trace.last
simulator_output = last.metadata.get("simulator_output")

if isinstance(simulator_output, LLMGeneratorOutput):
    print(f"Goal reached: {simulator_output.goal_reached}")
    print(f"Message: {simulator_output.message}")

Use goal_reached as an additional assertion:

if simulator_output and not simulator_output.goal_reached:
    print(f"Goal not reached: {simulator_output.message}")
else:
    print("Goal reached or no simulator output")

Output

Goal reached or no simulator output

6. Swap personas for A/B testing

With a single persona working, we can now run the same agent against multiple user types simultaneously. Each persona exercises a different interaction style, and running them concurrently with asyncio.gather means you get results for all three in roughly the time it takes to complete one.

Run the same agent against multiple user types to surface persona-specific failures.

import asyncio

personas = [
    (
        "impatient",
        "You are impatient. Keep messages short. Escalate quickly if not helped.",
    ),
    (
        "detailed",
        "You are thorough. Ask many follow-up questions before accepting any solution.",
    ),
    (
        "confused",
        "You are unsure what you need. Describe symptoms, not the actual problem.",
    ),
]


async def run_persona(name, instructions):
    sim = UserSimulator(persona=instructions, max_steps=6)
    scenario = Scenario(name).interact(
        inputs=sim,
        outputs=lambda inputs: support_agent(inputs),
    )
    return name, await scenario.run()


results = asyncio.run(asyncio.gather(*[run_persona(n, i) for n, i in personas]))

for name, result in results:
    print(f"{name}: {'PASSED' if result.passed else 'FAILED'}")

Output

impatient: PASSED detailed: PASSED confused: PASSED

Custom trace formatting

By default the trace prints interactions as raw inputs and outputs. You can write a simple formatting function to produce a human-readable transcript — for example, to log a simulated conversation or include it in a test failure message. For a subclass of Trace, Rich rendering, and how that interacts with print_report(), see Custom trace types.

def format_transcript(trace) -> str:
    """Format a trace as a human-readable chat transcript."""
    lines = []
    for turn in trace.interactions:
        lines.append(f"User:  {turn.inputs}")
        lines.append(f"Agent: {turn.outputs}")
    return "\n".join(lines)


result = await (
    Scenario("chat_trace_demo")
    .interact(
        inputs=customer,
        outputs=lambda inputs: support_agent(inputs),
    )
    .run()
)

print(format_transcript(result.final_trace))

Output

User: Hi, I’m trying to track a delayed order. Can you check the status of order #98765 and let me know when it will ship or arrive? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Thanks—can you confirm the updated delivery date for order #98765 and share any tracking number details I can use? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Hi again—could you please confirm the exact updated delivery date for order #98765 and provide the tracking number (or the link) so I can follow its progress? Also, please include whether there have been any recent delivery scans or delays. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Hi—thanks. My goal is just to get a clear, final update: please confirm the exact updated delivery date for order #98765 and provide the tracking number (or a tracking link). Also, can you tell me if there have been any recent delivery scans (what date/time) and whether there are any current delays? By the way, my name is Alex. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Hi—can you please provide the exact updated delivery date for order #98765, plus the tracking number or a direct tracking link? Also, what was the most recent delivery scan (date/time), and is there any current delay or exception that could change tomorrow’s arrival? My name is Alex. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Yes—please don’t send a generic response. I need a clear final update for order #98765: the exact updated delivery date, the tracking number (or a direct tracking link), and the most recent delivery scan date/time, plus whether there’s any exception/delay that could affect tomorrow’s arrival. My name is Alex. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Alex here again—can you please give the exact updated delivery date for order #98765 (month/day), and the actual tracking number or a direct tracking link? Also tell me the most recent delivery scan date/time and whether there are any exceptions/delays that could change tomorrow’s arrival. If you can’t access the tracking details, escalate and confirm what you will do next. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Hi—my name is Alex. I still don’t have the exact updated delivery date (month/day), the tracking number or a direct tracking link, or the most recent delivery scan date/time. Please provide those specifics for order #98765 now; if you can’t access them, escalate the case and tell me exactly what you’ll do next and by when. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

Next steps

• Generators reference — full UserSimulator parameter reference
• Multi-turn testing tutorial — multi-turn scenario basics
• Debug with Spy — inspect what happens inside each interaction