Simulate Users

Use UserSimulator to drive multi-turn tests with LLM-generated user inputs.

1. Configure a generator

To get started, provide the LLM that powers the simulator. UserSimulator uses the default generator (set_default_generator) — see Generators and judges. Set it once here, or pass a generator inline.

def support_agent(message: str) -> str:
    """Stub support agent for demonstration."""
    return "I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?"

from giskard.checks import set_default_generator

set_default_generator("openai/gpt-5.4-nano")

Output

Thank you for using Giskard open-source! 🐢 🙏 Giskard Enterprise adds deeper agent scans, audit reports with remediation guidance, test review interfaces for root-cause analysis & human feedback integration, and team collaboration — with flexible pricing. Learn more: https://giskard.ai

2. Create a UserSimulator with a persona

With the generator configured, we can now define who the simulated user is. The persona field acts as a system prompt for the simulator — it describes the user’s role, goal, and stopping condition. The more specific you are, the more deterministic and useful the generated conversation will be.

from giskard.checks.generators.user import UserSimulator

customer = UserSimulator(
    persona="""
    You are a customer trying to track a delayed order.
    - Start by asking about order #98765
    - Provide your name (Alex) when asked
    - Accept any resolution the support agent offers
    - Stop when the agent confirms a solution
    """,
    max_steps=8,
)

max_steps limits how many turns the simulator will generate before stopping.

3. Use the simulator as inputs in .interact()

Now we’ll wire the simulator into the scenario. Passing the UserSimulator as inputs tells the scenario to call it on each turn rather than using a fixed string — the scenario handles the loop automatically up to max_steps.

Pass the UserSimulator instance as the inputs argument. The scenario will call it repeatedly to generate each user turn.

from giskard.checks import Scenario, FnCheck

scenario = (
    Scenario("order_tracking")
    .interact(
        inputs=customer,
        outputs=lambda inputs: support_agent(inputs),
    )
    .check(
        FnCheck(fn=
            lambda trace: any(
                word in trace.last.outputs.lower()
                for word in ["resolved", "refund", "replacement", "shipped"]
            ),
            name="resolution_offered",
        )
    )
)

4. Run the scenario and inspect the trace

With the scenario built, run it and iterate over the trace to see the full conversation the simulator generated. This is especially useful when debugging a failing check — you can see exactly what the simulated user said at each step.

import asyncio
result = asyncio.run(scenario.run())

# Print every turn
for turn in result.final_trace.interactions:
    print(f"User:  {turn.inputs}")
    print(f"Agent: {turn.outputs}")
    print()

Output

User: Hi, I’m trying to track a delayed order—can you help me with order #98765? I’d like an update on its current status and expected delivery date. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Thanks—can you confirm the latest tracking details for order #98765 and whether tomorrow is still the expected delivery date? Also, my name is Alex. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Hi—thanks. Can you please confirm if tomorrow is still the expected delivery date for order #98765, and share the most recent tracking scan/location? If there’s been any delay or change in ETA, let me know what the new estimate is. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Yes—please confirm the latest ETA and the most recent tracking scan/location for order #98765. If tomorrow isn’t guaranteed, what’s the new estimated delivery date? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Sorry, but I need a concrete answer—can you share the most recent tracking scan (date/time and location) for order #98765 and confirm whether the ETA is still tomorrow or if there’s been a delay? If it changed, what’s the new estimated delivery date? My name is Alex. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: No—can you finally provide the most recent tracking scan details (exact date/time and location) for order #98765 and confirm in plain terms whether delivery is still guaranteed for tomorrow? If not, what is the updated estimated delivery date? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Alex here—please don’t repeat the generic status. Can you provide the exact date/time of the latest tracking scan and its location for order #98765, and confirm whether delivery tomorrow is guaranteed or give the new ETA if not? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Thanks—at this point I’m going to need escalation. Please provide the exact latest tracking scan details (date/time and location) for order #98765, and confirm clearly whether delivery tomorrow is guaranteed or give the updated ETA. If you can’t access the scan, escalate to a supervisor and let me know what resolution you can offer. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

5. Check goal_reached from simulator metadata

After the scenario finishes, the simulator writes a LLMGeneratorOutput into the last interaction’s metadata. This tells you whether the user’s stated goal was achieved, a stronger signal than just checking whether the scenario passed its checks, because it reflects the simulator’s own evaluation of the conversation outcome.

from giskard.checks.generators.base import LLMGeneratorOutput

last = result.final_trace.last
simulator_output = last.metadata.get("simulator_output")

if isinstance(simulator_output, LLMGeneratorOutput):
    print(f"Goal reached: {simulator_output.goal_reached}")
    print(f"Message: {simulator_output.message}")

Use goal_reached as an additional assertion:

if simulator_output and not simulator_output.goal_reached:
    print(f"Goal not reached: {simulator_output.message}")
else:
    print("Goal reached or no simulator output")

Output

Goal reached or no simulator output

6. Swap personas for A/B testing

With a single persona working, we can now run the same agent against multiple user types simultaneously. Each persona exercises a different interaction style, and running them concurrently with asyncio.gather means you get results for all three in roughly the time it takes to complete one.

Run the same agent against multiple user types to surface persona-specific failures.

import asyncio

personas = [
    (
        "impatient",
        "You are impatient. Keep messages short. Escalate quickly if not helped.",
    ),
    (
        "detailed",
        "You are thorough. Ask many follow-up questions before accepting any solution.",
    ),
    (
        "confused",
        "You are unsure what you need. Describe symptoms, not the actual problem.",
    ),
]


async def run_persona(name, instructions):
    sim = UserSimulator(persona=instructions, max_steps=6)
    scenario = Scenario(name).interact(
        inputs=sim,
        outputs=lambda inputs: support_agent(inputs),
    )
    return name, await scenario.run()


results = await asyncio.gather(*[run_persona(n, i) for n, i in personas])

for name, result in results:
    print(f"{name}: {'PASSED' if result.passed else 'FAILED'}")

Output

impatient: PASSED detailed: PASSED confused: PASSED

Custom trace formatting

By default the trace prints interactions as raw inputs and outputs. You can write a simple formatting function to produce a human-readable transcript — for example, to log a simulated conversation or include it in a test failure message. For a subclass of Trace, Rich rendering, and how that interacts with print_report(), see Custom trace types.

def format_transcript(trace) -> str:
    """Format a trace as a human-readable chat transcript."""
    lines = []
    for turn in trace.interactions:
        lines.append(f"User:  {turn.inputs}")
        lines.append(f"Agent: {turn.outputs}")
    return "\n".join(lines)


result = await (
    Scenario("chat_trace_demo")
    .interact(
        inputs=customer,
        outputs=lambda inputs: support_agent(inputs),
    )
    .run()
)

print(format_transcript(result.final_trace))

Output

User: Hi, I’m trying to track a delayed order. Can you please look into order #98765 and tell me the current status and expected delivery date? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Thanks—before I go, can you confirm the expected delivery date/time for order #98765 and whether there’s any tracking update I should watch for? Also, my name is Alex. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Got it—thanks. Since you’re saying it arrives tomorrow, can you confirm the exact expected delivery date/time window and whether there will be a new tracking scan today? If you can’t, just tell me what the next update should be and how I’ll know. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Thanks—just to make sure I understand: can you confirm the exact expected delivery date/time window for order #98765 and the next tracking update I should expect? If you can’t provide a window, tell me what the best estimate is and what event/scan will indicate it’s out for delivery. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Thanks—can you please confirm the best delivery estimate for order #98765 (date and approximate time window) and tell me what the next tracking scan should say when it’s out for delivery? If there’s anything you can do on your end to speed it up, please go ahead. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Hi—thanks. I’m okay with whatever resolution you can offer, but I still need the concrete next step. Can you provide the best estimated delivery date/time window for order #98765 and tell me exactly what I should look for in the tracking when it goes out for delivery? If you can’t speed it up, please confirm that and let me know the earliest date/time it should arrive. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Hi Alex—sorry, I mean I’m Alex. Can you please finally confirm the exact delivery date/time window for order #98765 and tell me what the next tracking update will say when it goes out for delivery? If you can’t speed it up, just confirm you can’t and give me the earliest expected arrival time. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Thanks. I just need a concrete answer: what is the earliest expected delivery time tomorrow for order #98765, and what exact tracking status should I look for to know it’s out for delivery (e.g., “Out for delivery” or “Departed facility”)? If you can’t speed it up, please confirm that and give the earliest time you can. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

Next steps

• Generators reference — full UserSimulator parameter reference
• Multi-turn testing tutorial — multi-turn scenario basics
• Debug with Spy — inspect what happens inside each interaction