Simulate Users

Use UserSimulator to drive multi-turn tests with LLM-generated user inputs.

1. Configure a generator

To get started, you need to provide the LLM that will power the simulator. UserSimulator uses a generator to produce each user turn, so the same model you use for your checks can also drive realistic user behavior.

UserSimulator uses an LLM to generate realistic user messages. Set a default generator once, or pass one inline.

def support_agent(message: str) -> str:
    """Stub support agent for demonstration."""
    return "I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?"

from giskard.checks import set_default_generator

set_default_generator("openai/gpt-5.4-nano")

Output

Thank you for using Giskard open-source! 🐢 🙏 Giskard Enterprise adds deeper agent scans, audit reports with remediation guidance, test review interfaces for root-cause analysis & human feedback integration, and team collaboration — with flexible pricing. Learn more: https://giskard.ai

2. Create a UserSimulator with a persona

With the generator configured, we can now define who the simulated user is. The persona field acts as a system prompt for the simulator — it describes the user’s role, goal, and stopping condition. The more specific you are, the more deterministic and useful the generated conversation will be.

from giskard.checks.generators.user import UserSimulator

customer = UserSimulator(
    persona="""
    You are a customer trying to track a delayed order.
    - Start by asking about order #98765
    - Provide your name (Alex) when asked
    - Accept any resolution the support agent offers
    - Stop when the agent confirms a solution
    """,
    max_steps=8,
)

max_steps limits how many turns the simulator will generate before stopping.

3. Use the simulator as inputs in .interact()

Now we’ll wire the simulator into the scenario. Passing the UserSimulator as inputs tells the scenario to call it on each turn rather than using a fixed string — the scenario handles the loop automatically up to max_steps.

Pass the UserSimulator instance as the inputs argument. The scenario will call it repeatedly to generate each user turn.

from giskard.checks import Scenario, FnCheck

scenario = (
    Scenario("order_tracking")
    .interact(
        inputs=customer,
        outputs=lambda inputs: support_agent(inputs),
    )
    .check(
        FnCheck(fn=
            lambda trace: any(
                word in trace.last.outputs.lower()
                for word in ["resolved", "refund", "replacement", "shipped"]
            ),
            name="resolution_offered",
        )
    )
)

4. Run the scenario and inspect the trace

With the scenario built, run it and iterate over the trace to see the full conversation the simulator generated. This is especially useful when debugging a failing check — you can see exactly what the simulated user said at each step.

import asyncio
result = asyncio.run(scenario.run())

# Print every turn
for turn in result.final_trace.interactions:
    print(f"User:  {turn.inputs}")
    print(f"Agent: {turn.outputs}")
    print()

Output

User: Hi—I’m trying to track a delayed order. Can you check the status for order #98765? Thanks. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Thanks—what’s the latest estimated delivery date/time for order #98765, and is there a tracking link or confirmation number you can share? Also, my name is Alex. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Okay—could you please share the latest tracking link or any confirmation number for order #98765, and let me know the exact estimated delivery date/time (including timezone)? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Hi—sorry, I’m still waiting on the actual tracking link/confirmation number and the exact delivery date/time with timezone for order #98765. Can you send that now? My name is Alex. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Thanks—can you please escalate this and provide the actual tracking link or confirmation number for order #98765, plus the exact estimated delivery date/time with timezone? My name is Alex. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Hi—following up again for order #98765. Please provide the actual tracking link and/or confirmation number, and the exact estimated delivery date/time including timezone. If it can’t be provided, tell me what the escalation status is and what the new expected resolution time is. My name is Alex. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Hi—this is Alex. I still don’t have the actual tracking link/confirmation number or the exact delivery date/time with timezone for order #98765. Can you escalate again and either (1) send the tracking link/confirmation now, or (2) tell me the escalation status and the exact new expected time when you’ll have this resolved? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

User: Hi—this is Alex. I need the actual tracking link/confirmation number for order #98765 and the exact estimated delivery date/time with timezone. If you can’t provide them, please confirm the escalation status and give the precise new expected resolution time. Can you send that now? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

5. Check goal_reached from simulator metadata

After the scenario finishes, the simulator writes a LLMGeneratorOutput into the last interaction’s metadata. This tells you whether the user’s stated goal was achieved, a stronger signal than just checking whether the scenario passed its checks, because it reflects the simulator’s own evaluation of the conversation outcome.

from giskard.checks.generators.base import LLMGeneratorOutput

last = result.final_trace.last
simulator_output = last.metadata.get("simulator_output")

if isinstance(simulator_output, LLMGeneratorOutput):
    print(f"Goal reached: {simulator_output.goal_reached}")
    print(f"Message: {simulator_output.message}")

Use goal_reached as an additional assertion:

if simulator_output and not simulator_output.goal_reached:
    print(f"Goal not reached: {simulator_output.message}")
else:
    print("Goal reached or no simulator output")

Output

Goal reached or no simulator output

6. Swap personas for A/B testing

With a single persona working, we can now run the same agent against multiple user types simultaneously. Each persona exercises a different interaction style, and running them concurrently with asyncio.gather means you get results for all three in roughly the time it takes to complete one.

Run the same agent against multiple user types to surface persona-specific failures.

import asyncio

personas = [
    (
        "impatient",
        "You are impatient. Keep messages short. Escalate quickly if not helped.",
    ),
    (
        "detailed",
        "You are thorough. Ask many follow-up questions before accepting any solution.",
    ),
    (
        "confused",
        "You are unsure what you need. Describe symptoms, not the actual problem.",
    ),
]


async def run_persona(name, instructions):
    sim = UserSimulator(persona=instructions, max_steps=6)
    scenario = Scenario(name).interact(
        inputs=sim,
        outputs=lambda inputs: support_agent(inputs),
    )
    return name, await scenario.run()


results = await asyncio.gather(*[run_persona(n, i) for n, i in personas])

for name, result in results:
    print(f"{name}: {'PASSED' if result.passed else 'FAILED'}")

Output

impatient: PASSED detailed: PASSED confused: PASSED

Custom trace formatting

By default the trace prints interactions as raw inputs and outputs. You can write a simple formatting function to produce a human-readable transcript — for example, to log a simulated conversation or include it in a test failure message. For a subclass of Trace, Rich rendering, and how that interacts with print_report(), see Custom trace types.

def format_transcript(trace) -> str:
    """Format a trace as a human-readable chat transcript."""
    lines = []
    for turn in trace.interactions:
        lines.append(f"User:  {turn.inputs}")
        lines.append(f"Agent: {turn.outputs}")
    return "\n".join(lines)


result = await (
    Scenario("chat_trace_demo")
    .interact(
        inputs=customer,
        outputs=lambda inputs: support_agent(inputs),
    )
    .run()
)

print(format_transcript(result.final_trace))

Output

User: Hi—I’m trying to track a delayed order. Can you check on order #98765 for me? Any updates on the status or delivery ETA would be great. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Thanks—can you confirm the delivery tomorrow and share the updated tracking link or latest scan time for order #98765? Also, my name is Alex. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Okay—thanks. Can you please share the latest tracking link or latest scan time for order #98765, and confirm it’s definitely arriving tomorrow? If you need any details from me to verify, tell me what to provide. Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Hi—thanks. I still need the actual latest tracking scan time (or the most recent event) and the updated tracking link for order #98765, plus confirmation that it will definitely arrive tomorrow. Please tell me what you can share, and if anything is missing, tell me what details you need from me to verify. (Name: Alex.) Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Thanks—can you please provide the actual latest tracking scan time/event and the most up-to-date tracking link for order #98765? Also, can you confirm with certainty that it will arrive tomorrow (not just “in transit”)? If you can’t see the link/scan time, tell me exactly what info you need from me to verify. (Alex) Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: No problem—then please escalate or re-check the shipment so you can give me the actual latest tracking scan time/event and the updated tracking link for order #98765, and confirm it will definitely arrive tomorrow. If you still can’t access that, tell me the exact verification steps or info you need from me (Alex). Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Hi—Alex here. Please don’t just repeat “arrives tomorrow.” I need (1) the exact latest tracking scan time/event and (2) the updated tracking link for order #98765, or—if you can’t access it—tell me precisely what you need from me to verify and escalate. Can you confirm with certainty it will arrive tomorrow? Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with? User: Hi—can you please confirm the order #98765’s latest tracking event with an exact timestamp and provide the updated tracking link? If you can’t access the link/timestamp, tell me exactly what you need from me to verify and then escalate the case. Also, please confirm it will arrive tomorrow with certainty. (Name: Alex.) Agent: I have located your order #98765. It is currently in transit and will arrive tomorrow. Is there anything else I can help you with?

Next steps

• Generators reference — full UserSimulator parameter reference
• Multi-turn testing tutorial — multi-turn scenario basics
• Debug with Spy — inspect what happens inside each interaction