Generate knowledge base tests

This section will guide you through generating knowledge base-focused test cases using the Hub SDK. Generative AI agents can face an endless variety of real-world scenarios, making it impossible to manually enumerate all possible scenarios. Automated, synthetic test case generation is therefore essential—especially when you lack real user chats to import as tests. However, a major challenge is to ensure that these synthetic cases are tailored to your business context, rather than being overly generic.

By generating domain-specific synthetic tests, you can proactively identify and address these types of failures before they impact your users or business operations.

Let’s start by initializing the Hub client or take a look at the Quickstart & setup section to see how to install the SDK and connect to the Hub.

from giskard_hub import HubClient

hub = HubClient()

You can now use the hub.datasets and hub.knowledge_bases clients to control the Giskard Hub!

Document-Based Testing

The generate_document_based method creates test cases from your knowledge base, making it ideal for testing business logic and accuracy.

Before generating test cases, you need to setup knowledge bases and create an agent.

# Generate document-based test cases for knowledge base testing
knowledge_base_dataset = hub.datasets.generate_document_based(
    model_id=model.id,
    knowledge_base_id=knowledge_base.id,
    dataset_name="Knowledge Base Tests",
    description="Test cases based on company knowledge base",
    n_questions=15, # total number of questions to generate
    topic_ids=["topic-uuid-1", "topic-uuid-2"]  # Optional: filter by specific topics
)

# Wait for the dataset to be created
knowledge_base_dataset.wait_for_completion()

# List the chat test cases in the dataset
for chat_test_case in knowledge_base_dataset.chat_test_cases:
    print(chat_test_case.messages[0].content)

Tip

You can also use the Giskard Hub UI to generate knowledge base test cases if you prefer a visual interface.

Next steps