Test suite

class giskard.Suite(name=None, default_params=None)

A test suite.

A class representing a test suite that groups a collection of test cases together. The Suite class provides methods to add new tests, execute all tests, and save the suite to a Giskard instance.

Create a new Test Suite instance with a given name.

Parameters:

• name (Optional[str]) – The name of the test suite.

• default_params (dict, optional) – Any arguments passed will be applied to the tests in the suite, if runtime params with the same name are not set.

__init__(name=None, default_params=None) → None

Create a new Test Suite instance with a given name.

Parameters:

• name (Optional[str]) – The name of the test suite.

• default_params (dict, optional) – Any arguments passed will be applied to the tests in the suite, if runtime params with the same name are not set.

run(verbose: bool = True, **suite_run_args)

Execute all the tests that have been added to the test suite through the add_test method.

Parameters:

• verbose (bool) – If set to True, the execution information for each test will be displayed. Defaults to False.

• **suite_run_args (Optional[dict]) – Any arguments passed here will be applied to all the tests in the suite whenever they match with the arguments defined for each test. If a test contains an argument that has already been defined, it will not get overridden. If any inputs on the test suite are missing, an error will be raised.

Returns:

containing test execution information

Return type:

TestSuiteResult

add_test(test_fn: GiskardTest | Callable[[...], TestResult | bool], test_id: int | str | None = None, display_name: str | None = None, **params) → Suite

Add a test to the suite.

Parameters:

• test_fn (Test) – A test method that will be executed or an instance of a GiskardTest class.

• test_id (Optional[Union[int, str]]) – A unique identifier used to track the test result. If None, the identifier will be generated based on the module and name of the test method. If the identifier already exists in the suite, a new unique identifier will be generated. (Default value = None)

• display_name (Optional[str]) – The name of the test to be displayed (Default value = None)

• **params – Default parameters to be passed to the test method. This parameter will be ignored if test_fn is an instance of GiskardTest.

Returns:

The current instance of the test suite to allow chained calls.

Return type:

Suite

add_test(test_fn: GiskardTest | Callable[[...], TestResult | bool], test_id: int | str | None = None, display_name: str | None = None, **params) → Suite

Add a test to the suite.

Parameters:

• test_fn (Test) – A test method that will be executed or an instance of a GiskardTest class.

• test_id (Optional[Union[int, str]]) – A unique identifier used to track the test result. If None, the identifier will be generated based on the module and name of the test method. If the identifier already exists in the suite, a new unique identifier will be generated. (Default value = None)

• display_name (Optional[str]) – The name of the test to be displayed (Default value = None)

• **params – Default parameters to be passed to the test method. This parameter will be ignored if test_fn is an instance of GiskardTest.

Returns:

The current instance of the test suite to allow chained calls.

Return type:

Suite

remove_test(arg)

remove_test(idx: int)

remove_test(test_name: str)

remove_test(giskard_test: GiskardTest)

Remove a test from the suite.

Parameters:

arg (int|str|GiskardTest) – If int: remove the test by index. If str: remove the test by name passed during the add_test method If GiskardTest: remove the test(s) by reference

Returns:

The current instance of the test suite to allow chained calls.

Return type:

Suite

upgrade_test(test: GiskardTest, migrate_params_fn: Callable[[Dict[str, Any]], Dict[str, Any]] | None = None) → Suite

Upgrade a test with a new version, the test being upgraded are matched using display_name tests property.

Parameters:

• test (GiskardTest) – The newest version of a test to be upgraded

• migrate_params_fn (Optional[Callable[[Dict[str, Any]], Dict[str, Any]]]) – An optional callback used to migrate the old test params into the new params

Returns:

The current instance of the test suite to allow chained calls.

Return type:

Suite

update_test_params(index: int, **params)

Update a test from the suite.

Parameters:

• index (int) – The index of the test to be updated

• **params – The params to be added/updated to the current one

Returns:

The current instance of the test suite to allow chained calls.

Return type:

Suite

upload(client: GiskardClient, project_key: str | None = None)

Saves the test suite to the Giskard backend and sets its ID.

Parameters:

• client (GiskardClient) – A GiskardClient instance to connect to the backend.

• project_key (str) – The key of the project that the test suite belongs to.

Returns:

The current instance of the test Suite to allow chained call.

Return type:

Suite

classmethod download(client: GiskardClient, project_key: str, suite_id: int) → Suite

Download the suite from the hub using project key and suite identifier.

Parameters:

• client (GiskardClient) – A GiskardClient instance to connect to the backend.

• project_key (str) – The key of the project that the test suite belongs to.

• suite_id (int) – identifier for the suite

Returns:

the downloaded suite

Return type:

Suite

class giskard.core.suite.SuiteInput(name: str, ptype: Any)

Represents an input parameter for a test suite.

Raises:

AssertionError – If the input type is not supported.

Examples

>>> input_param = SuiteInput("age", int)
>>> input_param.name
'age'
>>> input_param.type
<class 'int'>

class giskard.core.suite.DatasetInput(name: str, target: str | None = None)

Represents a dataset input parameter for a test suite.

Examples

>>> dataset_input = DatasetInput("data", target="label")
>>> dataset_input.name
'data'
>>> dataset_input.type
<class 'Dataset'>
>>> dataset_input.target
'label'

class giskard.core.suite.ModelInput(name: str, model_type: str | None = None)

Represents a model input parameter for a test suite.

Examples

>>> model_input = ModelInput("model", model_type="SKLearnModel")
>>> model_input.name
'model'
>>> model_input.model_type
'SKLearnModel'

class giskard.core.suite.TestSuiteResult(suite: Suite, inputs: Dict[str, Any], passed: bool, results: List[SuiteResult], execution_date: datetime, completion_date: datetime)

Represents the result of a test suite.

upload(client: GiskardClient, project_key: str | None = None, label: str | None = None)

class giskard.core.test_result.TestResult(passed: bool = False, messages: ~typing.List[~giskard.core.test_result.TestMessage] = <factory>, props: ~typing.Dict[str, str] = <factory>, metric_name: str = 'Metric', metric: float | None = None, missing_count: int = 0, missing_percent: float = 0, unexpected_count: int = 0, unexpected_percent: float = 0, unexpected_percent_total: float = 0, unexpected_percent_nonmissing: float = 0, partial_unexpected_index_list: ~typing.List[~giskard.core.test_result.PartialUnexpectedCounts] = <factory>, unexpected_index_list: ~typing.List[int] = <factory>, number_of_perturbed_rows: int = 0, actual_slices_size: ~typing.List[int] = <factory>, reference_slices_size: ~typing.List[int] = <factory>, output_df: bytes | None = None, output_ds: ~typing.List[~giskard.datasets.base.Dataset] = <factory>, details: ~giskard.core.test_result.TestResultDetails | None = None, is_error: bool = False)

Dataclass representing the result of a test

Parameters:

• passed – A boolean indicating whether the test passed or not

• messages – A list of TestMessage objects containing information about the test execution

• metric – A float representing the test metric