Test suite#

class giskard.Suite(name=None)#

A class representing a test suite that groups a collection of test cases together. The Suite class provides methods to add new tests, execute all tests, and save the suite to a Giskard instance.

id#

int An integer identifying the suite.

Type:

int

tests#

List[TestPartial] A list of TestPartial objects representing the test cases in the suite. A mapping of suite parameters with their corresponding SuiteInput objects.

Type:

List[giskard.core.suite.TestPartial]

name#

str A string representing the name of the suite.

Type:

str

Create a new Test Suite instance with a given name.

Parameters:

name (str, optional) – The name of the test suite.

__init__(name=None) → None#

Create a new Test Suite instance with a given name.

Parameters:

name (str, optional) – The name of the test suite.

run(verbose: bool = True, **suite_run_args)#

Execute all the tests that have been added to the test suite through the add_test method.

Parameters:

  • verbose (bool) – If set to True, the execution information for each test will be displayed. Defaults to False.

  • **suite_run_args (dict, optional) – Any arguments passed here will be applied to all the tests in the suite whenever they match with the arguments defined for each test. If a test contains an argument that has already been defined, it will not get overridden. If any inputs on the test suite are missing, an error will be raised.

Returns:

containing test execution information

Return type:

TestSuiteResult

add_test(test_fn: GiskardTest | Callable[[...], TestResult | bool], test_name: int | str | None = None, **params) → Suite#

Add a test to the suite.

Parameters:

  • test_fn (Test) – A test method that will be executed or an instance of a GiskardTest class.

  • test_name (Optional[Union[int, str]], optional) – A unique identifier used to track the test result. If None, the identifier will be generated based on the module and name of the test method. If the identifier already exists in the suite, a new unique identifier will be generated.

  • **params – Default parameters to be passed to the test method. This parameter will be ignored if test_fn is an instance of GiskardTest.

Returns:

The current instance of the test suite to allow chained calls.

Return type:

Suite

upload(client: GiskardClient, project_key: str)#

Saves the test suite to the Giskard backend and sets its ID.

Parameters:

  • client – A GiskardClient instance to connect to the backend.

  • project_key – The key of the project that the test suite belongs to.

Returns:

The current instance of the test Suite to allow chained call.

class giskard.core.suite.SuiteInput(name: str, ptype: Any)#

Represents an input parameter for a test suite.

name#

The name of the input parameter.

Type:

str

type#

The type of the input parameter.

Type:

Any

Raises:

AssertionError – If the input type is not supported.

Example

>>> input_param = SuiteInput("age", int)
>>> input_param.name
'age'
>>> input_param.type
<class 'int'>
class giskard.core.suite.DatasetInput(name: str, target: str | None = None)#

Represents a dataset input parameter for a test suite.

Inherits from SuiteInput.

name#

The name of the dataset input parameter.

Type:

str

target#

The target column of the dataset.

Type:

Optional[str]

Example

>>> dataset_input = DatasetInput("data", target="label")
>>> dataset_input.name
'data'
>>> dataset_input.type
<class 'Dataset'>
>>> dataset_input.target
'label'
class giskard.core.suite.ModelInput(name: str, model_type: str | None = None)#

Represents a model input parameter for a test suite.

Inherits from SuiteInput.

name#

The name of the model input parameter.

Type:

str

model_type#

The type or name of the model.

Type:

Optional[str]

Example

>>> model_input = ModelInput("model", model_type="SKLearnModel")
>>> model_input.name
'model'
>>> model_input.model_type
'SKLearnModel'
class giskard.core.suite.TestSuiteResult(passed: bool, results: List[Tuple[str, TestResult, Dict[str, Any]]])#

Represents the result of a test suite.

class giskard.ml_worker.testing.test_result.TestResult(passed: bool = False, messages: ~typing.List[~giskard.ml_worker.testing.test_result.TestMessage] = <factory>, props: ~typing.Dict[str, str] = <factory>, metric: float = 0, missing_count: int = 0, missing_percent: float = 0, unexpected_count: int = 0, unexpected_percent: float = 0, unexpected_percent_total: float = 0, unexpected_percent_nonmissing: float = 0, partial_unexpected_index_list: ~typing.List[~giskard.ml_worker.testing.test_result.PartialUnexpectedCounts] = <factory>, unexpected_index_list: ~typing.List[int] = <factory>, number_of_perturbed_rows: int = 0, actual_slices_size: ~typing.List[int] = <factory>, reference_slices_size: ~typing.List[int] = <factory>, output_df: bytes | None = None, output_df_id: str | None = None, is_error: bool = False)#

Dataclass representing the result of a test

Parameters:

  • passed – A boolean indicating whether the test passed or not

  • messages – A list of TestMessage objects containing information about the test execution

  • metric – A float representing the test metric