Test your ML model
How to get started with Automated Machine Learning testing
Giskard enables you to create test suites on AI models. It provides presets of tests so that you design and execute your tests in no time.
Here are the 3 steps to create and execute tests:
To create tests, you need first to create a test suite, here are the 3 steps:
1. Go to the test suite tab in Giskard and click on the button "create test suite"
2. Choose the inputs parameters of your test suite:
- Test suite name: A test suite name
- Model: The model that the test suite will test
- Actual dataset: A test dataset used to execute the tests inside the test suite. It could be any datasets that you've uploaded with 3.-upload-a-model-and-a-dataset
- Reference dataset (optional): An optional reference dataset used for the drift testing to assess the changes with the Actual dataset. It could be any datasets that you've uploaded with 3.-upload-a-model-and-a-dataset
3. Toggle on "the automatic test" to automatically pre-compute a batch of tests that is preloaded by Giskard according to your case.
- Metamorphic testing: Test if your model outputs behave as expected before and after input perturbation
- Heuristics testing: Test if your model output respect some business rules
- Performance testing: Test if your model performance is sufficiently high within some particular data slices
- Data drift testing: Test if your features don't drift between the reference and actual dataset
- Prediction drift testing: Test the absence of concept drift inside your model
To have detailed documentation on these tests, go to https://github.com/Giskard-AI/giskard/tree/main/python-client/giskard/ml_worker/testing
Once you have run the test you designed, Giskard provides the results (PASS or FAIL) of all your tests. You can then click on the test and further investigate the outputs of the tests to understand the issue.
1. Find the ids of the objects you created in Giskard
To find the ids of the tests and test suites you created in Giskard, use:
project.list_tests_in_suite(suite_id)
project.list_test_suites()
To find the ids of the model and datasets you uploaded in Giskard, use the *upload_* methods:
ds_id = upload_df(...)
model_id = upload_model(...)
model_id, ds_id = upload_model_and_df(...)
You can also find all your ids in the Giskard UI
2. Execute your tests & test suites externally
To execute your model externally and have your test results as a JSON file, use the following APIs:
project.execute_test(
test_id,
actual_ds_id=None,
reference_ds_id=None,
model_id=None)
project.execute_test_suite(
test_suite_id,
actual_ds_id=None,
reference_ds_id=None,
model_id=None)
When the optional arguments (model / actual_ds / reference_ds) are provided, they replace the existing ones in the test suites. If they're not provided, the old values (model / actual_ds / reference_ds) remain unchanged in the test suite settings.
Example:
A typical workflow can be:
new_model_id = upload_model(...)
test_result = project.execute_test_suite(
test_suite_id=123,
model_id=new_model_id)
Last modified 3mo ago