Contribute to Giskard

How to add your custom ML tests to Giskard’s open-source project

Push your tests in the Giskard repo

• Clone the Giskard repository

• Create a GitHub branch with the base as main, starting with test-contribution/name-of-your-branch

• From the root of the cloned repo run ./gradlew generateProto. This will generate the module`generated` that you will need to create your tests.

• Write your test inside one of the classes (MetamorphicTests, HeuristicTests, PerformanceTests or DriftTests) inside this repo. If your test does not fit these classes, you can also create a custom class in a new file.

• We recommend writing unit tests for your test functions: this is the way you can execute and debug your test! Unit tests should be placed in this directory.

Hint

Fixtures

A unit test is executed with a test model and test data provided as fixtures.

For example, in test_precision function, we use german_credit_data and german_credit_model as fixtures. If necessary, you can create your own fixtures; check this directory to know how.

• Create a Pull Request

Hint

Example

Let us guide you with an example where you want to create a heuristic test function:

• Since you are writing a heuristic test, you will select heuristic_tests.py from the list of files under https://github.com/Giskard-AI/giskard/tree/main/giskard-ml-worker/ml_worker/testing

• In heuristic_tests.py, write your code under class HeuristicTests(AbstractTestCollection) after the existing codes

• To write the unit test we select test_heuristic.py under the list of files https://github.com/Giskard-AI/giskard/tree/main/giskard-ml-worker/test

• Follow the pattern of parameterizing the inputs which helps you to test the function using different fixtures. You can read more about parameterizing unit tests here https://docs.pytest.org/en/7.1.x/example/parametrize.html

• If you want to create your own dataset, You can create your own fixture python file under https://github.com/Giskard-AI/giskard/tree/main/giskard-ml-worker/test/fixtures

• You can use https://github.com/Giskard-AI/giskard/blob/main/giskard-ml-worker/test/fixtures/german_credit_scoring.py for your reference. Make sure you return the model, data(with target) and test_data(without target) in the expected format.