Contribute to Giskard
How to add your custom ML tests to Giskard's open-source project
- Clone the Giskard repository
- Create a GitHub branch with the base as main, starting with
test-contribution/name-of-your-branch - From the root of the cloned repo run
./gradlew generateProto. This will generate the modulegeneratedthat you will need to create your tests. - Write your test inside one of the classes (
MetamorphicTests,HeuristicTests,PerformanceTestsorDriftTests) inside this repo. If your test does not fit these classes, you can also create a custom class in a new file. - We recommend writing unit tests for your test functions: this is the way you can execute and debug your test! Unit tests should be placed in this directory.
Fixtures
A unit test is executed with a test model and test data provided as fixtures.
For example, in test_precision function, we use german_credit_data and german_credit_model as fixtures. If necessary, you can create your own fixtures; check this directory to know how.
- To use this test on Giskard UI, we recommend you write the same code in the Giskard UI following these steps in our doc.
Even if your test works, it won’t appear in Giskard's frontend, unless you re-create it as a custom test in Giskard UI.
- Create a Pull Request
Example
Let us guide you with an example where you want to create a heuristic test function:
- Since you are writing a heuristic test, you will select heuristic_tests.py from the list of files under https://github.com/Giskard-AI/giskard/tree/main/giskard-ml-worker/ml_worker/testing
- In heuristic_tests.py, write your code under class HeuristicTests(AbstractTestCollection) after the existing codes
- To write the unit test we select test_heuristic.py under the list of files https://github.com/Giskard-AI/giskard/tree/main/giskard-ml-worker/test
- Follow the pattern of parameterizing the inputs which helps you to test the function using different fixtures. You can read more about parameterizing unit tests here https://docs.pytest.org/en/7.1.x/example/parametrize.html
- If you want to create your own dataset, You can create your own fixture python file under https://github.com/Giskard-AI/giskard/tree/main/giskard-ml-worker/test/fixtures
- You can use https://github.com/Giskard-AI/giskard/blob/main/giskard-ml-worker/test/fixtures/german_credit_scoring.py for your reference. Make sure you return the model, data(with target) and test_data(without target) in the expected format. </aside>
Last modified 3mo ago