🚀 Execute your test suite in your CI/CD pipeline#
First you’ll need to know how to run Giskard’s scan function see Scan your ML Model
By integrating Giskard into your CI/CD pipeline, you can run the scan function on every commit to your repository, ensuring that new models don’t introduce new vulnerabilities.
Create a script to scan your model#
By referring to the Scan your ML Model guide, you can create a script that will scan your model. We will refer to this python script as
scan.py in this guide.
from giskard import demo, Model, Dataset, scan model, df = demo.titanic() # By following previous user guides, you will be shown how to use your own model and dataset. # For example purposes, we will use the demo model and dataset. wrapped_model = Model(model=model, model_type="classification") wrapped_dataset = Dataset(df=df, target="Survived", cat_columns=['Pclass', 'Sex', "SibSp", "Parch", "Embarked"]) scan_results = scan(wrapped_model, wrapped_dataset) if scan_results.has_vulnerabilities: print("Your model has vulnerabilities") exit(1) else: print("Your model is safe") exit(0)
This is a basic example, but you can enhance the script by adding more logic to determine the safety of the model. For instance, you could consider the criticality or number of vulnerabilities and make the safety decision based on that.
Add the script to your CI/CD pipeline#
To include the model scanning script in your CI/CD pipeline, add a new job to your workflow file on GitHub.
name: Giskard CI/CD tutorial on: push: branches: - main jobs: automatic_scan: runs-on: ubuntu-latest steps: - name: checkout repo content uses: actions/checkout@v2 # checkout the repository content to github runner - name: setup python uses: actions/setup-python@v2 with: python-version: '3.8.16' # install the python version needed - run: pip install -r requirements.txt - name: execute test script run: | python scan.py id: test_output
Adding to this file, you will also need a
requirements.txt file that contains the dependencies needed to run your script. In this case, we will need to add
giskard to the file.
The above job will run on every push to the main branch of your repository. It will then run a simple script to scan your model for vulnerabilities, and the job will fail if any vulnerability has been found. You may also want to add a job that runs on pull requests to the main branch, to ensure that no new vulnerabilities are introduced by the pull request.
Using Test Suites
You can also use custom test suites to validate your model. Refer to the Test your ML Model guide, and modify the
scan.py script to run your test suite.