🚀 Execute your test suite in your CI/CD pipeline

Warning

First you’ll need to know how to run Giskard’s scan function see Scan your ML Model

By integrating Giskard into your CI/CD pipeline, you can run the scan function on every commit to your repository, ensuring that new models don’t introduce new vulnerabilities.

Create a script to scan your model

By referring to the Scan your ML Model guide, you can create a script that will scan your model. We will refer to this python script as scan.py in this guide.

from giskard import demo, Model, Dataset, scan

model, df = demo.titanic()

# By following previous user guides, you will be shown how to use your own model and dataset.
# For example purposes, we will use the demo model and dataset.
wrapped_model = Model(model=model, model_type="classification")
wrapped_dataset = Dataset(df=df, target="Survived", cat_columns=['Pclass', 'Sex', "SibSp", "Parch", "Embarked"])

scan_results = scan(wrapped_model, wrapped_dataset)
if scan_results.has_vulnerabilities:
    print("Your model has vulnerabilities")
    exit(1)
else:
    print("Your model is safe")
    exit(0)

This is a basic example, but you can enhance the script by adding more logic to determine the safety of the model. For instance, you could consider the criticality or number of vulnerabilities and make the safety decision based on that.

Add the script to your CI/CD pipeline

To include the model scanning script in your CI/CD pipeline, add a new job to your workflow file on GitHub.

name: Giskard CI/CD tutorial

on:
  push:
    branches:
      - main

jobs:
  automatic_scan:
    runs-on: ubuntu-latest
    steps:
      - name: checkout repo content
        uses: actions/checkout@v2 # checkout the repository content to github runner

      - name: setup python
        uses: actions/setup-python@v2
        with:
          python-version: '3.8.16' # install the python version needed

      - run: pip install -r requirements.txt

      - name: execute test script
        run: |
          python scan.py
        id: test_output

Adding to this file, you will also need a requirements.txt file that contains the dependencies needed to run your script. In this case, we will need to add giskard to the file.

requirements.txt

giskard

The above job will run on every push to the main branch of your repository. It will then run a simple script to scan your model for vulnerabilities, and the job will fail if any vulnerability has been found. You may also want to add a job that runs on pull requests to the main branch, to ensure that no new vulnerabilities are introduced by the pull request.

Using Test Suites

You can also use custom test suites to validate your model. Refer to the Test your ML Model guide, and modify the scan.py script to run your test suite.