Introduction

Machine Learning systems contain more than application code. Their behavior also depends on data, feature definitions, preprocessing, training configuration, dependency versions, model artifacts, decision thresholds, input schemas, infrastructure, and monitoring policies.

A traditional software CI/CD pipeline validates code and deploys an application build. An ML CI/CD pipeline must additionally validate data contracts, preprocessing logic, model quality, prediction consistency, artifact lineage, serving compatibility, and responsible-use requirements.

Continuous Integration provides rapid evidence about proposed changes. Continuous Delivery prepares an approved release that can be deployed safely. Continuous Deployment automatically promotes a release after all required controls pass.

Learning Objectives

Prerequisites

What Is CI/CD?

Software CI/CD vs ML CI/CD

End-to-End ML CI/CD Architecture

Source-Control Workflow

What Should Be Versioned?

Continuous Integration Workflow

ML-Specific CI Tests

Model-Quality Gate

def evaluate_release_gate(
    candidate_metrics,
    baseline_metrics,
    required_minimums,
    required_maximums=None
):
    failures = []

    required_maximums = (
        required_maximums
        or {}
    )

    for metric_name, minimum_value in (
        required_minimums.items()
    ):
        candidate_value = (
            candidate_metrics.get(
                metric_name
            )
        )

        if candidate_value is None:

            failures.append(
                f"Missing metric: {metric_name}"
            )

        elif candidate_value < minimum_value:

            failures.append(
                f"{metric_name} is below "
                "the required minimum."
            )

    for metric_name, maximum_value in (
        required_maximums.items()
    ):
        candidate_value = (
            candidate_metrics.get(
                metric_name
            )
        )

        if candidate_value is None:

            failures.append(
                f"Missing metric: {metric_name}"
            )

        elif candidate_value > maximum_value:

            failures.append(
                f"{metric_name} exceeds "
                "the permitted maximum."
            )

    baseline_f1 = baseline_metrics.get(
        "f1"
    )

    candidate_f1 = candidate_metrics.get(
        "f1"
    )

    if (
        baseline_f1 is not None
        and candidate_f1 is not None
        and candidate_f1 < baseline_f1
    ):
        failures.append(
            "Candidate F1 is below baseline F1."
        )

    return {
        "passed": len(
            failures
        ) == 0,
        "failures": failures
    }

Build Immutable Release Artifacts

docker build \
  --tag customer-churn-api:1.0.0 \
  .

docker image inspect \
  customer-churn-api:1.0.0

Continuous Delivery Workflow

Environment Promotion

Approval Gates

Deployment Strategies

Post-Deployment Validation

import numpy as np
import requests


def verify_deployed_prediction(
    endpoint_url,
    request_payload,
    expected_probability,
    authorization_header
):
    response = requests.post(
        endpoint_url,
        json=request_payload,
        headers={
            "Authorization": (
                authorization_header
            )
        },
        timeout=10
    )

    response.raise_for_status()

    response_payload = response.json()

    actual_probability = float(
        response_payload[
            "positive_class_probability"
        ]
    )

    np.testing.assert_allclose(
        actual_probability,
        expected_probability,
        rtol=1e-8,
        atol=1e-10
    )

    return response_payload

GitHub Actions CI Example

name: Validate ML Project

on:
  pull_request:
  push:
    branches:
      - main

permissions:
  contents: read

jobs:
  validate:
    runs-on: ubuntu-latest

    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          python -m pip install -r requirements.txt

      - name: Run static checks
        run: |
          python -m compileall app src tests

      - name: Run unit and integration tests
        run: |
          python -m pytest -v

      - name: Validate data contract
        run: |
          python -m src.validate_data \
            --configuration config/ci.json

      - name: Run training smoke test
        run: |
          python -m src.train \
            --configuration config/smoke-test.json

      - name: Validate generated artifact
        run: |
          python -m src.validate_artifact \
            --artifact pipeline-output/model.joblib

Container Build and Test Job

jobs:
  build-container:
    needs:
      - validate

    runs-on: ubuntu-latest

    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Build image
        run: |
          docker build \
            --tag customer-churn-api:test \
            .

      - name: Start container
        run: |
          docker run \
            --detach \
            --name customer-churn-api-test \
            --publish 8000:8000 \
            customer-churn-api:test

      - name: Wait for readiness
        run: |
          for attempt in $(seq 1 30); do
            if curl --fail \
              http://127.0.0.1:8000/ready; then
              exit 0
            fi
            sleep 2
          done

          docker logs customer-churn-api-test
          exit 1

      - name: Run smoke tests
        run: |
          python -m tests.container_smoke_test

      - name: Show logs on failure
        if: failure()
        run: |
          docker logs customer-churn-api-test

      - name: Remove test container
        if: always()
        run: |
          docker rm \
            --force \
            customer-churn-api-test

Azure Pipelines Example

trigger:
  branches:
    include:
      - main

pr:
  branches:
    include:
      - main

pool:
  vmImage: ubuntu-latest

stages:
  - stage: Validate
    displayName: Validate ML project

    jobs:
      - job: Test
        displayName: Run automated checks

        steps:
          - task: UsePythonVersion@0
            inputs:
              versionSpec: "3.12"

          - script: |
              python -m pip install --upgrade pip
              python -m pip install -r requirements.txt
            displayName: Install dependencies

          - script: |
              python -m pytest -v
            displayName: Run tests

          - script: |
              python -m src.validate_data \
                --configuration config/ci.json
            displayName: Validate data contract

          - script: |
              python -m src.train \
                --configuration config/smoke-test.json
            displayName: Run training smoke test

  - stage: Package
    displayName: Package release
    dependsOn:
      - Validate
    condition: succeeded()

    jobs:
      - job: Build
        steps:
          - script: |
              docker build \
                --tag customer-churn-api:$(Build.BuildId) \
                .
            displayName: Build immutable image

Continuous Training

Continuous Training automates a repeatable training pipeline when an approved trigger occurs. The output should be a candidate, not an automatically trusted production model.

Identity and Secret Management

Supply-Chain Security

Monitor the Release

Rollback

Rollback restores a previous approved combination of application, model, preprocessing, schema, threshold, environment, and infrastructure configuration.

GitOps for ML

GitOps stores declarative deployment configuration in version control and uses an authorized controller to reconcile the target environment with the approved repository state.

Responsible CI/CD for ML

Common CI/CD Mistakes

Recommended ML CI/CD Workflow

Hands-On Practice

Knowledge Check

CI/CD Readiness Checklist

Lesson Summary