04 · CI/CD Integration¶

Every fixture, marker, and report format from Levels 1–3 exists to be run somewhere other than your laptop. This module wires a pytest suite into GitHub Actions: install, run, capture JUnit XML, publish an HTML report, and fail the pipeline correctly when tests fail.

1. `--junitxml`: the format CI understands¶

CI dashboards don't parse pytest's terminal output — they parse a structured file. pytest produces one natively:

pytest -v --junitxml=report.xml

test_web_form.py::test_title PASSED
test_web_form.py::test_text_input_and_submit PASSED
test_web_form.py::test_auto_waiting_for_hidden_element PASSED
3 passed in 3.27s

<?xml version="1.0" encoding="utf-8"?>
<testsuites name="pytest tests">
  <testsuite name="pytest" errors="0" failures="0" skipped="0" tests="3" time="3.270">
    <testcase classname="test_web_form" name="test_title" time="1.128" />
    <testcase classname="test_web_form" name="test_text_input_and_submit" time="0.329" />
    <testcase classname="test_web_form" name="test_auto_waiting_for_hidden_element" time="1.685" />
  </testsuite>
</testsuites>

That XML was generated by an actual pytest run against the Playwright tests from Module 1. GitHub Actions, GitLab CI, and Jenkins all have built-in test reporters that read exactly this format to render a pass/fail table in the pipeline UI, no third-party plugin required on the CI side.

2. A minimal GitHub Actions workflow¶

# .github/workflows/tests.yml
name: Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          playwright install --with-deps chromium

      - name: Run tests
        run: pytest -v --junitxml=report.xml

      - name: Publish test results
        if: always()
        uses: dorny/test-reporter@v1
        with:
          name: pytest results
          path: report.xml
          reporter: java-junit

Two details matter more than they look:

playwright install --with-deps chromium — the --with-deps flag also installs the OS-level shared libraries Chromium needs on a bare Ubuntu runner. Without it you get a working local run and a mysteriously crashing CI run, the classic "works on my machine" gap this whole module exists to close.
if: always() on the reporting step — without it, a failing test suite causes the previous step to fail, which by default skips every step after it, including the one that would show you why it failed in the UI.

3. Failing the build for the right reason¶

pytest's own exit code already does the right thing — nonzero on any failure, zero only if everything passed or was explicitly skipped/xfailed. You don't need custom logic to "fail the pipeline"; you need to make sure nothing swallows that exit code. This is the most common CI mistake:

# WRONG — the shell always exits 0 because `| tee` is what's checked, not pytest
- run: pytest -v | tee output.log

# RIGHT — pytest's own exit code is what fails the job
- run: pytest -v --junitxml=report.xml

If you do need to post-process output, use set -o pipefail in a bash step, or write two separate steps instead of piping.

4. Caching to keep CI fast¶

Playwright's browser download in Module 1 took real wall-clock time even on a fast connection; on every CI run without caching, that's minutes added to every single push.

      - name: Cache Playwright browsers
        uses: actions/cache@v4
        with:
          path: ~/.cache/ms-playwright
          key: playwright-${{ runner.os }}-${{ hashFiles('requirements.txt') }}

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          playwright install --with-deps chromium

Playwright's own install step is smart enough to skip the download if the cached binary is already present and matches the version pinned in requirements.txt.

5. Splitting fast and slow suites¶

Reuse the markers from Level 1 Module 7 (@pytest.mark.smoke, @pytest.mark.regression) to run a fast subset on every push and the full suite on a schedule or before merge:

      - name: Smoke tests (every push)
        run: pytest -m smoke -v --junitxml=smoke.xml

      - name: Full regression (main branch only)
        if: github.ref == 'refs/heads/main'
        run: pytest -m "not smoke" -v --junitxml=regression.xml

6. Testing-specific traps¶

Trap 1 — tests that pass locally, hang forever in CI. A test waiting on a real network resource with no explicit timeout will pass locally (fast network, warm cache) and hang until the CI job's global timeout kills the entire job, not just the one test. Always set explicit per-test timeouts (pytest-timeout's @pytest.mark.timeout(30)) for anything that touches the network.

Trap 2 — headless-only bugs. A UI test can pass with headless=False locally and fail headlessly in CI because a browser dialog, viewport size, or font-rendering difference exposes a real layout bug. Always develop against headless=True locally too — Module 1 confirmed this environment runs Playwright headlessly with no code changes required, so there's no excuse to develop against a different mode than CI runs.

Trap 3 — secrets and environment parity. A test suite that reads os.environ["API_KEY"] locally via a .env file will KeyError in CI unless the same variable is set as a repository secret and explicitly passed into the job's env: block. This is a deployment gap, not a test bug, but it surfaces as a test failure and gets debugged as one.

Trap 4 — non-deterministic test order masking real bugs. pytest-xdist (Level 2 Module 8) runs tests in parallel and not necessarily in file order. A suite that only passes in a specific order has a hidden shared-state bug — CI parallelism is often what first exposes it, which is a feature of CI, not a flaw in it.

How It Actually Works¶

A CI pipeline's "job" is, underneath the YAML, a fresh container or VM: your test runner's exit code (see Level 1 — pytest's testsfailed counter driving process exit status) is the only signal the CI system's job scheduler actually understands. Every green checkmark or red X in a PR check ultimately traces back to that single process exit code being 0 or non-zero — the YAML config's "steps" are just a sequence of shell commands executed inside that ephemeral environment, and the pipeline stops (marks the job failed) the instant any step's process returns non-zero, unless it's explicitly marked to continue on error.

Build matrices (running the same job across Python 3.9/3.10/3.11, or multiple OSes) work by the CI scheduler templating your job definition once per matrix cell and dispatching each as an independent, parallel job — there's no shared state between matrix cells; each gets its own fresh container/VM, its own dependency install, its own full test run. This is why a matrix build's total wall-clock time is close to the slowest single cell's time (not the sum of all cells) when the CI provider has enough parallel runners available, but consumes CI-minutes proportional to the sum across cells — the parallelism is real, but the compute cost isn't free.

Caching dependency installs between runs (actions/cache, pip's wheel cache) works by hashing your lock file/requirements file and using that hash as a cache key —a cache hit skips the network-bound pip install resolution and download entirely and just restores a previously-saved site-packages directory, which is the concrete reason a well-cached pipeline's dependency step can drop from minutes to seconds.

Cheat sheet¶

Need	Command / config
Machine-readable results	`pytest --junitxml=report.xml`
Install Playwright in CI	`playwright install --with-deps chromium`
Don't swallow the exit code	avoid bare `\\| tee`; use `pipefail` or separate steps
Cache slow downloads	`actions/cache@v4` on `~/.cache/ms-playwright`
Fast subset every push	`pytest -m smoke`
Full suite before merge	`pytest -m "not smoke"` on `main`
Show results even on failure	`if: always()` on the reporting step
Kill hanging tests	`pytest-timeout`, `@pytest.mark.timeout(30)`

Exercise¶

Write a .github/workflows/tests.yml for a repo containing pytest + Playwright tests: checkout, setup-python, install deps, install Chromium with --with-deps, run pytest --junitxml=report.xml.
Add browser-binary caching keyed on your requirements.txt hash and explain, in a comment, what happens on a cache hit versus a cache miss.
Split the workflow into a smoke job that runs on every push and a regression job that only runs when the branch is main.
Deliberately introduce the pytest -v | tee output.log mistake from section 3 into a failing test suite, run it locally with echo $? afterward, and record the exit code you get versus the exit code from running pytest -v directly.
Add pytest-timeout to one intentionally-hanging test (time.sleep(120)), set @pytest.mark.timeout(5), and capture the exact failure message pytest produces when the timeout fires.