CI / CD

Run MCP health checks, conformance suites, and evals in GitHub Actions, GitLab CI, and other CI environments

Run mcpjam in CI to catch MCP server regressions on every push. The examples below cover GitHub Actions and GitLab CI, but the same commands work in any CI environment.

GitHub Actions

Authentication

There are three ways to authenticate in CI, depending on your server setup.

Option 1: Headless OAuth login

Best when your server supports OAuth with auto-consent (no interactive login page). The workflow obtains a fresh access token on every run.

Secrets needed:

Secret	Description
MCP_SERVER_URL	Your MCP server URL

name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: OAuth login (headless)
        run: |
          set -euo pipefail
          npx -y @mcpjam/cli@latest oauth login \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --protocol-version 2025-11-25 \
            --registration dcr \
            --auth-mode headless \
            --format json > /tmp/oauth-result.json
          TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
          rm -f /tmp/oauth-result.json
          if [ -z "$TOKEN" ]; then
            echo "::error::OAuth login did not return an access token"
            exit 1
          fi
          echo "::add-mask::$TOKEN"
          echo "MCP_TOKEN=$TOKEN" >> "$GITHUB_ENV"

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json

Option 2: Refresh token

Best when you already have a refresh token from a previous oauth login. Refresh tokens are long-lived and safe to store as secrets. The CLI handles the token exchange automatically.

Secrets needed:

Secret	Description
MCP_SERVER_URL	Your MCP server URL
MCP_REFRESH_TOKEN	OAuth refresh token from a previous login
MCP_CLIENT_ID	OAuth client ID (required with refresh tokens)
MCP_CLIENT_SECRET	OAuth client secret (if the client is confidential)

name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: |
          npx -y @mcpjam/cli@latest server doctor \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --refresh-token ${{ secrets.MCP_REFRESH_TOKEN }} \
            --client-id ${{ secrets.MCP_CLIENT_ID }} \
            --client-secret ${{ secrets.MCP_CLIENT_SECRET }} \
            --format json

To get a refresh token, run mcpjam oauth login locally with --format json and grab .credentials.refreshToken from the output.

Option 3: Static API key

Best when your server uses a non-expiring API key instead of OAuth.

Secrets needed:

Secret	Description
MCP_SERVER_URL	Your MCP server URL
MCP_API_KEY	Static API key

name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token ${{ secrets.MCP_API_KEY }} --format json

Option 4: No auth

Some servers don’t require authentication at all.

Secrets needed:

Secret	Description
MCP_SERVER_URL	Your MCP server URL

name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --format json

Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes (renamed parameters, changed descriptions, removed tools).

      - name: Snapshot before
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > before.json

      # your deploy step here

      - name: Snapshot after
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > after.json

      - name: Diff
        run: diff <(jq -S . before.json) <(jq -S . after.json)

OAuth conformance suite

Run the full registration x protocol version x auth mode matrix from a config file and output JUnit XML for test reporters.

      - name: OAuth conformance
        run: |
          npx -y @mcpjam/cli@latest oauth conformance-suite \
            --config ./oauth-matrix.json \
            --reporter junit-xml > report.xml

      - name: Upload test report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: oauth-conformance
          path: report.xml

See OAuth Conformance for details on the config file format.

Protocol conformance suite

Run a repeatable matrix of protocol check selections from a config file and publish JUnit XML.

      - name: Protocol conformance
        run: |
          npx -y @mcpjam/cli@latest protocol conformance-suite \
            --config ./protocol-conformance.json \
            --reporter junit-xml > protocol-report.xml

      - name: Upload protocol report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: protocol-conformance
          path: protocol-report.xml

MCP Apps conformance suite

Run the server-side MCP Apps surface checks from a config file and publish JUnit XML for CI dashboards.

      - name: MCP Apps conformance
        run: |
          npx -y @mcpjam/cli@latest apps conformance-suite \
            --config ./apps-conformance.json \
            --reporter junit-xml > apps-report.xml

      - name: Upload apps report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: apps-conformance
          path: apps-report.xml

Single-run protocol conformance, oauth conformance, and apps conformance also accept --reporter junit-xml when you only need one target/check selection instead of a suite config file.

---

GitLab CI

The same CLI commands work in GitLab CI. The examples below use GitLab CI/CD variables for secrets and .gitlab-ci.yml syntax.

Authentication

Headless OAuth login

mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
  script:
    - |
      npx -y @mcpjam/cli@latest oauth login \
        --url "$MCP_SERVER_URL" \
        --protocol-version 2025-11-25 \
        --registration dcr \
        --auth-mode headless \
        --format json > /tmp/oauth-result.json
      TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
      rm -f /tmp/oauth-result.json
      if [ -z "$TOKEN" ]; then
        echo "OAuth login did not return an access token"
        exit 1
      fi
      export MCP_TOKEN="$TOKEN"
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Refresh token

mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_REFRESH_TOKEN: $MCP_REFRESH_TOKEN
    MCP_CLIENT_ID: $MCP_CLIENT_ID
    MCP_CLIENT_SECRET: $MCP_CLIENT_SECRET
  script:
    - |
      npx -y @mcpjam/cli@latest server doctor \
        --url "$MCP_SERVER_URL" \
        --refresh-token "$MCP_REFRESH_TOKEN" \
        --client-id "$MCP_CLIENT_ID" \
        --client-secret "$MCP_CLIENT_SECRET" \
        --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Static API key

mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_API_KEY: $MCP_API_KEY
  script:
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_API_KEY" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes.

mcp-tool-diff:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_TOKEN: $MCP_TOKEN
  script:
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > before.json
    # your deploy step here
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > after.json
    - jq -S . before.json > /tmp/before-sorted.json
    - jq -S . after.json > /tmp/after-sorted.json
    - diff /tmp/before-sorted.json /tmp/after-sorted.json
    - rm -f /tmp/before-sorted.json /tmp/after-sorted.json

OAuth conformance suite

mcp-oauth-conformance:
  image: node:20
  script:
    - |
      npx -y @mcpjam/cli@latest oauth conformance-suite \
        --config ./oauth-matrix.json \
        --reporter junit-xml > report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml

See OAuth Conformance for details on the config file format.

---

Evals in CI

There are two ways to wire MCPJam evals into a pipeline: trigger a hosted eval run with the CLI, or run evals locally with the SDK and upload the results. Both authenticate with an MCPJam API key (sk_…) from Settings → API keys.

Trigger a hosted eval suite

mcpjam eval run starts an asynchronous run of a suite that lives in your MCPJam project. It prints a runId and returns immediately — the iterations execute on the MCPJam platform, not in your CI job. Check on the run with mcpjam eval status.

Secrets needed:

Secret	Description
MCPJAM_API_KEY	MCPJam API key (sk_…)

      - name: Start eval run
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          npx -y @mcpjam/cli@latest eval run \
            --suite "Nightly regression" \
            --project "My project" \
            --format json > eval-run.json
          echo "Started run $(jq -r '.runId' eval-run.json)"

In human format, eval run prints a View: line after the payload so you can open the run directly from the terminal:

View: https://app.mcpjam.com/evals/suite/<suiteId>/runs/<runId>?project=<projectId>

This line is only emitted in human format — --format json output is unchanged, so scripts that parse the JSON stream are unaffected.

Poll eval status until run.status is terminal (completed, failed, or cancelled), then gate the job on run.result:

      - name: Wait for eval result
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          RUN_ID=$(jq -r '.runId' eval-run.json)
          ATTEMPTS=0
          MAX_ATTEMPTS=60   # ~30 minutes at one poll per 30 s
          while :; do
            npx -y @mcpjam/cli@latest eval status \
              --run "$RUN_ID" \
              --project "My project" \
              --format json > eval-status.json
            STATUS=$(jq -r '.run.status' eval-status.json)
            case "$STATUS" in completed|failed|cancelled) break;; esac
            ATTEMPTS=$((ATTEMPTS + 1))
            if [ "$ATTEMPTS" -ge "$MAX_ATTEMPTS" ]; then
              echo "Timed out waiting for eval run (last status: $STATUS)"
              exit 1
            fi
            sleep 30
          done
          jq '.run.summary' eval-status.json
          [ "$(jq -r '.run.result' eval-status.json)" = "passed" ]

The eval verdict does not control the exit code — eval run and eval status exit non-zero on API, transport, and usage errors instead. The pass/fail verdict lives in the JSON payload (run.result), so the final assertion above is what fails the job. eval status also prints a View: line in human format, identical to the one eval run prints.

Hosted runs execute LLM iterations on the platform and consume your organization’s credits or configured provider keys. See the eval command reference for the rest of the surface (eval cancel, eval iterations, eval trace, eval cases, and more).

Upload SDK eval results

If you instead run evals inside your own CI job with @mcpjam/sdk (EvalTest / EvalSuite), set MCPJAM_API_KEY and results upload automatically to the CI Evals dashboard (pass-rate trends, per-model breakdowns, and a full trace per iteration):

      - name: Run SDK evals
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: npx vitest run evals/

See Save Results to MCPJam for auto-save, the manual reporting APIs, CI metadata (branch, commit SHA, run URL), and artifact upload (JUnit XML, Jest/Vitest JSON).