Azure Pipelines Setup Guide for Codity.ai

This guide covers all the important configuration points to ensure Codity.ai properly detects and monitors your Azure DevOps CI pipelines.

Critical Configuration Points

1. Pipeline Name

• Must contain one of these keywords: test, ci, or build
• Avoid generic names like "codity", "pipeline", "main"
• Why: Codity detects CI pipelines by checking if the pipeline name contains these keywords

Examples:

• codity-ci
• test-pipeline
• build-and-test
• codity (won't be detected)
• main-pipeline (won't be detected)

2. Job Display Name

• Must contain keywords: test, python, pytest, go, ruby, javascript, typescript, jest, rspec, junit
• Avoid default names like "Job", "__default"
• Why: The auto-fixer identifies test jobs by their display names to fetch logs

Examples:

• Run Python Tests
• Execute pytest
• Run Go Tests
• Job (logs won't be retrieved)
• Build (skipped as non-test job)

3. Working Directory

• Set workingDirectory if your project files are in a subdirectory
• Why: Azure DevOps runs from /home/vsts/work/1/s/ but your package.json, go.mod, etc. might be elsewhere
• How: Use workingDirectory: '$(System.DefaultWorkingDirectory)/your-subdir'

4. PYTHONPATH Configuration (Python Only)

• Always set PYTHONPATH for Python projects
• Why: Azure DevOps runs tests from /home/vsts/work/1/s/ which can break imports
• How: Use export PYTHONPATH="${PYTHONPATH}:$(System.DefaultWorkingDirectory)"

5. PR Triggers

• Include pr: section to trigger on pull requests
• Why: Without this, CI won't run on PRs and Codity can't monitor test results

6. Verbose Output

• Use -v or --verbose flags in test commands
• Why: Provides detailed logs for better error analysis and auto-fixing

Sample azure-pipelines.yml Files

Python with pytest

# Pipeline name MUST contain 'test', 'ci', or 'build'
name: python-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
# Job displayName MUST contain test-related keywords
- job: PythonTests
  displayName: 'Run Python Tests'
  steps:
  - task: UsePythonVersion@0
    displayName: 'Use Python 3.11'
    inputs:
      versionSpec: '3.11'

  - script: |
      pip install pytest
    displayName: 'Install dependencies'
    # Set working directory if package.json is in a subdirectory
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/backend' if needed

  # CRITICAL: Set PYTHONPATH to fix import issues
  - script: |
      export PYTHONPATH="${PYTHONPATH}:$(System.DefaultWorkingDirectory)"
      pytest -v
    displayName: 'Run pytest tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/backend' if needed

Go with testing

name: go-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: GoTests
  displayName: 'Run Go Tests'
  steps:
  - task: GoTool@0
    displayName: 'Install Go 1.21'
    inputs:
      version: '1.21'

  - script: |
      go test -v ./...
    displayName: 'Execute Go tests'
    # Set working directory if go.mod is in a subdirectory
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/services' if needed

JavaScript/TypeScript with Jest

name: javascript-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: JestTests
  displayName: 'Run Jest Tests'
  steps:
  - task: NodeTool@0
    displayName: 'Install Node.js'
    inputs:
      versionSpec: '18.x'

  # CRITICAL: Set working directory where package.json exists
  - script: |
      npm install
    displayName: 'Install dependencies'
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/frontend' if needed

  - script: |
      npm test -- --verbose
    displayName: 'Run Jest tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/frontend' if needed

Ruby with RSpec

name: ruby-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: RSpecTests
  displayName: 'Run RSpec Tests'
  steps:
  - task: UseRubyVersion@0
    displayName: 'Use Ruby 3.2'
    inputs:
      versionSpec: '3.2'

  - script: |
      gem install bundler
      bundle install
    displayName: 'Install dependencies'
    # Set working directory if Gemfile is in a subdirectory
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/api' if needed

  - script: |
      bundle exec rspec --format documentation
    displayName: 'Run RSpec tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/api' if needed

Java with JUnit/Gradle

name: java-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: JavaTests
  displayName: 'Run JUnit Tests'
  steps:
  - task: JavaToolInstaller@0
    displayName: 'Install Java 17'
    inputs:
      versionSpec: '17'
      jdkArchitectureOption: 'x64'
      jdkSourceOption: 'PreInstalled'

  - script: |
      chmod +x gradlew
      ./gradlew test --info
    displayName: 'Run Gradle tests'
    # Set working directory if build.gradle is in a subdirectory
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/service' if needed

Multi-language Monorepo

name: monorepo-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
# Backend (Python)
- job: BackendTests
  displayName: 'Run Python Backend Tests'
  steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.11'
  - script: |
      export PYTHONPATH="${PYTHONPATH}:$(System.DefaultWorkingDirectory)/backend"
      pip install pytest
      pytest -v
    displayName: 'Run backend tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)/backend'

# Frontend (JavaScript)
- job: FrontendTests
  displayName: 'Run JavaScript Frontend Tests'
  steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '18.x'
  - script: |
      npm install
      npm test -- --verbose
    displayName: 'Run frontend tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)/frontend'

# Services (Go)
- job: ServicesTests
  displayName: 'Run Go Services Tests'
  steps:
  - task: GoTool@0
    inputs:
      version: '1.21'
  - script: |
      go test -v ./...
    displayName: 'Run services tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)/services'

Project Structure Detection

Azure DevOps looks for these files to detect your project type:

Language	Files Detected	Common Locations
Python	requirements.txt, setup.py, pyproject.toml, pytest.ini	Root, /backend, /api
JavaScript/TypeScript	package.json, tsconfig.json, jest.config.js	Root, /frontend, /web
Go	go.mod, go.sum	Root, /services, /cmd
Ruby	Gemfile, Gemfile.lock, .gemspec, Rakefile	Root, /app, /api
Java	build.gradle, pom.xml, gradlew, mvnw	Root, /service, /app

How to Set Working Directory

Example 1: Frontend in subdirectory

my-repo/
├── frontend/
│   ├── package.json      ← Your package.json is here
│   ├── src/
│   └── tests/
└── backend/

Solution:

- script: |
    npm install
  workingDirectory: '$(System.DefaultWorkingDirectory)/frontend'

Example 2: Multiple services

my-repo/
├── services/
│   ├── api/
│   │   └── go.mod        ← Your go.mod is here
│   └── worker/
│       └── go.mod

Solution:

- script: |
    go test -v ./...
  workingDirectory: '$(System.DefaultWorkingDirectory)/services/api'

Example 3: Root-level project

my-repo/
├── package.json          ← Your package.json is in root
├── src/
└── tests/

Solution:

- script: |
    npm install
  workingDirectory: '$(System.DefaultWorkingDirectory)'  # Or just omit this line

Common Issues and Solutions

Issue: "ENOENT: no such file or directory, open '/home/vsts/work/1/s/package.json'"

Cause: npm/pip/bundle command running in wrong directory Solution: Add workingDirectory: '$(System.DefaultWorkingDirectory)/your-subdir' to the script step

Issue: "ModuleNotFoundError" in Python tests

Cause: Python can't find modules due to path issues Solution:

1. Set PYTHONPATH: export PYTHONPATH="${PYTHONPATH}:$(System.DefaultWorkingDirectory)/your-subdir"
2. Set workingDirectory: workingDirectory: '$(System.DefaultWorkingDirectory)/your-subdir'

Issue: "go.mod not found"

Cause: Go commands running in wrong directory Solution: Add workingDirectory: '$(System.DefaultWorkingDirectory)/your-go-dir'

Issue: "Could not find Gemfile"

Cause: Bundle commands running in wrong directory Solution: Add workingDirectory: '$(System.DefaultWorkingDirectory)/your-ruby-dir'

Issue: "CI detection result: has_ci=False"

Solution: Rename your pipeline to include test, ci, or build in the name

Issue: "No CI logs could be fetched"

Solution: Update job displayName to include keywords like test, python, pytest, etc.

Issue: "No CI pipelines found for PR"

Solution: Ensure pr: section exists in your YAML with branch triggers

Checklist

Before committing your azure-pipelines.yml:

• Pipeline name contains test, ci, or build
• Job displayName contains test-related keywords
• pr: trigger section is included
• For Python: PYTHONPATH is set in test script
• workingDirectory set correctly if project files are in subdirectory
• Test commands use verbose flags (-v, --verbose)
• Verified package.json/go.mod/Gemfile/build.gradle location
• Tested that dependencies install correctly

Quick Diagnosis

Run these commands locally to find where your project files are:

# Find package.json (JavaScript/TypeScript)
find . -name "package.json"

# Find go.mod (Go)
find . -name "go.mod"

# Find Gemfile (Ruby)
find . -name "Gemfile"

# Find build.gradle or pom.xml (Java)
find . -name "build.gradle" -o -name "pom.xml"

# Find requirements.txt (Python)
find . -name "requirements.txt"

Then set workingDirectory to match that location.

How Codity.ai Detects Your Pipeline

1. Pipeline Discovery: Checks if any pipeline name contains test, ci, or build
2. Build Detection: Looks for builds on your PR's source branch
3. Commit Matching: Compares build's commit SHA with PR's head commit
4. Job Filtering: Identifies test jobs by checking displayName for test keywords
5. Log Retrieval: Fetches logs from identified test jobs
6. Auto-Fix: Analyzes failures and automatically fixes tests (up to 3 attempts)

Testing Your Configuration

1. Update azure-pipelines.yml with recommended settings
2. Commit and push to Azure DevOps
3. Create a test PR
4. Check Codity logs for:

   ```

   [INFO] CI detection result: has_ci=True

   [INFO] Found 1 build(s) for PR #X

   [INFO] Processing build #Y: status=completed

   [INFO] Job found: 'Run Python Tests'

   [INFO] Fetching logs for build #Y

   [INFO] Successfully fetched logs

   ```

5. Check Azure DevOps pipeline logs for any path errors

Language-Specific Notes

Python

• Always set PYTHONPATH to include your source directory
• If using src/ layout, set: PYTHONPATH="${PYTHONPATH}:$(System.DefaultWorkingDirectory)/src"
• Common working directories: /, /backend, /api, /src

JavaScript/TypeScript

• Must have package.json in working directory
• If using monorepo, set workingDirectory for each app
• Common working directories: /, /frontend, /web, /client

Go

• Must have go.mod in working directory
• Use ./... to run all tests in subdirectories
• Common working directories: /, /services, /cmd, /pkg

Ruby

• Must have Gemfile in working directory
• Run bundle install before tests
• Common working directories: /, /app, /api, /lib

Java

• Must have build.gradle or pom.xml in working directory
• Make gradlew executable: chmod +x gradlew
• Common working directories: /, /service, /app, /backend

---