Azure Pipelines Setup Guide for Codity.ai

This guide covers all the important configuration points to ensure Codity.ai properly detects and monitors your Azure DevOps CI pipelines.

Critical Configuration Points

1. Pipeline Name

Must contain one of these keywords: test, ci, or build
Avoid generic names like "codity", "pipeline", "main"
Why: Codity detects CI pipelines by checking if the pipeline name contains these keywords

Examples:

codity-ci
test-pipeline
build-and-test
codity (won't be detected)
main-pipeline (won't be detected)

2. Job Display Name

Must contain keywords: test, python, pytest, go, ruby, javascript, typescript, jest, rspec, junit
Avoid default names like "Job", "__default"
Why: The auto-fixer identifies test jobs by their display names to fetch logs

Examples:

Run Python Tests
Execute pytest
Run Go Tests
Job (logs won't be retrieved)
Build (skipped as non-test job)

3. Working Directory

Set workingDirectory if your project files are in a subdirectory
Why: Azure DevOps runs from /home/vsts/work/1/s/ but your package.json, go.mod, etc. might be elsewhere
How: Use workingDirectory: '$(System.DefaultWorkingDirectory)/your-subdir'

4. PYTHONPATH Configuration (Python Only)

Always set PYTHONPATH for Python projects
Why: Azure DevOps runs tests from /home/vsts/work/1/s/ which can break imports
How: Use export PYTHONPATH="${PYTHONPATH}:$(System.DefaultWorkingDirectory)"

5. PR Triggers

Include pr: section to trigger on pull requests
Why: Without this, CI won't run on PRs and Codity can't monitor test results

6. Verbose Output

Use -v or --verbose flags in test commands
Why: Provides detailed logs for better error analysis and auto-fixing

Sample azure-pipelines.yml Files

Python with pytest

# Pipeline name MUST contain 'test', 'ci', or 'build'
name: python-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
# Job displayName MUST contain test-related keywords
- job: PythonTests
  displayName: 'Run Python Tests'
  steps:
  - task: UsePythonVersion@0
    displayName: 'Use Python 3.11'
    inputs:
      versionSpec: '3.11'

  - script: |
      pip install pytest
    displayName: 'Install dependencies'
    # Set working directory if package.json is in a subdirectory
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/backend' if needed

  # CRITICAL: Set PYTHONPATH to fix import issues
  - script: |
      export PYTHONPATH="${PYTHONPATH}:$(System.DefaultWorkingDirectory)"
      pytest -v
    displayName: 'Run pytest tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/backend' if needed

Go with testing

name: go-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: GoTests
  displayName: 'Run Go Tests'
  steps:
  - task: GoTool@0
    displayName: 'Install Go 1.21'
    inputs:
      version: '1.21'

  - script: |
      go test -v ./...
    displayName: 'Execute Go tests'
    # Set working directory if go.mod is in a subdirectory
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/services' if needed

JavaScript/TypeScript with Jest

name: javascript-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: JestTests
  displayName: 'Run Jest Tests'
  steps:
  - task: NodeTool@0
    displayName: 'Install Node.js'
    inputs:
      versionSpec: '18.x'

  # CRITICAL: Set working directory where package.json exists
  - script: |
      npm install
    displayName: 'Install dependencies'
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/frontend' if needed

  - script: |
      npm test -- --verbose
    displayName: 'Run Jest tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/frontend' if needed

Ruby with RSpec

name: ruby-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: RSpecTests
  displayName: 'Run RSpec Tests'
  steps:
  - task: UseRubyVersion@0
    displayName: 'Use Ruby 3.2'
    inputs:
      versionSpec: '3.2'

  - script: |
      gem install bundler
      bundle install
    displayName: 'Install dependencies'
    # Set working directory if Gemfile is in a subdirectory
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/api' if needed

  - script: |
      bundle exec rspec --format documentation
    displayName: 'Run RSpec tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/api' if needed

Java with JUnit/Gradle

name: java-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: JavaTests
  displayName: 'Run JUnit Tests'
  steps:
  - task: JavaToolInstaller@0
    displayName: 'Install Java 17'
    inputs:
      versionSpec: '17'
      jdkArchitectureOption: 'x64'
      jdkSourceOption: 'PreInstalled'

  - script: |
      chmod +x gradlew
      ./gradlew test --info
    displayName: 'Run Gradle tests'
    # Set working directory if build.gradle is in a subdirectory
    workingDirectory: '$(System.DefaultWorkingDirectory)'  # Change to '/service' if needed

Multi-language Monorepo

name: monorepo-test-ci

trigger:
  branches:
    include:
    - '*'

pr:
  branches:
    include:
    - '*'

pool:
  vmImage: 'ubuntu-latest'

jobs:
# Backend (Python)
- job: BackendTests
  displayName: 'Run Python Backend Tests'
  steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.11'
  - script: |
      export PYTHONPATH="${PYTHONPATH}:$(System.DefaultWorkingDirectory)/backend"
      pip install pytest
      pytest -v
    displayName: 'Run backend tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)/backend'

# Frontend (JavaScript)
- job: FrontendTests
  displayName: 'Run JavaScript Frontend Tests'
  steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '18.x'
  - script: |
      npm install
      npm test -- --verbose
    displayName: 'Run frontend tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)/frontend'

# Services (Go)
- job: ServicesTests
  displayName: 'Run Go Services Tests'
  steps:
  - task: GoTool@0
    inputs:
      version: '1.21'
  - script: |
      go test -v ./...
    displayName: 'Run services tests'
    workingDirectory: '$(System.DefaultWorkingDirectory)/services'

Project Structure Detection

Azure DevOps looks for these files to detect your project type:

Language	Files Detected	Common Locations
Python	`requirements.txt`, `setup.py`, `pyproject.toml`, `pytest.ini`	Root, `/backend`, `/api`
JavaScript/TypeScript	`package.json`, `tsconfig.json`, `jest.config.js`	Root, `/frontend`, `/web`
Go	`go.mod`, `go.sum`	Root, `/services`, `/cmd`
Ruby	`Gemfile`, `Gemfile.lock`, `.gemspec`, `Rakefile`	Root, `/app`, `/api`
Java	`build.gradle`, `pom.xml`, `gradlew`, `mvnw`	Root, `/service`, `/app`

How to Set Working Directory

Example 1: Frontend in subdirectory

my-repo/
├── frontend/
│   ├── package.json      ← Your package.json is here
│   ├── src/
│   └── tests/
└── backend/

Solution:

- script: |
    npm install
  workingDirectory: '$(System.DefaultWorkingDirectory)/frontend'

Example 2: Multiple services

my-repo/
├── services/
│   ├── api/
│   │   └── go.mod        ← Your go.mod is here
│   └── worker/
│       └── go.mod

Solution:

- script: |
    go test -v ./...
  workingDirectory: '$(System.DefaultWorkingDirectory)/services/api'

Example 3: Root-level project

my-repo/
├── package.json          ← Your package.json is in root
├── src/
└── tests/

Solution:

- script: |
    npm install
  workingDirectory: '$(System.DefaultWorkingDirectory)'  # Or just omit this line

Common Issues and Solutions

Issue: "ENOENT: no such file or directory, open '/home/vsts/work/1/s/package.json'"

Cause: npm/pip/bundle command running in wrong directory Solution: Add workingDirectory: '$(System.DefaultWorkingDirectory)/your-subdir' to the script step

Issue: "ModuleNotFoundError" in Python tests

Cause: Python can't find modules due to path issues Solution:

Set PYTHONPATH: export PYTHONPATH="${PYTHONPATH}:$(System.DefaultWorkingDirectory)/your-subdir"
Set workingDirectory: workingDirectory: '$(System.DefaultWorkingDirectory)/your-subdir'

Issue: "go.mod not found"

Cause: Go commands running in wrong directory Solution: Add workingDirectory: '$(System.DefaultWorkingDirectory)/your-go-dir'

Issue: "Could not find Gemfile"

Cause: Bundle commands running in wrong directory Solution: Add workingDirectory: '$(System.DefaultWorkingDirectory)/your-ruby-dir'

Issue: "CI detection result: has_ci=False"

Solution: Rename your pipeline to include test, ci, or build in the name

Issue: "No CI logs could be fetched"

Solution: Update job displayName to include keywords like test, python, pytest, etc.

Issue: "No CI pipelines found for PR"

Solution: Ensure pr: section exists in your YAML with branch triggers

Checklist

Before committing your azure-pipelines.yml:

Pipeline name contains test, ci, or build
Job displayName contains test-related keywords
pr: trigger section is included
For Python: PYTHONPATH is set in test script
workingDirectory set correctly if project files are in subdirectory
Test commands use verbose flags (-v, --verbose)
Verified package.json/go.mod/Gemfile/build.gradle location
Tested that dependencies install correctly

Quick Diagnosis

Run these commands locally to find where your project files are:

# Find package.json (JavaScript/TypeScript)
find . -name "package.json"

# Find go.mod (Go)
find . -name "go.mod"

# Find Gemfile (Ruby)
find . -name "Gemfile"

# Find build.gradle or pom.xml (Java)
find . -name "build.gradle" -o -name "pom.xml"

# Find requirements.txt (Python)
find . -name "requirements.txt"

Then set workingDirectory to match that location.

How Codity.ai Detects Your Pipeline

Pipeline Discovery: Checks if any pipeline name contains test, ci, or build
Build Detection: Looks for builds on your PR's source branch
Commit Matching: Compares build's commit SHA with PR's head commit
Job Filtering: Identifies test jobs by checking displayName for test keywords
Log Retrieval: Fetches logs from identified test jobs
Auto-Fix: Analyzes failures and automatically fixes tests (up to 3 attempts)

Testing Your Configuration

Update azure-pipelines.yml with recommended settings
Commit and push to Azure DevOps
Create a test PR
Check Codity logs for:
```
[INFO] CI detection result: has_ci=True
[INFO] Found 1 build(s) for PR #X
[INFO] Processing build #Y: status=completed
[INFO] Job found: 'Run Python Tests'
[INFO] Fetching logs for build #Y
[INFO] Successfully fetched logs
```
Check Azure DevOps pipeline logs for any path errors

Language-Specific Notes

Python

Always set PYTHONPATH to include your source directory
If using src/ layout, set: PYTHONPATH="${PYTHONPATH}:$(System.DefaultWorkingDirectory)/src"
Common working directories: /, /backend, /api, /src

JavaScript/TypeScript

Must have package.json in working directory
If using monorepo, set workingDirectory for each app
Common working directories: /, /frontend, /web, /client

Go

Must have go.mod in working directory
Use ./... to run all tests in subdirectories
Common working directories: /, /services, /cmd, /pkg

Ruby

Must have Gemfile in working directory
Run bundle install before tests
Common working directories: /, /app, /api, /lib

Java

Must have build.gradle or pom.xml in working directory
Make gradlew executable: chmod +x gradlew
Common working directories: /, /service, /app, /backend

codity doctor

Usage

What It Checks

Output