Sitecore: Creating a UI test pipeline on Azure DevOps

Recently I had to add an Azure pipeline for UI testing within a Sitecore solution. If you are brand new to pipelines as me let's, check what I struggled with to save you some time. 

The scenario is a Sitecore 10.2 solution with the CI/CD process is set up on Azure DevOps and we are using a self-hosted agent. 

In this case it was needed the pipeline was separated from the CI/CD process but depending on this process to get fired. Then it was created a new pipeline triggered by a certain different pipeline from a specific branch once a stage gets completed. Also, there is a constraint that a stage on the UI pipeline only can run if the corresponding stage on the CD pipeline has run already. 

Here is the resulting pipeline and at the end I will list the notes to take in account. 

uitest-pipeline.yml

resources:

  pipelines:

  - pipeline: <<PipelineName>>

    source: <<PipelineNameTriggeringThisOne>>

    trigger: 

      branches:

        - <<BranchNameWherePipelineIsTriggered>>

      stages:        

        - <<StageNameTriggeringThisone>>

variables: 

- template: <<VariablesPath>>

pool: <<AgentPool>>

stages: 

- stage: build_ui_test

  displayName: Building UI Test Project

  jobs: 

  - job: build

    displayName: Build Project

    continueOnError: false

    steps:

    - template: build-steps.yml

- stage: test_dev

  displayName: Test DEV

  variables:

    SITE: $(DEV_SITE)

    STAGE: $(DEVSTAGENAME)

  jobs:

  - job: GetStatus

    displayName: Get Stage State

    steps:

    - template: get-state-steps.yml

  - job: RunTests

    displayName: Run Tests

    dependsOn: GetStatus

    variables:

      STATE: $[ dependencies.GetStatus.outputs['GetStatusStep.State'] ]

    condition: eq(variables['STATE'], 'completed')

    steps:

    - template: build/pipelines/uitest/test-steps.yml

build-steps.yml

steps:

  - task: NuGetToolInstaller@1

    displayName: 'Install NuGet tool'

    inputs:

        versionSpec: '4.x'

        checkLatest: true

  - task: NuGetCommand@2

    displayName: 'Restore NuGet packages'

    inputs:

        command: 'restore'

        restoreSolution: << .sln file PATH>>

        feedsToUse: 'config'

        nugetConfigPath: 'nuget.config'

  - task: MSBuild@1

    displayName: 'Build UI tests project'

    inputs:

        solution: << .csproj FILE PATH >>

        configuration: Release

        clean: true

        createLogFile: true

  - task: PublishPipelineArtifact@1

    inputs:

        targetPath: $(Pipeline.Workspace)

        artifact: ui-tests-build

get-stage-state.ps1

Write-Host "Starting with Stage State Verification"

$stage="$env:STAGE"

Write-Host $stage

$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$env:CONNECTIONTOKEN"))

$buildURL = "https://dev.azure.com/$env:ORGANIZATION/$env:PROJECT/_apis/build/builds?definitions=$env:DEFINITIONID&api-version=6.0"

Write-Host $buildURL

$buildData= Invoke-RestMethod -Uri $buildURL -Headers @{authorization = "Basic $base64AuthInfo"} -Method Get

$buildTimeline= $buildData.value.url | Select-Object -first 1

$timelineURL= "$($buildTimeline)/Timeline"

$timeline= Invoke-RestMethod -Uri $timelineURL -Headers @{authorization = "Basic $base64AuthInfo"} -Method Get

$stageState=$timeline.records| Where-Object name -eq "$stage"|Select-Object state

$result=$stageState.state

Write-Host "State: $($result)"

Write-Host "##vso[task.setvariable variable=State;isOutput=true]$result"

get-state-steps.yml

steps:

      - task: PowerShell@2

        name: GetStatusStep

        displayName: Get Stage State

        inputs:

          filePath: 'build/pipelines/uitest/get-stage-state.ps1'

        env:

          CONNECTIONTOKEN: $(connectionToken)

test-steps.yml

steps:

- task: DownloadPipelineArtifact@2

  inputs:

      artifact: ui-tests-build

      targetPath: $(Pipeline.Workspace)

- task: replacetokens@3

  displayName: "Replace Token"

  inputs:

     rootDirectory: '$(Pipeline.Workspace)/s/$(UI_TEST_PROJECT_PATH)'

     targetFiles: |

        **/*.config

     encoding: 'auto'

     writeBOM: true

     actionOnMissing: 'warn'

     keepToken: true

     tokenPrefix: '$('

     tokenSuffix: ')'

     useLegacyPattern: false

     enableTelemetry: false

- task: VSTest@2

  displayName: Run UI Tests

  continueOnError: true

  inputs:

     testSelector: 'testAssemblies'

     testAssemblyVer2: |

        **\*test*.dll

        !**\*TestAdapter.dll

        !**\obj\**

     searchFolder: $(Pipeline.Workspace)

     rerunFailedTests: true

     rerunFailedThreshold: 90

     failOnMinTestsNotRun: true

     uiTests: true

     testRunTitle: 'UI Tests'

Key Notes

• It is possible to know every detail of the pipelines thanks to the API.
• Treating variables will have different behavior depending on if they are regular ones, secrets or if they come from a different stage
• Check the system variables on pipelines. Lately pipelines manage they own place to store artifacts $(Pipeline.Workspace)
• Artifacts will be downloaded automatically on the same stage they were created but on a different stage you will need to download it.
• Some of the constraints needed to control here could be solve with webhooks. For this scenario it was not possible, but I will be uploading a blog with it. 

Finally, hope these notes help you out and feel free to leave a comment if you have questions.