specflow - 在 DevOps Pipeline 上的 VSTest 任务中跳过所有 Specflow 场景

Question

我已经配置了一个 DevOps 管道来执行我的 SpecFlow Selenium 测试。一切似乎都运行良好，除了在 DevOps 代理上跳过了我的所有测试场景。

当我使用 VS2019 测试资源管理器在本地运行测试时，它们运行没有问题。

我尝试启用系统诊断以查看它们被跳过的原因，但看不到任何可能的原因。

这是 VSTest 任务的 YAML：

steps:
- task: VSTest@2
  displayName: 'VsTest - RetroSpectro.Tests.UI.Specflow.dll'
  inputs:
    testAssemblyVer2: |
     **\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
     !**\*TestAdapter.dll
     !**\obj\**
     !**\bin\**\ref\**
    uiTests: true
    runSettingsFile: RetroSpectro.Tests.UI.Specflow/Serial.runsettings
    pathtoCustomTestAdapters: packages
    runTestsInIsolation: true
    testRunTitle: 'SpecFlow Tests'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    failOnMinTestsNotRun: true
    rerunFailedTests: false

...以及任务执行的输出：

2021-06-25T09:02:15.0841271Z ##[section]Starting: VsTest - RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:15.1180446Z ==============================================================================
2021-06-25T09:02:15.1180820Z Task         : Visual Studio Test
2021-06-25T09:02:15.1181621Z Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2021-06-25T09:02:15.1182585Z Version      : 2.170.1
2021-06-25T09:02:15.1183892Z Author       : Microsoft Corporation
2021-06-25T09:02:15.1184298Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/vstest
2021-06-25T09:02:15.1184712Z ==============================================================================
2021-06-25T09:02:16.6157598Z SystemVssConnection exists true
2021-06-25T09:02:16.7413804Z SystemVssConnection exists true
2021-06-25T09:02:16.9135626Z Running tests using vstest.console.exe runner.
2021-06-25T09:02:16.9136452Z ======================================================
2021-06-25T09:02:16.9138232Z Test selector : Test assemblies
2021-06-25T09:02:16.9139517Z Test filter criteria : null
2021-06-25T09:02:16.9140779Z Search folder : D:\a\1\s
2021-06-25T09:02:16.9937367Z Action when minimum tests threshold not met : fail
2021-06-25T09:02:16.9938318Z Minimum tests expected to be run: 1
2021-06-25T09:02:16.9939834Z VisualStudio version selected for test execution : latest
2021-06-25T09:02:16.9943536Z Attempting to find vstest.console from a visual studio installation with version [16.0,17.0).
2021-06-25T09:02:16.9973962Z Run settings file : D:\a\1\s\RetroSpectro.Tests.UI.Specflow\Serial.runsettings
2021-06-25T09:02:16.9981237Z Run in parallel : false
2021-06-25T09:02:16.9984189Z Run in isolation : true
2021-06-25T09:02:16.9986763Z Path to custom adapters : D:\a\1\s\packages
2021-06-25T09:02:16.9991903Z Other console options : null
2021-06-25T09:02:16.9994735Z Code coverage enabled : false
2021-06-25T09:02:17.0001246Z Diagnostics enabled : false
2021-06-25T09:02:17.0008374Z SystemVssConnection exists true
2021-06-25T09:02:17.0368729Z Run the tests locally using vstest.console.exe
2021-06-25T09:02:17.0369728Z ========================================================
2021-06-25T09:02:17.0464264Z Source filter: **\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll,!**\*TestAdapter.dll,!**\obj\**,!**\bin\**\ref\**
2021-06-25T09:02:22.7152869Z SystemVssConnection exists true
2021-06-25T09:02:22.7157229Z [command]D:\a\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.170.1\Modules\DTAExecutionHost.exe --inputFile D:\a\_temp\input_0dad9d50-d594-11eb-bac6-a972b6c3989e.json
2021-06-25T09:02:22.7158487Z ======================================================
2021-06-25T09:02:22.8592004Z ##########################################################################
2021-06-25T09:02:22.8592761Z DtaExecutionHost version 18.170.30112.1.
2021-06-25T09:02:23.7498265Z Starting TestExecution Model...
2021-06-25T09:02:26.9047814Z Result Attachments will be stored in LogStore
2021-06-25T09:02:26.9297128Z Run Attachments will be stored in LogStore
2021-06-25T09:02:26.9433340Z Result Attachments will be stored in LogStore
2021-06-25T09:02:27.1235011Z Result Attachments will be stored in LogStore
2021-06-25T09:02:27.1235834Z Run Attachments will be stored in LogStore
2021-06-25T09:02:27.1470266Z Provided settings file:
2021-06-25T09:02:27.1485829Z <?xml version="1.0" encoding="utf-8"?>
2021-06-25T09:02:27.1486598Z <RunSettings>
2021-06-25T09:02:27.1487139Z    <SpecRun>
2021-06-25T09:02:27.1488252Z        <Profile>Serial.srprofile</Profile>
2021-06-25T09:02:27.1488944Z        <GenerateFeatureTrait>true</GenerateFeatureTrait>
2021-06-25T09:02:27.1490018Z    </SpecRun>
2021-06-25T09:02:27.1490506Z </RunSettings>
2021-06-25T09:02:27.1785620Z Updated Run Settings:
2021-06-25T09:02:27.1788682Z <RunSettings>
2021-06-25T09:02:27.1788992Z   <SpecRun>
2021-06-25T09:02:27.1789483Z     <Profile>Serial.srprofile</Profile>
2021-06-25T09:02:27.1789897Z     <GenerateFeatureTrait>true</GenerateFeatureTrait>
2021-06-25T09:02:27.1790256Z   </SpecRun>
2021-06-25T09:02:27.1790560Z   <RunConfiguration>
2021-06-25T09:02:27.1790961Z     <BatchSize>1000</BatchSize>
2021-06-25T09:02:27.1791385Z     <ResultsDirectory>D:\a\_temp\TestResults</ResultsDirectory>
2021-06-25T09:02:27.1791975Z   </RunConfiguration>
2021-06-25T09:02:27.1792294Z </RunSettings>
2021-06-25T09:02:27.1979593Z **************** Starting test execution *********************
2021-06-25T09:02:27.2197434Z [command]C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "@D:\a\_temp\y3jxhopis4l.tmp"
2021-06-25T09:02:27.4233968Z Microsoft (R) Test Execution Command Line Tool Version 16.10.0
2021-06-25T09:02:27.4235225Z Copyright (c) Microsoft Corporation.  All rights reserved.
2021-06-25T09:02:27.4413296Z vstest.console.exe "D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll"
2021-06-25T09:02:27.4414588Z /Settings:"D:\a\_temp\kpbetfmovlq.tmp.runsettings"
2021-06-25T09:02:27.4415655Z /InIsolation
2021-06-25T09:02:27.4416622Z /Logger:"trx"
2021-06-25T09:02:27.4417670Z /TestAdapterPath:"D:\a\1\s\packages"
2021-06-25T09:02:29.1247592Z Starting test execution, please wait...
2021-06-25T09:02:30.0040388Z A total of 1 test files matched the specified pattern.
2021-06-25T09:02:33.0429462Z SpecFlow+Runner execution started
2021-06-25T09:02:33.0431615Z SpecFlow+Runner 3.9.7 in Framework net50 in x64 mode execution started
2021-06-25T09:02:33.0433131Z TestAdapter Location: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\TechTalk.SpecRun.VisualStudio.TestAdapter.dll
2021-06-25T09:02:33.0433910Z Current UserName: 'VssAdministrator', MachineName: 'WIN-RNFTU0DQEPC'
2021-06-25T09:02:33.3384985Z SpecRun: running tests in D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.3386699Z Output folder configured to "D:\a\_temp\TestResults" (default)
2021-06-25T09:02:33.3387477Z Profile: RetroSpectro.Tests.UI.Specflow
2021-06-25T09:02:33.4638846Z Log file path: D:\a\_temp\TestResults\RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.log
2021-06-25T09:02:33.4749622Z Checking activation
2021-06-25T09:02:33.4854960Z Activation check completed
2021-06-25T09:02:33.4878329Z Starting test run
2021-06-25T09:02:33.7105584Z Discovering target: Chrome
2021-06-25T09:02:33.7106665Z Discovering tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.8841939Z Discovered tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.8843042Z Discovering target: Edge
2021-06-25T09:02:33.8843914Z Discovering tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9067409Z Discovered tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9068593Z Discovering target: Firefox
2021-06-25T09:02:33.9069400Z Discovering tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9193883Z Discovered tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9195007Z Discovering target: IE
2021-06-25T09:02:33.9196501Z Discovering tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9272484Z Discovered tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9277522Z Discovering target: Safari
2021-06-25T09:02:33.9278613Z Discovering tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:33.9352988Z Discovered tests from assembly: D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\RetroSpectro.Tests.UI.Specflow.dll
2021-06-25T09:02:56.8250888Z Scenario: Follow button is not available in header section on Add Item page in AddEditItemFeature (target: Chrome) -> Skipped on thread #-1
...
<!-- other chrome tests also skipped on thread #-1 -->
...
2021-06-25T09:02:56.8376857Z Scenario: Group description is available in header section on Edit Item page in AddEditItemFeature (target: Edge) -> Skipped on thread #-1
...
<!-- other edge tests also skipped on thread #-1 -->
...
2021-06-25T09:02:56.8518611Z Scenario: Group status is available in header section on Add Item page in AddEditItemFeature (target: Firefox) -> Skipped on thread #-1
...
<!-- other firefox tests also skipped on thread #-1 -->
...
2021-06-25T09:02:56.8587604Z Scenario: Group description is available in header section on Add Item page in AddEditItemFeature (target: IE) -> Skipped on thread #-1
...
<!-- other IE tests also skipped on thread #-1 -->
...
2021-06-25T09:02:56.8648632Z Scenario: Title label is displayed on Add Item page in AddEditItemFeature (target: Safari) -> Skipped on thread #-1
...
<!-- other Safari tests also skipped on thread #-1 -->
...
2021-06-25T09:02:56.8764035Z test run finished
2021-06-25T09:02:56.8765441Z publishing test results
2021-06-25T09:02:56.8766131Z Not publishing results.
2021-06-25T09:02:56.8766729Z test results published
2021-06-25T09:02:56.8767274Z generating reports
2021-06-25T09:02:56.9558142Z Starting external report generation process
2021-06-25T09:02:56.9590180Z Starting external program: "D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\SpecFlowPlusRunner\net461\SpecFlow.Plus.Runner.Reporting.exe" "C:\Users\VssAdministrator\AppData\Local\Temp\ca95d93f-da8f-41ce-9966-62cb15b332e9" in 
2021-06-25T09:02:58.4457850Z   Skipped Follow button is not available in header section on Add Item page [1 ms]
...
<!-- all tests were skipped in [< 1 ms] --!>
...
2021-06-25T09:03:00.2990896Z 'D:\a\1\s\RetroSpectro.Tests.UI.Specflow\bin\any cpu\release\net5.0\SpecFlowPlusRunner\net461\SpecFlow.Plus.Runner.Reporting.exe "C:\Users\VssAdministrator\AppData\Local\Temp\ca95d93f-da8f-41ce-9966-62cb15b332e9"' took 3337.3233ms
2021-06-25T09:03:00.2992974Z RAA6AFwAYQBcAF8AdABlAG0AcABcAFQAZQBzAHQAUgBlAHMAdQBsAHQAcwBcAFIAZQB0AHIAbwBTAHAAZQBjAHQAcgBvAC4AVABlAHMAdABzAC4AVQBJAC4AUwBwAGUAYwBmAGwAbwB3AF8AUgBlAHQAcgBvAFMAcABlAGMAdAByAG8ALgBUAGUAcwB0AHMALgBVAEkALgBTAHAAZQBjAGYAbABvAHcAXwAyADAAMgAxAC0AMAA2AC0AMgA1AFQAMAA5ADAAMgAzADMALgBoAHQAbQBsAA==
2021-06-25T09:03:00.2994487Z Successfully generated reports.
2021-06-25T09:03:00.3024068Z Successfully generated reports
2021-06-25T09:03:00.3157341Z Result: test framework error: At least one test thread aborted.
2021-06-25T09:03:00.3159973Z   Total: 185
2021-06-25T09:03:00.3160575Z   Succeeded: 0
2021-06-25T09:03:00.3161111Z   Ignored: 0
2021-06-25T09:03:00.3161627Z   Pending: 0
2021-06-25T09:03:00.3163981Z   Skipped: 185
2021-06-25T09:03:00.3164566Z   Failed: 0
2021-06-25T09:03:00.3165688Z   Execution Time: 00:00:22.9059275
2021-06-25T09:03:00.3166798Z Report file: file:///D:/a/_temp/TestResults/RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.html
2021-06-25T09:03:00.3168312Z Adding attachments to VSTest
2021-06-25T09:03:00.3169047Z adding Test Execution Report 'D:\a\_temp\TestResults\RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.html'
2021-06-25T09:03:00.3283768Z Log file: file:///D:/a/_temp/TestResults/RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.log
2021-06-25T09:03:00.3294371Z SpecFlow+Runner execution finished
2021-06-25T09:03:00.6411255Z Results File: D:\a\_temp\TestResults\VssAdministrator_WIN-RNFTU0DQEPC_2021-06-25_09_02_58.trx
2021-06-25T09:03:00.6488918Z Attachments:
2021-06-25T09:03:00.6490475Z   C:\Users\VssAdministrator\AppData\Local\Temp\0erzmwe3.ry2\RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.log
2021-06-25T09:03:00.6494999Z   C:\Users\VssAdministrator\AppData\Local\Temp\0erzmwe3.ry2\RetroSpectro.Tests.UI.Specflow_RetroSpectro.Tests.UI.Specflow_2021-06-25T090233.html
2021-06-25T09:03:00.6813251Z Test Run Successful.
2021-06-25T09:03:00.6814110Z Total tests: 185
2021-06-25T09:03:00.6814725Z     Skipped: 185
2021-06-25T09:03:00.6826934Z  Total time: 30.4837 Seconds
2021-06-25T09:03:00.7194885Z Vstest.console.exe exited with code 0.
2021-06-25T09:03:00.7196364Z **************** Completed test execution *********************
2021-06-25T09:03:00.7398059Z Test results files: D:\a\_temp\TestResults\VssAdministrator_WIN-RNFTU0DQEPC_2021-06-25_09_02_58.trx
2021-06-25T09:03:01.3258001Z Created test run: 1014788
2021-06-25T09:03:01.3261282Z Publishing test results: 185
2021-06-25T09:03:01.6120517Z Publishing test results to test run '1014788'.
2021-06-25T09:03:01.6165436Z TestResults To Publish 185, Test run id:1014788
2021-06-25T09:03:01.6272324Z Test results publishing 185, remaining: 0. Test run id: 1014788
2021-06-25T09:03:02.2543932Z Published test results: 185
2021-06-25T09:03:02.2556259Z Publishing Attachments: 3
2021-06-25T09:03:03.5514586Z ##[error]The specified minimum number of tests 1 were not executed in the test run.
2021-06-25T09:03:04.2147578Z Execution Result Code 1 is non zero, checking for failed results
2021-06-25T09:03:04.3270302Z Completed TestExecution Model...
2021-06-25T09:03:04.3691332Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2021-06-25T09:03:04.3697400Z ##[error]Error: The process 'D:\a\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.170.1\Modules\DTAExecutionHost.exe' failed with exit code 1
2021-06-25T09:03:04.3703872Z ##[error]Vstest failed with error. Check logs for failures. There might be failed tests.
2021-06-25T09:03:04.3933532Z ##[section]Finishing: VsTest - RetroSpectro.Tests.UI.Specflow.dll

看起来测试线程正在中止，但我不知道为什么。

.runsettings 比较简单：

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <SpecRun>
        <Profile>Serial.srprofile</Profile>
        <GenerateFeatureTrait>true</GenerateFeatureTrait>
    </SpecRun>
</RunSettings>

规格流配置文件是：

<?xml version="1.0" encoding="utf-8"?>
<TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5">
    <Settings projectName="RetroSpectro.Tests.UI.Specflow" />
    <Execution retryFor="Failing" stopAfterFailures="0" testThreadCount="1" testSchedulingMode="Random" retryCount="2" />
    <Environment framework="Net5.0" platform="x86" testThreadIsolation="AppDomain" apartmentState="STA"/>
    <TestAssemblyPaths>
        <TestAssemblyPath>RetroSpectro.Tests.UI.Specflow.dll</TestAssemblyPath>
    </TestAssemblyPaths>
    <VSTest testRetryResults="Unified" passRateAbsolute="1"/>
    <Targets>
        <Target name="Chrome">
            <Filter>!@Parallel</Filter>
            <DeploymentTransformationSteps>
                <EnvironmentVariable variable="RUNNER_TARGET" value="chrome" />
            </DeploymentTransformationSteps>
        </Target>
        <Target name="Edge">
            <Filter>!@Parallel</Filter>
            <DeploymentTransformationSteps>
                <EnvironmentVariable variable="RUNNER_TARGET" value="edge" />
            </DeploymentTransformationSteps>
        </Target>
        <Target name="Firefox">
            <Filter>!@Parallel</Filter>
            <DeploymentTransformationSteps>
                <EnvironmentVariable variable="RUNNER_TARGET" value="firefox" />
            </DeploymentTransformationSteps>
        </Target>
        <Target name="IE">
            <Filter>!@Parallel</Filter>
            <DeploymentTransformationSteps>
                <EnvironmentVariable variable="RUNNER_TARGET" value="ie" />
            </DeploymentTransformationSteps>
        </Target>
        <Target name="Safari">
            <Filter>!@Parallel</Filter>
            <DeploymentTransformationSteps>
                <EnvironmentVariable variable="RUNNER_TARGET" value="safari" />
            </DeploymentTransformationSteps>
        </Target>
    </Targets>
</TestProfile>

任何想法为什么要跳过测试或我可以尝试找出什么？

编辑：

我注意到在之前的（工作）运行中，VSTest 在线程#-0 中运行场景。崩溃后，它开始记录“skipped in thread #-1”消息。

标签： specflowvstestazure-devops-pipelinesazure-devops-hosted-agent