Skip to content

[BUG]: VSTest@3 TestPlan testSelector: not retrieving all test cases from the test suite #20912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
3 of 7 tasks
acostmig opened this issue Mar 7, 2025 · 0 comments
Open
3 of 7 tasks

Comments

@acostmig
Copy link

acostmig commented Mar 7, 2025

New issue checklist

Task name

VSTest

Task version

3

Issue Description

I've found a special case that when having 200+ test cases in a given test suite and the test configuration is mixed, the VSTest@3 task is only pulling up to the 201th test and leaving the rest out

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows

Relevant log output

no error message

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION] 

Repro steps

1. Create 2 test suites both of 300 test cases
2. let Test Suite 1 have 1 default configuration (assign to the test suite)
3. let Test Suite 2 have 1 default configuration (assign to the test suite)
4.  pick a handful of test cases to add a **different** configuration under Test Suite 2
     use order for this, lets say 
     Test Point (Test Case) of order 1
     Test Point (Test Case) of order 100
     Test Point (Test Case) of order 200
     Test Point (Test Case) of order 201
     Test Point (Test Case) of order 202
     Test Point (Test Case) of order 299
5. Use VSTask@3 to run the above
6. First run, run Test Suite 1 with the default configuration, all 300 will be executed here
7. Second run, run Test Suite 2 with **different** configuration

Expected: Second run with **different** configuration should run all 6 test cases
Actual: Second run with **different** configuration is only running the first 4 test cases on this configuration

Image

@acostmig acostmig added the bug label Mar 7, 2025
@acostmig acostmig changed the title [BUG]: [BUG]: TestPlan testSelector: not retrieving all test cases from the test suite Mar 7, 2025
@acostmig acostmig changed the title [BUG]: TestPlan testSelector: not retrieving all test cases from the test suite [BUG]: VSTest@3 TestPlan testSelector: not retrieving all test cases from the test suite Mar 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants