Add bail option to describe block

[maddie-j]

Clear and concise description of the problem

Re: #1459 (comment)

Also, should we have some escape hatch for specifying scope suite for some files, like describe.bail? Or that would be an overkill?

^^ Request for exactly that if possible please!

Use case: Am working in a repo with a lot of test files, and most of them I don't want to --bail on in a normal test run in CI/CD. However, there are some particular integration tests that I already explicitly specify as describe.sequential where I don't want the later tests to run if the earlier tests fail.

Specifically, this is when I want to test a multi-stage process and want to have one test for each stage in that process. Eg:

1. retrieve file from filestore (eg S3)
2. decrypt file
3. parse file as JSON
4. (several tests) check that certain values are available and valid in that JSON.

If I fail to retrieve the file from the remote store, I don't want to run the test to decrypt it. If I fail to decrypt it, I don't want to try parse it as JSON. And if the JSON parse fails, I can't exactly check if the values I want to find exist!

If I can bail on the later tests if the earlier ones fail, I can set up the tests within that describe block to save and reuse the output of the previous step and:

• keep the purpose of each test atomic so that the test failure only represents a failure in the aspect that test is explicitly testing for
• reduce network and storage load on the system I'm pulling from (save cost and avoid rate limiting)
• reduce time to run the test suite due to not having to rerun the fetching and parsing steps that have already passed in the earlier tests (especially if dealing with large files and/or far from the remote filestore location)
• reduce flakiness inherent to dealing with remote systems, especially if integrating with a downstream team whose system's unreliability isn't something you can directly improve.

And, most importantly, the bail configuration is only applied in the situation it's needed and doesn't affect any other test blocks or test suites that are run alongside your sequential, bailable code.

As an extension, it might be nice to have an option that tells tests to bail the run only in watch mode, which is likely only used while developing. This would be useful for tests that can take a long time to run under some circumstances (eg waiting for a file to show up in a filestore after triggering your service to run, something that'd hit whatever timeout limit for every single test if you broke your service). I'd like to set those possibly long-running tests to bail in watch mode so that I see the output from their failure ASAP, and so the next rerun is triggered sooner if I'm actively fixing the fault. But I don't want them to bail in CI/CD because I do want to see the result of all failures there (and to see which of those tests passed and which of them failed to aid with debugging).

I assume integration/component/end-to-end tests aren't as high of a priority to your team as unit tests with remote systems mocked, but these escape hatches would be super useful for those of us who do want to use vitest to run against deployments in our dev and test envs!

Suggested solution

Have a bail option available to put on both individual tests and describe blocks. If the describe block fails, the rest of that block is skipped. If the test fails, am not sure if it should just bail that block or that whole suite.

Optional: Allow .bail() to take some sort of "watch only" argument to specify that it should only cause the bailing behaviour while being run in watch mode but not during a normal run.

Alternative

No response

Additional context

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[hi-ogawa]

I think your use case is normally associated with something like Playwright's test.step feature https://playwright.dev/docs/api/class-test#test-step, which Vitest doesn't currently have.

Something similar was probably discussed somewhere and it can be achieved on user land with beforeEach/afterEach(ctx). Does something like this work for you? https://stackblitz.com/edit/vitest-dev-vitest-ytevyfla?file=test%2Frepro.test.ts

import { describe, test, beforeEach, afterEach } from 'vitest';

function setupBailSuite() {
  let bail = false;

  beforeEach((ctx) => {
    if (bail) {
      ctx.skip('bailed test');
    }
  });

  afterEach((ctx) => {
    bail ||= ctx.task.result?.state === 'fail';
  });
}

describe('step suite', () => {
  setupBailSuite();

  test('one', () => {});

  test('two', () => {
    throw new Error('boom2');
  });

  // this will be skipped
  test('three', () => {
    throw new Error('boom3');
  });
});

this will give following test results:

  ✓ step suite > one 1ms
   × step suite > two 1ms
     → boom2
   ↓ step suite > three [bailed test]