Travis flow optimization

[philpennock]

Description:

The current flow fires off a lot of jobs, even if there are trivial failures. Travis now has "build steps", to allow for chained dependencies. I've put together an experiment which might be interesting to you.

What I've run:

• https://travis-ci.org/philpennock/trav-test-stage-matrix/builds/356141984
• https://github.com/philpennock/trav-test-stage-matrix

In this, the lints run first; a failure in either terminates. Then the other tests are run.

In fact, I split the other tests up too, designating one as the "primary", the latest released version of Python, and run that first. Only once that gives the all clear do the rest fire. The 'nightly' job can fail, as before.

This has the downside that time to complete success increases, because the primary has to succeed before the soaks fire, but has the upside that it's clearer what to investigate, and keeps your hosting happier about resource-consumption by not burning containers doing things you know are going to fail. If you're already waiting 25+ minutes for a test, there's not that much difference.

But that's more contentious. I think the approach might be interesting to you, so have put together the test repo to try things out, and gotten the replacement matrix working, which should be enough for the maintainers to decide if there's anything worthwhile in this.

What didn't work?

Using an implicit 'test' stage, or anything pulling in a test stage, given an explicit matrix, just didn't seem to work. Build stages seem to interact properly with implicit matrix expansion to populate steps, but not with explicit matrices. But at least the allow_failure and fast_finish do work correctly.

[pradyunsg]

Hey @philpennock!

I've mentioned this in the past (#4478) but I never actually got to it.

I'll be happy to accept a PR for this. :)

[philpennock]

What aspects of the test repo setup do you like or dislike? The ticket above has links to a working setup for how things might be split, and the Travis results of that. The build steps themselves are dummy, but the framework should be 1:1 comparable, because I wrote it for double-checking how pip might behave.

I can go ahead and submit a PR, but I figure iterating on rough design on a repo where the tests take less than three minutes (and that long because of sleep invocations) is easier for everyone than iterating where the builds take ... "much longer".

[pradyunsg]

Hi @philpennock!

Sorry for the delay! Looking at the build matrix, I think something else would be nice:

• "Primary"
  • lint
  • docs
  • CPython 3.6 (latest stable)
  • CPython 2.7
• "Secondary"
  • everything else
    • in descending order of versions
      • we get to pop off the lowest item when dropping support. :P

[pradyunsg]

I do understand that this is longer than the 2 minutes from the linting only, but this information is definitely more useful and this will fail faster than status quo anyway.

Another idea would be to run the unit tests only but they aren't very useful (yet).

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.