Process spot directive

[adamrtalbot]

New feature

We have many directives for specific purposes, but not one to control spot behaviour. We frequently have to point specific processes at different queues to use on-demand machines for long running processes. If we had a dedicated spot directive, we could automate this process.

Note, this mainly applies to Google and Azure. For AWS it's trickier but could be managed via Seqera Platform.

Use case

processA <- spot
processB <- on demand
processC <- spot

The current implementation would be this configuration:

process {
    withName: processB {
        queue = "on-demand-queue"
    }
}

but the queue string is brittle and can change without the config reflecting this causing failures.

Suggested implementation

process.spot = true

Then it would use a spot machine if that executor supports it.

Google: Enables google.batch.spot in a process specific manner
Azure: Enables azure.batch.pools.<name>.lowPriority for the autopools feature
AWS: Not technically possible, although Seqera Platform could switch the queue under the hood.