Support for AWS Batch Consumable Resources

[exai-sukh]

New Feature

AWS has introduced a new feature for AWS Batch, called "Consumable Resources". It would be great if Nextflow's AWS executor could support this. You may see additional details in the AWS blog post How to use rate-limited resources in AWS Batch jobs with resource-aware scheduling.

Use Case

I have workflows which need staggered submissions to reduce the likelihood of throttling errors. This feature would resolve such issues.

Suggested Implementation

Specify ConsumableResourceProperties within the process directive. This could be added to the SubmitJobRequest in

nextflow/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy

Line 713 in 7d3cb2b

protected SubmitJobRequest newSubmitRequest(TaskRun task) {

[bentsherman]

If it's a new feature then it's likely only available in AWS SDK v2, which we haven't upgraded to yet. We plan to do that this year.

In the meantime, we do have some retry-with-backoff logic to handle rate limit errors. Is that not working for you?