How to implement a "while a statement fails sleep x seconds and retry"

[VasLem]

Hi, so there are cases where I am trying to avoid race conditions, particularly during publishing large files in Google Storage. I recently faced an issue with listFiles, that randomly failed (the issue) . In order to avoid this I could wrap listFiles in a while try and break otherwise catch and retry solution. However, while operations are not allowed, and I don't see how I can make this happen. The closest I have gotten is by using a find operation with e.g. max 10 retries... :

    def listFiles = {dir-> 
      if (dir instanceof String || dir instanceof org.codehaus.groovy.runtime.GStringImpl){
        dir = file(dir)
      } 
      1..10.find {
          try {
            dir.listFiles()
            return true
          } catch (Exception e) {
            sleep(5)
            return false 
          }
      }
      dir.listFiles()
    }

Obviously, this is ridiculously, reward-worth-level, bad programming :D. Do we have any alternative to implement such a thing in Nextflow?

[bentsherman]

While loops are generally not needed in Nextflow and I am still looking for actual use cases before I consider including it in the Nextflow language spec. In this case I think there is an underlying issue with the GCS integration that likely needs to be handled by the Nextflow runtime

However I could see an interesting generic solution for retry -- write a custom function that takes a closure and a map of retry params (max attempts, delay, jitter, etc) and performs the action with retries. If you write this function in a plugin, you could use a while loop. Then you can re-use the function throughout your pipeline, rather than using a process just for the retries.