Incorrect memory assign when use withLabels

[JiangyanYu]

Dear nextflow team,

I try to define the cpus, memory and time in the base.config file by using process lables (see below). This is a process using GPUs in a docker container. From the report file I notice the container indeed used 64 cpus, however only 6G memory, which is the default setting in my base.config. I used similar setting for other processes (only involved CPUS), which worked fine. All these CPU needed processed used the correct cpus and memory.

withLabel:process_gpu_long {
        cpus   = { check_max( 64    * task.attempt, 'cpus'     ) }
        memory = { check_max( 512.GB * task.attempt, 'memory'  ) }
        time   = { check_max( 24.h  * task.attempt, 'time'   ) }
    }

In the end, I defined the process itself to increase memory (see below). This could solve the problem. But I don't know why. Could you please give more information for this bug? Thanks a lot!

    withName:DEEPVARIANT {
        cpus   = { check_max( 64    * task.attempt, 'cpus'     ) }
        memory = { check_max( 512.GB * task.attempt, 'memory'  ) }
    }

Best,
Jiangyan

[bentsherman]

That check_max() function uses params to limit the maximum value of each resource. In fact I believe the default value of params.max_memory is 6 GB. I would check that param setting in the main nextflow.config

[JiangyanYu]

The max_memory is 562 GB in nextflow.config.

    max_memory                 = '562.GB'
    max_cpus                   = 128
    max_time                   = '24.h'

Below is the detailed setting in base.config. Two processes (DORADO_BASECALLER and DEEPVARIANT) need GPU and are with label process_gpu_long. However, although I set the memory as 512G in the label process_gpu_long, it only use 6GB as far as I can see from the pipeline_report. Therefore, I used withName to force these two processes to use 512gb memory. So far it works, but I don't know what was the issue when use withLable for GPU tasks.

process {
    // Important debug parameters: Check the defaults for all processes
    cpus   = { check_max( 1    * task.attempt, 'cpus'   ) }
    memory = { check_max( 6.GB * task.attempt, 'memory' ) }
    time   = { check_max( 4.h  * task.attempt, 'time'   ) }

    errorStrategy = { task.exitStatus in [143,137,104,134,139] ? 'retry' : 'finish' }
    maxRetries    = 1
    maxErrors     = '-1'

    // Process-specific resource requirements
    // NOTE - Please try and re-use the labels below as much as possible.
    //        These labels are used and recognised by default in DSL2 files hosted on nf-core/modules.
    //        If possible, it would be nice to keep the same label naming convention when
    //        adding in your local modules too.
    // Important debug parameters: Customise requirements for specific processes.
    // Note: the time = per CPU time x num of CPU. deepvariant (gpu_long) needs appr. 10h per CPU time.
    // This setting shall not exceed nextflow.cong max parameters!
    // See https://www.nextflow.io/docs/latest/config.html#config-process-selectors
    withLabel:process_single {
        cpus   = { check_max( 1                  , 'cpus'    ) }
        memory = { check_max( 6.GB * task.attempt, 'memory'  ) }
        time   = { check_max( 4.h  * task.attempt, 'time'    ) }
    }
    withLabel:process_low {
        cpus   = { check_max( 2     * task.attempt, 'cpus'    ) }
        memory = { check_max( 12.GB * task.attempt, 'memory'  ) }
        time   = { check_max( 8.h   * task.attempt, 'time'    ) }
    }
    withLabel:process_medium {
        cpus   = { check_max( 6     * task.attempt, 'cpus'    ) }
        memory = { check_max( 36.GB * task.attempt, 'memory'  ) }
        time   = { check_max( 24.h   * task.attempt, 'time'    ) }
    }
    withLabel:process_high {
        cpus   = { check_max( 18    * task.attempt, 'cpus'    ) }
        memory = { check_max( 512.GB * task.attempt, 'memory'  ) }
        time   = { check_max( 24.h  * task.attempt, 'time'    ) }
    }
    withLabel:process_gpu_long {
        cpus   = { check_max( 64    * task.attempt, 'cpus'     ) }
        memory = { check_max( 512.GB * task.attempt, 'memory'  ) }
        time   = { check_max( 24.h  * task.attempt, 'time'   ) }
    }
    withLabel:error_ignore {
        errorStrategy = 'ignore'
    }
    withLabel:error_retry {
        errorStrategy = 'retry'
        maxRetries    = 2
    }
    withName:CUSTOM_DUMPSOFTWAREVERSIONS {
        cache = false
    }
    withName:DORADO_BASECALLER {
        cpus   = { check_max( 64    * task.attempt, 'cpus'     ) }
        memory = { check_max( 512.GB * task.attempt, 'memory'  ) }
    }
    withName:DEEPVARIANT {
        cpus   = { check_max( 64    * task.attempt, 'cpus'     ) }
        memory = { check_max( 512.GB * task.attempt, 'memory'  ) }
    }
}

[bentsherman]

I'm not sure either. The withLabel should override the default setting just like withName