Skip to content

Support closure operand for pipe operator #3423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions docs/snippets/and-channel.nf
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
process foo {
input:
val begin
val middle
val end

output:
val result

exec:
result = "$begin $middle $end"
}

workflow {
ch_begin = channel.of('Hello')
ch_middle = channel.of('world')
ch_end = channel.of('!!!')

(ch_begin & ch_middle & ch_end)
| foo
| view
}
1 change: 1 addition & 0 deletions docs/snippets/and-channel.out
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Hello world !!!
29 changes: 29 additions & 0 deletions docs/snippets/and-process.nf
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
process foo {
input:
val message

output:
val result

exec:
result = "$message world"
}

process bar {
input:
val message

output:
val result

exec:
result = message.toUpperCase()
}

workflow {
channel.of('Hello')
| map { it.reverse() }
| (foo & bar)
| mix
| view
}
2 changes: 2 additions & 0 deletions docs/snippets/and-process.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
olleH world
OLLEH
19 changes: 19 additions & 0 deletions docs/snippets/pipe-with-closure.nf
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
process foo {
input:
val message
val suffix

output:
val result, emit: suffixed

exec:
result = "${message}${suffix}"
}

workflow {
suffix = ' world!'
channel.of('Hello','Hola','Ciao')
| map { it.toUpperCase() }
| { _ -> foo(_, suffix) }
| view
}
3 changes: 3 additions & 0 deletions docs/snippets/pipe-with-closure.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
HELLO world!
HOLA world!
CIAO world!
14 changes: 14 additions & 0 deletions docs/snippets/pipe.nf
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
process foo {
input:
val message

output:
val result

exec:
result = "$message world"
}

workflow {
channel.of('Hello','Hola','Ciao') | foo | map { it.toUpperCase() } | view
}
3 changes: 3 additions & 0 deletions docs/snippets/pipe.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
HELLO WORLD
HOLA WORLD
CIAO WORLD
18 changes: 18 additions & 0 deletions docs/snippets/process-named-stdout.nf
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
process sayHello {
input:
val cheers

output:
stdout emit: verbiage

script:
"""
echo -n $cheers
"""
}

workflow {
things = channel.of('Hello world!', 'Yo, dude!', 'Duck!')
sayHello(things)
sayHello.out.verbiage.view()
}
3 changes: 3 additions & 0 deletions docs/snippets/process-named-stdout.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Hello world!
Yo, dude!
Duck!
97 changes: 34 additions & 63 deletions docs/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,25 +150,8 @@ See {ref}`process-multiple-outputs` for more details.

The `emit` option can also be used to name a `stdout` output:

```groovy
process sayHello {
input:
val cheers

output:
stdout emit: verbiage

script:
"""
echo -n $cheers
"""
}

workflow {
things = channel.of('Hello world!', 'Yo, dude!', 'Duck!')
sayHello(things)
sayHello.out.verbiage.view()
}
```{literalinclude} snippets/process-named-stdout.nf
:language: groovy
```

:::{note}
Expand Down Expand Up @@ -243,7 +226,7 @@ Inputs can be specified like arguments when invoking the workflow:

```groovy
workflow {
my_pipeline( channel.from('/some/data') )
my_pipeline( channel.of('/some/data') )
}
```

Expand Down Expand Up @@ -334,72 +317,60 @@ The fully qualified process name can be used as a {ref}`process selector <config

The `|` *pipe* operator can be used to compose Nextflow processes and operators. For example:

```groovy
process foo {
input:
val data

output:
val result

exec:
result = "$data world"
}

workflow {
channel.from('Hello','Hola','Ciao') | foo | map { it.toUpperCase() } | view
}
```{literalinclude} snippets/pipe.nf
:language: groovy
```

The above snippet defines a process named `foo` and invokes it with the `data` channel. The result is then piped to the {ref}`operator-map` operator, which converts each string to uppercase, and finally to the {ref}`operator-view` operator which prints it.

:::{tip}
Statements can also be split across multiple lines for better readability:

```groovy
workflow {
channel.from('Hello','Hola','Ciao')
channel.of('Hello','Hola','Ciao')
| foo
| map { it.toUpperCase() }
| view
}
```

:::{versionadded} 23.12.0-edge
:::

### And `&`
When using the pipe operator, the right operand can also be a closure that receives the output of the left operand and returns the result of a process, workflow, or operator invocation, or any expression that produces a channel or multi-channel. This form is a useful way to define a custom mapping between the left-hand outputs and right-hand inputs, including the use of additional input channels aside from the left-hand outputs. For example:

The `&` *and* operator can be used to feed multiple processes with the same channel(s). For example:
```{literalinclude} snippets/pipe-with-closure.nf
:language: groovy
```

When the left operand is a process, the closure argument is equivalent to the `.out` of that process, and the output channels can be accessed by index or by name as described in [Process invocation](#process-invocation). For example, the `view` operation in the above example can be rewritten as `{ _ -> _.suffixed.view() }` to access the `suffixed` output of process `foo`.

You can also achieve the same behavior with the `then` operator:

```groovy
process foo {
input:
val data
channel.of('Hello','Hola','Ciao')
.map { it.toUpperCase() }
.then { _ -> foo(_, suffix) }
.view()
```

output:
val result
### And `&`

exec:
result = "$data world"
}
The `&` *and* operator can be used to feed multiple processes with the same channel(s). For example:

process bar {
input:
val data
```{literalinclude} snippets/and-process.nf
:language: groovy
```

output:
val result
In the above snippet, the initial channel is piped to the {ref}`operator-map` operator, which reverses the string value. Then, the result is passed to the processes `foo` and `bar`, which are executed in parallel. Each process outputs a channel, and the two channels are combined using the {ref}`operator-mix` operator. Finally, the result is printed using the {ref}`operator-view` operator.

exec:
result = data.toUpperCase()
}
:::{versionadded} 23.12.0-edge
:::

workflow {
channel.from('Hello')
| map { it.reverse() }
| (foo & bar)
| mix
| view
}
The and operator can also be used to compose channels into multi-channels. For example:

```{literalinclude} snippets/and-channel.nf
:language: groovy
```

In the above snippet, the initial channel is piped to the {ref}`operator-map` operator, which reverses the string value. Then, the result is passed to the processes `foo` and `bar`, which are executed in parallel. Each process outputs a channel, and the two channels are combined using the {ref}`operator-mix` operator. Finally, the result is printed using the {ref}`operator-view` operator.
Note that when a multi-channel is applied to an operator, the first channel is provided as the source and the other channels are applied as the arguments.
Loading