Skip to content

Proposal: New standard library functions #339

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 3 commits into
base: dev
Choose a base branch
from
Open

Proposal: New standard library functions #339

wants to merge 3 commits into from
+24 −44

Conversation

bentsherman
Copy link

@bentsherman bentsherman commented Mar 13, 2025 •
edited
Loading

This PR demonstrates some helper functions that I am thinking about adding to Nextflow, to reduce some of the boilerplate code that I see in nf-core pipelines.

For now I have added these functions to nf-boost so that they are easy to play with. If people like them, I'll try to add them to Nextflow.

These functions are meant to minimize the use of custom classes like Yaml and JsonOutput, providing a simpler functional interface.

Helper functions are documented in the nf-boost README.

@nf-core-bot

This comment was marked as off-topic.

Sign in to view
@github-actions GitHub Actions

This comment was marked as off-topic.

Sign in to view
SPPearce
SPPearce reviewed Mar 14, 2025
View reviewed changes
subworkflows/nf-core/utils_nextflow_pipeline/main.nf Outdated
Comment on lines 76 to 80
def temp_pf = file(".params_${timestamp}.json")
temp_pf.text = toJson(params, true)

nextflow.extension.FilesEx.copyTo(temp_pf.toPath(), "${outdir}/pipeline_info/params_${timestamp}.json")
temp_pf.copyTo("${outdir}/pipeline_info/params_${timestamp}.json")
temp_pf.delete()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to make the file and then copy it? Couldn't we just make it in the outdir?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We certainly could! 😆

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just rewriting this in nf-utils, and Idk why that's in the template either 🤔 @mirpedrol or @mashehu any ideas?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comes from nf-core/modules, I found the which that added the change, but I don't know the reason nf-core/modules@4c0a9b8

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @maxulysse

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't remember.
Pretty sure just me being stupid.
Sorry about that

@bentsherman
Simplify file writes
900588a 
Signed-off-by: Ben Sherman <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 19, 2025 •
edited
Loading

nf-core pipelines lint overall result: Failed ❌

Posted for pipeline commit c9e9ca4

+| ✅ 165 tests passed       |+
#| ❔  15 tests were ignored |#
!| ❗   2 tests had warnings |!
-| ❌   1 tests failed       |-

❌ Test failures:

❗ Test warnings:

  • pipeline_todos - TODO string in nextflow.config: Update the field with the details of the contributors to your pipeline. New with Nextflow version 24.10.0
  • pipeline_todos - TODO string in ro-crate-metadata.json: "description": "

    \n \n <source media="(prefers-color-scheme: dark)" srcset="docs/images/nf-core-fetchngs_logo_dark.png">\n <img alt="nf-core/fetchngs" src="docs/images/nf-core-fetchngs_logo_light.png">\n \n

    \n\nGitHub Actions CI Status\nGitHub Actions Linting StatusAWS CICite with Zenodo\nnf-test\n\nNextflow\nrun with conda\nrun with docker\nrun with singularity\nLaunch on Seqera Platform\n\nGet help on SlackFollow on TwitterFollow on MastodonWatch on YouTube\n\n## Introduction\n\nnf-core/fetchngs is a bioinformatics pipeline that ...\n\n TODO nf-core:\n Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the\n major pipeline sections and the types of output it produces. You're giving an overview to someone new\n to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction\n\n\n Include a figure that guides the user through the major workflow steps. Many nf-core\n workflows use the "tube map" design for that. See https://nf-co.re/docs/contributing/design_guidelines#examples for examples. \n Fill in short bullet-pointed list of the default steps in the pipeline \n\n## Usage\n\n> [!NOTE]\n> If you are new to Nextflow and nf-core, please refer to this page on how to set-up Nextflow. Make sure to test your setup with -profile test before running the workflow on actual data.\n\n Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.\n Explain what rows and columns represent. For instance (please edit as appropriate):\n\nFirst, prepare a samplesheet with your input data that looks as follows:\n\nsamplesheet.csv:\n\ncsv\nsample,fastq_1,fastq_2\nCONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz\n\n\nEach row represents a fastq file (single-end) or a pair of fastq files (paired end).\n\n\n\nNow, you can run the pipeline using:\n\n update the following command to include all required parameters for a minimal example \n\nbash\nnextflow run nf-core/fetchngs \\\n -profile <docker/singularity/.../institute> \\\n --input samplesheet.csv \\\n --outdir <OUTDIR>\n\n\n> [!WARNING]\n> Please provide pipeline parameters via the CLI or Nextflow -params-file option. Custom config files including those provided by the -c Nextflow option can be used to provide any configuration except for parameters; see docs.\n\nFor more details and further functionality, please refer to the usage documentation and the parameter documentation.\n\n## Pipeline output\n\nTo see the results of an example test run with a full size dataset refer to the results tab on the nf-core website pipeline page.\nFor more details about the output files and reports, please refer to the\noutput documentation.\n\n## Credits\n\nnf-core/fetchngs was originally written by Harshil Patel, Moritz E. Beber and Jose Espinosa-Carrasco.\n\nWe thank the following people for their extensive assistance in the development of this pipeline:\n\n If applicable, make list of people who have also contributed \n\n## Contributions and Support\n\nIf you would like to contribute to this pipeline, please see the contributing guidelines.\n\nFor further information or help, don't hesitate to get in touch on the Slack #fetchngs channel (you can join with this invite).\n\n## Citations\n\n Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. \n If you use nf-core/fetchngs for your analysis, please cite it using the following doi: 10.5281/zenodo.XXXXXX \n\n Add bibliography of tools and data used in your pipeline \n\nAn extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.\n\nYou can cite the nf-core publication as follows:\n\n> The nf-core framework for community-curated bioinformatics pipelines.\n>\n> Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.\n>\n> Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.\n",

❔ Tests ignored:

  • files_exist - File is ignored: .github/workflows/awsfulltest.yml
  • files_exist - File is ignored: .github/workflows/awstest.yml
  • files_exist - File is ignored: assets/multiqc_config.yml
  • files_exist - File is ignored: conf/igenomes.config
  • files_exist - File is ignored: conf/igenomes_ignored.config
  • files_exist - File is ignored: conf/modules.config
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
  • files_unchanged - File ignored due to lint config: assets/nf-core-fetchngs_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-fetchngs_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-fetchngs_logo_dark.png
  • actions_ci - actions_ci
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/fetchngs/fetchngs/.github/workflows/awstest.yml
  • multiqc_config - multiqc_config
  • modules_config - modules_config

✅ Tests passed:

Run details

  • nf-core/tools version 3.2.0
  • Run at 2025-05-19 12:11:30

@edmundmiller edmundmiller added this to the 1.13.0 milestone May 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maxulysse maxulysse maxulysse left review comments

@mashehu mashehu mashehu left review comments

@mirpedrol mirpedrol mirpedrol left review comments

@edmundmiller edmundmiller edmundmiller left review comments

@SPPearce SPPearce SPPearce left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.13.0
Development

Successfully merging this pull request may close these issues.
7 participants
@bentsherman @nf-core-bot @maxulysse @mashehu @mirpedrol @edmundmiller @SPPearce