Data lineage tracking (aka CID store)

[pditommaso]

Tentative implementation for addressable data store (very basic POC so far).

Update on 1 Mar 2025 from #5787 by @jorgee

M1 Implementation of CID store for provenance

Changes:

• CID store is specified by workflow.data.store.location
• Workflow Hash is created based on the workflow and parameters description
• workflow, tasks and outputs metadata are stored in <cid.store.location>/.meta
• references to other cid metadata are cid://<workflow_hash|task_hash/output_target_path
• CID NIO Filesystem to access data based on CIS URLs
• nextflow cid command to log, show and get lineage from CID store metadata

Known Limitations:

• Outputs which are not published in absolutePaths or URLs which are not subfolders both the outputDir, we can not infer the relative output target path. They are not currently tracked in the CID store. We could create a hash for the parent directory of the URL or absolute path and use it as relative folder.

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	36d5c82
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/680086091518c30008500621

[pditommaso]

@jorgee apologies, can latest changes be made as PR against this branch? so it will be much simpler do understand what's new for me

[jorgee]

@jorgee apologies, can latest changes be made as PR against this branch? so it will be much simpler do understand what's new for me

I have reverted the changes in this branch and created a new one in PR #5787

[jorgee]

Pushed minor fixes for the render. It was failing because of the change of the task inputs type (FileInParam is now path). There was also the problem of including forbidden characters in the Mermaid node id. Moreover, I have added a default value for the html file. Users only need to specify the data to render.

I have also checked what happens when you pass a task run or workflow run LID. In the case of task runs, it renders the task graph starting from the requested task and its predecessors based on file parameter dependencies. In the case of workflows, it renders the workflow and the input parameters. It was unintentional, but I will keep it, just an extra functionality for free.

Finally, I have added the closest property in the validation errors when parsing the fragments or query strings.

[pditommaso]

It looks in a good shape. I've made some tests and a a few little changes. Some notes:

1. better not use . in error messages. It may be confusing when preceding URIs or numbers.
2. Timestamps have been changes to OffsetDateTime using the current time zone. This allow keep track of the current location time. ✅
3. Added li command shortcut. ✅
4. We may want to change resolvedConfig to config for simplicity
5. We may want to change XxxOutputs and #outputs to XxxOutput. Along the same manner inputs to input
6. Find describe command verbose. What about view or print instead?
7. Likely TaskRun should include the resolved task script
8. Command lineage find 'type=DataOutput' work OK, but equivalent channel.fromPath('lid:///?type=DataOutput') seems not working 🔴
9. Likely makes no much sense to keep nf-lineage-h2. Planning to more an external plugin

[bentsherman]

Find describe command verbose. What about view or print instead?

view would be consistent with the view command, especially if we merge #5966

Command lineage find 'type=DataOutput' work OK, but equivalent channel.fromPath('lid:///?type=DataOutput') seems not working

I don't think this is meant to work? The query isn't really returning a single JSON but a collection of JSONs. This is why I suggested to remove the lid:// prefix when making a query, to not give the illusion that it's a LID path

[bentsherman]

Is this class used for file outputs? If so then I would call it FileOutput, I'm having a hard time understanding what it's used for

[bentsherman]

Should this be annotations instead of tags?

[bentsherman]

There is already intValue() for this, see below

[bentsherman]

If the only option under this scope is location, then I think we can shorten it to workflow.lineage.store

[bentsherman]

Continuing the discussion here about making workflow runs Merkle-compliant:

1. Rename WorkflowRun to WorkflowLaunch
2. Rename this class to WorkflowRun
3. Rename the workflowRun field in this class to launch
4. Use the hash of this class as the "workflow run id" in the lineage log

This way, the WorkflowRun represents the "final" record that is created and the entrypoint into the lineage from the runs log. It resolves the issue where lid://<hash>#outputs requires a reverse lookup. Even if we never go full CID, it still makes sense to do it

[bentsherman]

We could do a similar thing for tasks, i.e. TaskRun -> TaskLaunch and TaskOutputs -> TaskRun

[bentsherman]

If we rename describe to view, maybe we should also rename log to list to align more with the pipeline commands, and to not confuse with nextflow log or .nextflow.log

[bentsherman]

I tested with rnaseq-nf (workflow-outputs-3 branch) and received the following workflow output structure:

[
  {
    "type": "Collection",
    "name": "samples",
    "value": [ /* ... */ ]
  },
  {
    "type": "Collection",
    "name": "summary",
    "value": [
      "multiqc_report",
      "lid://494b7281ea2e02e985636dfbbcf6b8b3/multiqc_report.html"
    ]
  }
]

But this is not quite right. The summary output should just be a file. Wrapping in a list might be nice for a CSV file but it obscures the output here and also causes the LinObserver to infer the wrong type (Collection).

Applying this change produces the correct output structure:

[
  {
    "type": "Collection",
    "name": "samples",
    "value": [ /* ... */ ]
  },
  {
    "type": "Path",
    "name": "summary",
    "value": "lid://2265a814fd1c205ecc5b629070d759e2/multiqc_report.html"
  }
]

[bentsherman]

After playing around with the lineage command, I am skeptical about how much we are overloading this lid pseudo-filesystem. I thought it was just a nice add-on that we could experiment with, but now I think it's just getting in the way.

Currently there are three main uses for lid paths:

1. lid://<hash>[#props]: returns a metadata record or sub-path. This has no practical utility in a Nextflow script, not even for workflow outputs. Now that #outputs is a list, I can't access an output by name (e.g. #outputs.samples), which means I can't use channel.fromPath() to access an LID output in the same way as a samplesheet. So the LID output is no longer a drop-in replacement for samplesheets.

On the command line, it would be simpler to just provide the hash and use jq:

# before
nextflow li describe lid://<hash>#params

# after
nextflow li describe <hash> | jq .params

In a web interface like the platform, you'll use a graphical interface to navigate this metadata, so the fragment syntax is not needed there.

2. lid:///?<name>=<value>&...: used by the find command to retrieve a collection of metadata records. This also has no utility in a Nextflow script, because it is unrelated to domain-specific data like #outputs. It is only used by the find command, so the URI syntax is just getting in the way:

# before
# oops, forgot to escape the & ...
nextflow li find lid:///?type=DataOutput&workflowRun=lid://2265a814fd1c205ecc5b629070d759e2

# after
nextflow li find type=DataOutput workflowRun=2265a814fd1c205ecc5b629070d759e2

3. lid://<hash>/<path>: returns a content-addressed file. This is the original use case and the only one that still makes sense as far as I can tell. I think this works perfectly both on the command line and in the Nextflow script/runtime.

Based on this analysis, I think we should ditch (1) and (2) entirely and use lid:// only to refer to files.

Maybe we could use the fragment to refer to a specific output, e.g. lid://<hash>#samples. That would at least restore the original use case of passing a workflow output as input to a downstream pipeline.