[datasetio] register_datasets() does not identify remote nvidia provider

[raspawar]

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

For implemetation of remote::nvidia datasetio, the DatasetsRoutingTable does not allow nvidia provider due to below logic: The provider_id get's set to localfs instead.

The issue is being caused by:

# infer provider from source
        if source.type == DatasetType.rows.value:
            provider_id = "localfs"
        elif source.type == DatasetType.uri.value:
            # infer provider from uri
            if source.uri.startswith("huggingface"):
                provider_id = "huggingface"
            else:
                provider_id = "localfs"
        else:
            raise ValueError(f"Unknown data source type: {source.type}")

https://github.com/meta-llama/llama-stack/blob/main/llama_stack/distribution/routers/routing_tables.py#L434

Error logs

from llama_stack.distribution.library_client import LlamaStackAsLibraryClient
import os

os.environ["NVIDIA_PROJECT_ID"] = "experment@1"
client = LlamaStackAsLibraryClient("nvidia")
_ = client.initialize()

client.datasets.register(
    purpose="post-training/messages",
    dataset_id="sample-basic-test",
    source={
               "type": "uri",
               "uri": "hf://datasets/default/sample-basic-test"
           },
    metadata={"format": "json", "description": "This is an example of a dataset"},
)

result ==>
DatasetRegisterResponse(identifier='sample-basic-test', metadata={'format': 'json', 'description': 'This is an example of a dataset'}, provider_id='localfs', provider_resource_id='sample-basic-test', purpose='post-training/messages', source=SourceUriDataSource(type='uri', uri='hf://datasets/default/sample-basic-test'), type='dataset', access_attributes=None)

provider_id='localfs' which does blocks redirection to the NvidiaDatasetIOAdapter, hence

Expected behavior

Allowing provider_id as an argument to decide where to register dataset.

[raspawar]

cc: @dglogo, @mattf, @yanxi0830, @raghotham