refactor: extract pagination logic into shared helper function

[leseb]

What does this PR do?

Move pagination logic from LocalFS and HuggingFace implementations into a common helper function to ensure consistent pagination behavior across providers. This reduces code duplication and centralizes pagination logic in one place.

Test Plan

Run this script:

from llama_stack_client import LlamaStackClient

# Initialize the client
client = LlamaStackClient(base_url="http://localhost:8321")

# Register a dataset
response = client.datasets.register(
    purpose="eval/messages-answer",  # or "eval/question-answer" or "post-training/messages"
    source={"type": "uri", "uri": "huggingface://datasets/llamastack/simpleqa?split=train"},
    dataset_id="my_dataset",  # optional, will be auto-generated if not provided
    metadata={"description": "My evaluation dataset"},  # optional
)

# Verify the dataset was registered by listing all datasets
datasets = client.datasets.list()
print(f"Registered datasets: {[d.identifier for d in datasets]}")

# You can then access the data using the datasetio API
# rows = client.datasets.iterrows(dataset_id="my_dataset", start_index=1, limit=2)
rows = client.datasets.iterrows(dataset_id="my_dataset")
print(f"Data: {rows.data}")

And play with start_index and limit.

[leseb]

@booxter FYI.

[leseb]

If we agree on the approach, I'll update all the API lists with this.

[booxter]

This is a good step. Ultimately, I'd try to apply this logic automatically with a class factory / ABCMeta, to make sure all GETs that return list responses are covered by it without explicitly requesting the behavior with return paginate().

[booxter]

it seems redundant now that you paginate as the last step that receives all records.

[raghotham]

can you add a todo to add the url to fetch more?

[leseb]

Sure, I'll open a ticket for this.

[leseb]

#1847