-
Notifications
You must be signed in to change notification settings - Fork 7
Feature/tensor support #673
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
mdbenito
wants to merge
72
commits into
develop
Choose a base branch
from
feature/tensor-support
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… torch specializations
# Conflicts: # src/pydvl/utils/types.py # src/pydvl/valuation/scorers/supervised.py
# Conflicts: # src/pydvl/valuation/samplers/classwise.py
- Create array_ops.py with utilities for both numpy arrays and PyTorch tensors - Implement type-preserving functions for array creation and manipulation - Add proper type hints with Array protocol and TypeVar for type preservation - Add utility functions for library-specific operations - Import array_ops in utils/__init__.py 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add stratified_split_indices utility in array_ops.py to handle both numpy arrays and tensors - Update RawData.__post_init__ with improved type checking - Update Dataset.from_arrays to support tensors through type-agnostic operations - Add type hints and update docstrings for tensor support 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Update GroupedDataset to handle PyTorch tensors - Implement type-agnostic data_to_group and group_to_data mappings - Maintain tensor type in data_indices and logical_indices methods - Add comprehensive tests for tensor operations in GroupedDataset 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…sor support Extended test coverage to validate tensor support in Dataset and GroupedDataset classes: - Added tests for mixed input types and error handling - Added tests for edge cases like empty groups - Added tests for single vs multi-dimensional tensors - Added test for complex sequences of operations to verify type preservation - Verified factory methods maintain type consistency 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Updated Sample class to support PyTorch tensors - Modified IndexSampler to be tensor-agnostic - Added tests for tensor support in samplers - Updated hash and equality methods to work with both array types - Replaced numpy-specific operations with array_ops equivalents 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Fixed type errors in array_ops.py by adding proper type annotations and casts. - Added overloads for functions to maintain type precision - Fixed return type annotations for tensor operations - Added proper casting to ensure type safety - Fixed tensor-specific operations like .to() and .long() - Ensured consistent return types match input types 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…cumentation - Add comprehensive tests for tensor indices handling in samplers - Verify Sample.subset is always a numpy array - Test ClasswiseSample with tensor inputs for both subset and ooc_subset - Add error handling tests for invalid input types - Add documentation note about converting tensor indices to numpy arrays - Tests verify proper conversion and appropriate handling of tensor inputs This completes step 5.3 of the tensor support implementation plan.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds support for tensor data to
pydvl.valuation.dataset.Dataset
through generics, anArray
prototype and a collection of wrapper array functions inpydvl.utils.array
.array.py
and associated tests.Changes
Dataset
now supports instantiation with tensors or numpy arrays. The type is preservedDataset
indexingDataset
can take memmapped numpy arrays, or memmap them ifmmap=True
, reducing memory cost per-node.TorchSupervisedModel
, which is implemented e.g. byskorch.NeuralNetClassifier
, and used in the MSR notebook (not a new dependency)SkorchSupervisedScorer
to handle skorch models.TODO
array.py
Checklist
"tags": ["hide"]
or"tags": ["hide-input"]