Add nested struct casting support and integrate into SchemaAdapter #16371

kosiew · 2025-06-11T15:03:03Z

Which issue does this PR close?

This is the part of a series of PRs re-implementing #15295 to close #14757 by adding schema-evolution support for:

listing‐based tables
with nested structs
in DataFusion.

Rationale for this change

DataFusion’s existing casting logic could handle only primitive and top-level struct conversions via Arrow’s cast. Complex nested structs could not be projected or evolved safely across schema changes. This PR introduces a dedicated nested_struct module to recursively cast and validate arbitrary-depth structs, ensuring:

Missing target fields are filled with nulls
Extra source fields are ignored
Nested structs are handled depth-first
SchemaAdapter uses the new casting logic for file-to-table mappings

This enables robust schema evolution for Parquet, JSON, and other nested-schema file formats.

What changes are included in this PR?

New module datafusion/common/src/nested_struct.rs
- cast_struct_column & cast_column for recursive struct casting
- validate_struct_compatibility for compatibility checks
- Comprehensive unit tests covering simple, missing, extra, nested, and incompatible scenarios
Common imports updated to expose the nested-struct API
SchemaAdapter (datafusion/datasource/src/schema_adapter.rs)
- Use validate_struct_compatibility when comparing struct fields
- Inject cast_column into SchemaMapping for file → table conversions
- Update tests to cover nested struct adaptation and statistics mapping
Core datasource tests cleanup (datafusion/core/src/datasource/mod.rs)
- Reintroduced missing object_store import for Parquet tests

Are these changes tested?

Yes, this PR adds and updates multiple test suites:

Unit tests in nested_struct.rs for casting and validation
SchemaAdapter tests covering nested struct adaptation, batch mapping, and statistics translation
Core datasource tests ensuring Parquet integration still compiles and runs

All new and existing tests pass locally.

Are there any user-facing changes?

No breaking changes to public APIs; all new functions live under datafusion_common::nested_struct
Users leveraging custom SchemaAdapter can now transparently cast nested structs without manual null-filling
Documentation for nested struct casting will be added in a follow-up PR if needed

…nested structs

…ration

…n DataFusion

This reverts commit c1059e7.

…erFactory

Summary use the table's SchemaAdapterFactory in list_files_for_scan add a test for statistics mapping via a custom factory Testing cargo test -p datafusion --lib test_statistics_mapping_with_custom_factory -- --nocapture cargo test -p datafusion --lib --quiet (fails: missing test data)

allow SchemaMapping to use a custom per-column adaptation function update DefaultSchemaAdapter and NestedStructSchemaAdapter to pass closures remove NestedStructSchemaMapping adjust tests

…maAdapter

…streamline new methods

… consume self

…gTable for improved file source creation

…a error handling in ListingTable

… adapter.rs, and mapping.rs—to improve navigability and reduce merge conflicts.

…larity and functionality

timsaucer

In general, this code looks very well written and has excellent unit tests. My one concern is whether it is correct to assume that structs should always be allowed to cast to other structs. I can see two ways forward:

We continue with this assertion but add additional documentation to make this clear to users.
Provide a "strict" mode where we fail if we have missing fields or fields that cannot be cast (maybe the second half is already done here)

timsaucer · 2025-06-19T12:29:43Z

datafusion/common/src/nested_struct.rs

+    } else {
+        // If source is not a struct, return null array with target struct type
+        Ok(new_null_array(
+            &Struct(target_fields.to_vec().into()),
+            source_col.len(),
+        ))
+    }


From the function signature it seems like this is an improper usage - would it make sense that we would return an error in the case of calling this function for a non-struct?

I agree this should be an error. I think at this point (reading of the data) we should only be doing casting, not coercion or in this case replacement with nulls.

Valid point.
I'll make the change.

timsaucer · 2025-06-19T12:32:30Z

datafusion/common/src/nested_struct.rs

+};
+use std::sync::Arc;
+/// Adapt a struct column to match the target field type, handling nested structs recursively
+fn adapt_struct_column(


nit: would cast_struct_column be a more intuitive name? Same for cast_column below

I started with adapt_.. because these were created for schema_adapter.
Checking around, I do see a lot of fn cast..., and none of fn adapt...
I'll make the change

timsaucer · 2025-06-19T12:45:14Z

datafusion/datasource/src/nested_schema_adapter/adapter.rs

+                // Special handling for struct fields - always include them even if the
+                // internal structure differs, as we'll adapt them later
+                match (file_field.data_type(), table_field.data_type()) {
+                    (Struct(_), Struct(_)) => Ok(true),


Is it right that this presumes that we can always cast from struct to struct? It feels like that isn't necessarily true. For example, if I have an input struct that contains two fields, both of which are strings and I attempt to cast it to an output struct with one field of type u32 then I think the current approach would give me a pass but an empty array. Is that correct?

Thanks for pointing this out.
I'll add code to validate compatibility.

adriangb

Overall looks great! I would split the changes to ListingTable out and address https://github.com/apache/datafusion/pull/16371/files#r2156903806

adriangb · 2025-06-19T19:42:52Z

datafusion/common/src/nested_struct.rs

+    } else {
+        // If source is not a struct, return null array with target struct type
+        Ok(new_null_array(
+            &Struct(target_fields.to_vec().into()),
+            source_col.len(),
+        ))
+    }


I agree this should be an error. I think at this point (reading of the data) we should only be doing casting, not coercion or in this case replacement with nulls.

adriangb · 2025-06-19T19:52:20Z

datafusion/core/src/datasource/listing/table.rs

+    /// Optional [`SchemaAdapterFactory`] for creating schema adapters
+    schema_adapter_factory: Option<Arc<dyn SchemaAdapterFactory>>,


I feel like this part can be split into its own PR

I made the change.

Great, let's look at that but as part of it's own PR

…TableConfig

…truct casting

adriangb

This looks good on my end now. Let's have @xudong963 give their review and then we can merge.

alamb

Thank you @kosiew and @adriangb

I think the idea for a custom struct --> struct cast operator is really nice

However, I am concerned with how this is proposed to be wired in (it seems like it isn't connected to anything 🤔 )

alamb · 2025-06-26T17:44:59Z

datafusion/datasource/src/nested_schema_adapter/adapter.rs

+use datafusion_common::Result;
+use std::sync::Arc;
+
+/// A SchemaAdapter that handles schema evolution for nested struct types


What is the reason for adding a new NestedStructAdapter here rather than just extending the behavior of the existing DefaultSchemaAdapter?

It seems like the ability to coerce nested structs is a good one of everyone

Moved this into DefaultSchemaAdapter as suggested.

alamb · 2025-06-26T17:53:38Z

datafusion/common/src/nested_struct.rs

+/// - Arrow's cast function fails for non-struct types
+/// - Memory allocation fails during struct construction
+/// - Invalid data type combinations are encountered
+pub fn cast_column(source_col: &ArrayRef, target_field: &Field) -> Result<ArrayRef> {


I think the ability to cast from one struct to another type is really nice

However, I am worried that by only supporting struct -> struct casting in this module (and then in the NestedSchemaAdapter below) most DataFusion users won't be able to take advantage of this feature

Have you considered changing all uses of

use arrow::compute::cast;

To use this function instead?

I think that woudl nicely close #14757 as well as improve the situation with #14396?

For example

> select {'a': 1}; +----------------------------------+ | named_struct(Utf8("a"),Int64(1)) | +----------------------------------+ | {a: 1} | +----------------------------------+ 1 row(s) fetched. Elapsed 0.037 seconds. > select {'a': 1} = {'a': 2, 'b': NULL}; Error during planning: Cannot infer common argument type for comparison operation Struct(a Int64) = Struct(a Int64, b Null)

I logged this as issue in Expose and generalize cast_column to enable struct → struct casting in more contexts #16579
I will address it in another PR.

adriangb · 2025-06-26T17:56:51Z

However, I am concerned with how this is proposed to be wired in (it seems like it isn't connected to anything 🤔 )

@alamb I asked @kosiew to move that out of the PR to keep it a bit smaller. The plan is to wire it up to ListingTable. I know we want to keep ListingTable simple but it's a relatively small change: #16371 (comment)

extend DefaultSchemaAdapter to handle nested struct fields remove NestedStructSchemaAdapter module update re-exports and tests adjust nested stats mapping test

alamb

Thank you @kosiew and @adriangb

…pache#16371) This commit introduces a new `nested_struct` module to support recursive struct-to-struct casting in DataFusion. It enables safe and flexible schema evolution through: - Recursive casting of nested structs via `cast_column` and `cast_struct_column` - Filling missing target fields with null values - Ignoring extra fields present in the source but absent in the target - Validating field compatibility using `validate_struct_compatibility`, including nested levels Integration updates include: - Enhancing `SchemaAdapter` and `SchemaMapping` to use the new nested struct casting logic - Injecting a customizable `cast_column` function into `SchemaMapping` - Updating schema mapping to support nested structs for file-to-table schema projection Test coverage: - Unit tests for simple, nested, missing, and incompatible struct scenarios - Adapter tests for batch mapping and statistics mapping with nested fields - Ensures existing functionality remains intact These changes provide robust support for evolving complex data schemas and improve the safety of nested data transformations, especially when working with Parquet and JSON data sources.

kosiew added 26 commits June 10, 2025 18:13

feat: add NestedStructSchemaAdapter for handling schema evolution of …

2706808

…nested structs

Fix: unwrap result of schema adapter factory in ParquetSource configu…

dba76b0

…ration

feat: add example for handling schema evolution with nested structs i…

3b7223a

…n DataFusion

nested_struct2 <- optimizer_rule

c1059e7

Revert "nested_struct2 <- optimizer_rule"

9ffeec3

This reverts commit c1059e7.

add nested_struct2.rs for testing

b5b3ed3

feat: implement CustomParquetDataSource using NestedStructSchemaAdapt…

3c711a5

…erFactory

feat: add schema adapter factory support to ListingTableConfig

0e38592

feat: ListingTable apply schema adapter to file source

e8f5b95

feat: nested_struct2 - fix test

6fcf1a7

feat: remove nested_struct2 example file

f170169

refactor: reorganize imports in nested_schema_adapter.rs for clarity

f6997cd

refactor schema mapping with column adapter

11b7463

allow SchemaMapping to use a custom per-column adaptation function update DefaultSchemaAdapter and NestedStructSchemaAdapter to pass closures remove NestedStructSchemaMapping adjust tests

Remove unused create_adapter

368774f

evaluate-create_adapter-for-public-api into nested-struct-14757

6eb22af

refactor: reorganize imports and simplify closure in NestedStructSche…

a9661e7

…maAdapter

Merge branch 'main' into nested-struct-14757

d822cfc

refactor: enhance ListingTableConfig with Default implementation and …

8b87fc6

…streamline new methods

refactor: amend with_schema_adapter_factory method in ListingTable to…

71d06fe

… consume self

refactor: add create_file_source_with_schema_adapter method to Listin…

22ef8ad

…gTable for improved file source creation

refactor: implement FailingMapSchemaAdapterFactory for enhanced schem…

f99205e

…a error handling in ListingTable

split nested_schema_adapter.rs into smaller modules—e.g. factory.rs,…

1c887d7

… adapter.rs, and mapping.rs—to improve navigability and reduce merge conflicts.

refactor: reorder imports in tests module for improved clarity

e4913d1

refactor: enhance adapt_column and adapt_struct_column for improved c…

e70ce79

…larity and functionality

refactor: impl Debug for SchemaMapping adapt_column

dafb68e

github-actions bot added core Core DataFusion crate datasource Changes to the datasource crate labels Jun 11, 2025

kosiew added 2 commits June 11, 2025 23:03

Merge branch 'main' into nested-struct-14757

2c1dd3c

refactor: update imports in tests module for clarity and organization

fd8929c

timsaucer reviewed Jun 19, 2025

View reviewed changes

adriangb reviewed Jun 19, 2025

View reviewed changes

kosiew added 7 commits June 20, 2025 09:15

Merge branch 'main' into nested-struct-14757

2862a50

refactor: rename adapt_column to cast_column

07aa2aa

refactor: cast_struct_column - error if source is not struct

e392878

refactor: enhance struct compatibility validation for casting operations

267768e

remove unused import

53ab183

refactor: remove schema adapter factory and related code from Listing…

3a766fd

…TableConfig

Merge branch 'main' into nested-struct-14757

8874489

kosiew force-pushed the nested-struct-14757 branch from a2edcee to 8874489 Compare June 20, 2025 04:55

kosiew added 5 commits June 20, 2025 13:26

fix: tests

6fa8c89

refactor: rename AdaptColumnFn to CastColumnFn for clarity

96f97b7

feat: move and make validate_struct_compatibility public for nested s…

2bc8256

…truct casting

fix: update code block syntax in documentation

5be2fa7

Merge branch 'main' into nested-struct-14757

33f8bdc

adriangb approved these changes Jun 20, 2025

View reviewed changes

alamb reviewed Jun 26, 2025

View reviewed changes

alamb mentioned this pull request Jun 26, 2025

feature: adapt predicate pushdown for mismatched nested/struct schemas #16565

Open

github-actions bot added the development-process Related to development process of DataFusion label Jun 27, 2025

kosiew mentioned this pull request Jun 27, 2025

Expose and generalize cast_column to enable struct → struct casting in more contexts #16579

Open

kosiew force-pushed the nested-struct-14757 branch from ebd57af to 33f8bdc Compare June 27, 2025 04:28

github-actions bot removed the development-process Related to development process of DataFusion label Jun 27, 2025

Integrate nested schema support into DefaultSchemaAdapter

4456419

extend DefaultSchemaAdapter to handle nested struct fields remove NestedStructSchemaAdapter module update re-exports and tests adjust nested stats mapping test

kosiew force-pushed the nested-struct-14757 branch from 2b6fee2 to 4456419 Compare June 27, 2025 10:20

kosiew changed the title ~~Enable schema evolution for nested structs via adapt_column and custom adapter support in ListingTable~~ Add nested struct casting support and integrate into SchemaAdapter Jun 27, 2025

kosiew merged commit 0c30f13 into apache:main Jun 27, 2025
29 checks passed

alamb reviewed Jun 27, 2025

View reviewed changes

adriangb mentioned this pull request Jun 27, 2025

feat: implement predicate adaptation for nested structs #16589

Open

		/// Optional [`SchemaAdapterFactory`] for creating schema adapters
		schema_adapter_factory: Option<Arc<dyn SchemaAdapterFactory>>,

Add nested struct casting support and integrate into SchemaAdapter #16371

Add nested struct casting support and integrate into SchemaAdapter #16371

Uh oh!

Conversation

kosiew commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

timsaucer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kosiew Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kosiew Jun 27, 2025 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb commented Jun 26, 2025

Uh oh!

Uh oh!

alamb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kosiew commented Jun 11, 2025 •

edited

Loading

kosiew Jun 20, 2025 •

edited

Loading

adriangb Jun 20, 2025 •

edited

Loading

kosiew Jun 27, 2025 •

edited by alamb

Loading

alamb left a comment •

edited

Loading