Support computing statistics for FileGroup

[xudong963]

Which issue does this PR close?

• Follow up: Refactor: add FileGroup structure for Vec<PartitionedFile> #15379

Rationale for this change

Compute the FileGroup's statistics when computing all files' statistics

What changes are included in this PR?

Improving File Statistics Handling:

1. Replacing the get_statistics_with_limit function with get_files_with_limit and compute_all_files_statistics`, make code logic clearer.

2. Adding new functions to handle better statistics of FileGroup and all files, including compute_file_group_statistics and compute_all_files_statistics. Also adding a generic function compute_summary_statistics to compute statistics across multiple items that have statistics.

Enhancing Documentation:

1. Adding detailed comments and documentation for new functions to explain their purpose and usage.

Are these changes tested?

Yes, by existing tests

Are there any user-facing changes?

[jayzhan211]

Do you think it is a good idea to add another FileGroups struct and compute the statistics across all the files when we create FileGroups to be less error-prone (accidentally get the incorrect statistics)

[xudong963]

when we create FileGroups to be less error-prone (accidentally get the incorrect statistics)

Sorry, I don't get it. Why does adding a new FileGroups reduce error-prone?

[suremarc]

LGTM, as far as I can tell it is a refactor of existing logic, but made more flexible so we can get statistics at the global level as well as file group level. Nice 👍

[alamb]

Thank you @xudong963 @suremarc and @jayzhan211

I think this is a good refactor and the code is well structured and documented.

Would it be possible to add some unit tests for compute_summary_statistics? Something like:

1. Create a Vec
2. Call compute_summary_statistics
3. Verify the summarized statistics?

I think that would put us in a better position to continue working on the statistics collection code with confidence that we aren't breaking something, especially as we start relying on statistics more and more for correctness

[xudong963]

Would it be possible to add some unit tests for compute_summary_statistics? Something like:

Thanks @alamb ! I'm cooking it

Done in d8090fe

[alamb]

Thank you @xudong963 -- this looks good to me

[alamb]

👍

[alamb]

💯 for adding Arc

[xudong963]

Thanks for your review! Lets go

[alamb]

For anyone else following along, this PR is part of a larger plan top optimize ORDER BY queries operating on pre-sorted inputs. See this ticket for more detail

• [EPIC] Avoid sort for already sorted Parquet files that do not overlap values on condition #6672