You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In memory-limited queries, temporary results will be written to disk to reduce memory pressure. Now those spills are using plain arrow encoding, they can be applied some general-purpose compression like lz4 and zstd for more efficient disk space usage.
Describe the solution you'd like
Add a configuration for spill compression type
Pass an option to Arrow-IPC writer/reader for compression
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered:
I think this issue can be added to your current TODO list @ding-young.
In the future, if we introduce a dedicated spill format, we might revise the implementation to apply compression to the entire spill file.
But I think it's good to be implemented now since the code footprint is small (should be just passing an option into Arrow IPC reader/writer)
Is your feature request related to a problem or challenge?
Part of #16065
In memory-limited queries, temporary results will be written to disk to reduce memory pressure. Now those spills are using plain arrow encoding, they can be applied some general-purpose compression like lz4 and zstd for more efficient disk space usage.
Describe the solution you'd like
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: