Skip to content

Support compression in spill files #16130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Tracked by #16065
2010YOUY01 opened this issue May 21, 2025 · 4 comments
Open
Tracked by #16065

Support compression in spill files #16130

2010YOUY01 opened this issue May 21, 2025 · 4 comments
Assignees
Labels
enhancement New feature or request

Comments

@2010YOUY01
Copy link
Contributor

Is your feature request related to a problem or challenge?

Part of #16065

In memory-limited queries, temporary results will be written to disk to reduce memory pressure. Now those spills are using plain arrow encoding, they can be applied some general-purpose compression like lz4 and zstd for more efficient disk space usage.

Describe the solution you'd like

  1. Add a configuration for spill compression type
  2. Pass an option to Arrow-IPC writer/reader for compression

Describe alternatives you've considered

No response

Additional context

No response

@2010YOUY01 2010YOUY01 added the enhancement New feature or request label May 21, 2025
@2010YOUY01
Copy link
Contributor Author

I think this issue can be added to your current TODO list @ding-young.

In the future, if we introduce a dedicated spill format, we might revise the implementation to apply compression to the entire spill file.
But I think it's good to be implemented now since the code footprint is small (should be just passing an option into Arrow IPC reader/writer)

@ding-young
Copy link
Contributor

Sure. I also think that introducing compression option first is good to start though we may try more complicated techniques.

@ding-young
Copy link
Contributor

take

@Omega359
Copy link
Contributor

related/prior art: #14078 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants