Ability to chunk download from object store

[trungda]

Is your feature request related to a problem or challenge?

When downloading large objects (> 300MBs) using object_store crate, I often hit timeout using the default configuration (30 seconds connection timeout). Interestingly, when increasing the timeout, the download speed is actual lower (not sure if it's the same for everyone?).

Describe the solution you'd like

I am thinking if it makes sense to chunk a file into smaller ranges (say, 100MB each), and in parallel, download each range with different connection and reconcile them under the same interface.

Describe alternatives you've considered

Not sure if such a capability can be composed using the existing interfaces.

Additional context

No response

[alamb]

Thank you @trungda

I think it would be very interesting to build a "parallel downloader" ObjectStore implementation, though I am not sure it necessairly belongs in the core object_store crate (though it could be added if there is enough interest)

There might also be some interesting ideas to explore around "racing reads" to avoid latency

There are many good ideas in this paper, BTW: https://dl.acm.org/doi/10.14778/3611479.3611486

I think you could compose this kind of smart client from the existing interfaces

[trungda]

Thanks @alamb for the pointer. Also, I just realized that I filed this issue in the wrong repo. It should have belonged to arrow-rs :) Let me move it there to get discussion from other community members?

[trungda]

Closed in favor of apache/arrow-rs-object-store#274.