Support for back-pressurized, partially consuming Sink

[mzabaluev]

There may be a case for Sink implementations that can consume their items partially as internal capacity allows. A simple example is a buffered write adapter that accepts Bytes or slice items. In futures 0.1, start_send for such a sink could process a leading part of the slice that fills the output buffer, and return the remaining part in AsyncSink::NotReady. It is not implied by the documentation that the returned item might not be the same as the one passed into start_send, but technically there is nothing wrong with such an implementation, and it could be retroactively sanctioned.

In the 0.3 API under development, the signature of start_send does not allow returning the item, and ready/not ready decision making is separated in the poll_ready method which must operate solely on internal state of the receiver. While this simple and clean design may be meant to remove ambiguities like the one described above, I wonder if the use case allowing more flexible utilization of data buffers, or any likewise divisible data, was consciously discarded.

[tikue]

I think you can accomplish something similar with 0.3 Sinks by having an internal field to stage the buffer in your Sink impl. Then you return Ready only when the staging field has been fully written and can be replaced.

[mzabaluev]

@tikue The Sink impl still has to internalize or borrow the entire buffer, and there is a forced "pipeline break" where a partially consuming API could enable filling up the input buffer as it gets consumed by the writer. I think a solution with 0.3 may need a special adapter or a trait.

[tikue]

@mzabaluev I think I'm imagining this differently. In the case of Bytes, a value of type Bytes is a fixed size regardless of the size of the backing buffer on the heap. So a Sink should always be able to internally buffer the whole Bytes value.

[Matthias247]

I think that use-case is very special, and other Traits (e.g. AsyncWrite) make more sense for it. Sink is a trait which is dedicated for writing single Items in an "atomic" fashion. Maybe there could be a version of Sink that allows to send multiple items at once in a single call - for this this might be more applicable.

[mzabaluev]

@Matthias247 I agree in general. AsyncWrite, though, is a low-level trait for sending byte chunks. To address use cases like encoding a sequence of objects, or asynchronous incremental serialization of a complex object, a more generic solution may be needed, I think.

[dekellum]

See also #1665, which currently includes an external implementation of a TardySink trait and OneBuffer adapter to Sink.

[cramertj]

I don't have any planned changes related to this, and there hasn't been any discussion for over a year so i'm going to close this, but feel free to comment and I can reopen for discussion if there's a concrete proposal for improving the existing traits.