add @as_safe_channel

[jakkdl]

Mostly just shepherding @Zac-HD's implementation from #638 (comment)

I don't understand all the details of it, esp some of the code paths I'm completely failing how to cover, so gonna need some help. And I'm getting an exception that sometimes disappear.. which seems bad?
Feel free to add code suggestions and/or commit directly to the branch.

I don't know if this fully resolves #638, or if there's docs and/or other stuff that should be added.

I will make https://flake8-async.readthedocs.io/en/latest/rules.html#async900 suggest using this, and perhaps make it enabled by default, once released.

[A5rocks]

I'm not too sold on how some of this is done, but at least now it shouldn't fail CI.

edit: I'm also pretty sure the more correct way to fix the race condition would be using nursery.start instead of nursery.start_soon.

[jakkdl]

I'm not too sold on how some of this is done, but at least now it shouldn't fail CI.

edit: I'm also pretty sure the more correct way to fix the race condition would be using nursery.start instead of nursery.start_soon.

I tracked down the race condition to send_chan.aclose() getting cancelled, swallowing the exception, i.e. #1559

We still want to use start instead of start_soon to make sure the iterator gets properly closed.

The reason it works to move the send_chan closure to across the yield is simply because that allows the exception in _move_elems_to_channel to bubble up into the nursery without getting eaten (except if aclosing(aiterable) eats it), and the send channel can raise as much Cancelled as it wants in context_manager.
We do need to manually call send_chan.close() in _move_elems_to_channel though to halt any iterations over recv_chan.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00000%. Comparing base (737d96a) to head (58e03c9).
Report is 36 commits behind head on main.

Additional details and impacted files

@@               Coverage Diff                @@
##                 main        #3197    +/-   ##
================================================
  Coverage   100.00000%   100.00000%            
================================================
  Files             124          124            
  Lines           18848        19038   +190     
  Branches         1277         1287    +10     
================================================
+ Hits            18848        19038   +190     

Files with missing lines	Coverage Δ
src/trio/__init__.py	100.00000% <ø> (ø)
src/trio/_channel.py	100.00000% <100.00000%> (ø)
src/trio/_tests/test_channel.py	100.00000% <100.00000%> (ø)
src/trio/_tests/test_util.py	100.00000% <100.00000%> (ø)
src/trio/_util.py	100.00000% <100.00000%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jakkdl]

@Zac-HD if you can show why the inner loop is necessary it'd be great, but I'm kinda suspecting it's a remnant of previous implementations or something - because I can't come up with anything that would hit that code path.

[Zac-HD]

I think this'll help? But haven't run many tests yet...

[jakkdl]

I have no clue why sphinx fails to link the AbstractAsyncContextManager in the type hint, when it works perfectly fine for https://trio.readthedocs.io/en/stable/reference-core.html#trio.open_nursery

[TeamSpen210]

Looks good and useful.

This is unrelated, but while we're editing the async-gen docs, we might want to swap out the async_generator.aclosing reference to point to contextlib first, with async_generator as a backport.

[A5rocks]

The code itself looks good!

[jakkdl]

does anybody have any idea about the RTD fail?

I have no clue why sphinx fails to link the AbstractAsyncContextManager in the type hint, when it works perfectly fine for https://trio.readthedocs.io/en/stable/reference-core.html#trio.open_nursery

[Zac-HD]

Let's merge as-is then.

I don't want to split into multiple functions (linter recs etc get harder), we've already disabled .throw() by handing the user a channel instead of a generator object, and imo consistency with the obvious way to do similar things downstream is pretty nice.

[jakkdl]

sorry, with throwing I meant #3197 (comment)

for linter recs we'd just always recommend background_with_channel - and then in its docstring we'd mention the existence of the buffered version in case you want that

[Zac-HD]

#3197 (comment) doesn't bother me, because cancellation is always a possibility you have to handle. It "just so happens" that we always cancel "right before" the GeneratorExit would show up 😉

In practice most users are never going to touch the default, so maybe we should only supply the unbuffered version, and let power-users add their own buffering if desired? We could even share a recipe for the "await send_chan.send(x) instead of yield x" version, which is quite simple and avoids all the complexity of having a generator object in there...

[jakkdl]

In practice most users are never going to touch the default, so maybe we should only supply the unbuffered version, and let power-users add their own buffering if desired? We could even share a recipe for the "await send_chan.send(x) instead of yield x" version, which is quite simple and avoids all the complexity of having a generator object in there...

Okay yeah this seems fine. The only time this would be insufficient is if they're relying on an external asyncgen they can't modify and they want buffering, but if so they can still just write an equivalent version of buffer_it.

But god I am not looking forward to another go at the "oh just unwrap the exception from inside the group"; I'm tempted to go create a helper for that in a new PR, which would be pretty great for trio-websocket and probably others as well: https://github.com/python-trio/trio-websocket/blob/bec4232178700e53dccb887d028997f6746e91de/trio_websocket/_impl.py#L218

[Zac-HD]

OK, I think this is the last round of comments from me:

• Yeah, let's make the RecvChanWrapper clone-able. I don't have a use-case in mind but would prefer to avoid gaps vs the memory-channel interface. changed my mind on this; let's release without .clone() and follow up later if we get a feature request.
• Needs changelog entry etc., since I think it's worth releasing this after we merge (and then updating linters etc.)
• Now that there's no buffering, I think @trio.background_with_channel makes less sense as a name. Thoughts?
  • Current ideas: @trio.safe_(async_)?generator, @trio.generator(_to_|2|_as_)channel, variations. Prior art is named trio_async_generator, and includes buffering (size-zero channel, no semaphore).
  • "background" is just inaccurate now and so I'd fairly strongly prefer to rename it.

[Zac-HD]

false alarm, the simple idea leaks cancellations from the generator into the caller (cf PEP-789) 😞

[A5rocks]

Could we add a test for that in this PR too at least

[Zac-HD]

@generator_as_channel
async def agenfn():
    with trio.CancelScope() as cscope:
        cscope.cancel()
        yield
        
async def test_doesn't_leak_cancellation():
    with pytest.raises(AssertionError):
        async with agenfn() as recv_chan:
            async for _ in recv_chan:
                pass
        raise AssertionError("should be reachable")

[Zac-HD]

Three things below - I'm very keen to merge and thus deferred .clone(), but I also think we need to get the core interface right the first time 😅

• The big-but-easy one is renaming the function @as_safe_channel; see below for why
• We can drop some docs now that the interface is simpler 🎉
• Two small but important correctness changes.

(once more into the diff, dear friends!)

[Zac-HD]

I suggest we ship an initial version without .clone() support, and come back later if anyone asks for it. This will already be super valuable, and I'd rather not wait any longer!

[jakkdl]

I was mostly thinking if we have any strong reasons not to offer it, implementing it should be straightforward.
Though looking at the interface of MemoryReceiveChannel we also don't have receive_nowait on the wrapper and I'm not seeing how that one would make sense to offer - so let's stick to the interface of ReceiveChannel

[Zac-HD]

This looks great to me!

• tiny tweaks for changelog and coverage below
• ✅ Switch from PurePath#as_uri to PurePath#joinpath for our test that PurePath methods get inherited #3249 has fixed the unrelated test failures

then let's ship it!

[jakkdl]

Okay I added a testcase for multiple cancellations, and current implementation will break code that does try: <agen stuff>; except Cancelled: ..., so I'll change raise_single_exception_from_group to only unwrap one layer of a group (and then handle multi-cancellation/multi-keyboardinterrupt) and make use of it

[Zac-HD]

Phew! That was more of a saga than I expected, but I'm very very happy to have it in. Thanks again to @jakkdl, and to everyone else who helped out with design and code review - I think we got to a much better solution than my original proposal.

[jakkdl]

turns out this shit is complicated at times 😅

But yes thank you all for initial design, reviews, and other feedback!