Added PressedButtons to PickingInteraction::Pressed.

[mintlu8]

Objective

Fixes #15983. PickingInteraction is preferable to events in some use cases for code organization. This PR adds the keys pressed to PickingInteraction::Pressed, allowing the user to ignore certain keys like the RMB in such use cases.

Solution

• Added field PressedButtons to PickingInteraction::Pressed.

Testing

• Added a test for merging PickingInteractions.
• Added an example for PickingInteraction.

Migration Guide

PickingInteraction::Pressed now has a field, code matching against it should use matches!(interaction, PickingInteraction::Pressed(..)) instead.

[alice-i-cecile]

I like this. It makes PickingInteraction much more useful. The use of bitflag packing to reduce memory bloat is clever too.

[alice-i-cecile]

This will be useful as part of #15597, and I'd like to get this merged first.

[BenjaminBrienen]

Many computer mice have a "Back" and a "Forward" button, but I don't know if those affect focus.

[aevyrie]

I'm not sure this is correct. Touch inputs are probably akin to a primary press, and some touch devices, like pens, can have a primary secondary, tertiary press, just like a mouse.

[BenjaminBrienen]

I've had something like that before: my Note 9 has a pen and it supports hovering over something with the pen in the air and clicking a side-button to perform a secondary action.

[NthTensor]

A stylus is not a touch pointer. Winit does not currently support stylus input, so neither do we.

[aevyrie]

It definitely works, I've tested it. It's treated like a touch input. This was on a MS surface with a surface pen.

Regardless, if you are matching on whether a button was pressed, why would you not want it to be considered pressed if it was touched by a different pointing device?

[aevyrie]

This is why I specifically used the wording "primary/secondary". A primary press can come from many types of devices.

[mintlu8]

The PR just maps the input from picking in a form that makes the most sense to me. Definitely would like to see more detailed information and more buttons, but that has to come from picking first.

[mintlu8]

Renamed to left and right.

[aevyrie]

Tying an interaction struct to hardware is asking for people to just default to the mouse, and suddenly a bunch of software doesn't work with other pointing devices for no good reason.

I would take a look at the JS pointer event APIs - they were designed to be hardware agnostic for this reason: https://developer.mozilla.org/en-US/docs/Web/API/Pointer_events#determining_button_states

I would strongly prefer we follow this model, and make the easy path one that is device agnostic. E.g. under the hood, this struct might be a huge bitflag that encodes which device buttons are down, but the easiest-to-access methods are things like is_primary_pressed, which return true if the primary button of any pointing device is down.

I'd also like to ask whether this structure should even exist anymore. It was originally added to match Interaction in bevy_ui. However, with observers, we can do the same thing with less memory and cpu, because we don't need to update the state of every interactive entity even if no one is interested in reading it. That might mean refactoring the pointer events a bit to coalesce button information.

[alice-i-cecile]

I'd also like to ask whether this structure should even exist anymore. It was originally added to match Interaction in bevy_ui. However, with observers, we can do the same thing with less memory and cpu, because we don't need to update the state of every interactive entity even if no one is interested in reading it. That might mean refactoring the pointer events a bit to coalesce button information.

Yeah, this is increasingly my opinion. It's nice for the "change widget style based on interaction", but that's the only use case where it's nicer than observers.

[NthTensor]

That might mean refactoring the pointer events a bit to coalesce button information.

Which I am more than happy to do if we get specs for the events we need. Enter/Leave is already on the radar.

[aevyrie]

Yeah, this is increasingly my opinion. It's nice for the "change widget style based on interaction", but that's the only use case where it's nicer than observers.

We definitely have a need to coalesce multiple events to determine state. You can't, for example, use Over and Out to determine if an entity is hovered, that entity might receive multiple of these events from multiple pointers and bubbled from children. I'm not sure what the right approach is though. I'm kinda leaning towards keeping this struct for that reason though.

I think this PR would be okay to merge with some modifications to make it match the pointer API I linked. It currently mixes the concept of abstract pointer buttons and device button.

1. Change the name to PointerDevicePressed or something similar. This struct tracks which device buttons are down, but presents a public API that aggregates them, to help push users toward correct code. PressedButtons is probably fine too, but we should disambiguate from, say keyboard buttons. PointerPressed?
2. Change PRIMARY/SECONDARY to something like LEFT_MOUSE/RIGHT_MOUSE/MIDDLE_MOUSE. E.g. every bit field should be device specific, this would also include mouse fwd/back and pen barrel when we have them. This structure would become an aggregate of all types of devices, if there are multiple devices (e.g. touch) all states should be OR'ed (like present).
3. Make the fields private if possible, and expose methods primary(), secondary(), middle(). To expose the internal details, add any(button: PointerDeviceButtons) and all(button: PointerDeviceButtons) so they can check if a specific button(s) was pressed.
   • This would also support chording, because you could pass in buttons.any(PointerDeviceButtons::LMB | PointerDeviceButtons::RMB ) or buttons.all(PointerDeviceButtons::LMB | PointerDeviceButtons::RMB ).
   • LMB and touches should both be considered a primary press. RMB and pen barrel (once it is supported) would be secondary presses. Again, follow the spec used on the web.
4. Virtual (custom) pointers should be forced to decide what kind of input they are mocking, e.g. touch/mouse/pen.

[alice-i-cecile]

See here for the mapping from Touch. I think that this is the right approach for the current PR.

[mintlu8]

1. Change the name to PointerDevicePressed or something similar. This struct tracks which device buttons are down, but presents a public API that aggregates them, to help push users toward correct code. PressedButtons is probably fine too, but we should disambiguate from, say keyboard buttons. PointerPressed?

Do you think it is fine to just reserve X1 and X2 on the bitflags? I don't see a reason to encapsulate it further.

https://developer.mozilla.org/en-US/docs/Web/API/Pointer_events#determining_button_states

According to this spec the current implementation is a upgrade on touch input since usually we treat touch as left click, but with this implementation you can potentially treat them differently.