Proposal for V2: occlusion detection

[szager-chromium]

I propose that the IntersectionObserver spec be extended to allow for detection of occlusion by other content.

Here's an early draft of the proposed spec change:

http://szager-chromium.github.io/IntersectionObserver/

Special attention should be paid to this section, which describes the heuristics for occlusion detection:

http://szager-chromium.github.io/IntersectionObserver/#calculate-visibility-algo

The intention with that language is to make it possible to implement the feature efficiently; and to make it maximally useful for the anticipated primary use cases. Here's a small slide deck explaining the motivation and anticipated use cases for this feature:

https://docs.google.com/presentation/d/13-M2eqNKnClEPXiEQK2iwvnk3njqssj4OzDuZCSe_jQ/edit?usp=sharing

[chrishtr]

In terms of efficient implementation, we at Chrome Rendering think it should be relatively straightforward
to implement this spec. We're going to start prototyping shortly.

Here is some pseudocode:

bool TestVisibility(element))  {
  if (root viewport transform of element is not identity or translation)
    return false;

  if (any filters or non-1.0 opacity is applied on any ancestor of element)
    return false;
  Rect root_viewport_rect = RootViewportVisualRect(element)

  if (HitTest(root_viewport_rect) != [element])
    return false;
  return true;
}

Rect RootViewportVisualRect(element) {
  Return the bonuding box of: border box rect of |element|, transformed and clipped, in the root frame's viewport space. In other words, a rect that encloses all pixels which are drawn into by the element.
}

ElementList HitTest(root_viewport_rect) {
  Return list of elements that are the hit test results for a hit test at any integer point within the bounds of root_viewport_rect
}

A rect-based HitTest and RootViewportVisualRect are already implemented in Chrome, and should hopefully already be present in other browsers as well. Edge even has a rect-based
hit testing API exposed to script (msElementsFromRect) already.

In Chrome, the rect-based hit testing is already needed for gesture disambiguation on Android, for example. RootViewportVisualRect is used for raster invalidation.

[ojanvafai]

Bikeshedding: s/isVisible/isGuaranteedVisible/ to make it more clear what it's actually measuring.

[dbaron]

So I don't think Gecko has rect-based hit-testing code, so we'd need to implement it in order to implement this feature. That makes it seem like a pretty substantial feature from our end. (What other web features does your rect-based hit testing support?)

It would be interesting to hear about the use cases for this. Is this something that sites (ads?) are widely simulating today using JS APIs, in ways that perform poorly? Or is it rarely simulated but more requested, or just requested?

Some of the use cases seem to be about checking visibility for various security use cases. If that's a use case, then it creates a new class of security vulnerability: false positive reports in this API. How confident are you that that space is easy to secure? It seems like there are a lot of web features that can occlude or clip things.

[szager-chromium]

We use rect-based hit testing to figure out the target of touch events.

It's not necessary to implement rect-based hit testing, but it's a convenient path to implementation if it's present. My original implementation idea -- and we may yet return to it -- is to first walk the graphics layers and look for overlap with graphics layers that paint on top of the target element; and then to walk the list of painted objects in the target graphics layer, starting from the target element, and check for overlaps.

The primary motivation is to eliminate common patterns of fraud and abuse on the web, and to enable trust relationships between embedded third-party iframes and their host documents. Specifically, it gives the iframe a strong guarantee that its content is visible on screen, and has not been painted over or altered in any way by the embedding document.

Some of the likely use cases are shown in the slide deck, including:

• Click-jacking. Any iframe that receives a click can never be sure that it was visible to the user when it was clicked. Think of embedded 'like' or 'share' or 'sign in with' or 'pay with' widgets.

• ad stacking, a form of ad fraud where multiple ads are displayed stacked on top of each other. Only the top-most ad is visible, but all of the advertisers are charged for the impression.

Given the motivation, the idea is that a conforming implementation may sometimes give false negatives (i.e., report a target as "possible occluded" when it's actually visible), but must never give a false positive (i.e., report a target as "definitely visbile" when it's actually occluded). We're pretty sure that's achievable in chromium, but I don't know enough about other browser implementations to know how difficult it would be for them.

[szager-chromium]

I would also add that, to my knowledge, there is no existing functionality that comes close to what V2 would offer. This is in contrast to V1, which mostly offers information that can be had by other means. I believe V2 would be an entirely novel capability.

[chrishtr]

It would be interesting to hear about the use cases for this. Is this something that sites (ads?) are widely simulating today using JS APIs, in ways that perform poorly? Or is it rarely simulated but more requested, or just requested?

We have spoken with engineers who work on ad tracking systems. It is clear that all large ad tracking
libraries today try to detect occlusion and other click jacking techniques. In addition, there is almost
certainly an arms race between them and those who try to manipulate the system.

For these reasons, even with IOv1, ad trackers still need to run a setTimeout polling operation in order to try to detect occlusion. And in the setTimeout callback they run a lot of expensive code. One example we have seen is that they use the elementFromPoint API at the center point of an ad to simulate a hit test. (This is both expensive and also inaccurate because it can't cross non-same-origin
frame boundaries).

Ad tracking systems which perform such a hit test will of course only get faster if they use the
built-in hit testing we plan to implement in IOv2, which we can highly optimize (as szager mentioned), and run at convenient times that don't impact UX, such as even on different threads or processes
if necessary.

[dakami]

I can confirm with absolute confidence that the clickjacking problem is a mess, there's no existing way to solve it, and people are deploying terrible code all the time in an attempt to try. I ended up getting pretty involved in the subject, building an early PoC to show something could be done graphically to address the issue (code at https://github.com/dakami/ironframe , slides at https://dankaminsky.com/2015/08/09/defcon-23-lets-end-clickjacking/ ).

This is very cool.

[jakearchibald]

For others following this issue: the spec is currently at http://w3c.github.io/IntersectionObserver/v2/

[szager-chromium]

The V2 spec proposal shipped in chromium earlier this year:

https://chromestatus.com/features/5878481493688320

[zcorpan]

AFAIK there still isn't a spec for hit testing (see https://w3c.github.io/csswg-drafts/css-ui/#issue-bdab65a4), although https://w3c.github.io/csswg-drafts/cssom-view/#dom-document-elementsfrompoint exists which depends on hit testing and is implemented across the board https://caniuse.com/mdn-api_document_elementsfrompoint

[zcorpan]

Hmm however elementsFromPoint only does point-based hit testing, not rect-based hit testing.

[chrishtr]

AFAIK there still isn't a spec for hit testing (see https://w3c.github.io/csswg-drafts/css-ui/#issue-bdab65a4), although https://w3c.github.io/csswg-drafts/cssom-view/#dom-document-elementsfrompoint exists which depends on hit testing and is implemented across the board https://caniuse.com/mdn-api_document_elementsfrompoint

You are right that there is no definition of hit testing yet in a specification. Likewise, there is not yet a precise definition in a spec of ink overflow (see w3c/csswg-drafts#8649). Previous discussions of IntersectionObserver occlusion testing have blocked on this problem, because making it precisely defined impacts interoperability.

If we can agree on the usefulness of occlusion detection and the definition in the "v2" IntersectionObserver spec, then we can go and define hit testing and visual overflow elsewhere as needed in order to support specifying occlusion.

Hmm however elementsFromPoint only does point-based hit testing, not rect-based hit testing.

Yes. However, even before this feature, Chromium already needed to do rect-based hit testing for the purpose of mobile hit testing, where user touches are not precise enough to use a single point. The definition of the rect-based hit test is a simple generalization of a point-based hit test.