Creating Complex UI in Background

[gebodal]

Hello all,

I'm having an issue with an application ported from WPF, where the application freezes while a complicated UI element is being constructed.

The UI element is essentially one big DrawingGroup, with many nested sub-elements, mostly GeometryDrawing and other DrawingGroup objects. In the WPF app, these are built in a background thread while the main app continues to be responsive for editing (imagine something a bit like an Overleaf-style editor, with text box on one side and a visual representation on the other).

In WPF, this worked well enough, as it allows you to construct UI elements on a different thread as long as they're not active, but now in Avalonia I can't construct DrawingGroup, GeometryDrawing, or PathGeometry objects in this background thread. Otherwise I get something like:

System.InvalidOperationException: Call from invalid thread
   at Avalonia.Threading.Dispatcher.<VerifyAccess>g__ThrowVerifyAccess|16_0()
   at Avalonia.AvaloniaObject..ctor()
   at Avalonia.Media.Geometry..ctor()
   at Avalonia.Media.StreamGeometry..ctor()
   at Avalonia.Media.PathGeometry..ctor()
   at MyApp.DesignerGenerator.Run() in C:\Path\To\File\DesignerGenerator.cs:line 140

The issue is that the engine backing up this layout is doing quite a lot of work while building the UI elements - as it's quite an involved layout system. In theory I could build an intermediate representation of the geometry, but this seems like it's just going to shorten the freezing problem rather than really get around it, as this representation will still have to be converted into Avalonia elements at some point.

Now, I know Avalonia is constrained to be mono-threaded for the UI due to hardware restrictions for some devices, so I get why I can't build high level UI components on another thread. But why can't I build PathGeometry objects? Or, alternatively, how can I build my geometry objects on a background thread so that the UI remains responsive while building this large nested structure?

I wondered about a custom rendering pass of some kind, but the UI would still pause while it was doing this, right? Can I use this "render to an off-screen location" trick I've been hearing about from a different thread? But then I'm rendering to a bitmap, right, and will need to deal with resolution issues (as this canvas needs to be zoomable)?

Or is the solution to build an intermediate representation and just try and make the conversion from representation to DrawingGroup as efficient as possible? This would preserve the vector representation for zooming with a LayoutTransform, but maybe is inefficient in other ways I'm not appreciating?

Your suggestions and wisdom are much appreciated.

[timunie]

You may need to use Dispatcher.UiThread.Post after you have the PathData. Also you may want to check how Svg.Skia hanldes it for Avalonia: https://github.com/wieslawsoltes/Svg.Skia

[maxkatz6]

Now, I know Avalonia is constrained to be mono-threaded for the UI due to hardware restrictions for some devices, so I get why I can't build high level UI components on another thread. But why can't I build PathGeometry objects? Or, alternatively, how can I build my geometry objects on a background thread so that the UI remains responsive while building this large nested structure?

You are right, that some simpler primitives like geometry is possible to be built on secondary threads in theory. It wasn't implemented in supported in Avalonia yet, as we weren't prioritizing it.
It might not be too difficult to implement at this point.

If you need to build really large primitives and render it on secondary threads, you might need to skip whole PathGeometry layer.
Build your geometry on background threads and pass it to ICustomDrawOperation implementation. It will always be executed on render thread, meaning whole construction-to-rendering phase can skip UI thread altogether.
Although it's only efficient, if your geometry can be represented with a series of ImmediateDrawingContext draw operations.

[gebodal]

Thank you for the feedback, I have a few follow-up questions, if you don't mind!

1. Does Avalonia have a pre-built way of constructing and storing ImmediateDrawingContext draw operations, or would I have to roll my own here? If so, do you have any thoughts on important design aspects that should be taken into account here?
2. When you say "render thread" as opposed to "UI thread", does that mean there are at least two threads governing the UI overall, one for the interactions (I assume) and one for the actual pixel rendering? And if so, are the draw operations done entirely on the rendering thread?
3. You're mentioning Skia separately to the ICustomDrawOperation, which I'm guessing means I don't have use the context.TryGetFeature<ISkiaSharpApiLeaseFeature>() call, and can instead do everything using the ImmediateDrawingContext passed to ICustomDrawOperation.Render. What are the pros and cons of these two approaches (Skia vs ImmediateDrawingContext)? Is it just a matter of execution vs implementation efficiency?

(As an aside, I did implement an intermediate representation of the geometry, and with a little caching and delayed execution, it's not too bad, but there's still a noticeable lag in the whole application when the document is first displayed. So maybe getting a bit more involved with the drawing context might be a nice touch...)

[maxkatz6]

1. RenderTargetBitmap can be considered as one, as it will store operation recorded into it, and output them when drawn, without rasterizing them ahead. But AFAIK, it's CPU bound only, which might or might not affect your app performance.
2. Yes, at least two threads (on most of platforms at least, browser is normally single threaded). Control.Render(DrawingContext) is called on the UI thread as it's more convenient - this method will record draw operations that later will be sent to the render thread. And yes, actual rendering will be done on render thread, based on draw operations recorded on the UI thread. A hint - whenewer you see ImmediateDrawingContext in the API (like ICustomDrawOperation) - this drawing context is operating with render target directly.
3. Yes, you don't have to use ISkiaSharpApiLeaseFeature with ICustomDrawOperation. ImmediateDrawingContext has limited API surface, limited to basic Draw operations, while ISkiaSharpApiLeaseFeature gives you way more flexibility with direct access to SKCanvas (custom shaders, complex geometry, whatever Skia supports). Both are only supported from the render thread.

[gebodal]

OK, let's see if I got this right:

• We have Control.Render(DrawingContext) and ICustomDrawOperation.Render(ImmediateDrawingContext).
• Control.Render is called from the UI thread, and can (must always?) pass an ICustomDrawOperation object to the DrawingContext.
• The draw operation object holds the information needed to draw the control to the ImmediateDrawingContext (quite possibly via SkCanvas, as ImmediateDrawingContext really is limited!), and performs those operations when ICustomDrawOperation.Render is called.

1. Alright, so, how often is Control.Render called? I'm assuming not too often, if a new ICustomDrawOperation object is being created for each call of Control.Render in the CustomSkiaPage?
2. Are there any memory/concurrency issues with creating a single ICustomDrawOperation object (all readonly fields, say) to be stored on the Control itself, and simply passed to the DrawingContext for each call to Control.Render?
3. But then how do I manually trigger a new call to Render so that a new draw operation object is created (with potentially new data)?
4. How about storing SkPaint or SkPath objects on a Control?
5. I'm also assuming that the pixel values that are calculated from ICustomDrawOperation.Render are then cached until something like InvalidateVisual is called, and just translated as needed until then? Or is this not actually determing pixel values, and simply building a data structure which itself will calculate the pixel values at some later point? I'm trying to work out where the efficiency is coming from for rendering large geometry collections - as once my geometry/drawing data structure has been created on the UI thread, it renders very responsively even with a large number of window resizes. And given that the example ICustomDrawOperation.Render in CustomSkiaPage contains a lot of new objects, I'm guessing that isn't happening every frame?

That was a lot of questions, so my thanks if you find the time for a reply!

[maxkatz6]

Alright, so, how often is Control.Render called?

When visual is invalidated and needs to be re-rendered. You also can cache/reuse ICustomDrawOperation object. It doesn't have to be new each time.

Are there any memory/concurrency issues with creating a single ICustomDrawOperation

You just need to be careful updating mutable ICustomDrawOperation properties, as you deal with multithreading there.

to be stored on the Control itself

It can be dangerous, if any AvaloniaProperty is used from the ICustomDrawOperation, as AvaloniaProperties can only be accessed on the UI thread.

But then how do I manually trigger a new call to Render

InvalidateVisual(). If you want to invalidate it to be rendered on the next frame, then TopLevel.GetTopLevel(this).RequestAnimationFrame(t => InvalidateVisual()) or just window.RequestAnimationFrame(t => InvalidateVisual()). You can call RequestAnimationFrame from the inside of Control.Render method, if you want repeating rendering on each display frame.

How about storing SkPaint or SkPath objects on a Control?

Not sure if I understand. You can store them there and reuse, sure. If it's mutable, some locks might be required.

I'm also assuming that the pixel values that are calculated from ICustomDrawOperation.Render are then cached until something like InvalidateVisual is called

No. When ICustomDrawOperation.Render is called Avalonia is rendering new frame on the render thread, these operations are going to be sent to the rendering target and flushed to be displayed on the screen during the same frame.

Or is this not actually determing pixel values, and simply building a data structure which itself will calculate the pixel values at some later point

Technically, the latter. GPU will rasterize and get pixel values later after rendering target was flushed. It will happen during the same rendering frame, but after all the visuals and operations have completed.

And given that the example ICustomDrawOperation.Render in CustomSkiaPage contains a lot of new objects, I'm guessing that isn't happening every frame?

It does happen every frame. This skia page could be written better, yeah. Especially with skia sharp, these objects ideally should be reused/pooled.

[gebodal]

OK, I think that sounds like a solution, then.

So, if I subclass ICustomDrawOperation and store all the Skia data objects needed to draw the content in that object, which I then reuse until the content changes, that should more-or-less work as desired.

Follow-on (apologies): How does this work with LayoutTransform? Currently I have a Canvas nested inside a LayoutTransform, with the transform providing zooming, and all of this inside a ScrollViewer. Presumably Avalonia/Skia will deal with the viewport and rasterising of the vector paths given to Skia, and I'll get nicely zoomable vector graphics with my custom control...? (He says, hopefully)