Vulkan based 3D rendering in Avalonia iOS app using MoltenVK

[llfab]

In this thread I want to explain the important aspects that allowed me to make vulkan based 3D rendering work on iOS devices.

Starting Point

We have a desktop app that targeted Windows and Linux at first. Also we have our own scene graph based 3D renderer that has a vulkan rendering backend only. The integration between the 3D renderer and Avalonia was done through Avalonia's GPU interop API that allows for texture sharing between the Avalonia composition renderer and the 3D renderer even if Avalonia is driven by OpenGL and the 3D renderer is driven by vulkan.
As the GPU interop only supports Windows and Linux and not macOS we have created an additional integration between the 3D renderer and Avalonia that renders into a bitmap that then is put to the screen with DrawingContext.DrawImage(_texture, sourceRect, targetRect);. This approach introduces an additional cycle between CPU and GPU which still leads to ~180fps on Apple silicon. So good enough for now.
The pure vulkan 3D renderer works once you install VulkanSDK on macOS and LibraryImport from the "libvulkan.dylib" instead of vulkan-1.dll on Windows or similar on Linux.
Hence, as a starting point we had a working solution not only on Window and Linux but also on macOS using MoltenVK.

Goal

As a goal we wanted the very same app work on iOS (iPad and iPhone), even if the existing app layouts would not fit the reduced screens well

The way it worked

Native AOT

We base our apps on our own core libraries that have grown to ~400K LoC. There are some dynamics like System.Reflection.Emit being used, hence, I had to find alternatives to support Native AOT

Assets / Resources

In many apps we just use project settings like:

<ItemGroup>
  <None Include="*.ttf" CopyToOutputDirectory="PreserveNewest"/>
</ItemGroup>

which do the job for desktop apps. However, for app bundled apps on iOS you need to package stuff correctly:

• do not reference anything outside your app folder
• do not use <None ... > because that is not packaged into your app bundle (Warnings during build)
• some Xamarin magic tries to guess where your file belong to based on file extensions. In most cases you do not want that guess, hence, you need to provide PublishFolderType manually

<ItemGroup>
  <Content Include="*.ttf" CopyToOutputDirectory="PreserveNewest" CopyToPublishDirectory="Always" PublishFolderType="Resource" />
</ItemGroup>

Note: if you have a shared (or other transitive) assembly like with the default Avalonia x-plat template (MyProject, MyProject.Desktop, MyProject.iOS), some of the asset features in the project file only work if you multi-target that assembly:

<TargetFrameworks>net9.0;net9.0-ios</TargetFrameworks>

Above PublishFolderType="Resource" would e.g. not work if this transitive assembly is not multi-targeted containing the -ios target.

Make MoltenVK part of the app bundle

Obviously, on iOS you cannot install VulkanSDK, hence, you must packages the binaries (and other files) into your app bundle. Note that the packaging for iOS is different than the one for macOS. On macOS you can package the *.dylib files whereas on iOS you should package.(xc)framework bundles.

Most of it is explained here: https://www.khronos.org/blog/developing-with-vulkan-on-apple-ios
I just found it a bit more difficult to translate that into a .NET based application.

Here is what has worked for me:

• Take below files from the installed VulkanSDK (on macOS) and put it into your repo
• make sure that the .(xc)framwork bundles are put into the ./Frameworks folder of your app bundle
• make sure the vulkan loader json file folder structure is put into ./vulkan folder of your app bundle

Therefore, the project settings look as follows:

<!-- Package the reference to MoltenVK -->
 <ItemGroup>
   <NativeReference Include="..\..\..\Deploy\ThirdParty\Vulkan\macos-ios-any\1.3.290\iOS\lib\MoltenVK.xcframework">
     <Kind>Framework</Kind>
   </NativeReference>
 </ItemGroup>

 <!-- For non iOS simulator builds add the vulkan loader and validation layer -->
 <ItemGroup Condition="!$(RuntimeIdentifier.Contains('simulator'))">
   <NativeReference Include="..\..\..\Deploy\ThirdParty\Vulkan\macos-ios-any\1.3.290\iOS\lib\vulkan.framework">
     <Kind>Framework</Kind>
   </NativeReference>
   <NativeReference Include="..\..\..\Deploy\ThirdParty\Vulkan\macos-ios-any\1.3.290\iOS\lib\VkLayer_khronos_validation.framework">
     <Kind>Framework</Kind>
   </NativeReference>
   <None Include="..\..\..\Deploy\ThirdParty\Vulkan\macos-ios-any\1.3.290\iOS\share\vulkan/**/*" CopyToOutputDirectory="PreserveNewest" PublishFolderType="RootDirectory" Link="vulkan/%(RecursiveDir)%(Filename)%(Extension)" />
 </ItemGroup>

(Note: vulkan loader and validation layers are not supported for the iOS simulater)

LibraryImport library path (for your own vulkan binding)

Where you would use "vulkan-1.dll" on Windows or "libvulkan.dylib" on macOS, that is different for the app bundled .(xc)framworks. Here, the rpath syntax is used to specify the lib to be loaded. Also you need to account for using vulkan loader or just the pure MoltenVK. Again: there is no vulkan loader on iOS simulator and in your release version you want to use only MoltenVK without the vulkan loader for performance reasons.

These library names have worked for me on the different platforms:

        private const string VulkanWindowsLibName = "vulkan-1";
        private const string VulkanLinuxLibName = "libvulkan.so.1";
        private const string VulkanMacOSVulkanLoaderLibName = "libvulkan.dylib";
        private const string VulkanMacOSMoltenVkLibName = "MoltenVK.dylib";
        private const string VulkanIosVulkanLoaderLibName = "@rpath/vulkan.framework/vulkan";
        private const string VulkanIosMoltenVKLibName = "@rpath/MoltenVK.framework/MoltenVK";

I then use NativeLibrary.SetDllImportResolver to set the proper library for the given OS platform.

Conclusion

Vulkan on macOS and iOS is supported by Khronos (so it does not only work by accident) and works nicely in an Avalonia app that consists on a single source with stubs for .Destkop and .iOS.
The important tricks are:

• make Native AOT work
• get your assets right
• package MoltenVK with your app and use the right library path

[danipen]

Thanks for the detailed explanation. I think it will be super useful for someone trying to do the same in the future. Some questions...

• Do you use metal or OpenGL rendering in macos?
• Do you have a minimal app/sample that demonstrate the techniques you used for 3d rendering in all the platforms?

Thanks!

[llfab]

I cannot release a full sample app because it is closed source. But I can certainly put together a sample that shows the 3 projects (Sample, Sample.Desktop, Sample.iOS) and its settings. Good idea. I will try to work something out.

Avalonia rendering backend:
There is as a short and a longer answer to this. The short one is: we use the Avalonia default settings. I believe that is OpenGL on macOS and Metal on iOS, as far as I understand.

The long answer:
On Windows wie must use OpenGL (which also is default) because of the nature of GPU interop API that makes us of OpenGL/Angle and DXGI to create a shareable texture that we can use in vulkan. There are some vk esension methods used for this.
On macOS there currently is not GPU interop, hence, no shared texture. As we are using the "cheep" workaround to vk render into a bitmap and the draw this in an Avalonia control, no specific Avalonia / Skia render backend is necessary. The same goes for iOS. However, we are working on a GPU interop for macOS too, wich may bring such a dependency to use a specific Skia render backend. We are also thinking about such a GPU interop for iOS which may introduce a similar dependency on Skia render backend for iOS.

Hope that answers what you wanted to know.

[danipen]

Please let us know if you find the time to build a sample app!

[llfab]

As I will not be able to release the closed source, I still wanted to provide the main content of the project files.

The shared project for both desktop and iOS

<Project Sdk="Microsoft.NET.Sdk">
  
  <PropertyGroup>
    <OutputType>Library</OutputType>
    <TargetFrameworks>$(DotnetDefaultVersion);$(DotnetDefaultVersion)-ios</TargetFrameworks>
    <Platforms>AnyCPU;x64</Platforms>
    <AvaloniaUseCompiledBindingsByDefault>true</AvaloniaUseCompiledBindingsByDefault>
  </PropertyGroup>

  <Import Project="..\..\..\BuildConfig\AvaloniaLibrary.props" />
  <Import Project="..\..\..\BuildConfig\AvaloniaFluentTheme.props" />
  
  <ItemGroup>
    <Content Include="Assets\**" CopyToOutputDirectory="PreserveNewest" CopyToPublishDirectory="Always" PublishFolderType="Resource" />
  </ItemGroup>

  <ItemGroup>
    <AvaloniaResource Include="AssetsEmbedded\**" />
  </ItemGroup>

</Project>

The props files just contain the references to the avalonia packages

• Target both net9.0 and net9.0-ios
• Package as Content or AvaloniaResource.

The Desktop Project

<Project Sdk="Microsoft.NET.Sdk">
  
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>$(DotnetDefaultVersion)</TargetFramework>
    <Platforms>AnyCPU;x64</Platforms>
  </PropertyGroup>

  <PropertyGroup>
    <ApplicationManifest>app.manifest</ApplicationManifest>
  </PropertyGroup>

  <Import Project="..\..\..\BuildConfig\AvaloniaApplication.props" />

  <ItemGroup>
    <ProjectReference Include="..\AvaloniaIosApp\AvaloniaIosApp.csproj" />
  </ItemGroup>

  <!-- Include the Vulkan dynamic libraries for macOS -->
  <ItemGroup Condition="'$(Platform)' == 'AnyCPU'">
    <Content Include="..\..\..\Deploy\ThirdParty\Vulkan\macos-ios-any\1.3.290\macOS\lib\libMoltenVK.dylib">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Always</CopyToPublishDirectory>
      <Link>runtimes\osx\native\libMoltenVK.dylib</Link>
    </Content>
  </ItemGroup>

</Project>

• Reference the shared project
• Reference Avalonia Desktop
• Get the libMoltenVK.dylib from Vulkan SDK and package as content. Putting it into runtimes\osx... made it to be found on MacOS when running the app from IDE. For publishing there may be better folders.

The iOS project

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>$(DotnetDefaultVersion)-ios</TargetFramework>
    <Platforms>AnyCPU</Platforms>
    <SupportedOSPlatformVersion>13.0</SupportedOSPlatformVersion>
  </PropertyGroup>

  <PropertyGroup>
    <!--<RuntimeIdentifier>iossimulator-x64</RuntimeIdentifier>-->
    <RuntimeIdentifier>ios-arm64</RuntimeIdentifier>
    <CodesignKey>...</CodesignKey>
    <CodesignProvision>iOS Team Provisioning Profile: ...</CodesignProvision>
  </PropertyGroup>

  <Import Project="..\..\..\BuildConfig\AvaloniaApplication.Ios.props" />

  <ItemGroup>
    <ProjectReference Include="..\AvaloniaIosApp\AvaloniaIosApp.csproj" />
  </ItemGroup>

  <!-- Package the reference to MoltenVK -->
  <ItemGroup>
    <NativeReference Include="..\..\..\Deploy\ThirdParty\Vulkan\macos-ios-any\1.3.290\iOS\lib\MoltenVK.xcframework">
      <Kind>Framework</Kind>
    </NativeReference>
  </ItemGroup>

  <!-- For non iOS simulator builds add the vulkan loader and validation layer -->
  <ItemGroup Condition="!$(RuntimeIdentifier.Contains('simulator'))" >
    <NativeReference Include="..\..\..\Deploy\ThirdParty\Vulkan\macos-ios-any\1.3.290\iOS\lib\vulkan.framework">
      <Kind>Framework</Kind>
    </NativeReference>
    <NativeReference Include="..\..\..\Deploy\ThirdParty\Vulkan\macos-ios-any\1.3.290\iOS\lib\VkLayer_khronos_validation.framework">
      <Kind>Framework</Kind>
    </NativeReference>
    <None Include="..\..\..\Deploy\ThirdParty\Vulkan\macos-ios-any\1.3.290\iOS\share\vulkan/**/*" CopyToOutputDirectory="PreserveNewest" PublishFolderType="RootDirectory" Link="vulkan/%(RecursiveDir)%(Filename)%(Extension)" />
  </ItemGroup>

</Project>

• Reference Avalonia iOS
• Reference the shared project
• Package MoltenVK from its xcframework from Vulkan SDK
• Package Vulkan loader, validation layours and necessary json files and folders from Vulkan SDK
  • the libs will be in the "framework" folder of the app bundle
  • the json files and folders wiil be in the "vulkan" folder of the app bundle
  • Note: vulkan loader does not work on iOS simulator, hence, do not install it there

Hope that helps

[danipen]

Thank you very much for sharing this setup. I'm pretty sure it will help for people starting this kind of project. Please also if you find the time to build a "hello world" (drawing a triangle or so) vulkan app with Avalonia to understand how things are initialized it would be really useful I think

[dbriard]

Thank you so much for the explanations!
And I can't agree more, a minimal hello world would be super helpful.
If you could do that, that would be awesome :)

[llfab]

I will not be able to provide a running hello world. The 3D renderer is closed source as are all our base libs that also help to the wiring.

I may be able to provide something that would not build but just shows the principle. But that will consume some effort :-)

[llfab]

You can find a non-working sample here: https://github.com/llfab/Samples/tree/main/AvaloniaIosApp

Again it is heavily dependent on closed code. Hence, you will not be able to build. Anyhow, I will show all 3 assemblies XY, XY.Desktop, XY.iOS