Questions about physical memory management and memory management for passthrough devices

[hky1999]

Physical memory allocation for guest VM

Currently, in axaddressspace, we use alloc_frame from PagingHandler trait to allocate backend physical address for MemoryArea from memory_set, which allocates at the unit of single frame and may not be contiguous.

As a result, a contiguous MemoryAreas backend memory may spread into different physical frames. Which make it complicate for accessing the guest VM's address space within the hypervisor. A guest MemoryArea with a contiguous GPA (Guest Physical Address) may be non-contiguous on the HPA (Host Physical Address). For operations like loading the guest image, we cannot simply access the guest memory by converting the image load address (GPA) to HPA and then using phys_to_virt to convert it to HVA (Host Virtual Address), because contiguous GPAs may be mapped to non-contiguous HPAs.

Two possible solutions:

1. allocate contiguous physical frames for MemoryArea

One possible solution is that we make MemoryArea’s physical frames contiguous, through methods like alloc_contiguous in GlobalPage of axalloc, in this case we can easily translate GPA into HPA, and then access corresponding HVA through raw pointer dereference within the limit of MemoryArea size.

However, current PagingHandler trait provided by page_table_multiarch does not support this semantics; it can only allocate one page frame at a time (through alloc_frame()). To use the alloc_contiguous method of GlobalPage, we need to rely on axalloc. However, I only want axaddresspace to hold a PagingHandling generic.

So, the first method might require modifying the page_table_multiarch crate, which seems like a significant change.

Furthermore, allocating a large block of contiguous physical memory at once is not flexible enough, making it difficult to support demand paging.

2. Manually concatenate non-contiguous physical address segments

The other solution is more complicated but more flexible. We can keep current frame by frame allocation strategy, while use methods like translated _byte_buffer (like rCore-Tutorial do) to obtain physical page frames corresponding to virtual pages one by one, convert them into byte arrays (&'static mut [u8]), and store them individually in a Vec.

Current implementation:

    /// Translate&Copy the given `VirtAddr` with LENGTH len to a mutable u8 Vec through page table.
    ///
    /// Returns `None` if the virtual address is out of range or not mapped.
    pub fn translated_byte_buffer(
        &self,
        vaddr: VirtAddr,
        len: usize,
    ) -> Option<Vec<&'static mut [u8]>> {
        if !self.va_range.contains(vaddr) {
            return None;
        }
        if let Some(area) = self.areas.find(vaddr) {
            if len > area.size() {
                return None;
            }

            let mut start = vaddr;
            let end = start + len;

            let mut v = Vec::new();
            while start < end {
                let (start_paddr, _, page_size) = self.page_table().query(start).unwrap();
                let mut end_va = start.align_down(page_size) + page_size.into();
                end_va = end_va.min(end);

                v.push(unsafe {
                    core::slice::from_raw_parts_mut(
                        H::phys_to_virt(start_paddr).as_mut_ptr(),
                        (end_va - start.as_usize()).into(),
                    )
                });
                start = end_va;
            }
            Some(v)
        } else {
            None
        }
    }

The downside of this approach is undoubtedly increased complexity. And the translated_byte_buffer or other similar methods need to know the size of the corresponding GPA region in advance. During image loading, we need to know the size of the image file. Currently, the image file size is not included in the VM configuration information and is only obtained when reading the image (though this is not an problem).

We might need to rely on an Enum to wrap the GPA data type from the hypervisor's (HPA & HVA) perspective, encapsulating the complexity of the Vec<&'static mut [u8]> structure within the Enum. Additionally, the Enum would need to implement the std::io::Read and Write traits.

something like:

enum GPAType {
    Linear(&'static mut [u8]),
    Concat(Vec<&'static mut [u8]>),
}

Memory Management for passthrough device & other emulated devices

My second question is relatively simpler and more straightforward.

How do we design the memory management part in AxVMConfig ?

We can use current design of VMMemConfig for normal memory region like RAM of guest VM

#[derive(Debug, Default, Clone, serde::Serialize, serde::Deserialize)]
pub struct VmMemConfig {
    pub gpa: usize,
    pub size: usize,
    pub flags: usize,
}

Should we place the MMIO memory spaces for passthrough devices and emulated devices together with the normal memory regions, or should we place them in the device configuration part ?

Note:

• for passthrough devices, we need to map the GPA to HPA (and possibly configure the SMMU if there are offsets)
• for emulated devices, we do not allocate physical memory for them.

[hky1999]

@aarkegz

[aarkegz]

The downside of this approach is undoubtedly increased complexity. And the translated_byte_buffer or other similar methods need to know the size of the corresponding GPA region in advance.

Yes, it looks awful.

---

I think the key issue is that: how often will we write to guest memory in the hypervisor mode? If it's not uncommon, with large data chunks, then the overhead of the second approach could be intolerable, not to mention the complexity. So generally I think the first one is better. Missing functions like alloc_contiguous in PagingHandler can be introduced in a new sub-trait of PagingHandler, or by some other ways, and it should be a big deal (changing page_table_multiarch may not be that horrible).

However, there IS something to worry: dynamic memory allocation. It's a feature we do not need to implement now but clearly good to have in the future, but obviously, the first approach effectively eliminated the possibility of it.

Luckily, we have ⬇️

We might need to rely on an Enum to wrap the GPA data type from the hypervisor's (HPA & HVA) perspective, encapsulating the complexity of the Vec<&'static mut [u8]> structure within the Enum. Additionally, the Enum would need to implement the std::io::Read and Write traits.

something like:

enum GPAType {
    Linear(&'static mut [u8]),
    Concat(Vec<&'static mut [u8]>),
}

It's a good design. Some extra methods about memory copying allow us to isolate the extra complexity in AddrSpace (with some extra works in MemorySet and MemoryArea).

So, with everything mentioned above, my preference is:

• A sub-trait for alloc_contiguous, or modifying PagingHandler, or something else.
• Statically allocated MemoryAreas with large size, and (in the future) dynamically allocated MemoryAreas with intermediate size.
• A enhanced translated_byte_buffer, which can handle GPAs cross the border of neighboring MemoryAreas (for better performance).

---

Another possible solution (though it may be kind of strange and not very applicable) is to mirror the ept in a reserved range of the page table of arceos. For example, if we have GPA: [x, x + len) -> HPA: [y, y + len) in EPT, then we can map HVA: [base + x, base + x + len) -> HPA: [y, y + len). It strikes me about 10 mins ago and I have not assessed it carefully.

[hky1999]

Missing functions like alloc_contiguous in PagingHandler can be introduced in a new sub-trait of PagingHandler, or by some other ways, and it should be a big deal (changing page_table_multiarch may not be that horrible).

I've discussed with @equation314 about this problem. a new sub-trait of PagingHandler seems redundant, we can just extent alloc_frame method to alloc_frames and add a new parameter named num_frames. It's the sphere of influence that makes me concerned, alloc_frame is widly used in page table related functions.

[aarkegz]

It's the sphere of influence that makes me concerned

That's exactly the reason why I suggested a new subtrait.

[aarkegz]

It sounds like "Linear" mapping, I'm kind of confused with it.

Yes. It means map the guest address space to a reserved range of host virtual address space linearly.

[aarkegz]

Will there be a need for cross-MapArea memory access for the Guest VM?

There may be if we have dynamic memory allocation.

[hky1999]

"dynamic" refers to MemoryArea, where the backend of a MemoryArea may consist of non-contiguous physical page frames.

Typically, a MemoryArea represents a specific type of memory space in the Guest VM, such as device memory or RAM.

Semantically, there are no access requests across MemoryArea for the Guest VM,
so only cross-page frame access within a MemoryArea needs to be considered.