Introduction
In today's post, I'll explain the technique used in the initialisation of my personal type 2 (SVM) hypervisor. This technique, which I shall refer to as "ghost mapping", allows me to load my hypervisor without its driver appearing in any global lists or leaving any memory/page table artefacts.
Disclaimer
The individual techniques discussed are not necessarily novel, but the way they are combined and applied here may be of interest. Note that these techniques will not work on systems with HVCI or VBS enabled.
Why would I want this?
Tools such as debuggers are very useful to malware analysts, vulnerability researchers, defenders and offenders alike. They provide the user with the ability to step through instructions, view process memory, CPU registers, and the stack at any given time, pause the execution of a process at any arbitrary location and more. These abilities allow the user to figure out exactly how some piece of software works, and exactly how it doesn't. This is incredibly useful to say, a malware analyst who is tasked with researching and sampling some new found threat that has yet to be publicly documented. These debuggers can be implemented in a variety of ways, and at a variety of levels, for the most advanced threats you're going to need transparent/invisible tools to debug them as they're actively monitoring for these exact tools. This is where a hypervisor comes in to the rescue, but not without hard work. There are many ways to detect the presence of an unwanted listener, one of which is finding its driver loaded.
Background
PTE Format
An x86/x64 PTE is a 64 bit entry. The key fields relevant to us are: bit 0 (Valid), bit 1 (Write), bit 63 (NX), and bits 12:47 which store the Page Frame Number (PFN), the physical page this entry translates to.
Nested Paging
A very useful feature within AMD's SVM is Second Level Address Translation (SLAT), what this does is add an extra translation layer between guest physical addresses (GPAs) and host physical addresses (HPAs). On a normal system the translation is as follows: system page tables -> physical address -> memory contents, on a virtualised core with SLAT enabled it's instead like this: guest page tables -> guest physical address -> nested page tables -> host physical address -> memory contents. With this second level of translation, address 0x1A000 on the guest does not have to translate to address 0x1A000 on the host, meaning we can have guest's 0x2C000 GPA translate to host's 0x1A000 HPA changing the bytes the guest would've under normal circumstances seen reading that address. To enable nested paging in your own hypervisor you must set the NP_ENABLE bit in your VMCB and set the nCR3 address which is the base of your nested page tables:
vp->GuestVmcb.ControlArea.NpEnable |= SVM_NP_ENABLE_NP_ENABLE;
vp->GuestVmcb.ControlArea.NCr3 = pml4_base_pa.QuadPart;Hiding your hypervisor
Stage One: PTE Hijacking
We need our driver to be invisible, so obviously we can't load it through conventional means, we are going to have to manual map it. Manual mapping comes with its own detections, for instance your allocation usually resides in memory regions unbacked by any legitimate module, and if you do live inside the address range of legitimate module any memory scan is going to come up dirty. We are going to be hijacking the PTEs of a legitimate driver and tackling the mentioned detection in stage two, the hypervisor's NPT initialisation. We need a target to hijack, during my development and testing I went with fvevol.sys (BitLocker), this is because this driver is always loaded on boot by Windows but when BitLocker is disabled the driver never runs and is just dead code taking up unnecessary space. This is perfect because that means that we won't risk any sort of race condition when messing with the page tables. First things first, we must allocate some memory to place our driver, I chose to do this by allocating ImageSize of RW data with MmAllocateIndependentPages. We then proceed with the usual steps of manual mapping and fix up our image. When we hijack the target's PTEs we must align the end of our .text section with theirs, this is because the PTEs we hijack must hold permissions required for the page it points to. Any page we intend to execute must be pointed to by a PTE with the NoExecute (NX) bit zeroed, and any page we intend to read/write must be pointed to by a PTE with the Write bit set. This makes sure that our executable and data sections are stored in PTEs with the necessary permissions. Identify the Virtual Address (VA) of the target's .text section edge, and the VA of your own. We take the VirtualSize of our .text section subtract that from the edge of our target's .text section to get the starting point of our hijack. For example if fvevol's .text ends at VA 0xFFFFF8001234A000 and our .text is 0x3000 bytes, we begin hijacking at 0xFFFFF80012347000. This way our .text section ends with theirs, and our data sections begin with theirs, we are not mapping our PE headers. The bits to determine what physical page a PTE points to are bits 12:47 or the PFN (bits 51:48 are included but typically zero/reserved). To hijack a PTE we take our physical page aligned address and shift it to the right by 12 to get its PFN and we write the result into the PFN bits by first masking out the original PFN and then ORing in our new one, preserving the existing permission and status flags. The pages we don't hijack remain pointing to the driver's original contents at those locations. Finally we execute the entry point of our hypervisor. Below I've attached a diagram to visualise our PTE hijack:
Stage Two: NPT Cloaking
So we've hijacked legitimate address space and are good on that front, but if anyone were to check what memory is actually there we'd be caught instantly. Lucky for us, we're getting virtual. When you're building your Nested Page Tables (NPTs) generally you want to map them with 1GB PDPT huge pages for optimisation purposes, but since we need 4KB granularity we must do what we call "page table splitting" where we are able to keep most PDPTEs as simple 1GB huge pages, but when required we split it down to 2MB large pages with 4KB page tables where required. Doing this allows us to keep as much of the performance benefits of large/huge pages as possible whilst still having 100% control of our memory and page tables. To properly perform your NPT cloaking when building your NPT entries, if the GPA corresponds to a physical page belonging to your hypervisor, you set the NPT PTE's PFN to the original driver's HPA instead of your own. By default NPTs are mapped 1:1 GPA = HPA so when we change the PFN of only our specific pages everything else still maps to the expected locations. With this in place any guest side physical memory read, whether it's through APIs such as MmCopyMemory or manually mapping PTEs, translates through our NPT to the original driver's contents, while our hypervisor continues executing from the real host physical pages. Below I've attached a diagram to visualise our NPT Cloaking:
Detection surface & limitations
This post has intentionally not covered every possible inconsistency. If you were to follow these steps without additional cleanup, you would be left open to detectable artefacts such as inconsistencies within the MmPfnDatabase. Another thing worth mentioning is that in this post I've only talked about hiding the hypervisor's image, but not the allocations it makes. Your VMCB and NPT are still accessible via guest physical memory, if you'd like to mitigate this I'd suggest adding some code to your allocation wrapper that adds every allocated page to a map where you can iterate over and cloak in your NPT setup.
Conclusion
With both stages in place, our hypervisor resides in a legitimate driver's virtual address space and is invisible to guest physical memory inspection. From here we can implement debugging primitives such as hardware breakpoints via intercepts, single step tracing, memory watchpoints and all from a position that's significantly harder to detect than a conventionally loaded driver.