Fixes #29. Ensure cr0/cr4 fixed bit settings prior vmxon. #49

sduverger · 2018-05-02T14:41:34Z

Hi,

Sorry for the delay. Holidays :) It fixes the vmxon issue #29 with code refactoring as discussed in PR46 (you can now ignore).

raphaelning

Thanks a lot! I've made quite a few comments, but I wouldn't ask you to address all of them, since most of them are out of the scope of this patch ("refactoring") and can be addressed later. I'd pick out the one at cpu.c:723 as the only required change.

If you don't have time to improve this patch, let me know if it would be easier if we took over and updated the PR directly.

raphaelning · 2018-05-04T10:16:51Z

core/cpu.c

@@ -636,11 +719,12 @@ uint32 load_vmcs(struct vcpu_t *vcpu, preempt_flag *flags)
            }
        }
 #endif
+        check_vmxon_coherency(cpu_data);
+        restore_host_cr(cpu_data);


I believe we should only restore the CRs when fatal == true. In the !fatal case, we should just act as if __vmxon() were successful (even though it failed because another Mac VMM had enabled VMX for this CPU).

raphaelning · 2018-05-04T10:47:55Z

core/cpu.c

+
+    smp_mb();
+
+    err = __vmxoff();


Here's an idea to avoid exposing restore_host_cr() without moving all __vmxon() error handling to vmxroot_enter():

Basically, we ensure that it's safe to call vmxroot_leave() at all times:

err = vmxroot_enter(); // Do something // Always call vmxroot_leave(), even if vmxroot_enter() failed err = vmxroot_leave();

For this to work, we just need to make sure we do not call __vmxoff() unless we know VMX is already on for the current CPU, or even better, unless it was HAXM that turned VMX on for the current CPU. I think this is exactly what the VMXON_HAX flag is for, so we can check (cpu_data->vmm_flag & VMXON_HAX), assuming vmxroot_enter() always sets this flag after a successful __vmxon(). Of course, we also need to clear the flag after __vmxoff().

raphaelning · 2018-05-07T08:31:23Z

core/include/vmx.h

+struct per_cpu_data;
+extern vmx_error_t vmxroot_enter(struct per_cpu_data *cpu_data);
+extern vmx_error_t vmxroot_leave(struct per_cpu_data *cpu_data);
+extern void restore_host_cr(struct per_cpu_data *cpu_data);


nit: Is there a good reason to use extern in function declarations? It has no effect, and people would expect the function to be defined elsewhere (in a .c file).

To me, restore_host_cr() is too low-level to be exposed as part of the interface. If it has to be exposed, how about using the same prefix for consistency, e.g. vmxroot_restore_host_cr()? Ideally, vmxroot_enter() should take care of error handling, so its caller doesn't have to.

raphaelning · 2018-05-07T08:55:28Z

core/cpu.c

+        ia32_wrmsr(IA32_FEATURE_CONTROL,
+                   fc_msr | FC_LOCKED | FC_VMXON_OUTSMX);
+
+    return __vmxon(hax_page_pa(cpu_data->vmxon_page));


As I said in another comment, it would be nice if we handled__vmxon() errors in vmxroot_enter() and did not expose restore_host_cr(). In fact, there are pieces of logic currently implemented in load_vmcs() that should eventually be moved here:

Updating the global variables for debugging, e.g. log_vmxon_xxx.

Setting the VMXON_HAX bit of cpu_data->vmm_flag if __vmxon() was successful.

Handling the Mac-specific !fatal case.

Handling the fatal case.

Making all these changes would probably be too ambitious for this refactoring effort. But maybe we could get 1 and 2 done first?

raphaelning · 2018-05-07T09:43:14Z

core/cpu.c

+ * Restore host control registers value before entering vmxroot
+ * operations. Must be smp_func_call() safe.
+ */
+void restore_host_cr(struct per_cpu_data *cpu_data)


Let's make this function static when we remove its declaration from vmx.h.

raphaelning · 2018-05-07T09:43:40Z

core/cpu.c

+void restore_host_cr(struct per_cpu_data *cpu_data)
+{
+    if (cpu_data->vmm_flag & VMXON_RESTORE_CR0)
+        set_cr0(cpu_data->host_cr0);


Also clear the VMXON_RESTORE_CR0 bit?

yeah, no debate :)

raphaelning · 2018-05-07T09:43:50Z

core/cpu.c

+        set_cr0(cpu_data->host_cr0);
+
+    if (cpu_data->vmm_flag & VMXON_RESTORE_CR4)
+        set_cr4(cpu_data->host_cr4);


Also clear the VMXON_RESTORE_CR4 bit?

this one too

raphaelning · 2018-05-07T10:14:41Z

core/cpu.c

+    uint64 cr0, cr0_f0, cr0_f1;
+    uint64 cr4, cr4_f0, cr4_f1;
+
+    smp_mb();


I'm not really familiar with memory barriers, and I'm curious why you use it here. Because of this line, smp_mb() is called twice for both load_vmcs() and invept_smpfunc() after the refactoring, but only once in the original code.

It's only compiler stuff optimization to prevent memory reordering operations ... Please refer to https://yarchive.net/comp/linux/memory_barriers.html, if you want to know more about it. You don't have to see it as a real function call.
In this situation we have to be safe toward smp function calls (like it's the case in HAXM code when you meet xxx_smpfunc()). Even if the 2 functions use to be called along side, if one day someone calls them appart ... it may fail.
As for all of your previous comments, i tried to keep the philosophy of the existing code without rewriting everything actually.

raphaelning · 2018-05-07T10:16:30Z

core/cpu.c

+{
+    vmx_error_t err;
+
+    smp_mb();


Same here (see my comment on line 574).

raphaelning · 2018-05-07T10:20:39Z

core/cpu.c

@@ -502,7 +502,15 @@ uint8 is_vmcs_loaded(struct vcpu_t *vcpu)

 int debug_vmcs_count = 0;

-void restore_host_cr4_vmxe(struct per_cpu_data *cpu_data);
+void check_vmxon_coherency(struct per_cpu_data *cpu_data)


Make this function static? Actually, a better idea is to merge this function into vmxroot_leave() (after modifying vmxroot_leave() as I propose below).

sduverger · 2018-06-11T08:29:47Z

Hello,

I'm really sorry for the answer delay. I had no more time slot to work on the subject :( and i think it will be difficult by the end of the year. I'll answer point by point to your comments, but please feel free to do what ever you think it's best with the PR. I did it a little bit in a hurry, to help, but based on your comments it tends to become a full-time dev job :) and i totally agree some part would have to be completely rewritten.

There is coexistence issue in MacOS when some Mac VMMs are running at the same time. invept might not be invoked due to coexistence issue. This change is to fix the issue that invept is not invoked. It also refactors the existed logic for VMXON/VMXOFF into two functions. The function names come from PR #49.

raphaelning · 2018-06-29T08:33:37Z

Thanks, Stephane. I guess we should have kept this PR simple and left the refactoring to another PR. Now we have created #69 for the latter. When that is merged, it should be easy to fix #29. But I understand that you may not have time to rebase your original patch and resubmit it. We can do that later and will give you credit in the commit message.

Fixes intel#29. Ensure cr0/cr4 fixed bit settings prior vmxon.

47abf96

raphaelning reviewed May 7, 2018

View reviewed changes

junxiaoc mentioned this pull request Jun 29, 2018

core: fix bug that INVEPT is not invoked on Mac #69

Merged

intel deleted a comment from xianxiaoyin Aug 23, 2019

intel deleted a comment from yiqisiben Aug 23, 2019

wcwang force-pushed the master branch from 44f450a to 4e0e920 Compare November 6, 2022 11:22

wcwang force-pushed the master branch 2 times, most recently from 563eb1b to 6b942e3 Compare November 25, 2022 03:23

wcwang force-pushed the master branch 2 times, most recently from b73a231 to da1b8ec Compare January 26, 2023 02:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #29. Ensure cr0/cr4 fixed bit settings prior vmxon. #49

Fixes #29. Ensure cr0/cr4 fixed bit settings prior vmxon. #49

sduverger commented May 2, 2018

raphaelning left a comment

raphaelning May 4, 2018

raphaelning May 4, 2018

raphaelning May 7, 2018

raphaelning May 7, 2018

raphaelning May 7, 2018

raphaelning May 7, 2018

sduverger Jun 11, 2018

raphaelning May 7, 2018

sduverger Jun 11, 2018

raphaelning May 7, 2018

sduverger Jun 11, 2018

raphaelning May 7, 2018

raphaelning May 7, 2018

sduverger commented Jun 11, 2018

raphaelning commented Jun 29, 2018 •

edited

Loading

Fixes #29. Ensure cr0/cr4 fixed bit settings prior vmxon. #49

Are you sure you want to change the base?

Fixes #29. Ensure cr0/cr4 fixed bit settings prior vmxon. #49

Conversation

sduverger commented May 2, 2018

raphaelning left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sduverger commented Jun 11, 2018

raphaelning commented Jun 29, 2018 • edited Loading

raphaelning commented Jun 29, 2018 •

edited

Loading