Add support for multicore Windows 10 guests #272

nevilad · 2020-03-10T19:33:51Z

When passing host cache values to the guest in cpuid 4, change maximum number of addresable IDs according to the number of virtual CPUs.

There was a comment:
// [31:26] cores per package - 1
Are there any comments why hax assumed 1 core per package?

wcwang · 2020-03-11T02:46:45Z

Thanks for your patch. We have verified that the Windows guest can boot up with single core as below.

qemu-system-x86_64.exe -L C:\Binaries\qemu_pr204 -name win10x86 -cpu Broadwell -machine pc-i440fx-3.0 -usb -device usb-tablet -m 4G -vga std -rtc base=localtime -accel hax -cdrom C:\Binaries\windows\Win10_1903_V1_English_x86.iso C:\Binaries\windows\win1032.qcow2

haxm_warning: CPUID 4.0, eax 0x1c004121, ebx 0x1c0003f, ecx 0x3f, edx 0x0
haxm_warning: CPUID 4:0, eax 0x4121, ebx 0x1c0003f, ecx 0x3f, edx 0x0 AFTER

However, when specifying multiple cores, the Windows guest ran into BSOD as before.

qemu-system-x86_64.exe -L C:\Binaries\qemu_pr204 -name win10x86 -cpu Broadwell -smp 2,cores=2,threads=1 -machine pc-i440fx-3.0 -usb -device usb-tablet -m 4G -vga std -rtc base=localtime -accel hax -cdrom C:\Binaries\windows\Win10_1903_V1_English_x86.iso C:\Binaries\windows\win1032.qcow2

haxm_warning: CPUID 4.0, eax 0x1c004121, ebx 0x1c0003f, ecx 0x3f, edx 0x0
haxm_warning: CPUID 4:0, eax 0x40004121, ebx 0x1c0003f, ecx 0x3f, edx 0x0 AFTER

nevilad · 2020-03-11T20:01:36Z

The original version, which did not modify host values, did not work for both 1 core and multiple cores?
This version, which modifies addresable IDs, works on 1 core, but not multiple?
The code in this PR is done as in qemu. This means that Windows on multicore systems may check other things. I was unable to figure it out yet and need time to investigate it.

wcwang · 2020-03-12T01:28:50Z

The original version, which did not modify host values, did not work for both 1 core and multiple cores?

We verified that the original version also had same issue, which will result in BSOD on multiple cores.

This version, which modifies addresable IDs, works on 1 core, but not multiple?

Whether it is patched with PR #272 or not, the current version on master branch works on 1 core, but not on multiple cores.

The code in this PR is done as in qemu. This means that Windows on multicore systems may check other things. I was unable to figure it out yet and need time to investigate it.

Thanks for your effort.

of virtual CPUs Signed-off-by: Alexey Romko <nevilad@yahoo.com>

nevilad · 2020-03-14T10:03:42Z

Now it works on 2 cores.
There was a comment:
// [31:26] cores per package - 1
Are there any comments why hax assumed 1 core per package?

wcwang · 2020-03-16T09:03:27Z

We had searched the development history but found no clues. I think maybe the earlier versions considered the complexity for this case. From vCPU perspective, each time a vCPU may run as a different physical host CPU. The comments here may be to simplify implementation. We are verifying your update and will continue to discuss it in details. Thanks.

wcwang · 2020-03-19T11:06:55Z

We have tested the patch and it did work on Windows. Multiple cores can be supported and Windows guest can boot up. However, when we tested HAXM on macOS, since merging PR #204, whether pulling the patch in PR #272 or not, all guests cannot be launched with multiple cores, even running an empty QEMU as below.

$ .\qemu-system-x86_64 -chardev stdio,id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios -no-reboot -smp cores=2 -accel hax

The output of QEMU has been attached here for your reference. Thanks.

nevilad · 2020-03-19T19:46:24Z

I have no MacOs to debug, so will ask some questions:

since merging PR #204, whether pulling the patch in PR #272 or not, all guests cannot be launched with multiple cores, even running an empty QEMU as below.

Does an empty QEMU with multiple cores work without PR #204?

The output of QEMU has been attached here for your reference. Thanks.

Whats happening with the VM - it exits or writes something to the screen? Or crashes? Can you attach hax log with HAX_LOGD output? The problem should be in SeaBios, since it is the only code that runs.

wcwang · 2020-03-20T13:53:00Z

Does an empty QEMU with multiple cores work without PR #204?

Yes, it can boot up without PR #242

Whats happening with the VM - it exits or writes something to the screen? Or crashes?

The results were random. Seldom QEMU exited quickly after launching. The guest screen flashed once and there was no impact on host. In most cases, QEMU launched and the text "Guest has not initialized the display (yet)" was printed in the QEMU window.

Can you attach hax log with HAX_LOGD output?

The log mechanism on macOS still has some problems. We will attach the available logs after fixing the issue. Thanks.

nevilad · 2020-03-20T14:08:36Z

Yes, it can boot up without PR #242

You mean PR #272?

The log mechanism on macOS still has some problems. We will attach the available logs after fixing the issue. Thanks.

When it will be fixed?
Can you run hax on windows host with an empty guest and same BIOS (seaBios)?

nevilad · 2020-03-22T20:40:35Z

core/vcpu.c

+                struct vm_t *vm = vcpu->vm;
+                hax_list_head *list;
+                int vcpu_count = 0;
+
+                hax_mutex_lock(vm->vm_lock);
+                hax_list_for_each(list, (hax_list_head *)(&vm->vcpu_list)) {
+                    vcpu_count++;
+                }
+                hax_mutex_unlock(vm->vm_lock);


This should be moved to a function.

wcwang · 2020-03-23T14:14:25Z

You mean PR #272?

Sorry for typo. It is PR #204. QEMU can only boot up without PR #204. Even after applying the latest update of PR #272, QEMU cannot boot up with multiple cores on macOS yet.

When it will be fixed?

As we are working on the next release, maybe this issue will be postponed to next month.

Can you run hax on windows host with an empty guest and same BIOS (seaBios)?

Yes, QEMU can boot up for multiple cores with an empty guest and same BIOS on Windows host under below cases:

Without PR Windows10 guest support #204
With PR Windows10 guest support #204
With PR Windows10 guest support #204 plus PR Add support for multicore Windows 10 guests #272

Furthermore, with PR #204, after I reverted the patch 329dc40 (Add cr8 to hax ioctl interface), QEMU can boot up for multiple cores with an empty guest on macOS.

According to our release schedule, we will consider to resolve the multiple cores issue after the next release. In order to avoid the regressions on macOS, maybe I have to revert the cr8 patch first. Thanks for your understanding.

nevilad · 2020-03-23T15:05:25Z

In order to avoid the regressions on macOS, maybe I have to revert the cr8 patch first.

I think it is better not to merge this PR to have working MacOS. This will lead to nonworking multicore WIndows 10 guest, but it did not work in previous versions too. Cr8 support is vitable for supporting Win10 x64 and MacOS, so I recommend not to remove it.

When do you plan to release new haxm version (how many time do I have to debug this issue)?

wcwang · 2020-03-24T01:28:17Z

I think it is better not to merge this PR to have working MacOS. This will lead to nonworking multicore WIndows 10 guest, but it did not work in previous versions too.

Yes, macOS did not work even after applying this patch.

Cr8 support is vitable for supporting Win10 x64 and MacOS, so I recommend not to remove it.

Not true. As I mentioned, after merging PR #204 and reverting the patch 329dc40 , QEMU can boot up for multiple cores with an empty guest on both Windows and macOS. Originally, QEMU can boot up for multiple cores before merging PR #204 . Therefore, there is a regression after merging CR8 patch.

When do you plan to release new haxm version (how many time do I have to debug this issue)?

Today is the last day to deliver release and we also need about 2 weeks to complete release test. Never mind, once the multiple cores issue resolved, we can make another release for you. Thanks for your understanding and your future contribution will be welcome.

wayne-ma · 2020-03-28T03:57:33Z

Ok to verify

nevilad · 2020-04-21T21:28:09Z

Haxm is released, we can continue with this PR?
We found the problem is in cr8 handling commit 329dc40. I think we can find the problem by commenting out some parts of the commit.
@wcwang can you build hax from #288 and ry to run under MacOS? I removed cr8 from vcpu there, I think structure layout changes may be the source of the problem, since cr8 in the mentioned commit is only saved from user mode, but not used.

wcwang · 2020-04-22T11:38:26Z

Thanks for your tracking. Currently, our team is busy in improving the performance on Windows. I think we can continue with this PR but the time may be delayed a bit. I will try to build HAXM from #288 on macOS later. Thanks again.

nevilad · 2020-06-10T12:20:55Z

@wcwang when does your team finish with improving the performance on Windows and continue with this PR?

wcwang · 2020-06-12T03:22:43Z

@nevilad, thanks for your tracking. Recently, we put lots of effort on debugging the performance issue on Windows but have not found the solution. We expect HAXM will not be impacted even when L1TF mitigation is enabled. Sorry for not collaborating with you to make the this pull request move on. Thanks for your understanding.

hyuan3 · 2020-06-22T02:10:21Z

core/vcpu.c

            // Bit mask to identify the reserved bits in paging structure high
            // order address field
            physical_address_size = (uint8_t)state->_eax & 0xff;
            pw_reserved_bits_high_mask =
                    ~((1 << (physical_address_size - 32)) - 1);

            state->_ebx = state->_ecx = state->_edx = 0;
+
+            if (vcpu_count > 1)
+                state->_ecx |= vcpu_count - 1;


According to Intel SDM, when the initial EAX value is 0x80000008, ECX is reserved to 0. Why set state->_ecx here?

You are right, SDM says it's reserved to 0. I've done this since qemu does the same:

case 0x80000008: /* virtual & phys address size in low 2 bytes. */ if (env->features[FEAT_8000_0001_EDX] & CPUID_EXT2_LM) { /* 64 bit processor */ *eax = cpu->phys_bits; /* configurable physical bits */ if (env->features[FEAT_7_0_ECX] & CPUID_7_0_ECX_LA57) { *eax |= 0x00003900; /* 57 bits virtual */ } else { *eax |= 0x00003000; /* 48 bits virtual */ } } else { *eax = cpu->phys_bits; } *ebx = env->features[FEAT_8000_0008_EBX]; *ecx = 0; *edx = 0; if (cs->nr_cores * cs->nr_threads > 1) { *ecx |= (cs->nr_cores * cs->nr_threads) - 1; } break;

nevilad · 2020-08-06T19:32:23Z

@wcwang how is your work on the performance issue on Windows going?

nevilad · 2021-01-06T17:40:00Z

Can we continue with this PR?

wcwang · 2021-01-07T13:26:20Z

Thanks for @nevilad's tracking. Currently we are working on some new features in CPUID module. Anyway, I would like to collaborate with you to continue with the PR if possible. Have you looked through the new implementation of CPUID module? Could you re-implement this feature based on the latest code? Then I will try to review and verify it when ready. Thanks a ton again.

nevilad · 2021-01-08T12:05:24Z

The new code is OK, but it does not allow me to use subleafs. Do you plan to add this support or should I do that? It's about changing cpuid_manager_t - add subleaf and flags.

wcwang · 2021-01-08T13:12:38Z

Thanks for your review. Maybe currently we do not have such plan to enhance CPUID features. Your idea sounds quite great, but maybe it will change a lot of framework code. Our team is working on another feature of Android, I think we'd better choose another time to continue this PR. During this period, I think you may consider your implementation first and then we can discuss later. Thanks for your understanding.

nevilad · 2021-01-08T13:20:03Z

OK, will wait till you're ready with the feature!

nevilad · 2021-04-12T16:32:34Z

I see a new haxm is released, can we continue with this PR?

wcwang · 2021-04-13T02:37:50Z

@nevilad, thanks for your tracking. You may also notice that we still have a release to be delivered, i.e., v7.7.0. Our current task priority is still in the feature of the Android guest OS. When we have plans to optimize the Windows guest, we will consider this PR at the first time. Thanks for your understanding.

nevilad · 2021-04-13T09:22:23Z

When do you plan to release v7.7.0? Can we continue with this PR after v7.7.0 release?

nevilad · 2021-04-21T08:39:42Z

@wcwang When do you plan to release v7.7.0? Can we continue with this PR after v7.7.0 release?

wcwang · 2021-04-21T10:21:21Z

@nevilad, sorry for late response. Maybe we will postpone the release v7.7.0 as there is another snapshot issue of Android guest. After this release, our task priority may still be in the feature of the Android guest OS. I am not sure when we can continue this PR. Anyway, thanks for your tracking and contributions.

nevilad mentioned this pull request Mar 10, 2020

Windows10 guest support #204

Merged

Change cpuid 1, 4 and 0x80000008 returned values according to number

3a0654a

of virtual CPUs Signed-off-by: Alexey Romko <nevilad@yahoo.com>

nevilad force-pushed the cpuid_4_ids branch from 6dbef99 to 3a0654a Compare March 14, 2020 09:34

nevilad changed the title ~~Correct cpuid 4 output~~ Add support for multicore Windows 10 guests Mar 20, 2020

nevilad commented Mar 22, 2020

View reviewed changes

nevilad mentioned this pull request Apr 16, 2020

qemu + haxm does not work to boot and run Mac OS X High Sierra #149

Open

nevilad mentioned this pull request May 12, 2020

QEMU/HAXM does not work with -cpu max #38

Open

hyuan3 reviewed Jun 22, 2020

View reviewed changes

HaxmCI added CI:Build Pass CI:Build Pass CI:Mac Test Pass CI:Mac Test Pass labels Mar 2, 2021

HaxmCI added CI:Build Fail CI:Build Fail and removed CI:Build Pass CI:Build Pass CI:Mac Test Pass CI:Mac Test Pass labels May 24, 2021

wcwang force-pushed the master branch from 44f450a to 4e0e920 Compare November 6, 2022 11:22

wcwang force-pushed the master branch 2 times, most recently from 563eb1b to 6b942e3 Compare November 25, 2022 03:23

wcwang force-pushed the master branch 2 times, most recently from b73a231 to da1b8ec Compare January 26, 2023 02:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for multicore Windows 10 guests #272

Add support for multicore Windows 10 guests #272

nevilad commented Mar 10, 2020

wcwang commented Mar 11, 2020

nevilad commented Mar 11, 2020

wcwang commented Mar 12, 2020

nevilad commented Mar 14, 2020

wcwang commented Mar 16, 2020

wcwang commented Mar 19, 2020

nevilad commented Mar 19, 2020

wcwang commented Mar 20, 2020

nevilad commented Mar 20, 2020

nevilad Mar 22, 2020

wcwang commented Mar 23, 2020

nevilad commented Mar 23, 2020

wcwang commented Mar 24, 2020 •

edited

Loading

wayne-ma commented Mar 28, 2020

nevilad commented Apr 21, 2020

wcwang commented Apr 22, 2020

nevilad commented Jun 10, 2020

wcwang commented Jun 12, 2020

hyuan3 Jun 22, 2020

nevilad Jun 22, 2020

nevilad commented Aug 6, 2020

nevilad commented Jan 6, 2021

wcwang commented Jan 7, 2021

nevilad commented Jan 8, 2021

wcwang commented Jan 8, 2021

nevilad commented Jan 8, 2021

nevilad commented Apr 12, 2021

wcwang commented Apr 13, 2021

nevilad commented Apr 13, 2021

nevilad commented Apr 21, 2021

wcwang commented Apr 21, 2021

Add support for multicore Windows 10 guests #272

Are you sure you want to change the base?

Add support for multicore Windows 10 guests #272

Conversation

nevilad commented Mar 10, 2020

wcwang commented Mar 11, 2020

nevilad commented Mar 11, 2020

wcwang commented Mar 12, 2020

nevilad commented Mar 14, 2020

wcwang commented Mar 16, 2020

wcwang commented Mar 19, 2020

nevilad commented Mar 19, 2020

wcwang commented Mar 20, 2020

nevilad commented Mar 20, 2020

nevilad Mar 22, 2020

Choose a reason for hiding this comment

wcwang commented Mar 23, 2020

nevilad commented Mar 23, 2020

wcwang commented Mar 24, 2020 • edited Loading

wayne-ma commented Mar 28, 2020

nevilad commented Apr 21, 2020

wcwang commented Apr 22, 2020

nevilad commented Jun 10, 2020

wcwang commented Jun 12, 2020

hyuan3 Jun 22, 2020

Choose a reason for hiding this comment

nevilad Jun 22, 2020

Choose a reason for hiding this comment

nevilad commented Aug 6, 2020

nevilad commented Jan 6, 2021

wcwang commented Jan 7, 2021

nevilad commented Jan 8, 2021

wcwang commented Jan 8, 2021

nevilad commented Jan 8, 2021

nevilad commented Apr 12, 2021

wcwang commented Apr 13, 2021

nevilad commented Apr 13, 2021

nevilad commented Apr 21, 2021

wcwang commented Apr 21, 2021

wcwang commented Mar 24, 2020 •

edited

Loading