Update:
Hey man I think I caught your problem.
1) Please Open VirtualBox and and Then right click on your Virutal Machine.
2) Select Settings.
3) Open General
Setting from the left panel.
4) Open the Basic tab.
5) Choose Linux from the Type
option.
6) Then choose Other Linux(64-bit) in the Version Section.
I have tried the option of 32-bit mode in my PC and the same problem happens as you asked in your question. So, select 64-bit mode to make it run.
Note: Android 5.x and 6.x are based on the 64-bit architecture. If you
selected the Other Linux(32-bit) option. Then Virtual Box will try to
emulate a 64-bit program in a 32-bit virtual machine type, which is
not possible.
So, Please make sure that you have selected the Other Linux(64-bit)
option for Android Versions higher than 5.x. (You can run Android
versions upto 4.x in the 32-bit mode.)

Old Answer:
No, You don't need to reinstall windows. It is not an operating system
fault.
I faced this problem.
It is due to the Antivirus Program.
If you have any antivirus program then just disable it and TRY
AGAIN.
You also don't need to set any compatibility mode.
Currently, I am using Genymotion(Version 2.6.0) and
VirtualBox(Version 5.0.12 r104815) in Windows 10 Pro without any error.