Although the issue seems to be with ModelSim, it really is an issue with Windows. As you noted, you have associated a file type (in this case, .vhd) with a given program, and now you need to undo it in order to get ModelSim to work as it did initially.
The following is the manual method for removing file associations for the .vhd extension in Windows 10:
- (Search for and) run regedit command.
- In the left sidebar of the regedit GUI, navigate to \Computer\HKEY_CLASSES_ROOT.
- Delete the ".vhd" directory.
- Navigate to \Computer\HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts.
- Delete the ".vhd" directory.
- Navigate to \Computer\HKEY_CURRENT_USER\Software\Microsoft\Windows\Roaming\OpenWith\FileExts.
- Delete the ".vhd" directory.
- You may or may not need to restart ModelSim and/or your computer.
UPDATE
Performing just steps 6 and 7 may fix the issue by itself, so try those first, then proceed with steps 4 and 5, check again if that fixes the issue, and so on.
You will need to repeat the fix every time that you perform Open With
on a .vhd file, even if you Open With
and then cancel (i.e. do not select an application and do not open said file).