I am currently reading into building a custom UI using Wix#, but since I have more experience with (WPF) MVVM than with WinForms i was wondering if it was possible to use that instead.
Is it possible?
You can actually use WPF UI with Wix#. See example in wix# repo.
If you want to use WinForms mandatory, you can also try to utilize winforms bindings to write ui-related stuff in a wpf-style. See here about WinFroms bindings. But in my opinion, WinForms bindings is not so good and I personally prefer not to use them in simple UI.