I think I understand where the OP is coming from.
When using MediatR in web apps, it's very easy to see the individual pieces that make up the application. There would be one request / handler / optional for each controller action, each cleanly separated from each other.
In a Windows Service, there's no concept of individual actions. There's only a single entry point, so it can be confusing to know where to use MediatR.
I believe a good way to go about it is to use a single request / handler / response that corresponds to the single entry point. That would mean putting all the windows service logic in the handler.
If that results in a very long handler, then continuously refactor as new abstractions and opportunities for eliminating code duplication are discovered. This would be exactly the same approach as you would do in a MediatR-powered web application.