If you have control over building the service, just change it to target the .NET 3.5 framework and you should be able to install it.
Otherwise, if the service was built to target the .NET framework 4.0 and the server only has .NET 3.5 installed, I don't think it will run unless you explicitly override it in a config file to run under .NET 3.5 (which I wouldn't recommend).
As an aside, I always recommend that people use the WiX Toolset to write installers for Windows services, instead of using InstallUtil, because InstallUtil is a custom, non-standard Visual Studio tool to install windows services, whereas Windows Installer (MSI), the approved standard for windows installations has built-in support for installing Windows services that is tried-and-tested, and gives you more control, e.g. stopping and starting services etc.