It appears that there is no native way to do this, as yet again, M$ treats its users like incompetent idiots that shouldn't be allowed to perform tasks that might actually make the entire system more stable or productive.
The only workaround might be what was suggested for running a batch script in background (technically that falls under 3rd party, as the script is written and run unofficially by another party), but this wouldn't "limit" the process per se, but would instead restart the process when it hits its max limit you impose.
The issues with a restarting script that you should consider are:
You will be killing the process unexpectedly, which means it will be
interrupted in whatever task it was performing.
Corrupted files could be generated if a file were being written to
when the process gets restarted. This can be alleviated if the
process has its own file verification and rewrite contingencies
(such as online games and Steam).
Other sub-processes and threads will also be killed as a result, and
could propagate the other issues listed here.
If the process eats up RAM rather quickly, restarting the process
might occur too often, rendering the process useless (file a
critical bug report and stop using the software).
Although the Software process is experiencing an unintentional
memory leak, it is possible that the process has an internal policy
of behavior when the system is low/depleted in RAM, such as when it
ends up using the Pagefile. If it changes its behavior to reduce its
footprint in this case, restarting the process ensures it will never
enter this kind of "reduced" state of operation.
Another reason why the Linux Team got things right, but alas, Windows got the world's support in drivers and games first, so we have to outlive it.
BTW, I personally experience a MASSIVE memory leak in the Razer GameScannerService.exe process, where of my 16GB of 1800MHz memory (and my 5GHz CPU), within a few minutes it has already allocated nearly 2GB of RAM, and I have seen it climb all the way up to 9GB if I allowed it to.
Razer does nothing to fix this, despite the massive number of bug reports. Our only option is to disable the service in some way and live without our ability to have Razer auto-apply profiles to games or auto-discover new games that get installed.