I'm not sure if it's safe to have virtual memory (pagefile.sys) in servers?
It absolutely is. Even better, it is highly recommended to employ some reasonable sized swapping space.
Would it affect the data in the cache that is being handled by RAM?
The is no such thing as "the cache"; usually you have L1/L2/L3 Cache, a filesystem cache, storage cache, cached libraries and much more.
Calling conflicts in a filesystem cache (usually resides in RAM), as I understand you are asking for, are almost impossible. The kernel (of modern operatin systems) redirects logical calls to the "right" physical objects, as he is the one asked about anyway. I am positive that you don't have to worry about this in almost any ususual use cases.