An SSL certificate has two parts; a public key and a private key.
.CER files contain nothing secret. It's the public key that the web server sends to every client connecting to the SSL enabled site. There's no reason to delete it from a security standpoint. In fact, you probably want to have it backed up in case the server dies and need to re-use it on whatever is replacing it.
However, the .CER file is not useful without its matching private key. Windows generates a private key automatically when you create a certificate request. Your .PFX file definitely contains a copy of the .CER. But it may or may not contain a copy of the private key. It entirely depends on who generated it and how.
If the PFX file does contain the private key, then yes. Lock that bad boy away off site and delete the copy from the server. Someone with that file could potentially use it to impersonate your site.
If the PFX file does not contain the private key, it's no more useful than the raw CER file. And if possible, you probably want to export a new PFX file that does contain the private key for DR/backup purposes (assuming the certificate allows private key export).