If you are REALLY paranoid about security, you do not store the private key on the filesystem, you store it in hardware. A Hardware Security Module (HSM) from a company like SafeNet or Thales/nCipher is a physically hardened security appliance that is designed to product encryption keys. Once the keys are generated on the HSM they cannot be removed. That is, you can not extract the key and copy it off somewhere else to be used for malicious purposes. Additionally, they have hardened operating systems and software designed to prevent an attacker from being able to hack into the appliance. Too many unsuccessful logins? The device securely wipes its contents. Try to pry open the box to remove the flash memory holding the keys? There is epoxy on the physical hardware components and you will end up destroying them.
These devices also typically include cryptographic accelerators to increase the speed at which cryptographic operations are performed. They also tend to include standards-based APIs, such as PKCS#11 or JCE so it is easy to integrate with other software, such as Apache or your Java JVM.
An HSM is not cheap, but if you are really concerned about the security of your cryptographic material, this is how you protect it.