Last night my server(centos5) had unusual outgoing traffic, About 12kGB and right now my server is down because I have to pay that traffic's cost. My server is a small server for a small App. I don't know whats happened but the support of hosting says it can be your server hacked and used for DDOS attack or can be amplification bug etc. They don't help much.
My questions is: How I can check that traffic is what about and used for what? How I can check my server hacked? Hosting staff had my root password for some tasks i gave them already. and how I can prevent this? Is there any way to set limit to outgoing traffic?