-1

We have a two server Exchange environment and they've been hacked in some way. get-queue displays tens of thousands of emails in the queues. Most of the email was being sent to yahoo.com.tw, yaahoo.com.tw, yahoo.com.hk, and yatw.

After finding a Taiwan IP that was connect to port 25 on one of our mail servers, I added the IP to an ACL on our firewall, and the queues stopped filling up.

I cleared the nearly 100k queues on both servers and the issue has seemed to resolved.

My question is, without an open relay (which I've confirmed we do not have), how should I go about investigating where the security hole was/is? I would like to see if a certain user's account was authenticating to do this, and if so, change passwords of course. Is there another way they could have been relaying if we don't have an open relay?

Steven Sedory
  • 61
  • 8
  • If you're not an open relay have you checked whether or not you allow relaying from any web or application servers? – joeqwerty Aug 07 '15 at 22:22
  • Yes, we do. We have a handful of authorized servers that can use the relay. I don't think that is where the traffic came from. – Steven Sedory Aug 07 '15 at 22:33
  • The source of the problem and how it was being used could be two different things. Exposed credentials on one service could allow a hacker to authenticate with exchange and send emails. – Drifter104 Aug 07 '15 at 22:47
  • Understood. Is there a way to tell what the source was? Either one of my allowed servers for relay, or an authenticated user account? – Steven Sedory Aug 07 '15 at 22:50
  • 1
    `I don't think that is where the traffic came from.` - Better to audit them and confirm that then to speculate. – joeqwerty Aug 07 '15 at 22:54
  • You got the symptom of an open relay. In powershell you can secure your server more. Check the last powershell command from that article; https://exchangemaster.wordpress.com/2013/03/08/checking-for-open-relay-in-exchange-20072010/ Be sure you accept only email for your domain. Activate the antispam feature for that – yagmoth555 Aug 07 '15 at 23:04
  • possible duplicate of [How do I deal with a compromised server?](http://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server) – kasperd Aug 08 '15 at 10:46

1 Answers1

0

Mystery solved. Though a bit embarrassing, perhaps this will help someone else out in the future:

As mentioned, we have two Exchange servers (version 2013). They're in a database availability group (DAG) for redundancy and performance. HAProxy on a Linux box acts as a load balancer for the DAG.

Before the load balancer was in place, we would switch the NAT rules on our firewall from one server to the other in order to test each server, essentially simulating what the load balancer would be doing once in place. When the load balance was put in place, everything still worked well, except one thing: our anonymous SMTP relay.

Some of our other servers use SMTP to send messages and notifications. One or two of them only allow a simple server address and port when pointing to an SMTP server, and do not allow for more settings like user/pass, which could be used to setup connection to an authenticated relay. So, we had setup anonymous receive connectors (not an "open relay") which were scoped to only allow the servers that needed access based on their IP addresses.

Well, when the load balancer got put in place, the SMTP relay wasn't relaying traffic for our other servers that were trying to connect to it any more. This was because the load balance changed the source address of SMTP request that pass through it to it's own IP. Well, I quickly and foolishly added the load balancer's IP to the anonymous relay scope, and WALA!...our servers could use the relay again......AND SO COULD THE REST OF THE INTERNET!

So, contrary to what I had be concered about in the original post, no one had our credentials, nor had anything malicious been installed on our servers, I had simply accidentally made our anonymous relay and open relay, all because I allowed the IP of the load balancer to be part of the relay scope.

My issue now is finding out a way for my other servers to send email using a relay that is based of off IP address and not authentication (the way I had it originally), since their software does not allow the latter. Of course I could just point them to one of the Exchange servers directly, bypassing the load balancer, but that would defeat the purpose of load balancing between our two servers. Any ideas on that? Can the load balancer somehow be more transparent and not change the source address of IP packets that pass through it?

Steven Sedory
  • 61
  • 8
  • A load balancer configured for direct server return (which excludes the open source version of HAProxy) would do, but this brings with it its own complexities and drawbacks. – Michael Hampton Aug 08 '15 at 18:40