Is this a botnet?

Question

I manage networking equipment at a small ISP for 60 highrise buildings. Each building has anywhere from 1 to 60 24 port switches, or some DSLAMs, or some coax gateways, with p2p radios on the roof, with a main router at a central building.

I recently got an email from a guy working for SiteTruth that explains that there's phishing emails coming from a phony email address, but that the IP associated with the domain of the email is one of ours. He goes on to say that a traceroute to the IP bounces around a bunch of our IPs before hitting the trace IP, and hints that we may be hosting an entire botnet. I did the same traceroute and saw the same route he included in the email.

So the route hits on about 60 of our dynamic IPs before hitting the target IP. We use dhcp for clients on our entire network and all these hops were off dynamically assigned IPs, so hops off client boxes(sorry if I seem redundant).

I'm an aspiring network admin so I have no clue what this is. What is the reason for so many random hops to random customer routers/computers? Is it evidence of a botnet if a domain is registered to one of our IPs and a tracert to it hops across 60 other client devices on our network? I could find the MAC of the offending IP on the router and track it down to a particular port on a switch in a building and shut off that port, but if the client has no idea he's part of a botnet and has no idea this is occurring than I'll have to re-enable the port as they are paying customers.

Interestingly enough I've just performed the same tracert as I did a few days ago when I got the email and now there's only about 20 hops on our IPs before the target of the trace is hit. I'm guessing this is just due to dhcp being involved? I really have no clue.

Does anyone know what the heck this is? And if it's a botnet how am I supposed to cripple it? I'm guessing I would need to block only particular traffic?

Any more info on exactly what's happening and how to deal with it would be much appreciated. I don't understand why there's so many hops between dynamic IPs on our network during a simple tracert, does that confirm a botnet? I'd really like to do something so that our IPs don't end up being blacklisted.

Answer 1

Seeing a the route "hop around" on your network is really bizarre. You should see a traffic flow that hops form your edge router through your distribution routers. You definitely shouldn't be seeing traffic being routed through end user Customer addresses / devices. I tend to think you're seeing some kind of artifact of your configuration in these traceroute outputs, rather than evidence that routing is somehow being done by your end user Customers, but I can't say for sure without seeing it. (I also wonder what protections you've got enabled to prevent one Customer from spoofing the IP addresses assigned to another. Typical low-end layer 2 switches aren't going to provide a lot of protection on that front.)

Your network is different than a typical corporate network in that you're providing access to the Internet for paying Customers. Were I in your shoes I'd err on the side of not filtering any traffic if at all possible. (Your other question re: outbound TCP port 25 is an example of where it has become acceptable to filter traffic. Egress and ingress filtering to prevent spoofed IPs from entering / leaving your network are also good filtering rules.)

I think it's perfectly reasonable to shut down a client sourcing malicious traffic (and it should be noted in your service agreements that you have such a policy). Obviously, you should try to contact the Customer in question to let them know why they've been cut off. It would be nice if you provided them some packet captures to allow them to locate the source and eliminate it.

I don't have experience running an ISP to tell you what you should be monitoring for, in terms of oddball traffic patterns from Customers. Botnets try to mimic "normal" user traffic to circumvent monitoring and filtering anyway. I'd be more worried about monitoring your own equipment for unauthorized access attempts, locking down management interfaces to allow access with secure protocols from only authorized hosts, and employing routing and switching gear that prevents Customers from interfering with other Customers, spoofing their source addresses, exhausting your DHCP pool, putting up rogue DHCP servers, etc.