-2

I've seen several similar questions - but none that truly answer the question. So here goes:

PROBLEM: A script is being run on one of our servers that is making outgoing post (and other) requests that we suspect is using CURL. It is attempting to find holes in other servers - in this case specifically in servers running WordPress, but there have been other requests.

QUESTION: Is there a GUI tool that can be used to monitor and view the source of these requests so that we can isolate the script and shut it down?

We are running an AWS instance, we do have Webmin available (if that's helpful) and this is a production machine that cannot be significantly slowed or have its normal traffic impeded.

Thanks to all, in advance, for the help!

Lee Fuller
  • 127
  • 6
  • Ultimately, you've been hacked, and can't really trust the server. :-/ – ceejayoz Aug 20 '15 at 14:38
  • @ceejayoz Yes.. agreed. But we need to start somewhere. :( – Lee Fuller Aug 20 '15 at 15:51
  • You need to start by turning the server **off**. – Michael - sqlbot Aug 21 '15 at 02:35
  • @Michael-sqlbot Unfortunately that's not possible. This is a production server. So far I've located the spamming bots that were installed. There are still attempt (HTTP POSTs) that are reaching out to other servers. There aren't many. But enough to be concerned. My guess is that it's simply bots trying to find weaker systems. – Lee Fuller Aug 21 '15 at 03:55

3 Answers3

2

I suggest Burp suite Fiddler and Wireshark for deeper analyzing.

FargolK
  • 121
  • 2
2

I know that you are looking for GUI, but there is no GUI with magic button "SHOW ME WHO HACKED ME". This isn't TV, this is system administration. You need to use proper tools.

For starters, you can block all outgoing communication its destination port is 80 via iptables:

iptables -t filter -A OUTPUT -p tcp --dport 80 -j DROP

This will drop ALL communication no matter which process tried to start it. Then you can start playing with netstat and ps to find, which process does the bad stuff on your precious machine:

netstat -np | grep ^tcp | grep ":80"

On my machine, the result of command above is this:

tcp        0      0 192.168.1.2:34831       185.31.17.246:80        ESTABLISHED 22640/spotify   
tcp        0      0 192.168.1.2:48809       104.16.105.85:80        ESTABLISHED 10572/iceweasel

As you can see, only two processes communicate via HTTP with some servers - iceweasel and spotify. The last column is [process_number]/[process_name]. With this, you can query ps and get the actual process:

ps axu | grep 22640

Again, on my machine, it says (shortified) this:

mkudlac+ 22682  0.2  2.0 1003656 123440 ?      Sl   09:12   0:31 /opt/spotify/spotify-client/Data/SpotifyHelper --type=renderer --js-flags=--harmony-proxies --no-sandbox --lang=en-US --lang=en-US --locales-dir-path=/opt/spotify/spotify-client/Data/locales --log-severity=disable --resources-dir-path=/opt/spotify/spotify-client/Data --disable-accelerated-2d-canvas --disable-accelerated-video-decode --channel=22640.1.2031916850

Now I know path to executable and user it runs under.

To combine all this to "simple" one liner:

netstat -np | awk '/^tcp/{print $5 "/" $7}' | grep ":80" | awk -F'/' '{print $1; if ($2 != "-") system("ps axu | grep " $2 " | grep -v grep"); print "================"; }'

The result on my machine shows this:

104.16.104.85:80
mkudlac+ 10572  6.3  6.5 1016592 401108 ?      Sl   11:38   2:18 iceweasel http://serverfault.com/questions/715556/is-there-a-gui-tool-to-log-and-view-outgoing-curl-requests-from-a-linux-server
================

First line is destination IP address. Second line is full information about rogue process. Third line is delimiter to optically divide huge output.

These commands (at least netstat and ps) needs to be executed under root. When you clear your machine, you can delete the blocking iptables command with:

iptables -t filter -D OUTPUT -p tcp --dport 80 -j DROP

EDIT:

To be able to leave this script unattended and logging into file, you can alter it this way:

while (true); do netstat -np | awk '/^tcp/{print $5 "/" $7}' | grep ":80" | awk -F'/' '{print $1; if ($2 != "-") system("ps axu | grep " $2 " | grep -v grep"); print "================"; }' | tee -a hack.log; sleep 30; done

This will check for rogue connections every 30 seconds and write it to hack.log file.

mkudlacek
  • 1,677
  • 1
  • 11
  • 15
  • Oh yes.. understand that there isn't a "magic" fix. Not looking for that. Only looking for a tool that helps monitor external connections initiated by our system. And the tools you've listed here are just a few of what I've used. Yet - when the connectivity is so sporadic that I would need to be staring at my monitor to catch it in the act, I was looking for a low-impact tool that would log and allow me to view initiated outgoing http post connections. – Lee Fuller Aug 21 '15 at 14:38
  • This is a low impact tool, consumes very little resources. I expanded the answer to log the results into file. You can run it in detached `screen` session. – mkudlacek Aug 21 '15 at 16:10
  • Thanks, that's a great example. About the only weakness I see is that it polls every 30 seconds, so items could slip between that window. Of course, could always make it every 5 seconds, or something. Lots of data that way though. :) Thanks! – Lee Fuller Aug 21 '15 at 18:24
  • No problem Lee, I'm glad it helped. If you are satisfied with this answer, please accept it with clicking on tick mark. – mkudlacek Aug 22 '15 at 06:46
1

I would also like to propose "iftop", this can be usefull to get existing connections from your machine: It listens to network traffic on a named interface and displays a table of current bandwidth usage by pairs of hosts

Tom
  • 616
  • 8
  • 13
  • This looks quite useful. Does it log? – Lee Fuller Aug 21 '15 at 14:40
  • I never used it this way, but looking at this topic it seems to be now possible to have a text output: http://serverfault.com/questions/379469/iftop-how-to-generate-text-file-with-its-output – Tom Aug 21 '15 at 14:49