Starting program once leads to 10 instances

Question

I have an old SUSE 8.1 server which just recently started acting crazy.

If I start a program once:

/usr/local/jdk1.6.0_21/bin/java -cp example-ser-0.8.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/Translator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=logback-test.xml example.ser.Scanner abc12345

There will be 10 instances running (same user, same arguments) as shown by ps ax -l | grep Scanner:

  F S   UID   PID  PPID  C PRI  NI ADDR SZ WCHAN  TTY        TIME CMD
000 S   502 20228 20227  0  75   0 - 98222 rt_sig pts/11     0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345
002 S   502 20229 20228  0  75   0 - 98222 schedu pts/11     0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345
002 S   502 20230 20229  0  75   0 - 98222 schedu pts/11     0:14 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345
002 S   502 20231 20229  0  75   0 - 98222 schedu pts/11     0:01 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345
002 S   502 20232 20229  0  76   0 - 98222 rt_sig pts/11     0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345
002 S   502 20233 20229  0  76   0 - 98222 rt_sig pts/11     0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345
002 S   502 20234 20229  0  80   0 - 98222 rt_sig pts/11     0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345
002 S   502 20235 20229  0  75   0 - 98222 rt_sig pts/11     0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345
002 S   502 20236 20229  0  80   0 - 98222 rt_sig pts/11     0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345
002 S   502 20237 20229  0  75   0 - 98222 schedu pts/11     0:00 /usr/local/jdk1.6.0_21/bin/java -cp /usr/local/lib/java/example-ser-0.7.jar:/usr/local/lib/java/slf4j-api-1.6.1.jar:/usr/local/lib/java/logback-core-0.9.24.jar:/usr/local/lib/java/logback-classic-0.9.24.jar:/usr/local/lib/java/jcl-over-slf4j-1.6.1.jar:/usr/local/lib/java/commons-cli-1.2.jar:/usr/local/lib/java/commons-lang-2.5.jar:/usr/local/lib/java/mysql-connector-java-5.1.13-bin.jar:/usr/local/lib/java/xBaseJ-20091203.jar:/usr/local/lib/java/CampaignRulesTranslator-0.7.0.jar:/usr/local/lib/java/antlr-3.2.jar:/usr/local/lib/java/joda-time-1.6.jar -Dlogback.configurationFile=/usr/local/lib/java/logback.xml example.ser.Scanner abc12345
000 S     0 25155 19094  0  80   0 -   406 pipe_w pts/2      0:00 grep Scanner

If I ^C the terminal where I started the program, all 10 disappear. This program doesn't do any forking or create any extra threads. I also tried several older versions of this program going back years before this problem and they're all experiencing this now as well. What could be going on here?

PPID (parent process id) on the process list might help a lot — Daniel Prata Almeida, Aug 01 '12 at 19:33
As you see the PPID of each process is the PID of the process before, so JVM is forking. Check the libs, that's my hunch. Did that happened before? Or you had just 1 process? — Daniel Prata Almeida, Aug 01 '12 at 19:49

score 1 · Accepted Answer · answered Aug 01 '12 at 19:31

1

The program itself might not fork but the libs/jars provided on the classpath might be forking , I would first try to check those libs code for forks. If that fails, try to execute it with a different JVM version and/or environment and see what happens.

answered Aug 01 '12 at 19:31

Daniel Prata Almeida

702
4
5

Looks like this wasn't the source of the problem after all then. I must have been using a different view before. – Brad Mace Aug 01 '12 at 19:54

Starting program once leads to 10 instances

1 Answers1