12

I have created a function doWork() that's scheduled to run every day at 1:00 am, the function is as follows:

@Schedule(hour = "1", persistent = false)
    public void doWork()
    {
        System.out.println("Starting .....\nTIME: " + System.currentTimeMillis());
        System.out.println("this : " + this);

        //Some code here, if-conditions and try/catch blocks. No loops

        System.out.println("Exiting .....\nTIME: " + System.currentTimeMillis());
        System.out.println("this : " + this);
    }

The problem is that this function runs more than once, not as scheduled.

Once I created it, it ran exactly as expected (everyday at 1:00:00 am exactly). A few days later, it started running at 1:03:00 (which doesn't make any sense since it's nonpersistent and there was no downtime in the server anyway). After that the function started running more than once with very short intervals in between (seconds difference)

Does anyone know what might cause this, or tell me what I can do to fix it?

[EDIT]: Environment details

Application Server: WebSphere Application Server 8.5.5

IDE: Rational Application Developer 9.1

Database Management System: IBM DB2 10.1

Ahmed Anwar
  • 688
  • 5
  • 25

1 Answers1

1

I believe that you are observing the container attempting to retry a failed doWork() call.

The EJB timer service is transactional.

If the execution of a timeout method throws any runtime exception then the transaction will be rolled back and the container will attempt to execute the timeout method again. See §18.2.8 Transactions of the EJB 3.1 Specification.

Additionally, if the transaction timeout expires, some implementations will just mark the current transaction for rollback and continue with processing. This will result in the timer call failing eventually and a retry will be attempted.

This retry mechanism is not well specified and the actual behaviour varies from implementation to implementation. Some will retry forever and others will give up after some number of attempts. WebSphere for example provides a way to specify the retry strategy. See Creating timers using the EJB timer service for enterprise beans.

Finally if your application is running across multiple nodes then you probably have a timer running per server instance. According to §18.2.3 Non-persistent Timers of the spec:

For automatic non-persistent timers, the container creates a new non-persistent timer during application initialization for each JVM across which the container is distributed.

If these are all performing the same task then that is likely to cause enough chaos to generate errors, rollbacks and subsequent retries.

Steve C
  • 18,876
  • 5
  • 34
  • 37
  • According to the logs, the doWork() method runs successfully the first time, then it runs again and fails and keeps retrying. What is does is that it takes a csv file, parses it, deletes everything from the database and then persists the parsed data. According to the logs, on the first attempt the database is emptied and re-populated successfully, then the method runs again and keeps retrying and I end up with an empty database. Since the first try is successful, I don't see why the method will be called again after the first attempt? – Ahmed Anwar Aug 15 '16 at 11:06
  • You still need to provide more details of your environment before anyone can speculate further. Additionally, how can you be sure that the initial transaction completed successfully - i.e.after `doWork()` exits? – Steve C Aug 15 '16 at 12:57
  • I am sure it completed successfully on the first try because of the logging. By checking the database logs I could see that the records from the parsed .csv file were persisted into the database successfully. It then proceeds to retry, deletes everything in the database and I end up with an empty database. And please check the edit for environment details – Ahmed Anwar Aug 15 '16 at 13:07
  • How many server instances do you have? Is the EJB @Singleton or @Stateless? – Steve C Aug 15 '16 at 13:27
  • How long does the method take to execute? – Steve C Aug 15 '16 at 13:33
  • It is @Stateless. And there are 2 server instances on 2 nodes. The method takes around an hour because it takes a huge csv file (with over 100,000 rows of data), parses and persists it into the database – Ahmed Anwar Aug 16 '16 at 16:28
  • Does this mean that you have changed the default transaction timeout to something ridiculously long? Are you sure that doWork() only runs on one instance at a time? – Steve C Aug 17 '16 at 05:22
  • This was an error with the database server, it now runs in almost 30 seconds, but still runs multiple times. And how do I know how many instances it runs on? – Ahmed Anwar Aug 18 '16 at 11:44