8

I have a UpStart service job that has many (~100) instance that need to be started. Each of them is a resource-heavy process that does a lot of disk reading/writing during startup. When all of them start or respawn at the same time, they cause trouble due to excessive disk read/write requests.

I need a way to limit the number of instances that UpStart tries to start or respawn simultaneously. For example, is there a way to let UpStart hold off launching another instance until, say 30 seconds, after the startup or respawning of another instance has begun?

Shanqing Cai
  • 3,756
  • 3
  • 23
  • 36
  • If your job is a shell script, you could insert either a random pause, so that when 10 start at the same time they each pause a different number of seconds and space themselves out -- or you could have all jobs lock a certain file when doing the intensive startup. But that is a shell scripting solution that has nothing to do with upstart. – Paul May 16 '15 at 03:46
  • 1
    can you show a upstart script? are they al somewhat the same? – Alex May 16 '15 at 07:11
  • I think the lock method mentioned by @Paul is cool. But that'll require changes to the app code itself (btw, it is not a shell script. It is a full compiled application). It is surprising that UpStart doesn't have some builtin functionality to automatically hold and queue service instances (?). – Shanqing Cai May 22 '15 at 17:59

2 Answers2

5

You can start them in sequence by using

start on started otherUpstartService
Alex
  • 5,759
  • 1
  • 32
  • 47
  • When is a service considered "started" by UpStart? As soon as the process is launched? If it is too soon, it might not help with spreading the load out in time. If it is too late, there may be too much delay. – Shanqing Cai May 22 '15 at 17:55
1

You can use pre-start or post-stop to pause after each job. E.g post-stop exec sleep 5

Kremnev Sergey
  • 332
  • 1
  • 8
  • If they all sleep for the same amount of time at start up, then the issue is still there. Is there a way to make the sleep time random? – Shanqing Cai May 22 '15 at 17:56
  • You could do something like `post-stop exec sleep $(( ( RANDOM % 10 ) + 1 ))` to get sleeps between 1 and 10 seconds. Just change the 10 to any other integer to get a wider range – Kremnev Sergey May 25 '15 at 06:26