0

I would like some tools or approaches to measure how much bandwith, cpu, memory is being used by each instances of ruby script (ex. a web spider).

The goal here is trying to see the maximum number of ruby scripts that can run on my ubuntu web server. Multiple users will be running the script.

basically i have an app which can launch multiple instances of ruby script. multiple users use this app. so i need to figure out a threshold.

2 Answers2

1

Is just running a bunch of them and seeing when the machine's load average starts to shoot up not an option?

womble
  • 96,255
  • 29
  • 175
  • 230
  • yes. i would need to get a rough idea of the max threshold. doesn't need to be exact, it should give a rough idea of what kind of servers i need to purchase, how many users to put on each server. –  Nov 08 '09 at 09:27
  • 1
    So how would running an increasing number of instances of the script not allow you to do this? – womble Nov 08 '09 at 09:49
1

Well, the absolute maximum is however many can fit in to your system memory (RAM + swap). Of course by that point it may be that your machine is already being ground to a halt, so that's not necessarily a good metric to use.

The "max number" is more likely defined as how many instances can be running while your machine is still performing acceptably. The only way to find this out is to benchmark. Use one server and start loading it up with instances of your script. Find the ideal number by measuring the metrics that are important to you. It could be execution time, latency, network throughput, or whatever it is in your case. There are many variables and you need to identify the ones that matter. Try to find out where the bottlenecks are by profiling your script, and monitor your system I/O and network throughput to see if you can optimize your hardware for your application.

Also make sure you leave enough resources free on the server for administration overhead (being able to SSH in, run snmpd or other monitoring software).

If this sounds somewhat vague and general, that's because it is. You're basically asking people to engineer a complete hardware system for an application they know nothing about.

Kamil Kisiel
  • 12,184
  • 7
  • 48
  • 69