2

My boss told me we can get a new web server if it is really necessary. I don't know if this is really necessary, so I decided to ask the ones who probably do know.

The website has the IIS 6.0 as well the Apache installed. The IIS is hosting only one application, which is a long polling application (consumes a lot of threads).

The current machine is a Windows 2003 with two CPUs (probably one physical, but two logical).

We have around 3,000 unique visitors per day.

When you type in the address and hit enter, sometimes it takes even more than a minute for an initial load.

I tried to fine tune the IIS by increasing the maxWorkerThreads, maxIoThreads, and other process model attributes but to not much avail.

Is there any use in obtaining a better machine? How much better? Any suggestions?

Any suggestions that could help and do not include a purchase of a new machine?

EDIT: Here's the current system information

Computer: Intel(R) Xeon(R) CPU 5130 @ 2.00 GHz, 2.00 GB RAM

Operating system: Microsoft Windows Server 2003, Standard Edition, Service Pack 2

Please let me know if you need other information in regards of the system.

pzelenovic
  • 133
  • 5
  • Seems like this is better suited for ServerVault than Stack Overflow. –  Apr 20 '11 at 22:30
  • thank you, I will go there... sorry for any inconvenience :) –  Apr 20 '11 at 22:35
  • 3000 uniques per day doesn't seem like a heavy load. How does your application handle long polling? –  Apr 20 '11 at 22:36
  • it relies on the IHttpAsyncHandler, which instantiates an object implementing IAsyncResult. Then the method calls for the startAsyncTask, which in the beginning of the execution is set to wait with the use of manual reset event mre.WaitOne(). Once the new data, coming in in the form of csv files is uploaded to the source folder, the file system watcher object fires an onChange event which calls for mre.Set() (releases the async thread to complete the response. I'm not sure if I'm using this the right way but any suggestions are more than welcome! – pzelenovic Apr 20 '11 at 22:52
  • 1
    Yeah, figure out what is broken in the software that is causing it to go slow? Figure what what specifically is the bottleneck. RAM/CPU/IO and upgrade the component that is the bottleneck. – Zoredache Apr 20 '11 at 23:00
  • cThank you for your response, but Can you please suggest a way on how to figure out what exactly is the bottleneck in my situation? For example, I know that the CPU usage never exceeds 50% and that is only when I set the minFreeThreads to 40 and the minLocalRequestFreeThreads to 40. When I leave those values as default the CPU usage is even lower. – pzelenovic Apr 20 '11 at 23:06
  • @noob-foreva, without knowing a lot more detail about your environment it is difficult for me to guess. Performance monitor may reveal some detail if you collect the right couters. Perhaps you need to run a network capture (tcpdump/wireshark). Perhaps it is partly on the client side and you need to add a browser plugin to monitor what is happening. You really have to solve the problem as a system. A good example of troubleshooting can be seen in the SF blog. http://blog.serverfault.com/post/views-of-the-same-problem-network-admin-dba-and-developer/ – Zoredache Apr 20 '11 at 23:16
  • @Zoredache - Thank you very much! I will proceed on that path... May the force be with you :) I would mark you as an answerer but there seems to be no option when the answer is provided as a comment. – pzelenovic Apr 20 '11 at 23:27
  • noob, if tweaking those values makes the CPU usage increase, it means that they were limiting the throughput at first (higher CPU usually = more work getting done). You may have hit another barrier now, though; I'd suggest looking at perfmon counters for PhysicalDisk\%Idle Time and Current Disk Queue Length - you may have hit a disk barrier, or a memory barrier, or a networking barrier (less likely from the description). Or, just run the PAL tool (from codeplex) against a wide range of counters and see what it suggests. Or Server Performance Advisor. Or another automated perf utility... – TristanK Apr 21 '11 at 05:02
  • You've posted no relevant details at all. On what basis do you expect an answer to be formed? – John Gardeniers Apr 21 '11 at 05:11
  • @TristanK - Thank you very much for your response. As my nickname suggests, I am tryly a noob, especially in the area of server tweaking and will follow your advice and come back with more details and results. – pzelenovic Apr 21 '11 at 06:57
  • @John Gardeniers, I am sorry, I am not really experienced in this area, so I was not sure what exactly I was supposed to post. I will edit my original post with the code and test results suggested by TristanK and maybe that helps. Honest thanks to your replies. – pzelenovic Apr 21 '11 at 06:59
  • @noob, you need to tell us more about your current server and the load being placed on it by the applications. To put things into perspective, what you have done so far is equivalent to asking whether or not you should get a new car, without telling us anything about your current car (other than that it has an engine) or what you use it for. More information means better answers, which is after all what you're after. – John Gardeniers Apr 21 '11 at 08:26
  • @John Gardeniers, OK that I can provide right away... Please see the edit in the original post. – pzelenovic Apr 21 '11 at 10:11

5 Answers5

1

If you can move to a new 6 core or more server, with say 12GB of modern memory, new disks, a newer 64-bit server OS and the latest versions of all your code requirements you should see around a 2-to-20 times speed increase overall, perhaps more in fact.

Chopper3
  • 101,299
  • 9
  • 108
  • 239
  • OK, thank you for your reply, but can you please explain what is your answer based on? For example I know that there is more CPUs in the 6 core, which translates (correct me if I'm wrong) to more available threads, but why is more modern memory and new disks and the 64-bit OS going to increase the speed? And also, the range of multiplication you have given me is quite wide... 2 to 20? – pzelenovic Apr 20 '11 at 23:01
  • 1
    @noob-foreva - you've answered your own question: "More", "Modern" and "new" are always faster than "less" "antique" and "old". We can't give you an exact figure because the fact of the matter is, *we know almost nothign about your app*. Even what you've told us doesn't even start to scratch the surface of making an estimate. – Mark Henderson Apr 20 '11 at 23:35
  • 1
    Modern servers are nice. But just blindly recommending hardware isn't going to do anyone any good. You need to read up on using Perfmon to monitor CPU, memory, disk and network and then use that information to start digging in more to find out where the constraints are exactly. Buying a shiny new server is going to do you no good at all if you find the box isn't particularly constrained but your app has issues or that it is using little CPU and memory but tons of IO and your new server doesn't have enough. – icky3000 Apr 21 '11 at 00:31
  • 3
    I disagree strongly with whoever downvoted this answer. If he is running 2003 it is a good guess that his hardware is antiquated, and also that he is hardware constrained: spending $500-$2000 is going to drastically improve the performance of his app. It's going to be cheaper too than spending tons of man-hours (or woman hours) trying to squeeze blood from a turnip. – Antonius Bloch Apr 21 '11 at 01:20
  • Antonius hits the nail on the head: you can usually solve a performance problem by throwing hardware or time at it, which is more cost effective is really up to you to decide. – gravyface Apr 21 '11 at 02:51
  • 2
    Lots of shoulds and usuallys here. You *usually* can solve a performance problem by throwing hardware or time at it, but without measuring your performance problem you don't know where to throw the money or time. Profiling your problem first so you know where to spend your money makes sense, rather than blindly throwing money at it. – Daniel Lawson Apr 21 '11 at 04:43
  • Thank you all for your replies. I did think that hardware might resolve the issue, but I also thought that maybe it is still unnecessary and I don't want to advise anybody to spend money for hardware if I can resolve this issue without that. I will run the previosly mentioned performance tests and will post them back here. Thank you all for your great help! – pzelenovic Apr 21 '11 at 07:08
  • -1: Big iron is not the way to address webserver scalability – symcbean Apr 21 '11 at 13:01
  • @symcbean - yes it often is, especially when the question is very specifically about buying new hardware and titled "Getting a new server" – Chopper3 Apr 21 '11 at 13:29
1

This is a little bit cheeky, but having seen Noob's software issues in this post on StackOverflow I would actually suggest against any real need to upgrade the server. Particularly as noob has mentioned he doesn't exceed 50% cpu times.

The problem you face (I think) is that your first requests move into your softwares' 60,000 ms timeout async callback code.

I think what you need to do is change the software so that on first request you pass results back directly, no file system watch. Retrieve the current data and post back as fast as you can get the code to do so.

On the async postbacks, implement your software as is.

You can find out if the page is being requested the first time, or as a result of a postback, using the Page.IsPostBack property.

This should allow your visitors to get the page within around 1-5 seconds from what I've seen of your code, and then only hold the async callbacks for the 60 sec timeout.

Apologies for hijacking a serverfault thread with a stackoverflow answer =(

Community
  • 1
Smudge202
  • 111
  • 3
  • Thank you for your suggestions, but this I've already done. When a user navigates to the live score page, the first thing I do is check whether it's the first load or the page postback, and if not postback I load the data regularly, without asynchronous threading. Once the page is finished loading, the javascript function makes an ajax call to the separate async http handler and goes on with the procedure described on stack overflow post. Thanks once again though :) – pzelenovic Apr 22 '11 at 14:11
0
  1. Find out your bottleneck, check CPU utilization, RAM, network, disk I/O
  2. Is there a DB for this application, is it local to the server / on the network, or a remote network?

Post this information for better feedback from others.

mwojski
  • 1
  • Kind thanks to your reply. There is a MS SQL database server, running locally, and with only one simple connection just for the home page, with only around 20 records from one table to pull... So I wouldn't think that's the problem. I will run some tests today and come back with results, and I will also post code for more info. Thank you once again! Best regards, noob-foreva – pzelenovic Apr 21 '11 at 07:06
  • Based on the hardware spec you posted, the server could be as old as 5-6 years. I would run its management software (such as OpenManage if it's a Dell that is) or check SNMP while you're at it to cover all your bases to make sure there aren't underlying hardware issues or overheating due to failed components. – mwojski Apr 21 '11 at 20:18
0

It doesn't seem to me you have a high enough load due to traffic.

I would first see if there is anything I can do to improve the code behind or find out whats really slowing things down. A minute of response usually indicates a long execution time. Buying a new server right away may be shooting yourself in the foot. A new server may or may not improve your performance by much.

I would also check your connection to the server if you are testing remotely and not over a LAN/Localhost and getting such response times.

These days people are more likely to recommend a hardware upgrade (cheaper and easier way out) which is quite all right. However it's important to see if we have done everything else right so a hardware upgrade would benefit more.

If you can't really fix the application (or it seems fine) for whatever reason or all optimizations have been done, and network seems ok then your only choice is to upgrade hardware for which you need to see your resource usage for a page load. Does the server use more memory or CPU. Its possible that other applications are competing for resources, can you start the resource/performance monitor and see the pattern?

hope it helps to give you more things to check

Abhishek Dujari
  • 567
  • 2
  • 5
  • 17
  • Thank you kindly for your suggestions. As suggested by you and other members, I will read into performance monitors, and see if I can find out what exactly is the problem. What I do know is that when it is not a peak hour, the website performs just fine. No errors, nothing... But that doesn't provide you with any relevant details, so I will run the tests today and come back with more relevant data. Thank you once again! – pzelenovic Apr 21 '11 at 07:04
  • noob, don't worry. just take your time and post what you can find. we'll try our best to put our heads together and find a good solution before you ask your boss to sign the check for more hardware :) – Abhishek Dujari Apr 21 '11 at 11:32
0

Do not buy a bigger box

Web serving (unlike database serving) gets much more benefit from adding lots of small servers in place of one large server - and since you're already running a lot of long-polling over HTTP, this is especially true. Increasingly the problems become those of the number of open connections, the number of running processes, the size of the network buffers, the rate at which the system can switch contexts....

So going out and spending, say $4k on 2 servers will give you a significantly faster solution than spending even quite a lot more on a single server.

There is a huge additional benefit - say the availability for your server is 99% - assuming no shared dependencies, then the availabilty for 2 nodes is 100-100*(0.01*0.01) = 99.99% - its nearly a hundred times more reliable.

While its possible that you could speed up your current system massively by tuning it we'd need to know a LOT more about the nature of the workload and the current config - and even then it would require a lot of work and testing to get an optimal solution - there are a few good books on the topic - go read some of them.

Having said that, sometimes upgrading (or preferably adding) hardware is the only solution - and often it can be the most cost-effective solution long before you've exhausted the tuning effort.

symcbean
  • 21,009
  • 1
  • 31
  • 52