41

Can you please explain the two methodologies which has been implemented in various servlet implementations:

  1. Thread per connection
  2. Thread per request

Which of the above two strategies scales better and why?

Geek
  • 26,489
  • 43
  • 149
  • 227
  • 1
    http://stackoverflow.com/questions/7457190/how-threads-allocated-to-handle-servlet-request - duplicate? – Mikhail Mar 05 '13 at 10:38

5 Answers5

34

Which of the above two strategies scales better and why?

Thread-per-request scales better than thread-per-connection.

Java threads are rather expensive, typically using a 1Mb memory segment each, whether they are active or idle. If you give each connection its own thread, the thread will typically sit idle between successive requests on the connection. Ultimately the framework needs to either stop accepting new connections ('cos it can't create any more threads) or start disconnecting old connections (which leads to connection churn if / when the user wakes up).

HTTP connection requires significantly less resources than a thread stack, although there is a limit of 64K open connections per IP address, due to the way that TCP/IP works.

By contrast, in the thread-per-request model, the thread is only associated while a request is being processed. That usually means that the service needs fewer threads to handle the same number of users. And since threads use significant resources, that means that the service will be more scalable.

(And note that thread-per-request does not mean that the framework has to close the TCP connection between HTTP request ...)

Having said that, the thread-per-request model is not ideal when there are long pauses during the processing of each request. (And it is especially non-ideal when the service uses the comet approach which involves keeping the reply stream open for a long time.) To support this, the Servlet 3.0 spec provides an "asynchronous servlet" mechanism which allows a servlet's request method to suspend its association with the current request thread. This releases the thread to go and process another request.

If the web application can be designed to use the "asynchronous" mechanism, it is likely to be more scalable than either thread-per-request or thread-per-connection.

FOLLOWUP

Let's assume a single webpage with 1000 images. This results in 1001 HTTP requests. Further let's assume HTTP persistent connections is used. With the TPR strategy, this will result in 1001 thread pool management operations (TPMO). With the TPC strategy, this will result in 1 TPMO... Now depending on the actual costs for a single TPMO, I can imagine scenarios where TPC may scale better then TPR.

I think there are some things you haven't considered:

  • The web browser faced with lots of URLs to fetch to complete a page may well open multiple connections.

  • With TPC and persistent connections, the thread has to wait for the client to receive the response and send the next request. This wait time could be significant if the network latency is high.

  • The server has no way of knowing when a given (persistent) connection can be closed. If the browser doesn't close it, it could "linger", tying down the TPC thread until the server times out the connection.

  • The TPMO overheads are not huge, especially when you separate the pool overheads from the context switch overheads. (You need to do that, since TPC is going to incur context switches on a persistent connections; see above.)

My feeling is that these factors are likely to outweigh the TPMO saving of having one thread dedicated to each connection.

aholbreich
  • 4,951
  • 2
  • 23
  • 38
Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • You wrote "(And note that thread-per-request does not mean that the framework has to close the HTTP connection after each request ...)" .So when the connection is reused later,does the same thread get attached to it or is it a different thread? – Geek Mar 05 '13 at 17:34
  • "*Thread-per-request scales better than thread-per-connection.*" ... What about the scenario where there is a **significant number** of HTTP requests served in a **single** HTTP connection? For this scenario, I can imagine that the thread-per-request strategy causes a lot of server load due to thread pool management overhead, while the connection-per-request strategy wouldn't have this overhead. – Abdull Jul 22 '13 at 23:27
  • I don't understand what you are saying. This question is about thread-per-request versus thread-per-connection. If you make the restriction that a connection can make only one request (i.e. non-persistent connections) then thread-per-request and thread-per-connection have the same characteristics. – Stephen C Jul 23 '13 at 01:23
  • @StephenC Let's assume a single webpage with 1000 images. This results in 1001 HTTP requests. Further let's assume *HTTP persistent connections* is used. With the *TPR* strategy, this will result in 1001 thread pool management operations (*TPMO*). With the *TPC* strategy, this will result in 1 TPMO... Now depending on the actual costs for a single TPMO, I can imagine scenarios where TPC may scale better then TPR. But maybe I am missing a point? – Abdull Jul 23 '13 at 09:53
  • **Java threads are rather expensive, typically using a 1Mb memory segment each, whether they are active or idle** Do you have a reference to confirm it? I just created 1,000 active threads running and the program takes 12m. – serg.nechaev Jan 09 '14 at 02:17
  • @serg.nechaev - I've seen the evidence in the OpenJDK source-code and in the Oracle JVM documentation. How are you actually measuring the memory usage? (12M sounds implausible to be ... if the threads are really active.) – Stephen C Jan 09 '14 at 06:37
  • 1
    @serg.nechaev - here's a specific reference: http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html ... and look for "ThreadStackSize" – Stephen C Jan 09 '14 at 06:40
  • _"the thread-per-request model is not ideal when there are long pauses during the processing of each request."_ But how is TPC better in this case? – garci560 Nov 25 '17 at 20:54
  • @nprensen - In that situation, TPC would avoid the situation where one connection (i.e. one user) is tying down LOTS of threads for a long period. However, the real solution for that scenario is asynchronous model. As I said in the sentences following the phrases that you quoted! – Stephen C Nov 26 '17 at 01:55
7

HTTP 1.1 - Has support for persistent connections which means more than one request/response can be received/sent using the same HTTP connection. So to run those requests received using the same connection in parallel a new Thread is created for each request.

HTTP 1.0 - In this version only one request was received using the connection and the connection was closed after sending the response. So only one thread was created for one connection.

Narendra Pathai
  • 41,187
  • 18
  • 82
  • 120
2

Thread per connection is the Concept of reusing the same HTTP Connection from multiple requests(keep-alive).

Thread per requestwill create a thread for each request from a client.Server can create a number of threads as per request.

Suresh Atta
  • 120,458
  • 37
  • 198
  • 307
  • 1
    Just to clarify for others, keep-alive will not necessarily allocate a thread for the entire connection, for instance when made against a servlet container. That's the difference between Keep-Alive and TPC. – tunesmith Jun 27 '15 at 07:29
1

Thread per request will create a thread for each HTTP Request the server receives .

Thread per connection will reuse the same HTTP Connection from multiple requests(keep-alive) .AKA HTTP persistent connection but please note that this supported only from HTTP 1.1

Thread Per Request is faster as most web container use Thread Pooling.

The number of maximum parallel connections you should set on the number of cores on your server. More cores => more parallel threads .

See here how to configure... Tomcat 6: http://tomcat.apache.org/tomcat-6.0-doc/config/executor.html

Tomcat 7: http://tomcat.apache.org/tomcat-7.0-doc/config/executor.html

Example

Sudhakar
  • 4,823
  • 2
  • 35
  • 42
0

Thread per request should be better, because it reuses threads, while some client may be idle. If you have a lot of concurrent users, they could be served with a less number of threads and having equal number of thread would be more expensive. There is also one more consideration - we do not know if user is still working with application, so we cant know when to destroy a thread. With a thread per request mechanism, we just use a thread pool.

Mikhail
  • 4,175
  • 15
  • 31
  • Does it use thread pool or spawns new threads and joins the old ones? – LtWorf Mar 05 '13 at 10:45
  • Spawning a new thread is very expensive operation. I assume that any adequate implementation reuses threads through pool. – Mikhail Mar 05 '13 at 10:51
  • I know it's expensive, that's why I asked. Without thread pool using one thread per request is much more expensive than using one thread per connection. And assumptions are bad, that's why I asked. – LtWorf Mar 05 '13 at 11:01
  • Which exact implementation you are talking about, javax.servlet.* is just an API, it does not specify the way requests are served. – Mikhail Mar 05 '13 at 11:15
  • Yup, but usually API are implemented by some software. – LtWorf Mar 05 '13 at 12:28