0

I checked the gemfire server log and found the following statements in my log file. :

    Rejected connection from Server connection from [client host address=XXX.XXX.XXX.XX; client port=XXXX1] because incoming request was rejected by pool possibly due to thread exhaustion

    Rejected connection from Server connection from [client host address=XXX.XXX.XXX.XX; client port=XXXX2] because incoming request was rejected by pool possibly due to thread exhaustion

    ....

What are the possible causes? How do i find the root cause?

I using Gemfire 9.8.6, and most of the regions are replicated. Clients are connected to the server regions through Caching Proxy by Spring Data Gemfire.

gemfire.properties [Server]

enter image description here

Based on the Cache Server Log File, i found out that my Handshaker max Pool Size: 4 and max-connections=800 and max-threads=0

    Handshaker max Pool size: 4
    CacheServer Configuration: port=51XX max-connections=800 max-threads=0 notify-by-subscription=true socket-buffer-size=1250000

I changed the file descriptors for redhat soft limit to 8192, and the hard limit to 81920, and number of processes (nproc) soft limit to 501408, with an unlimited hard limit.

  1. Total Number of Server : 2
  2. Total Number of Locator : 2
  3. Total Number of Client: 15

Thank you for your help

Jack
  • 89
  • 6

1 Answers1

0

This message is generally logged by the GemFire server whenever it doesn't have enough resources to handle the amount of requests coming in. I'd suggest having a look at Fine-Tuning Your Client/Server Configuration and Making Sure You Have Enough Sockets. Hope this helps. Cheers.

Juan Ramos
  • 1,421
  • 1
  • 8
  • 13
  • 1) How do i know what is my current max socket and thread? 2) What is the Handshaker max Pool size is equaled to 4? Based on the https://gemfire.docs.pivotal.io/98/geode/managing/monitor_tune/system_member_performance_connection_thread_settings.html, i thought the Hanshaker max pool size is equalled to 10. 3) Why the max-connection is equaled to 800? i thought the defaults for max-connections is -1 meaning the connection count can grow to accommodate the number of active threads performing region operations? – Jack May 14 '20 at 02:25
  • The defaults have those specific values because they have proven to be sufficient for the general use case, you can certainly tune them to match your SLA requirements, but I'd strongly suggest to execute several cycles of tunning and testing before applying the changes to production environments. The default values for the different parameters can be seen [here](https://gemfire.docs.pivotal.io/910/geode/reference/topics/cache_xml.html) and [here](https://gemfire.docs.pivotal.io/910/geode/reference/topics/gemfire_properties.html). – Juan Ramos May 14 '20 at 10:26
  • Where can I find out the maximum number of thread and socket ? In the gfsh or pulse? – Jack May 14 '20 at 15:28