1


I am building TCP Proxy: client <-> proxy <-> Vertica
I have a net.TCPListener, which takes incoming requests by AcceptTCP() and creating connections, then, making connection to destination socket by net.DialTCP("tcp", nil, raddr). Looks like a bridge. Default proxy model.

Firstly, at first version, i have a trouble: if i have 59 parallel incoming request, everything is fine. But if i have one more (60), i have a trouble: 1-59 connections are OK, but 60 and newer are fault. I cant catch error properly. Looks like some socket unexpectedly closes

Secondly, i tried to set queue for listener. It helps me a lot: but if i have more than 258 requests, i get error again.

My question: is there any limit of connections in net package? May be it is system limitation?

For external info: Vertica running in docker container, hw/system: macbook, vertica limit connection pool: 5, but pool logic implemented into proxy. I also tried set "raw" proxy without pool logic (thats why i set queue for listener: i must not exceed threshold of Vertica User's pool), result is 258 requests..

UPDATED: (05.04.2020)
Looks like it is system limitations fault. Did I mention anywhere that I trying to run the whole system on one PC?
So, what I had:

  • 300 parallel processes as requests (making by multiprocessing.Pool Python) (300 sockets)
  • Listener that creates 300 connections (once more 300 sockets)
  • And series of rapidly creating/closing sockets in deep of proxy (according to queue and Vertica pool)

What I have now:

  • 300 python requests making from another PC in my local network (on Windows)
  • Proxy works fine
  • But I have several errors on Windows PC, which creating requests to my proxy. Errors like low memory in "swap file".

I still need to make some stress test for proxy. Adding less memory for swap file didn't solve my problem on Windows PC. I will be grateful for any suggestions and ideas. Thanks!

Denis Newbie
  • 11
  • 3
  • 2
    The `net` package does not have a connection limit. Many systems do have a limit. Write the application to handle and report the errors returned from AcceptTCP, DialTCP and the net.Conn methods. The errors will be useful in debugging the issue. – Charlie Tumahai Apr 04 '20 at 17:31
  • @CeriseLimón yeah looks like you are absolute right about systems limit. Question was updated a few secs ago. – Denis Newbie Apr 05 '20 at 16:11

1 Answers1

0

How does the proxy connect to Vertica? There is by default a maximum of 50 ordinary mortal users to be connected to one Vertica node at any one time. The superuser "dbadmin" always has 5 connections in addition to that.

So if I try to connect 60 times as dbadmin, I get this on a default Vertica configuration:

Connection attempt failed: FATAL 4060:  New session rejected due to limit, already 55 sessions active

You can increase the Vertica config item MaxClientSessions from its default of 50 per node.

Command is : ALTER NODE <_node_name_> SET MaxClientSessions = 100, for example.

I suppose you are always connecting to the same Vertica node, and that you have set ConnectionLoadBalancing to FALSE. So you always connect to the same node, and soon reach the default maximum of 50.

Hope that's the reason found ....

marcothesane
  • 6,192
  • 1
  • 11
  • 21
  • Proxy just builds a bridge between Client and Vertica. User connect via vertica-python, jdbc, etc... As i mentioned at end of my question, I have vertica limit connection = 5 for my user (not dbadmin), and purpose of my proxy is handling a lot of connection, managing connections and provide slots for clients. It's hard to explain, english is not my native, so there is example: Proxy: User limit: 5 Parallel connections to proxy: 100 All: passed Without proxy only 5 will be passed, other will failed Hope you got it – Denis Newbie Apr 05 '20 at 15:57