I'm building a Firebird DB transaction manager in Python on Linux with JS+PHP clients. Javascript sends all necessary information to PHP; PHP encodes this and sends it via socket to Python, which has a socket bound to a port constantly listening and creates a new thread using threading to handle that request asynchronously.
The accept loop in python:
while 1:
conn, addr = s.accept()
req = conn.recv(1024)
ret = read_headers(req)
threading.Thread(target=client_thread, args=(conn, addr, ret, smphr,)).start()
s.close()
The send/read block in php:
$sock = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
$sockconnect = socket_connect($sock, $host, $port);
$msg = urldecode(http_build_query($params));
socket_write($sock, $msg, strlen($msg));
$received;
while(socket_recv($sock, $buf, 1024, 0) >= 1){
$received .= $buf;
}
echo $received;
socket_close($sock);
It all seems to be working properly until we started testing with larger number of connections. I have a loop in the JS client that sends several (25-100 are the numbers I've used so far) queries request with a select first random-number-of-lines from a large table.
The first few requests that the server receives are processed simultaneously but then it seems to become synchronous.
After much logging, found out that only 7/8 threads are active at any given time. New requests are only accepted and processed after one of the 7 current ones finishes.
If I comment the socket_recv while loop in php, python will then run everything simultaneously and return as soon as is available, which is exactly what I want, but since I've commented the block that actually gets the result, nothing is shown (obviously).
Every request/queries is logged as a different script call (according to chrome's network dev tool) so I don't know why they're blocking each other.
I'm fairly new at php/python and I can't for the life of me figure out what's going on.
Any suggestions?
Edit: I've also tried different bits of code to read the response in php (none worked as intended):
$buf = 'buffer';
socket_recv($sock, $buf, 1024, MSG_WAITALL);
echo $buf;
Same as previous implementation, 7/8 thread 'limit'
$buf = 'buffer';
socket_recv($sock, $buf, 1024, MSG_DONTWAIT);
echo $buf;
As the flag implies, doesn't wait for a response, therefore has no response
while ($out = socket_read($sock, 1024, PHP_NORMAL_READ)) {
echo $out;
}
Same thread 7/8 thread limit.
Second edit:
Added Python prints, in case it helps.
With read in php:
starting select first 3000 * from receb_quotas on tr1
starting select first 1 * from receb_quotas on tr0
starting select first 1 * from receb_quotas on tr2
starting select first 1 * from receb_quotas on tr4
starting select first 3000 * from receb_quotas on tr3
starting select first 3000 * from receb_quotas on tr5
finishing tr4 (count: 1) | remaining threads: 7
finishing tr0 (count: 1) | remaining threads: 7
starting select first 150 * from receb_quotas on tr8
starting select first 3000 * from receb_quotas on tr6
finishing tr2 (count: 1) | remaining threads: 7
starting select first 1 * from receb_quotas on tr7
finishing tr7 (count: 1) | remaining threads: 7
starting select first 3000 * from receb_quotas on tr9
finishing tr8 (count: 150) | remaining threads: 7
finishing tr1 (count: 3000) | remaining threads: 6
finishing tr3 (count: 3000) | remaining threads: 5
finishing tr6 (count: 3000) | remaining threads: 4
finishing tr5 (count: 3000) | remaining threads: 3
finishing tr9 (count: 3000) | remaining threads: 2
Without php read:
starting select first 3000 * from receb_quotas on tr1
starting select first 15 * from receb_quotas on tr0
starting select first 15 * from receb_quotas on tr3
starting select first 3000 * from receb_quotas on tr4
starting select first 1500 * from receb_quotas on tr2
starting select first 150 * from receb_quotas on tr5
starting select first 1 * from receb_quotas on tr6
starting select first 1500 * from receb_quotas on tr7
starting select first 150 * from receb_quotas on tr8
starting select first 15 * from receb_quotas on tr9
finishing tr0 (count: 15) | remaining threads: 11
finishing tr3 (count: 15) | remaining threads: 10
finishing tr6 (count: 1) | remaining threads: 9
finishing tr9 (count: 15) | remaining threads: 8
finishing tr8 (count: 150) | remaining threads: 7
finishing tr5 (count: 150) | remaining threads: 6
finishing tr7 (count: 1500) | remaining threads: 5
finishing tr2 (count: 1500) | remaining threads: 4
finishing tr1 (count: 3000) | remaining threads: 3
finishing tr4 (count: 3000) | remaining threads: 2
It really does seem that without the read in php, the queries are all started at the same time and returned as soon as ready.
