My application is crawling a website. I use this crawler :
On my dev environment, the latency is acceptable (3-4 sec) but on my prod server (ec2 T2.micro Linux/Amazon) there is a 30 seconde latency (wait ttfb).
I tested the same code in another server (1&1) and there is a acceptable latency (3-4 sec). So the problem is not inside the code, but about the server config.
I have tried :
- check the php conf (php.ini)
- add cache to php (Zend OPCache)
- add apache 2 vary: accept-encoding header
- test a t2.medium instance
- change the region of my instance
I found this similar question : PHP + Apache long wait time but I think that my ec2 volume is an EBS SSD so there is no access time issue.
This unsolved question is also link to mine : https://stackoverflow.com/questions/34299038/deployed-web-application-long-ajax-wait-ttfb-time
An extract of vmstat during the request:
[ec2-user@ip-xxx-xx-xx-xx ~]$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 1580 534472 29696 240356 0 0 2 31 14 41 0 0 100 0 0
0 0 1580 534456 29696 240356 0 0 0 0 19 43 0 0 100 0 0
0 0 1580 534456 29696 240356 0 0 0 0 12 37 0 0 100 0 0
0 0 1580 534456 29696 240356 0 0 0 0 12 38 0 0 100 0 0
0 0 1580 534456 29696 240356 0 0 0 0 15 45 0 0 100 0 0
0 0 1580 534456 29696 240356 0 0 0 0 18 49 0 0 100 0 0
0 0 1580 534456 29696 240356 0 0 0 0 12 38 0 0 100 0 0
0 0 1580 533544 29696 240356 0 0 0 0 32 60 0 0 100 0 0
0 0 1580 533544 29696 240356 0 0 0 0 13 39 0 0 100 0 0
0 0 1580 533544 29696 240356 0 0 0 0 17 45 0 0 100 0 0
0 0 1580 533544 29696 240356 0 0 0 4 16 49 0 0 100 0 0
0 0 1580 533544 29696 240356 0 0 0 0 13 40 0 0 100 0 0
0 0 1580 533544 29696 240356 0 0 0 0 13 38 0 0 100 0 0
0 0 1580 533544 29696 240356 0 0 0 0 13 38 0 0 100 0 0
0 0 1580 533544 29696 240356 0 0 0 0 16 47 0 0 100 0 0
0 0 1580 533544 29700 240352 0 0 0 4 17 48 0 0 99 1 0
0 0 1580 533544 29700 240356 0 0 0 0 15 39 0 0 100 0 0
0 0 1580 533544 29700 240356 0 0 0 0 13 39 0 0 100 0 0
0 0 1580 533544 29700 240356 0 0 0 0 13 39 0 0 100 0 0
0 0 1580 533544 29700 240356 0 0 0 0 17 46 0 0 100 0 0
0 0 1580 533544 29700 240356 0 0 0 4 19 50 0 0 100 0 0
0 0 1580 533544 29700 240356 0 0 0 0 15 41 0 0 100 0 0
[UPDATE]
Right now I know that this latence is due to a GuzzleHttp\Client request :
(string) $this->httpClient->get($url)->getBody();
My security group seems ok, all is allowed from outbound and HTTP is allowed from inbound. My network out also seems ok (40,000 - 50,000 bytes when I request).
I'm thinking that this should be a problem with mpm_prefork_module...
[UPDATE 2]
From SSH, If I curl http://google.fr, I have no latency. If I curl my specific url (http://www.leboncoin.fr) the latency is >30 sec. It's specific to my AWS Ec2 with this URL.
