We tried this POC to deploy code via AWS Code deploy on 20 live servers, which are behind Load balancer. We are having nginx running in front of Hiphop. We tried hot deployment, i.e. deploying while nginx was running.
As soon as the deployment process moves the new file to the designated place in the production servers, we start getting the following errors, which continues indefinitely on some servers, and the Jenkins jobs times out after polling for 50 minutes -
\nFatal error: syntax error, unexpected $end in /path/to/file.php on line 19477
It appears like only a part of the file gets loaded and read, even though the file in its entirety has no syntax errors.
Restarting nginx on such servers manually fixes the problem, but that does not seem to be a good solution.
We are trying to find out the reason behind this issue.
HHVM version being used - HipHop VM 3.12.0-dev (rel)
Nginx version - 1.8.0
Alternative approach
We are now trying to do cold deployment (shut down nginx then do deployment and then turn on nginx again), but that too is throwing its own issues. I will not post those details here, but the idea is to take the advantage of the large number of servers we have, and do cold deployment in such a way that only a small percentage of the servers behind LB have nginx off at a time, so that it does not cause too much load on the running servers.
