So before I give you a recommended solution for your question, I want to add really quick that I would probably find a php library which can achieve what you are looking to do, instead of executing shell commands from PHP, which can be a bad idea for many reasons, but if you intend on going that route, please look at this answer for some advice on security when executing shell commands via PHP:
https://stackoverflow.com/a/4535900/4660602
Now, to get back to your answer, anytime you are executing a big task in PHP that requires some extra time, it is never really a good idea to keep the user waiting at the browser. That is where building a Queue can become crucial to your application. You begin to work on some task AFTER you explain to your user via your user interface that the work is being done in the background and they will be notified when it is completed, etc.
There are ways to create a queue WITHOUT using 3rd party software, but there are some excellent tools out there such as RabbitMQ, IronMQ or Beanstalkd which can be extremely helpful to performing tasks in the background. These services push your task into a queue and these items in the queue are processed in a timely manner, but the user does not have to wait for a response until it finishes working, hence no more 504 timeouts.
OR you could try a much dirtier solution and just increase the script timeout value in php and on your server, but this can have some unexpected results. For nginx & for Apache
Best of luck!