5

What I want to do :
User hit a url (ex: BASE-URL/download/json).
Code fetch huge data from database and stream it into a JSON file and user is prompted to download that file.

What I tried :
Created a large (700 MB) JSON file. Tried opening it with code, but error was thrown.

file_get_contents($file)

Error :

Allowed memory size of XXX bytes exhausted (tried to allocate YYY bytes)

I am working in Symfony2. Data can be upto 700MB in size. Can't increase the allocated memory. How can I achieve this? Please comment if any further information is required.

Anurag Rana
  • 1,429
  • 2
  • 24
  • 48
  • hava you try to increase php memory limit `ini_set('memory_limit', '512M')` ?? – egig Sep 29 '15 at 09:00
  • 3
    I haven't used it for large files so YMMV, but Symfony has `StreamedResponse` and `BinaryFileResponse` objects. Can't actually remember which one I used... Have a look anyway. Docs: http://symfony.com/doc/current/components/http_foundation/introduction.html#streaming-a-response – Darragh Enright Sep 29 '15 at 09:05
  • @egig Can't do that. Lets say I do not have access/permission to do that. – Anurag Rana Sep 29 '15 at 09:05
  • 4
    Your first problem is not streaming the file, it's reading the file. That's where PHP chokes; You should read your file line by line and stream the response as @Darragh suggests, so as to avoid memory exhaustion problems – tchap Sep 29 '15 at 09:09
  • @tchap - Got it. I will fetch data from DB in chunks. But how to stream it and prompt user to download it as a continuous file. – Anurag Rana Sep 29 '15 at 09:36
  • I suggest you to use a [streamedresponse](http://www.layh.com/2014/04/18/symfony2-download-filestream-as-streamedresponse/) in conjunction on a buffered file reading as discussed [here](http://stackoverflow.com/questions/2603807/how-to-save-memory-when-reading-a-file-in-php/2603923#2603923) – Matteo Sep 29 '15 at 09:40

1 Answers1

12

I was able to achieve what I wanted with the help of all the above comments.
Here is the code for the same.

$response = new StreamedResponse();  
$i = -999999;  
$response->setCallback(function () {  
    while($i < 999999){  
        echo '{"G1ello":"hualala"}';  
        $i = $i + 1;  
    }  
});  
$filename = "Data.json";  
$contentDisposition = $response->headers->makeDisposition(ResponseHeaderBag::DISPOSITION_ATTACHMENT, $filename);
$response->headers->set('Content-Type', 'application/json');
$response->headers->set('Content-Disposition', $contentDisposition);
return $response;

It created a large file which I was able to download. Exactly what I wanted. Posting the solution so that it can help someone with same issue.

Anurag Rana
  • 1,429
  • 2
  • 24
  • 48