Downloading CSV with StreamedResponse stops after several rows

Question

I have a MongoDB collection with a few thousand entries which I would like to download as a CSV file.

I have some code which basically does the following; A controller method queries the database using Doctrine ODM for all records. The returned cursor is then fed to a StreamedResponse. In the StreamedResponse I loop over the cursor and output each record as a line in the CSV.

The following code works, and a file gets downloaded. Only, it contains not more than 60 rows. What could be the reason that it stops streaming at that point, while the cursor count indicates that there are 2670 results?

// Get all available bookings
$cursor = $this
        ->get('doctrine_mongodb')
        ->getRepository('FooBundle:Booking')
        ->getQueryBuilder()
        ->getQuery()
        ->execute();

$response = new StreamedResponse(function () use ($cursor) {

    $handle = fopen('php://output', 'r+');

    $headerPrinted = false;

    // Calling $cursor->count() here returns 2670

    foreach ($cursor as $result) {
        // Transform the Booking object to a flat array.
        $data = $this->constructRow($result);

        // Print CSV header
        if (!$headerPrinted) {
            fputcsv($handle, array_keys($data), ';', '"');
            $headerPrinted = true;
        }

        // Add a line in the csv file.
        fputcsv($handle, $data, ';', '"');
    }

    fclose($handle);
});

$response->headers->set('Content-Type', 'application/force-download');
$response->headers->set('Content-Disposition', 'attachment; filename="bookings.csv"');

return $response;

And as far as I can see there is no code that accumulates data in memory while streaming. I use Symfony 2.7, Doctrine MongoDB ODM 1.0.3 and PHP 5.5.9 with a memory_limit of 256M.

@Matteo, There was nothing added to the logs, which thinking of it, is quite suspicious. Turns out that I had a wrongly configured log handler. After fixing this I got a ``Doctrine\ODM\MongoDB\DocumentNotFoundException``. Apparently a referenced document was deleted without removing the reference from the Booking object. I think that fixing this will my fix my problem. :) — Sander Toonen, Dec 10 '15 at 08:37
@Matteo, Well, for now I manually removed the reference from the document and tried again. I'm looking into using [DoctrineExtensions](https://github.com/l3pp4rd/DoctrineExtensions/blob/master/doc/reference_integrity.md) to avoid dead references in the future. But catching the error in case it occurs again might not be a bad idea as well. But how could I catch the error and keep the foreach loop going? If I wanted to catch it I would have to put the whole foreach block in a ``try { }`` right? — Sander Toonen, Dec 10 '15 at 09:05
for chatching the exception you need to redesing your loop (the exception now is raised in the loop condition) and set the loop condition with the `hasNext ` nethod of the curson then you can catch the exception in the loop body where you call the `getNext ` method that hydrate the object (see the soruce code https://github.com/doctrine/mongodb-odm/blob/master/lib/Doctrine/ODM/MongoDB/Cursor.php#L303) — Matteo, Dec 10 '15 at 09:19
@Matteo, I was thinking of creating wrapper class around the cursor that implements the Iterator interface and that has the ``try {} catch (){}`` inside the next method. But your idea is much simpler. I like it! — Sander Toonen, Dec 10 '15 at 09:39
i think a do while statement do the work, let me know if you need an example of it. — Matteo, Dec 10 '15 at 09:41
I managed it :) But if you provide the example as an answer I will accept it. — Sander Toonen, Dec 10 '15 at 09:45

score 1 · Accepted Answer · answered Dec 10 '15 at 10:10

As described in the comment, the problem is related to a not idratable object on the result from the cursor. For catching the exception and write the correct data, you can catch the excection. The loop need to be rewrited in order to catch the exception. As Example, you can do a do while loop as follow:

do {
        try 
        {
            // this could rise an exception
            $result = $cursor->getNext()
            // Transform the Booking object to a flat array.
            $data = $this->constructRow($result);

            // Print CSV header
            if (!$headerPrinted) {
                fputcsv($handle, array_keys($data), ';', '"');
                $headerPrinted = true;
            }

            // Add a line in the csv file.
            fputcsv($handle, $data, ';', '"');
        } catch (\Exception $e) 
        {
          // do nothing and process next element
        } 
 }
while($cursor->hasNext())

I would suggest also to use a library for managing the CSV writing, such as:

hope this help

score -1 · Answer 2 · answered Dec 10 '15 at 00:10

-1

Sounds like its to do with your memory limit in your php.ini

ini_set('memory_limit','16M');

Bump that up and see how many lines you get

answered Dec 10 '15 at 00:10

NickOS

764
1
7
18

I've edited my post to add some information about the memory settings. I have quite some memory available and the whole point of using a StreamedResponse is not to keep the sended data in memory. – Sander Toonen Dec 10 '15 at 08:27

Downloading CSV with StreamedResponse stops after several rows

2 Answers2