Sending a lot of API requests quickly with php?

Question

I'm looking to make a bunch of REST api calls as quickly as possible. I currently have about 1,000 requests to make.

The results of these calls are not needed for any processing or any thing like that. I just simply need to post all of them to the api url.

I of course tried inside my loop which is very slow. I have also tried using curl_multi_exec but that is just about as slow. Here is that code.

foreach($users as $user){
                $ch = curl_init();
                curl_setopt($ch, CURLOPT_USERAGENT, 'Mandrill-PHP/1.0.36');
                curl_setopt($ch, CURLOPT_POST, true);
                curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
                curl_setopt($ch, CURLOPT_HEADER, false);
                curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
                curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
                curl_setopt($ch, CURLOPT_TIMEOUT, 600);
                curl_setopt($ch, CURLOPT_URL, 'https://mandrillapp.com/api/1.0/messages/send.json');
                curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
                curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
                curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
                curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($mandril_message));

                curl_multi_add_handle($mh,$ch);
}
        $active = null;
        //execute the handles
        do {
            $mrc = curl_multi_exec($mh, $active);
        } while ($mrc == CURLM_CALL_MULTI_PERFORM);

        while ($active && $mrc == CURLM_OK) {
            if (curl_multi_select($mh) != -1) {
            do {
                $mrc = curl_multi_exec($mh, $active);
            } while ($mrc == CURLM_CALL_MULTI_PERFORM);
            }
        }

Any thoughts on how I can do this quickly. the rest of my code only takes a second or so to complete without the curl code.

does the api allow for batching, e.g. send over `1,2,3,4,...` as a single request, rather than individual requests of `1`, then `2`, then `3`, etc...? — Marc B, Jul 18 '13 at 17:46
I'm currently looking into that. So far I have not found any thing like that. They will let you send the same email to multiple people but in this case every email is unique to each user. I'm working with mandril's api btw. — hcker2000, Jul 18 '13 at 17:48
I have emailed mandril (the api provider) to see what they say about sending them lots of emails via one transaction. — hcker2000, Jul 18 '13 at 21:02

score 2 · Accepted Answer · answered Jul 30 '13 at 20:51

Ok so I finally found some time to get this working correctly using the mandril api. I went from about 2000 api request to 1. Processing time went from 10 minutes to 6.8 seconds.

So basically you just need to make sure you use there merge vars and specify a recipient for each set of merge vars.

Here is an example of how to do this.

"to": [
        {
            "email": "recipient1.email@example.com"
        },
        {
            "email": "recipient2.email@example.com"
        }
    ],
"merge_vars": [
        {
            "rcpt": "recipient1.email@example.com",
            "vars": [
                {
                    "name": "FNAME",
                    "content": "John"
                },
                {
                    "name": "FEED",
                    "content": "Your personalized feed content here"
                }
            ],
            "rcpt": "recipient2.email@example.com",
            "vars": [
                {
                    "name": "FNAME",
                    "content": "Jane"
                },
                {
                    "name": "FEED",
                    "content": "Your personalized feed content here"
                }
            ]
        }
    ]

what to do if we have different email content for different users? I mean I have to pass their names on the top of the email content (e.g. "hello xyz......").. than how to do it?? — Nishant Solanki, Mar 31 '14 at 08:10
Uh, unless I am reading this wrong, you are sending the merge vars for recipient2 only... would you mind clarifying? — Matt, Oct 02 '15 at 08:36

score 0 · Answer 2 · answered Jul 18 '13 at 17:48

0

A thousand REST requests is going to be slow. The trick here is to run the job asynchronously so that the user isn't impacted. You can send an AJAX request to the PHP script in question and enhance the PHP script to write progress data somewhere ($_SESSION, flat file, database, whatever.) You could then send separate AJAX calls to fetch the progress of the job and show updates to the user as the job runs.

This is all assuming that this is a web page we are talking about here and not a shell script.

answered Jul 18 '13 at 17:48

Alfred Fazio

956
7
10

Its a command line script. I really dont even need to know when it done as I will be able to tell if it sent correctly via the api dashboard. The lazy thing to do would be to just set the php max execution time to infinity and let it churn but that dosn't scale very well. – hcker2000 Jul 18 '13 at 17:53

Sending a lot of API requests quickly with php?

2 Answers2