Big Foreach Loop with Google Maps API

Question

i have a bigger issue with a foreach() Loop and the Google Maps API class.

Currently i have 1500 datasets in my MySQL Table, i sort them out by "latitude IS NULL". Then i want to foreach() them and call the Google Maps API (KM6_Geo) each time and Save the new geocodes to the MySQL Database.

BUT: After about 250/300 Calls / Updates my Server replies with an "timeout".

Now this is my current script:

    foreach(KM6_Addresses::getGeoCodes() as $item):

        $geocode = KM6_Geo::generate($item['street'].' '.$item['zipcode'].' '.$item['city']);
        $data = array(
            'lat'       => $geocode['lat'],
            'lng'       => $geocode['lon']

        );

        $where = array('company_id' => $item['company_id']);        
        KM6_Addresses::updateGeoCode($data,$where);
    endforeach;

Now my Question: How can i foreach() with packages or something like that. So i do not try to update 1500 data sets.

Maybe someone can help me out.

Thanks! ;)

Are you running this through the web server or as a console script? You may be running into the php time limit. Or your browser is terminating the connection as its not receiving any content within the time limit. — datasage, Dec 19 '12 at 18:04

score 1 · Answer 1 · edited May 23 '17 at 11:56

It appears that you are reaching the API quotas.

This is triggered by too many requests from the same IP in a given day or too many in a period of time.

I would look at doing the following.

Try implementing a pause every 50-100 datasets to see if that helps with the first set of quotas.

The other would be to look at the API premier which might have higher quotas.

Check this out for another response on the same lines.

-- EDIT --

I did come across this then.

If you are able to pick up on the status codes, if you get the TOO_MANY_Queries, it appears that in their Python code, they execute a 2.0s delay.

Perhaps looking at doing something like that where you apply a 2s delay between each ForEach, or loop until you hit the limit, sleep(2) and then resume and see if that works.

The big thing here is you do not want get placed on the black list. Do you need to run all 1500 in 10-30 seconds? or if you implement a 2s delay between each, you would be looking at 3000s, or for your code to complete in 50 minutes and you would still have 1,000 calls left in your quota for that 24 hour period.

I would HIGHLY recommend building in safeguards from approaching the 2500 limit in a 24 hour period.

I'm not so sure. The limit per day is 2500. Its also not clear if the timeout has to do with script execution, or with the API. — datasage, Dec 19 '12 at 18:06

Big Foreach Loop with Google Maps API

1 Answers1