2

I am trying to call the information from an API to retrieve the data for items within the $baseURL. 

$url1

is used to GET the availability json data.

$url2

is used to GET the json data for an individual item. e.g: Content, images, multimedia etc.

<?php

 $APIKEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
 $baseURL = 'http://feeds.example.com/v1/';
 $ext =  '.json';

 // calls latest updated items
 $url1   = 'folder';
 // calls specific ID of item with corresponding details
 $url2   = 'folder/item{id}';
 $ch1 = curl_init($baseURL . $url1 . $ext);
 $ch2 = curl_init($baseURL . $url2 . $ext);

 $headers = array();
 $headers[] = 'x-auth-token:' . $APIKEY;
 $headers[] = 'X-Affiliate-Authentication:'. $APIKEY;
 $headers[] = 'Content-Type: application/x-www-form-urlencoded; charset=utf-8';

 curl_setopt($ch1, CURLOPT_USERPWD, $APIKEY);
 curl_setopt($ch1, CURLOPT_HTTPHEADER, $headers);
 curl_setopt($ch1, CURLOPT_RETURNTRANSFER, true);

 curl_setopt($ch2, CURLOPT_USERPWD, $APIKEY);
 curl_setopt($ch2, CURLOPT_HTTPHEADER, $headers);
 curl_setopt($ch2, CURLOPT_RETURNTRANSFER, true);

 $mh = curl_multi_init();

 curl_multi_add_handle($mh,$ch1);
 curl_multi_add_handle($mh,$ch2);

 $active = null;

 do {
   curl_multi_exec($mh, $active);
 } while ($active);

 $response_1 = curl_multi_getcontent($ch1);
 $response_2 = curl_multi_getcontent($ch2);
 echo "$response_1 $response_2";
 // Add function here

 // Close function here
 curl_multi_remove_handle($mh, $ch1);
 curl_multi_remove_handle($mh, $ch2);

 curl_multi_close($mh);

 ?>

'response_1' outputs (which is as expected):

{"data":[{"id":2,"last_update":"2016-08-26 15:43:39"},
{"id":6,"last_update":"2016-08-26 08:57:53"},
{"id":7,"last_update":"2016-08-23 16:49:15"},
{"id":8,"last_update":"2016-08-19 10:20:00"},
{"id":9,"last_update":"2016-08-28 23:44:25"},
{"id":28,"last_update":"2016-08-27 12:45:16"},
{"id":29,"last_update":"2016-08-12 07:55:32"},
{"id":30,"last_update":"2016-07-29 11:00:23"},
{"id":31,"last_update":"2016-07-05 09:48:50"}
. 
.. 
...

'response_2' outputs (which is expected):

{"data":[
 {
  "id":2,"last_update":"2016-08-26 15:43:39",
  "address":
  {
  "address1":"Foo Bar",
  "address2":"Foo","city":"Bar","state":"FooBar","zip_code":"F00 84R","country":"Foo Bar",
 },
 "details":
 {
 "dwelling_name":"FooBar Hotel","hotel_type":"Big",
 "maximum_capacity":16,
 "base_capacity":16,
 "bedrooms":6,
 "bathrooms":3,
 "currency":"GBP"
 },
 .
 ..
 ...

I need to determine the best method to dynamically update the $url[]. The following is my logic and won't work if you ran it because of course it is just my notes...

 // folder/item{id}     
 $url[];
 // Retrieved from 3rd party API for comparison
 $storedUpdate;

  foreach {
   item{id} 
    if $storedUpdate and item{id} matches item{id} from $url1 {
      do nothing
     else {
       POST item{id} data to 3rd party API
       POST $latestUpdate to 3rd party API
     }
   };

I chose curl_multi_init due to its ability (from reading up on it) to parse data in a asynchronized manner. With item{ids} easily exceeding 3500, it seems illogical to manually code these into my work which is why I am looking at the plausibility of doing it dynamically.

I have been scouring the internet, forums and php curl manual to find a means of dynamically updating $url2 so that it GET's all data.

I believe I can use $url[] to input multiple urls, the problem seems to remain that I would therefore have to manually input these URLs. 

feeds.example.com/v1/folder/{id}.json

I won't beat around the bush with this, I have learned this in a very short time frame and am at my whits end trying to determine the best practise for achieving my aim.

Any help would be much appreciated.

Granted
  • 53
  • 8

1 Answers1

1

If I'm understanding you correctly, then your logic is slightly off. Your second request depends on information returned in the first request, therefore you require a synchronous flow not an asynchronous one. To do this asynchronously would require... time travel.

If you still want to reap the benefits of the curl_multi... functions, this will have to be limited to the second request(s). For example:

<?php

 $APIKEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';
 $baseURL = 'http://feeds.example.com/v1/';
 $ext =  '.json';

 // calls latest updated items
 $url   = 'folder';

 $ch1 = curl_init($baseURL . $url . $ext);

 $headers = array();
 $headers[] = 'x-auth-token:' . $APIKEY;
 $headers[] = 'X-Affiliate-Authentication:'. $APIKEY;
 $headers[] = 'Content-Type: application/x-www-form-urlencoded; charset=utf-8';

 curl_setopt($ch1, CURLOPT_USERPWD, $APIKEY);
 curl_setopt($ch1, CURLOPT_HTTPHEADER, $headers);
 curl_setopt($ch1, CURLOPT_RETURNTRANSFER, true);

 $response_1 = curl_exec($ch1);
 $response_1_json = json_decode($response_1, true);
 $response_1_data = $response_1_json['data'];

 $mh = curl_multi_init();
 $multiHandles = [];

 foreach ($response_1_data as $key => $item) {
     $id = $item['id'];
     $multiHandles[$key] = curl_init($baseURL . 'folder/item' . $id . $ext);
     curl_setopt($multiHandles[$key], CURLOPT_USERPWD, $APIKEY);
     curl_setopt($multiHandles[$key], CURLOPT_HTTPHEADER, $headers);
     curl_setopt($multiHandles[$key], CURLOPT_RETURNTRANSFER, true);
     curl_multi_add_handle($mh, $multiHandles[$key]);
 }

 $active = null;

 do {
   curl_multi_exec($mh, $active);
 } while ($active);

 $response_2 = "";

 foreach ($response_1_data as $key => $item) {
     $response_2 .= curl_multi_getcontent($multiHandles[$key]);
     curl_multi_remove_handle($mh, $multiHandles[$key]);
 }

 echo $response_2;

 curl_multi_close($mh);

 ?>

I've not had chance to test this so it's likely something is off, but hopefully you get the gist of what I'm doing here. Hope this helps :)

Sworrub Wehttam
  • 588
  • 4
  • 14
  • Cheers for the help here, gutted about the time travel. I updated a few bits in my editor '$url1' to '$url' when defining '$ch1' which was straight forward. The foreach loop seemed to output an error like: `Invalid argument supplied for foreach()` I added an if statement derived from the post: [link](http://stackoverflow.com/questions/2630013/invalid-argument-supplied-for-foreach) by Andy Shellem – Granted Aug 31 '16 at 09:58
  • Comment Voided as you replied as I was writing my comment which is no longer valid other than the fact that I have been learning this within the last 9 days from no prior knowledge of PHP curl. I will implement what you said and return in a moment. – Granted Aug 31 '16 at 10:19
  • Ah, I wasn't paying enough attention to your JSON. The line `$response_1_data = $response_1_json['data'];` should infact be `$response_1_data = $response_1_json->data;` as it's an object, not an array. I'll edit the code – Sworrub Wehttam Aug 31 '16 at 10:19
  • We posted that roughly the same time, just try altering the line as I just suggested... that should be the root of your new errors. – Sworrub Wehttam Aug 31 '16 at 10:25
  • `Notice: Trying to get property of non-object` is an error for the above mentioned change `Warning: Invalid argument supplied for foreach()` was because I removed my IF statements previously mentioned `curl_multi_getcontent() expects parameter 1 to be resource, null given` `Undefined variable: ch2` which is due to the fact that $url1 was changed to `$url` and therefore `$url2` does not exist thus not having `$ch2` set with the `$baseURL . $url2 . $ext` etc – Granted Aug 31 '16 at 10:43
  • Sorry, I made a lot more mistakes than I realised. I've edited the code again, try it with those changes. The other errors should be gone, but you may still get `Notice: Trying to get property of non-object` or `Warning: Invalid argument supplied for foreach()` if your JSON is different - let's see what happens first though. – Sworrub Wehttam Aug 31 '16 at 10:58
  • Both errors you mention above still exist unfortunately. In what way do you mean if my JSON is different? No data is being returned after notice / warning messages – Granted Aug 31 '16 at 11:21
  • I have placed: `var_dump($response_1);` after `$response_1 = curl_exec($ch1);` and `$response_1_json = json_decode($response_1, true);` to determine what is coming out. The result includes: `string(78583)` before outputting the desired output (shown in the initial question): `"{"data":[{"id":2,"last_update":"2016-08-26 15:43:39"},{"id":6,"last_update":"2016-08-26 08:57:53"},{"id":7,"last_update":"2016-08-23 16:49:15"}.... etc` : After the output finishes, it returns to the `Notice: Trying to get property of non-object` – Granted Aug 31 '16 at 11:42
  • Have you `var_dump`ed $response_1_json also? It should show something like `object(stdClass)#1 (...) {` take a look at the [documentation for json_decode](http://php.net/manual/en/function.json-decode.php) too, it's possible that the data is deeper than the recursion limit. I'm also gonna quickly make a test script too see if I'm going wrong anywhere – Sworrub Wehttam Aug 31 '16 at 12:11
  • Ok, I think your JSON is invalid. I tested it using [this validator](http://jsonlint.com/) and it points out the trailing comma at the end of the `address` object. With your JSON like this, `json_decode` won't work (it will return null) hence your later errors. I'll see if there's some sort of *JSON cleaner/fixer* function... – Sworrub Wehttam Aug 31 '16 at 12:19
  • There are many attempts at fixing JSON using regex etc, but none really fit your exact problem. I can adapt one to fit if you need it - but at the end of the day this will be merely masking the problem of your badly formatted JSON. Where does this data come from? Is it something you're in control of? If so, then use the proper built in functions to encode the JSON. If it's from a 3rd party report it as a bug. Anyway, mock-out the $response_1 with properly formatted JSON as a test, if it works then I'll leave this answer as is and we can continue this in chat (but let's see what happens first). – Sworrub Wehttam Aug 31 '16 at 12:52
  • Re your question about `var_dump`ing on the `$response_1_json`: it outputs `array(1) { ["data"]=> array(1648) { [0]=> array(2) { ["id"]=> int(2) ["last_update"]=> string(19) "2016-08-26 15:43:39" } [1]=> array(2) { ["id"]=> int(6) ["last_update"]=> string(19) "2016-08-26 08: ...` which is the correct output I would need. – Granted Aug 31 '16 at 13:41
  • Interesting, in that case you can disregard a lot of what I said. The first "fix" I mentioned is now irrelevant, change it back to `$response_1_data = $response_1_json['data'];` as it is in fact an array... I don't know *how* it's an array, but that's what it says so we'll go with that. The `JSON` you posted in your question btw *is* invalid, was that additional comma just a result of you pasting it here? – Sworrub Wehttam Aug 31 '16 at 13:50
  • Ah, it's an array because of the second parameter set to `true` in `json_decode`. I'll change the code in the answer back. – Sworrub Wehttam Aug 31 '16 at 13:55
  • @ Sworrub The additional comma may have been down to me cutting off the rest of the output as it does go on for quite a bit and I thought pasting the first couple of lines would allow people to get the idea. Now I have updated the code my xampp server is timing out `Fatal error: Maximum execution time of 30 seconds` on **Line 40** which is: `curl_multi_exec($mh, $active);` – Granted Aug 31 '16 at 14:09
  • Setting `set_time_limit(0);` has given the server enough time to work out an error which I have seen all too much in this lovely venture of mine: `{"error":{"code":404,"message":"Invalid URL."}}{"error":{"code":404,"message":"Invalid URL."}}` to rectify this I changed the `$multiHandles[$key] = curl_init($baseURL . 'folder/item' . $id . $ext);` specifically `'folder/item'` to `'folder/'` and now it returns 100 results of id's up to 2393. NOTE: Removing `set_time_limit(0);` instantly crashes XAMPP. Can I keep it there? or should I set it to 2 minutes perhaps? – Granted Aug 31 '16 at 14:16
  • "*and now it returns 100 results of id's up to 2393*" is that not your desired result? To get the details of each id returned in your first response? – Sworrub Wehttam Aug 31 '16 at 14:29
  • My apologies, I have made so many iterations of this code I can honestly say learning it in such a short time makes it difficult to keep track of things. But yes, having retrieved the data of each individual dwelling (folder/ = dwelling/) using the ID's retained from the first GET request is what I was aiming for. This means I am now ready to set up a second connection (with APIKEY) to POST the data accordingly: This of course does not fall under my initial request for help and therefore should I need assistance, I will likely create a second question and make references to this one. – Granted Aug 31 '16 at 14:43
  • Excellent, I'm happy to have helped. – Sworrub Wehttam Aug 31 '16 at 15:06
  • Thank you very much for your help and your patience. Hopefully the troubling shooting will also help others trying to achieve something similar to. – Granted Aug 31 '16 at 15:10