0

I am having an almost identical problem to this one, posted in '09. It doesn't appear to be successfully solved, and I have been poring over related questions all day & implementing suggestions I have found.

I am attempting to pull dynamic (changes daily) data from an XML source (not a file) and insert into a mySQL database. When I attempt to pull the data without fwrite, the data returns without tags/keys in the output. Thus, I cannot properly assign the data to specific database fields.

I decided to write the information to a file and go from there, but the file aborts/"completes" at 334kb every single time. Unfortunately, I am a cURL novice and don't have the chops to see exactly what the issue is. Also, I am using vqMod for OpenCart to do this, otherwise I'd be using straight up PHP.

What am I missing?

Also, is there a way to pull the xml as an array instead of as a string (thereby potentially bypassing the fwrite step)? Or, should I be writing to some other file type?

Any suggestions or a redirect to a solved question I've missed are appreciated.

This is the relevant code, commented portions are fixes I've attempted: 

$curl = curl_init();
    $fp = fopen('dir/file.xml' , "w" );
    //w+ will not download any information from the url - file is created but empty.

    //AS IS, downloads first 334KB of file then lands on a blank page
    //and a 500 error when any of these other options are implemented. 
    curl_setopt($curl, CURLOPT_URL, 'http://www.url.com');
    curl_setopt($curl, CURLOPT_FILE, $fp);
    curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);

    //  curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    //  curl_setopt($curl, CURLOPT_TIMEOUT,  300);
    //  curl_setopt($curl, CURLOPT_NOPROGRESS, false);
    //  curl_setopt($curl, CURLOPT_RANGE, '0-1000');


    //  $data = array();
    $data = curl_exec($curl);

    fwrite($fp, $data);

    curl_close($curl);
    fclose($fp);

Update: Attempted to use simplexml_load_string instead of fwrite() to pull one product's information, but am still having limited success. Example of XML I am using:

        <?xml version="1.0"?>
      <response>
        <root>
          <part>
            <![CDATA[PARTNUM]]>
          </part>
          <errorcode>0</errorcode>
          <errormsg></errormsg>
          <special>N</special>
          <description>
            <![CDATA[]]>
          </description>
          <price>75</price>
          <weight>1.02</weight>
          <webpurch>Y</webpurch>
          <altnum>
            <![CDATA[ALT-NUM]]>
          </altnum>
          <active>Y</active>
          <redo>
            <![CDATA[]]>
          </redo>
          <codes> 
              <code>
                <fieldname>
                  <![CDATA[Mfr Part No]]>
                </fieldname>
                <fieldvalue>
                  <![CDATA[PARTNUM]]>
                </fieldvalue>
               </code>
               <code>
                <fieldname>
                  <![CDATA[Special Code 1]]>
                </fieldname>
                <fieldvalue>
                  <![CDATA[XYZ123]]>
                </fieldvalue>
               </code> 
            </codes>
          <customtag>N</customtag>
          <onhand>0</onhand>
          <notes>
            <![CDATA[PRODUCT-SPECIFIC NOTE]]>
          </notes>
          <mfr>
            <mfr_name>
              <![CDATA[MFR]]>
            </mfr_name>
          </mfr>
          <altpartnums>
            <altnum>
              <![CDATA[PARTNUM.12]]>
            </altnum>
          </altpartnums>
          <gtrue>N</gtrue>
          <group>
            <![CDATA[GROUP NAME]]>
          </group>
          <categories>
            <cat>294</cat>
            <cat>475</cat>
          </categories>
        </root>
      </response>

This is an example of the $data return for multiple products when I do not use fwrite(): 0 N 75 1.02 Y Y N 0 N 294 475 0 N 288 12 Y Y Y 18 Y 222 456 3786

Community
  • 1
  • 1
Jane
  • 1
  • 6
  • how about `simplexml_load_file` / `simplexml_load_string`, http://www.php.net/manual/en/function.simplexml-load-file.php, it might do it, it doesn't give you an array per se but it does gives you a pretty easy to handle object, how big is the file? – gmaliar Mar 01 '13 at 19:54
  • Would file_get_contents not also work here? – Sturm Mar 01 '13 at 19:55
  • @NathanMann, it would but it still needs parsing, simplexml_load_file and the likes of it in PHP solve it. – gmaliar Mar 01 '13 at 19:56
  • @Guy: The file varies in length - it is a list of active (30,000+) products with some related data coming from a Unix server. Every day, the list is slightly different. On the old system I'm converting from, I have a similarly created .csv that is usually about 23MB. – Jane Mar 01 '13 at 20:03
  • @Jane, and do you control the list as well? Is it possible to just send the deltas each day? – gmaliar Mar 01 '13 at 20:10
  • 1
    @Guy: No, I don't control the actual data that is included in the list. It is always the same detail set about each product that comes through (which is why I'd normally use the tags for allocation), but the products themselves are input/edited by other people and there is no way to know when someone has changed a minor detail. Essentially, I'd be updating manually all day every day if I had to make incremental changes to products in the DB. I'm testing the simplexml_load_file, will add results to initial post. – Jane Mar 01 '13 at 20:18

2 Answers2

0

Remove this line:

fwrite($fp, $data);

I'm quite sure $data is empty because you've specified not to return any string data, instead write it to a file. Not so surprising the the file is empty too.

silkfire
  • 24,585
  • 15
  • 82
  • 105
  • That's the thing - the file is not always empty. The information returned for $data is a list of characters (e.g. '1 Y N N 1333.4 55.45 MFR') without fwrite. There are at least 30,000 products to parse, and not all of them have a value in all fields of my xml. Thus, it is imperative that I have the matching xml tags to be able to properly put my product information from my server directly to my database. As I stated initially, I had tried it without fwrite but then decided to go that route to see if I could pull intact xml. – Jane Mar 01 '13 at 20:34
  • What do you mean by "matching tags"? Why are the contents not the same when you choose to return it as a string and not save to file? Doesn't make sense to me... – silkfire Mar 01 '13 at 20:49
  • I am pulling from xml with various trees. I need information for more than one tree AND need to know exactly which tree the info came from. The info is returned as an unbroken string, with no indicators as to what information is being presented. 1y vs. '1 Y' ---- There are further problems with nested trees, empty trees, etc. – Jane Mar 01 '13 at 20:55
  • ^^ I meant "roots" not 'trees'. Either way, I've got roots, parents, children, sub-children, even sibling elements (tags) that may or may not contain data for each specific product. – Jane Mar 01 '13 at 21:03
  • Your problem right now is failing to download the file right? One step at a time, you can't parse the file before you've succeeded to download it completely. Could you perhaps send a link to it unless it's private? – silkfire Mar 01 '13 at 21:09
  • The problem is that the download initiates but then ends at 334kb - this final file is incomplete because it only grabs about 300/30,000 products, yet the opening root tag is closed as if it is EOF. I am trying to figure out *why* the file stops downloading at 334kb. Yes, this is sensitive information, so I cannot send a link. :-/ – Jane Mar 01 '13 at 21:16
  • Are you on shared hosting? Maybe you have some limitations with file downloads. – silkfire Mar 01 '13 at 21:22
  • No, on a dedicated server. I regularly generate a file using this same XML and it ends up at about 23MB, so that's not the problem. That's why I was thinking it's some issue with the cURL I'm using to pull this info - I would prefer just to use PHP, but am limited by the framework I am using. It does not make a connection unless I use cURL... – Jane Mar 01 '13 at 21:31
0

I got it to work sans fwrite() using this code:

        if(extension_loaded('curl')){
                    $curl = curl_init();
                    curl_setopt($curl, CURLOPT_URL, 'http://www.url.com');
                    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
                    curl_setopt($curl, CURLOPT_TIMEOUT, 30);
                    curl_setopt($curl, CURLOPT_NOPROGRESS, FALSE);
                    curl_setopt($curl, CURLOPT_MAXCONNECTS, 1);
                    curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 0); 

                    $newparts = curl_exec($curl);

                    $xml = simplexml_load_string($newparts);

                foreach($xml->modpart as $item){

                      if(($item->active == 'Y') || ($item->active == 'YES')){
                          $status = '1';
                      } else {
                          $status = '0';
                      }


                if(!empty($item->mfr)){ 
                      $rs   = $this->db->query("SELECT manufacturer_id FROM ".DB_PREFIX."manufacturer WHERE name = '".$item->mfr->mfr_name."'");
                      $mfr_id = $rs->row['manufacturer_id'];
                }

                 if(!empty($item->codes)){
                          if($item->codes->code->fieldname == 'Mfr Part No'){
                              $mpn = $item->codes->code->fieldvalue;
                          } 
                  }
                      $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product (model, sku, mpn, quantity, date_available, sort_order, manufacturer_id, price, weight, status, date_added) 
                          VALUES ( 
                              '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."', 
                              '".mysql_real_escape_string($item->sku)."', 
                               '".$mpn."', 
                              '".mysql_real_escape_string($item->mpn)."', 
                               NOW(), 1000,
                              '".$mfr_id."',
                               '".mysql_real_escape_string($item->price)."', 
                               '".mysql_real_escape_string($item->weight)."', 
                               '".$status."', 
                                 NOW() 
                              )");    

                    $pr = $this->db->query("SELECT product_id FROM ".DB_PREFIX."product WHERE sku = '".$item->part."'");
                    $product_id = $pr->row['product_id'];

                    if(isset($product_id)){

                              $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_description (product_id, language_id, name, meta_keyword, meta_description, description, tag)
                                      VALUES(                                                       
                                              '" . (int)$product_id . "',
                                              '1',
                                              '" . $item->part. "',
                                              '" . $item->part. "',
                                              '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."', 
                                              '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."', 
                                              '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."' 
                                      )");


                            $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_to_store SET product_id = '" . (int)$product_id . "', store_id = '0'");

                            $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "url_alias SET query = 'product_id=" . (int)$product_id . "', keyword = '" .$item->part . "'");


                        foreach($item->cats as $cats){
                            $category = $cats->cat;
                                foreach($category as $category_id){
                                $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_to_category SET product_id = '" . (int)$product_id . "', category_id = '" . (int)$category_id . "'");
                                } 
                        } 
                             $this->cache->delete('product');
                    } 
            } 

            curl_close($curl);

    }

I still have had no luck with the timeout issue, but at least I can get my products into the database by some means. I will post the timeout solution if/when I find it.

Jane
  • 1
  • 6