I have been trying to load the USDA nutritional data into a mysql database from the ascii version of the data. I used the scripts from here and everything came in except the NUT_DATA.txt file which is ~34MB in size. The file loads correctly using the php script (so memory limits should not be an issue, they are set at 128M anyways) using the file_get_contents function. However, when it gets to the preg_split('#\n#', $file) line, the script ends without further comment. If I break the file into smaller pieces, it runs fine. Why does it not work on the full file? I have memory_limit set to 128M and upload_max_filesize set to 128M. The whole file seems to have been read in, it is just the preg_split that seems to be having problems.
Is there something obvious I'm missing?
Thanks!
Here is the php code I have been testing with:
<?php
// from http://drupal.org/node/107806
//$dbh=mysql_connect(/*db info*/);
//mysql_select_db(/*db info*/);
try
{
$dbh = new PDO('mysql:host=127.0.0.1;dbname=XXXXXXX', 'XXXXXXX', 'XXXXXXX');
echo "PDO access worked! <br />";
} catch (PDOException $e)
{
echo "PDO error: " . $e->getMessage() . "<br />";
die();
}
//List of SR20 filenames with associated field counts.
$filenames = array(//"DERIV_CD" => 2,
// "FD_GROUP" => 2,
// "NUTR_DEF" => 6,
// "SRC_CD" => 2,
// "WEIGHT" => 7,
// "FOOTNOTE" => 5,
// "DATA_SRC" => 9,
// "DATSRCLN" => 3,
// "FOOD_DES" => 14,
"NUT_DATA" => 17
// "NUT_DATA2" => 17,
// "NUT_DATA_test" => 17
);
foreach($filenames as $filename => $count) {
echo "inside foreach: $filename ";
$file = file_get_contents($filename.'.txt');
echo " read in the file, length = " . strlen($file) . "<br />";
$lines = preg_split('#\n#', $file);
echo "made it past preg_split <br />";
// print_r($lines);
echo " lines = " . count($lines). "<br />";
for($i=0;$i<count($lines);++$i) {
$line = $lines[$i];
// echo "Inside for loop, line = '" . $line . "'<br />";
//Some text fields are split over several lines. Concatenate them.
while(substr_count($line, '^') < $count-1 || (substr_count($line, '~')%2) == 1) {
++$i;
if($i>=count($lines)) { break; }
$line .= '<br />'.$lines[$i];
}
$fields = trim($line);
if(strlen($fields) == 0) continue;
$fields = str_replace(array("'", '~', '^'), array("''", "'", ','), $fields);
$sql = "INSERT INTO `$filename` VALUES($fields);";
//Insert zeroes for unfilled values.
$sql = str_replace(array(',,', ',)'), array(',0,', ',0)'), $sql);
$sql = str_replace(array(',,', ',)'), array(',0,', ',0)'), $sql);
$sql = str_replace(array(',,', ',)'), array(',0,', ',0)'), $sql);
//Some single-char fields aren't quoted.
$sql = preg_replace('#,([A-Z])\);$#', ",'$1');", $sql);
// comment out the database interface until things are working
/* $stmt = $dbh->prepare($sql);
if(!$stmt->execute())
{
echo $sql;
}
*/
}
}
echo "done";
?>
