0

I’m working with the royal mail PAF database in csv format (approx 29 million lines), and need to split the data into sql server using php.

Can anyone recommend the best method for this to prevent timeout?

Here is a sample of the data: https://gist.github.com/anonymous/8278066

Sofia Rose
  • 203
  • 2
  • 8

3 Answers3

1

To disable the script execution time limit, start your script off with this:

set_time_limit(0);

Another problem you will likely run into is a memory limit. Make sure you are reading your file line-by-line or in chunks, rather than the whole file at once. You can do this with fgets().

Brad
  • 159,648
  • 54
  • 349
  • 530
  • I tried using set time limit and fgetcsv, but it still timed out, is there a major difference between fgetcsv and fgets? – Sofia Rose Jan 06 '14 at 04:14
  • Your host has probably disallowed you from using `set_time_limit()`. See if you can edit your PHP.ini file's entry for `max_execution_time`, or talk to your host to see if they can allow you to use `set_time_limit()`. – Brad Jan 06 '14 at 04:39
0

Start your script with 

ini_set('max_execution_time', 0);
chanchal118
  • 3,551
  • 2
  • 26
  • 52
0

The quickest way I found was to use SQL Servers BULK INSERT to load the data directly and unchanged, from the csv files, into matching db import tables. Then do my own manipulation and population of application specific tables from those import tables.

I found BULK INSERT will import the main CSVPAF file, containing nearly 31 million address records in just a few minutes.

Mark Worrall
  • 290
  • 3
  • 9