Replace duplicates in a text file with PHP?

Question

I have text files that have list of thousands of names like this

DallasWebJobs
DallasWebJobs
DallasWebJobs
php_gigs
brotherjudkins
goldbergwb
SanDiegoWebJobs
brinteractive
muracms
browan85
php_gigs
php_gigs
php_gigs
php_gigs

1 name per line, 1 file may have up to 30,000 names on it though and I need to replace all duplicate names because probably as many as half are duplicates.

I would like to do this in PHP, 1 though was importing each line into a MySQL database and then doing it but that seems like overkill, i'm sure there is an easier way.

Please help if you can

Update I found this for emails, it should work too

$list = file('./Emailist.txt');
$list_unique = array_unique($list);
foreach ($list_unique as $mail) {
    echo $mail;
}

What about case sensitivity? Would **bob** be a duplicate of **Bob**? — Bot, Feb 29 '12 at 02:04
There is a trivial way of doing that in shell `sort yourfile | uniq`. Are you sure you want use php facilities? — zerkms, Feb 29 '12 at 02:06
@Computer I updated my question above, I just found an example that was used for emails, I think it will work for this. The names are Twitter names so I don't think the bob vs Bob will be an issue — CodeDevelopr, Feb 29 '12 at 02:07
following the previous comment I woulddo this in excel/open office if anyway possible. — jribeiro, Feb 29 '12 at 02:08
I had a simple program in C# that was made just for this task but I can't find it — CodeDevelopr, Feb 29 '12 at 02:09

score 1 · Accepted Answer · answered Feb 29 '12 at 02:11

From php.net: serg dot podtynnyi at gmail dot com 06-Feb-2009 11:21

//Remove duplicates from a text files and dump result in one file for example: emails list, links list etc

<?php 

$data1 = file("data1.txt");  

file_put_contents('unique.txt', implode('\n', array_unique($data1))); 
?>

This will remove all duplicates and save it as a file of unique.txt

or

<?php 

$data1 = file("data1.txt"); 

$uniqueArray = array_unique($data1)); 
?>

Will store it in $uniqueArray

score 0 · Answer 2 · answered Feb 29 '12 at 02:09

0

$lines = file("test-file");

foreach($lines as $line)
{
    $new[str_replace(array("\n","\r"),"",$line)] = 1;
}

print_r(array_keys($new));

answered Feb 29 '12 at 02:09

GregSchoen

404
3
9

score 0 · Answer 3 · edited Feb 29 '12 at 02:21

0

$file = file_get_contents($filename);
$arr = array();
$arr = split('\n',$file);
$arr = array_unique($arr);

Then write contents of $arr to textfile again

edited Feb 29 '12 at 02:21

Bot

11,868
11
75
131

answered Feb 29 '12 at 02:12

everconfusedGuy

2,709
27
43

Replace duplicates in a text file with PHP?

3 Answers3