Removing lines that contain the same thing in a list

Question

I have a list that contains something like this:

C12 0402 123456 90
C133 0402 123456 90
C9 0402 123456 90
C132 0603 abcd 0
C54 0603 abcd 0
R FID1 fiducial 0
R FID2 fiducial 0
R FID3 fiducial 0
R FID4 fiducial 0

I would like to check each line in the List and make sure that the duplicates are not added... I tried this but I know there is a logic error in here.

List<string> noDuplicatesList = new List<string>();

foreach (var line in theList)
{
    if (!noDuplicatesList.Contains(line.PartNumber)) //This is doing nothing...?
        noDuplicatesList.Add(line.Name + " " + line.PartDescription + " " line.PartNumber + " " + line.Rotation);
}

foreach (var line in noDuplicatesList)
{
    var splitLine = line.Split(' ');
    //Print out statements...
}

Question

How do I remove/skip lines that contain the same value (in the example above 123456, abcd, fiducial) so that it will only print something like this:

C12 0402 123456 90
C132 0603 abcd 0
R FID1 fiducial 0

Will duplicates always be grouped together like in the example? Or could a line with e.g. "123456" also appear later in the list? — Igby Largeman, Sep 13 '11 at 22:33
@dario: no, that's just removing basic duplicate lines. This is removing lines with duplicate tokens... very different. — Igby Largeman, Sep 13 '11 at 22:34
See: http://stackoverflow.com/questions/1300088/distinct-with-lambda for an idea, specifically Jon Skeet's answer (DistinctBy) — Anthony Pegram, Sep 13 '11 at 22:37

Jimmy · Accepted Answer · 2011-09-13T22:50:59.233

Do you just use the 3rd column as the grouping condition?

var noDuplicatesList = lines
    .GroupBy(l => l.PartNumber)
    .Select(group => group.First())

foreach(var item in noDuplicatesList)
    Console.WriteLine("{0} {1} {2} {3}", 
        item.Name,
        item.PartDescription,
        item.PartNumber,
        item.Rotation
    );

lines:

C12 0402 123456 90
C133 0402 123456 90
C9 0402 123456 90
C132 0603 abcd 0
C54 0603 abcd 0
R FID1 fiducial 0
R FID2 fiducial 0
R FID3 fiducial 0
R FID4 fiducial 0

lines.GroupBy(l => l.PartNumber):

{ C12 0402 123456 90, C133 0402 123456 90, C9 0402 123456 90}
{ C132 0603 abcd 0, C54 0603 abcd 0 }
{ R FID1 fiducial 0, R FID2 fiducial 0, R FID3 fiducial 0, R FID4 fiducial 0 }

lines.GroupBy(l => l.PartNumber).Select(group => group.First()):

C12 0402 123456 90
C132 0603 abcd 0
R FID1 fiducial 0

okay, just editted the code to make it a bit more explanatory. — Jimmy, Sep 13 '11 at 22:51
Wow that's great! Thank you for the breakdown of what it looks like too. I never knew what the `GroupBy` looked like :) — theNoobGuy, Sep 13 '11 at 23:02

score 0 · Answer 2 · answered Sep 13 '11 at 22:42

(On reflection, this answer is assuming that order counts in your list. If order doesn't matter, then just use GroupBy as cited in the answer below.)

The typical "modular" way to accomplish this would be to write a PartitionBy function that works something like this:

private static IEnumerable<ICollection<T>> PartitionBy<T, U>(
    IEnumerable<T> sequence, Func<T, U> selector)
{
    var buffer = new List<T>();

    using (var e = sequence.GetEnumerator())
    {
        if (!e.MoveNext())
            yield break;

        var priorValue = selector(e.Current);
        buffer.Add(e.Current);

        while (e.MoveNext())
        {
            var newValue = selector(e.Current);

            if (!object.Equals(selector(e.Current), priorValue))
            {
                priorValue = newValue;
                yield return buffer;
                buffer = new List<T>();
            }

            buffer.Add(e.Current);
        }
    }

    if (buffer.Any())
        yield return buffer;
}

Then your problem can be solved like this:

var results = PartitionBy(objs, x => /* ... */).Select(list => list.First());

where the ... are however you get the third value out of an object in your list (e.g. x.Split()[2].)

Removing lines that contain the same thing in a list

Question

2 Answers2