use linq to aggregate text file contents so that they are grouped

Question

Given the following query.

var query = files
            .SelectMany(file => File.ReadAllLines(file))
            .Where(_ => !_.StartsWith("*"))
            .Select(line => new {
                Order = line.Substring(32, 7),
                Delta = line.Substring(40, 3),
                Line = new String[] { line }
            });

This clearly produces a list of objects with the properties Order: string, Delta: string and Line: string[]

I have a list of items that looks like this.

{ 1, 'A', {'line1'} }, 
{ 1, 'A', {'line2'} }, 
{ 2, 'B', {'line3'} }, 
{ 1, 'B', {'line4 } }

is it possible to use the Linq Aggregate or similar functional construct to collect all the adjacent Order and Delta combinations together whilst accumulating the lines.

So that the aggregate is a list of items containing all it's 'lines'

{ 1, 'A', {'line1', 'line2'} }
{ 2, 'B', {'line3'} }
{ 1, 'B', {'line4'} }

Since aggregation iterates sequentially it should be possible to collect all the adjacent lines that have the same fields equal.

It's easy to do in a loop, but I am trying to do it with a set of lambdas.

I think you could use `Enumerable.Aggregate` to do this. I'd have to play around with it to get it working though. If you don't get an answer by the time I get back from lunch, I'll see what I can come up with. If this has to pass through Entity Framework, it's probably a no-go though. Are you using EF? — Bradley Uffner, Feb 26 '19 at 15:34
`Aggregate` works by taking the current element and the accumulator, and creating a new accumulator. This isn't what you're doing, so I don't think it's a good match (although you could make it work, but in an ugly and hard-to-read way). What you want is a linq method which yields consecutive "equal" elements, where equality is determined by a delegate. There's nothing built in which does this, although you could write your own (which is just generalising the loop that's easy for you to write). Alternatively, it looks like MoreLinq's GroupAdjacent does this — canton7, Feb 26 '19 at 15:38
Sorry, the example I linked to earlier was for grouping by *consecutive* numbers, not *adjacent*. Deleted to avoid confusion. — Bradley Uffner, Feb 26 '19 at 15:44
Maybe this can help: https://stackoverflow.com/a/14879567/4499267 — Phate01, Feb 26 '19 at 15:50

Chris Pickford · Answer 1 · 2019-02-26T16:00:22.603

1

Note: Does not group items by adjacency

You can produce the desired results using a simple GroupBy combined with a SelectMany:

var query = new[] {
  new { order = 1, delta = "A", line = new[] { "line1" } },
  new { order = 1, delta = "A", line = new[] { "line2" } },
  new { order = 2, delta = "B", line = new[] { "line3" } },
  new { order = 1, delta = "B", line = new[] { "line4" } },
};

query
  .GroupBy(q => new { q.order, q.delta })
  .Select(q => new {
    order = q.Key.order,
    delta = q.Key.delta,
    lines = q.SelectMany(l => l.line)
});

Produces:

edited Feb 26 '19 at 16:00

answered Feb 26 '19 at 15:46

Chris Pickford

8,642
5
42
73

1

I don't think this is right - he wanted *adjacent* groupings. Add a new element with `order = 1, delta = "A"` at the end. – canton7 Feb 26 '19 at 15:47
Hmm, not exactly sure what this adjacent requirement means. Will remove answer if OP confirms this doesn't address his issue. – Chris Pickford Feb 26 '19 at 15:53
My understanding, and I think this is pretty clear from the question, is that if two *consecutive* elements have the same order and delta, then they should be grouped. If there are elements with the same order and delta which are not consecutive, then they should not be grouped. – canton7 Feb 26 '19 at 15:54
@Chris meaning only group elements next to eachother that match on order and delta, not all orders and deltas in the collection – Daniel Feb 26 '19 at 15:54

Vlad · Accepted Answer · 2019-02-27T10:57:55.200

You'll need the following variation of GroupBy:

public static class EnumerableExtensions
{
    public class AdjacentGrouping<K, T> : List<T>, IGrouping<K, T>
    {
        public AdjacentGrouping(K key) { Key = key; }
        public K Key { get; private set; }
    }

    public static IEnumerable<IGrouping<K, T>> GroupByAdjacent<T, K>(
                            this IEnumerable<T> sequence, Func<T, K> keySelector)
    {
        using (var it = sequence.GetEnumerator())
        {
            if (!it.MoveNext())
                yield break;
            T curr = it.Current;
            K currKey = keySelector(curr);
            var currentCluster = new AdjacentGrouping<K, T>(currKey) { curr };
            while (it.MoveNext())
            {
                curr = it.Current;
                currKey = keySelector(curr);
                if (!EqualityComparer<K>.Default.Equals(currKey, currentCluster.Key))
                {
                    // start a new cluster
                    yield return currentCluster;
                    currentCluster = new AdjacentGrouping<K, T>(currKey);
                }
                currentCluster.Add(curr);
            };
            // currentCluster is never empty
            yield return currentCluster;
        }
    }
}

Having this adjacent grouping, your code can be the same as in Chris's answer:

var query = files
    .SelectMany(file => File.ReadAllLines(file))
    .Where(_ => !_.StartsWith("*"))
    .Select(line => new
    {
        Order = line.Substring(32, 7),
        Delta = line.Substring(40, 3),
        Line = new String[] { line }
    })
    .GroupByAdjacent(o => new { o.Order, o.Delta })
    .Select(g => new { g.Key.Order, g.Key.Delta, Lines = g.Select(o => o.Line).ToList() });

Disclaimer: the function GroupByAdjacent is from my own pet project and not copied from anywhere.

Thanks @vlad I was hoping there would be a simpler way... but this is great thanks for sharing — Jim, Feb 27 '19 at 07:01
@Jim: I was hoping it too some time ago, but after some fruitless research found out that it's actually simpler to impelment it myself. — Vlad, Feb 27 '19 at 10:56

use linq to aggregate text file contents so that they are grouped

2 Answers2