Create dictionary lookup of headers found in HTTP (< 2.0) string without LINQ

Question

I'm currently parsing some HTTP request headers from a log file, I need to split them up and create a dictionary for easier lookups. The code I'm using is:

public static Dictionary<string, string> CreateLookupDictionary(string input)
    {
        Debug.WriteLine(input);
        return input.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries)
            .Select(x => x.Split(new string[] {": "}, StringSplitOptions.None))
            .ToDictionary(x => x[0], x => x[1], StringComparer.InvariantCultureIgnoreCase);
    }

This is working for 99% of the headers, but then...

...
Keep-Alive: timeout=20
Expires: Sat, 04 Jun 2011 18:43:08 GMT
Cache-Control: max-age=31536000
Cache-Control: public
Accept-Ranges: bytes
...

Now the key Cache-Control already exists, so it's throwing an exception about the key already existing.

Is there an elegant way to overwrite the value that's there, I don't want to have to rewrite the LINQ unless I really have to.

Thanks

Answer 1

• .ToDictionary requires each key be unique, by design.
• Linq doesn't have a .DistinctBy( x => x.y ) method, but we can get the same behaviour with .GroupBy( x => x.y ).Select( grp => grp.Last() ). This has the effect of discarding all previous results with the same y value.

So if you group by the HTTP header name first and then select the last item in each group then that will get you what you want:

// Using cached static fields to avoid unnecessary array allocation:
static readonly String[] _splitOnLines = new String[] { "\r\n" };
static readonly String[] _splitHeader  = new String[] { ": " };

public static Dictionary<String,String> CreateLookupDictionary(String input)
{
    Debug.WriteLine(input);
    return input
        .Split( _splitOnLines , StringSplitOptions.RemoveEmptyEntries )
        .Select( line => line.Split( _splitHeader, StringSplitOptions.None ) )
        .Where( arr => arr.Length == 2 ) // filter out invalid lines, if any
        .Select( arr => ( name: arr[0], value: arr[1] ) ) // using C# 7 named tuples for maintainability
        .GroupBy( header => header.name )
        .Select( duplicateHeaderGroup => duplicateHeaderGroup.Last() )
        .ToDictionary( header => header.name, header.value, StringComparer.InvariantCultureIgnoreCase );
}

Alternatively, use a custom aggregation which uses the key-indexed Item setter property which always succeeds. This approach may have faster performance if duplicates are rare compared to my previous example.

public static Dictionary<String,String> CreateLookupDictionary(String input)
{
    Debug.WriteLine(input);
    return input
        .Split( _splitOnLines , StringSplitOptions.RemoveEmptyEntries )
        .Select( line => line.Split( _splitHeader, StringSplitOptions.None ) )
        .Where( arr => arr.Length == 2 )
        .Select( arr => ( name: arr[0], value: arr[1] ) )
        .Aggregate(
            new Dictionary<String,String>( StringComparer.InvariantCultureIgnoreCase ),
            ( d, header ) =>
            {
                d[ header.name ] = header.value;
                return d;
            }
        );
}

Answer 2

.Net Core 2.0 has a solution: see https://stackoverflow.com/a/54075677/887092

Use Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpParser class:

If you use this approach, you will be able to handle many unforeseen edge cases.

Here's the pseudocode (untested) - and this will only work with .Net Core (apparently)

using Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http;
using System;
using System.Buffers;
using System.Collections.Generic;
using System.Text;

public class ExampleUsage 
{
    public static void Main(string[] args)
    {
        string requestString =
        @"POST /resource/?query_id=0 HTTP/1.1
        Host: example.com
        User-Agent: custom
        Accept: */*
        Connection: close
        Content-Length: 20
        Content-Type: application/json

        {""key1"":1, ""key2"":2}";

        var headerResult = Parser.Parse(requestString);
    }
}

public class Parser : IHttpHeadersHandler
{
    private Dictionary<string, string> result = null;

    public Dictionary<string, string> Parse(string requestString)
    {
        result = new Dictionary<string, string>();  

        byte[] requestRaw = Encoding.UTF8.GetBytes(requestString);
        ReadOnlySequence<byte> buffer = new ReadOnlySequence<byte>(requestRaw);
        HttpParser<Program> parser = new HttpParser<Program>();

        parser.ParseRequestLine(this, buffer, out var consumed, out var examined);
        buffer = buffer.Slice(consumed);

        parser.ParseHeaders(this, buffer, out consumed, out examined, out var b);
        buffer = buffer.Slice(consumed);
    }

    public void OnHeader(Span<byte> name, Span<byte> value)
    {
        result.Add(Encoding.UTF8.GetString(name), Encoding.UTF8.GetString(value));
    }
}

Answer 3

Personally I would use .ToLookup instead to preserve the multiple values for the same key. .ToLookup will not error on duplicate keys, it will create an IEnumerable<V> of the value, this case being IEnumerable<string>:

public static ILookup<string, string> CreateLookup(string input)
{
    Debug.WriteLine(input);
    return input.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries)
        .Select(x => x.Split(new string[] {": "}, StringSplitOptions.None))
        .ToLookup(x => x[0], x => x[1], StringComparer.InvariantCultureIgnoreCase);
}