Group and Select First From Two Tables Linq

Question

I'm trying to create a simple query using EFCore, returning the list of people i'm conversing with, and the last message that was sent between the two of us (pretty much like how it's displayed on Facebook Messenger or Whatsapp). I created the linq query but its generating one hell of an sql query. I'm trying to optimize the linq query to generate a better sql, so here comes the full story:

0. The Two Entities: Visitors and ChatMessages

The Visitor contains the visitor information, and the ChatMessages contains the actual chat.

1. First Try

I tried the first query as follows:

from c in ChatMessages
orderby c.CreatedAt descending 
group c by c.VisitorId  into x
select x.First()

Which got me the list of latest messages grouped by the visitor id:

which is cool, specially with the short sql query generated:

SELECT [t3].[test], [t3].[Id], [t3].[Message], [t3].[UserId], [t3].[VisitorId], [t3].[isDeleted] AS [IsDeleted], [t3].[CreatedAt], [t3].[CreatedBy], [t3].[LastUpdatedAt], [t3].[LastUpdatedBy], [t3].[isFromVisitor] AS [IsFromVisitor]
FROM (
    SELECT [t0].[VisitorId]
    FROM [ChatMessages] AS [t0]
    GROUP BY [t0].[VisitorId]
    ) AS [t1]
OUTER APPLY (
    SELECT TOP (1) 1 AS [test], [t2].[Id], [t2].[Message], [t2].[UserId], [t2].[VisitorId], [t2].[isDeleted], [t2].[CreatedAt], [t2].[CreatedBy], [t2].[LastUpdatedAt], [t2].[LastUpdatedBy], [t2].[isFromVisitor]
    FROM [ChatMessages] AS [t2]
    WHERE (([t1].[VisitorId] IS NULL) AND ([t2].[VisitorId] IS NULL)) OR (([t1].[VisitorId] IS NOT NULL) AND ([t2].[VisitorId] IS NOT NULL) AND ([t1].[VisitorId] = [t2].[VisitorId]))
    ORDER BY [t2].[CreatedAt] DESC
    ) AS [t3]
ORDER BY [t3].[CreatedAt] DESC

2. Second Try, Joining the Visitor table as well

Now I want to return the visitor information as well, so I have to join the visitors table:

from c in ChatMessages
join v in Visitors on  c.VisitorId equals v.Id 
orderby c.CreatedAt descending 
group new {Message = c, Visitor = v} by c.Visitor.Id  into x
select x

Which generated what I want:

Problem is, the generate SQL query got very messy:

SELECT [t2].[Id] AS [Key]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
GROUP BY [t2].[Id]
GO

-- Region Parameters
DECLARE @x1 BigInt = 1
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 2
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 3
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 4
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 5
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 6
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 7
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 8
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 9
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 10
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 11
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 12
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 13
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 14
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 15
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 16
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 17
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 18
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 19
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC
GO

-- Region Parameters
DECLARE @x1 BigInt = 20
-- EndRegion
SELECT [t0].[Id], [t0].[Message], [t0].[UserId], [t0].[VisitorId], [t0].[isDeleted] AS [IsDeleted], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[isFromVisitor] AS [IsFromVisitor], [t1].[Id] AS [Id2], [t1].[Email], [t1].[Name], [t1].[Phone], [t1].[isDeleted] AS [IsDeleted2], [t1].[CreatedAt] AS [CreatedAt2], [t1].[CreatedBy] AS [CreatedBy2], [t1].[LastUpdatedAt] AS [LastUpdatedAt2], [t1].[LastUpdatedBy] AS [LastUpdatedBy2], [t1].[Fingerprint], [t1].[IP]
FROM [ChatMessages] AS [t0]
INNER JOIN [Visitors] AS [t1] ON [t0].[VisitorId] = ([t1].[Id])
LEFT OUTER JOIN [Visitors] AS [t2] ON [t2].[Id] = [t0].[VisitorId]
WHERE @x1 = [t2].[Id]
ORDER BY [t0].[CreatedAt] DESC

Which does not seem like a query i would want to fire at the database. Moreover, when executing this code inside the asp.net core app, im getting an exception EF.Property called with wrong property name., not sure why:

crit: converse_app.Controllers.VisitorsController[0]
      There was an error on 'GetVisitorsAsync' invocation: System.InvalidOperationException: EF.Property called with wrong property name.
         at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
         at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.VisitBinary(BinaryExpression binaryExpression)
         at Microsoft.EntityFrameworkCore.SqlServer.Query.Internal.SqlServerSqlTranslatingExpressionVisitor.VisitBinary(BinaryExpression binaryExpression)
         at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.Translate(Expression expression)
         at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.WeakEntityExpandingExpressionVisitor.Expand(Expression source, MemberIdentity member)
         at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.WeakEntityExpandingExpressionVisitor.VisitMember(MemberExpression memberExpression)
         at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.WeakEntityExpandingExpressionVisitor.VisitMember(MemberExpression memberExpression)
         at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.WeakEntityExpandingExpressionVisitor.Expand(SelectExpression selectExpression, Expression lambdaBody)
         at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.RemapLambdaBody(ShapedQueryExpression shapedQueryExpression, LambdaExpression lambdaExpression)
         at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateGroupBy(ShapedQueryExpression source, LambdaExpression keySelector, LambdaExpression elementSelector, LambdaExpression resultSelector)
         at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
         at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
         at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
         at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
         at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
         at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
         at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
         at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass9_0`1.<Execute>b__0()
         at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
         at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
         at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression)
         at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetEnumerator()
         at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
         at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
         at converse_app.Controllers.VisitorsController.GetVisitorsAsync(Int32 pageSize, Int32 pageNumber

Sorry for the long for the post, so my question is how can i optimize the linq query for a better sql output, as well as the reason this error might be firing.

I'm using .NET Core 3 preview8 with EF Core 3 preview8, and running against MSSQL.

Answer 1

The query you are looking for standardly is expressed in LINQ to Entities (EF) with something like this (no joins, no GroupBy, use navigation properties):

var query = context.Visitors
    .Select(v => new
    {
        Visitor = v,
        Message = v.VisitorChatMessages
            .OrderByDescending(m => m.CreatedAt)
            .FirstOrDefault()
    });

But here is the trap. EF6 creates quite inefficient SQL query, and EF Core until now produces (again quite inefficient) N + 1 SQL queries.

But this is changing in EF Core 3.0 in a positive direction! Usually (and still) I don't recommend using the preview (beta) versions of EF Core 3.0, because they are rewriting the whole query translation/processing pipeline, so many things don't work as expected.

But today I've updated my EF Core test environment to EF Core 3.0 Preview 9 and I'm pleased to see that the above query now nicely translates to the following single SQL query:

  SELECT [v].[Id], [v].[CreatedAt], [v].[CreatedBy], [v].[Email], [v].[Fingerprint], [v].[IP], [v].[IsDeleted], [v].[LastUpdatedAt], [v].[LastUpdatedBy], [v].[Name], [v].[Phone], [t0].[Id], [t0].[CreatedAt], [t0].[CreatedBy], [t0].[IsDeleted], [t0].[IsFromVisitor], [t0].[LastUpdatedAt], [t0].[LastUpdatedBy], [t0].[Message], [t0].[UserId], [t0].[VisitorId]
  FROM [Visitors] AS [v]
  LEFT JOIN (
      SELECT [t].[Id], [t].[CreatedAt], [t].[CreatedBy], [t].[IsDeleted], [t].[IsFromVisitor], [t].[LastUpdatedAt], [t].[LastUpdatedBy], [t].[Message], [t].[UserId], [t].[VisitorId]
      FROM (
          SELECT [c].[Id], [c].[CreatedAt], [c].[CreatedBy], [c].[IsDeleted], [c].[IsFromVisitor], [c].[LastUpdatedAt], [c].[LastUpdatedBy], [c].[Message], [c].[UserId], [c].[VisitorId], ROW_NUMBER() OVER(PARTITION BY [c].[VisitorId] ORDER BY [c].[CreatedAt] DESC) AS [row]
          FROM [ChatMessages] AS [c]
      ) AS [t]
      WHERE [t].[row] <= 1
  ) AS [t0] ON [v].[Id] = [t0].[VisitorId]

Note the beautiful utilization of the ROW_NUMBER() OVER (PARTITION BY ORDER BY) construct. This is the first time EF query translation does that ever. I'm excited. Good job, EF Core team!

---

Update: The exact equivalent of your first query (which btw fails with runtime exception in Preview 9)

from c in context.ChatMessages
orderby c.CreatedAt descending 
group c by c.VisitorId  into x
select x.First()

but with additional information is

from v in context.Visitors
from c in v.VisitorChatMessages
    .OrderByDescending(c => c.CreatedAt)
    .Take(1)
orderby c.CreatedAt descending
select new
{
    Visitor = v,
    Message = c
})

The generated SQL is pretty much the same - just the LEFT OUTER JOIN becomes INNER JOIN and there is additional ORDER BY at the end.

Looks like that to make this work, it's essential to avoid GroupBy and use GroupJoin (which collection navigation property represents in LINQ to Entities queries) or correlated SelectMany to achieve the desired grouping.

Answer 2

You should be able to get your visitor information via association without explicitly trying to join/group on it.

Apologies for switching to Fluent syntax, I really dislike the Linq QL, it always seems so forced to do anything through it... :)

Your original query:

from c in ChatMessages
orderby c.CreatedAt descending 
group c by c.VisitorId  into x
select x.First()

or

var groupedMessages = context.ChatMessages
    .OrderByDescending(c => c.CreatedAt)
    .GroupBy(c => c.VisitorId)
    .First();

To group by Visitor:

var groupedMessages = context.ChatMessages
    .OrderByDescending(c => c.CreatedAt)
    .GroupBy(c => c.Visitor)
    .First();

This will give you a Key as a Visitor entity, with the messages for that visitor. However, this somewhat begs the question, why?

var visitorsWithMessages = context.Visitors.Include(v => v.VisitorChatMessages);

This loads the visitors and eager-loads their associated chat messages. Entities satisfy queries against the data relationships. For consumption though we care about details like ensuring the chat messages are ordered, or possibly filtered.

To project that into a suitable structure I'd use view models for the visitor and chat message to optimize the query to cover just the details I care about, and present them in the way I care about:

var visitorsWithMessages = context.Visitors
   // insert .Where() clause here to filter which Visitors we care to retrieve...
   .Select(v => new VisitorViewModel
   {
      Id = v.Id,
      Name = v.Name,
      RecentChatMessages = v.VisitorChatMessages
         .OrderByDescending(c => c.CreatedAt)
         .Select(c => new ChatMessageViewModel
         {
            Id = c.Id,
            Message = c.Message,
            CreatedAt = c.CreatedAt,
            CreatedBy = c.User.UserName ?? "Anonymous"
         }).Take(10).ToList()
    }).ToList();

This uses projection and EF's mapped relationships to get a list of Visitors and up to 10 of their most recent chat messages. I populate simple POCO view model classes which contain the fields I care about. These view models can be safely returned from methods or serialized to a view/API consumer without risking tripping up lazy loading. If I just need the data and don't need to send it anywhere, I can use anonymous types to get the fields I care about. We don't need to explicitly join entities together, you only need to do that for entities that deliberately do not have FK relationships mapped. We also do not need to deliberately eager-load entities either. The Select will compose an optimized SQL statement for the columns we need.

You can consume those results in the view and render based on Visitor + Messages or even dive deeper if you want to display a list of each visitors 10 most recent messages /w visitor details as a flattened list of messages:

Edit: The below query may have had an issue by accessing "v." after the .SelectMany. Corrected to "c.Visitor."

var recentMessages = context.Visitors
   .SelectMany(v => v.VisitorChatMessages
      .OrderByDescending(c => c.CreatedAt)
      .Select(c => new VisitorChatMessageViewModel
      {
          Id = c.Id,
          VisitorId = c.Visitor.Id,
          Message = c.Message,
          CreatedAt = c.CreatedAt,
          CreatedBy = c.User.UserName ?? "Anonymous",
          Visitor = c.Visitor.Name
      }).Take(10)
    }).ToList();

No idea how to do that in Linq QL though. :)

Edit: That last example will give you the last 10 messages with their applicable Visitor detail. (Name)

To get the last 100 messages for example and group them by their visitor:

var recentMessages = context.Visitors
   .SelectMany(v => v.VisitorChatMessages
      .OrderByDescending(c => c.CreatedAt)
      .Select(c => new VisitorChatMessageViewModel
      {
          Id = c.Id,
          VisitorId = c.Visitor.Id,
          Message = c.Message,
          CreatedAt = c.CreatedAt,
          CreatedBy = c.User.UserName ?? "Anonymous",
          Visitor = c.Visitor.Name // Visitor Name, or could be a ViewModel for more info about Visitor...
      }).Take(100)
    }).GroupBy(x => x.Visitor).ToList();

If you select a VisitorViewModel for Visitor instead of ".Visitor.Name" then your grouping key will have access to Name, Id, etc. whatever you select from the associated Visitor.