1

I have a case where I need to send tens of thousands of ids to the graphql server in the filtering query.

The query now generated by the HT is something like this:

_dbContext.
  Forms
  .Where(c=>staticLiistOfIds.Contains(c.Id))
  .Select(c=>new {C.Name,C.Age});

I have two problems with this:

  1. slow performance
  2. SQL Server Limit I guess is around 32K

I have found a Nuget library to convert this static list to a temp table,so now I want to override the HT middle to rewrite the above query generated to the following:

_dbContext.
  Forms
  .Where(c=>_dbContext..AsQueryableValues(staticLiistOfIds).Contains(c.Id))
  .Select(c=>new {C.Name,C.Age});

This will create a temp table for this static list of ids so I will be able to solve the above two problems that I have.

Joundill
  • 6,828
  • 12
  • 36
  • 50
Mahamad Husen
  • 229
  • 1
  • 13

1 Answers1

0

So since i didn't get answers, I had to ask from the Slack of HotChocolate's Team and hopefully, they provided me with the documentation extending-filtering/extending-iqueryable:

in case the link was broken, here is

Extending IQueryable The default filtering implementation uses IQueryable under the hood. You can customize the translation of queries by registering handlers on the QueryableFilterProvider.

The following example creates a StringOperationHandler that supports case-insensitive filtering:

// The QueryableStringOperationHandler already has an implemenation of CanHandle
// It checks if the field is declared in a string operation type and also checks if
// the operation of this field uses the `Operation` specified in the override property further
// below
public class QueryableStringInvariantEqualsHandler : QueryableStringOperationHandler
{
    // For creating a expression tree we need the `MethodInfo` of the `ToLower` method of string
    private static readonly MethodInfo _toLower = typeof(string)
        .GetMethods()
        .Single(
            x => x.Name == nameof(string.ToLower) &&
            x.GetParameters().Length == 0);

    // This is used to match the handler to all `eq` fields
    protected override int Operation => DefaultFilterOperations.Equals;

    public override Expression HandleOperation(
        QueryableFilterContext context,
        IFilterOperationField field,
        IValueNode value,
        object parsedValue)
    {
        // We get the instance of the context. This is the expression path to the propert
        // e.g. ~> y.Street
        Expression property = context.GetInstance();

        // the parsed value is what was specified in the query
        // e.g. ~> eq: "221B Baker Street"
        if (parsedValue is string str)
        {
            // Creates and returnes the operation
            // e.g. ~> y.Street.ToLower() == "221b baker street"
            return Expression.Equal(
                Expression.Call(property, _toLower),
                Expression.Constant(str.ToLower()));
        }

        // Something went wrong 
        throw new InvalidOperationException();
    }
}

This operation handler can be registered on the convention:

public class CustomFilteringConvention : FilterConvention
{
    protected override void Configure(IFilterConventionDescriptor descriptor)
    {
        descriptor.AddDefaults();
        descriptor.Provider(
            new QueryableFilterProvider(
                x => x
                    .AddDefaultFieldHandlers()
                    .AddFieldHandler<QueryableStringInvariantEqualsHandler>()));
    }
}

// and then
services.AddGraphQLServer()
    .AddFiltering<CustomFilteringConvention>();

To make this registration easier, Hot Chocolate also supports convention and provider extensions. Instead of creating a custom FilterConvention, you can also do the following:

services
    .AddGraphQLServer()
    .AddFiltering()
    .AddConvention<IFilterConvention>(
        new FilterConventionExtension(
            x => x.AddProviderExtension(
                new QueryableFilterProviderExtension(
                    y => y.AddFieldHandler<QueryableStringInvariantEqualsHandler>()))));

but I was suggested that doing this way(sending up to 100k list of string ids to the graphQL server) is not a good approach. so I decided to take another approach by writing a custom simple dynamic LINQ generates.

Thanks.

Mahamad Husen
  • 229
  • 1
  • 13
  • There was no answer because the question is problematic. `slow performance` means little without details like indexes and the execution plan. There's no other way to run such a query with EF Core to begin with. The very idea of filtering by 1000 client-side rows is wrong, never mind 100K. Such a long list can only come from either another query or a data source. In both cases it's a *lot* easier to write a plain old SQL query that JOINs the query that produces IDs with the query you want to filter – Panagiotis Kanavos Dec 19 '22 at 08:21
  • I downvoted because the link is already broken. Link-only queries are bad for precisely this reason - if the link gets broken, the answer is meaningless. At the very least add a summary of what that link points to – Panagiotis Kanavos Dec 19 '22 at 08:23
  • Thanks for the reply, well by `slow performance` I was referring to statically sending the list of a string id (thousands of items) through EF will be translated to a `select in ('','', etc)` , so I was trying to insert this list to a temp table then do the select in from this table using EF LINQ. – Mahamad Husen Dec 19 '22 at 08:36
  • @panagiotis-kanavos i can't use plain SQL query, since the data structure I have has hundreds of fields from different tables, and the clients want to dynamically query whatever fields she/he wants with the filtering by those long list. – Mahamad Husen Dec 19 '22 at 08:41
  • Yes you can use SQL. Because you already do - there's no other way to query a database. LINQ is translated to SQL, not executed by itself. If you want to filter arbitrary tables by 100K IDs something very weird is going on with the application design. – Panagiotis Kanavos Dec 19 '22 at 08:44
  • `IN` is as fast if not faster because SQL Server can calculate statistics for this statement. A temporary table will only be faster if it's indexed. In any case, this kind of querying simply isn't suitable for GraphQL, OData or even ORMs like EF Core or NHibernate. If you have so much data you should *import* it into a staging table with proper indexes, JOIN with the other reporting query and do whatever you want with the results. To import so much data you should use something like SqlBulkCopy. – Panagiotis Kanavos Dec 19 '22 at 08:45
  • If you want to import 100K rows you probably shouldn't use a temporary table but a staging table stored in a location you can control, with the indexes you want. 100K isn't a lot of data for any table but temporary tables are created in `tempdb` which is used for a *lot* of jobs in SQL Server. – Panagiotis Kanavos Dec 19 '22 at 08:47
  • @panagiotis-kanavos the link is not broken, I just double-checked. anyways, I also added a snapshot for the doc link in case it got broken in the future. – Mahamad Husen Dec 19 '22 at 08:51
  • @panagiotis-kanavos filter arbitrary columns from by 100K IDs is weird I know, but the requirement for the system is like this, it's not something I want, the nature of the project I work on demand this, where the users want to generate a report based on up to 100K ids from arbitrary fields. i hope you get my point. – Mahamad Husen Dec 19 '22 at 08:56
  • @panagiotis-kanavos I have summarized my use case as to why I do it in the chat discussion. I appreciate it if you could improve this answer and I won't lift the ban I got(which I don't think is fair because the questions are somehow not well-formatted and not _extremely_ – Mahamad Husen Dec 19 '22 at 09:12