1

How can I apply a filter accent-insensitive? In OData the "eq" operator is case and accent sensitive. The case is easy to fix, because the "tolower" but relative to the accent I'm not getting a simple solution. I know contains is supposed to be accent-insensitive but if I use contains filtering by "São José" I am only getting these responses "São José" and "São José dos Campos", it is missing "Sao Jose".

The following example filtering by "Florianopolis" is expected to return "Florianópolis", but it does not:

url: api/cidades/get?$filter=contains(nome, 'Florianopolis')

[HttpGet] 
[EnableQuery] 
public ActionResult<IQueryable<CidadeDTO>> Get() 
{ 
    try
    {
        return Ok(_mapper.Map<IEnumerable<CidadeDTO>>(_db.Cidades)); 
    } 
    catch (System.Exception e) 
    { 
        return BadRequest(e.GetBaseException().Message);
    }
} 

It should bring aswell, like entity framework.

Chris Schaller
  • 13,704
  • 3
  • 43
  • 81
Tarsis F
  • 51
  • 3
  • Only to mention, "contains" is also "accent-sensitive" – Tarsis F Feb 08 '22 at 13:48
  • What framework or library are you using and what is the backing store? In many SQL Server backed OData APIs this comparison will be determined by the collation set on the database connection itself. Your example is confusing though please list out a small table of records that shows some examples of the values you are not seeing that you are expecting. – Chris Schaller Feb 09 '22 at 15:10
  • Thank you for your reply. I'm using SQL Server and my collation is case insensitive and accent insensitive. If I make a query with entity framewor it works. An example, if I ask to EF to bring me all the cities with name "florianopolis" it will give me the records with "florianópolis", "florianopolis", "FLORIANOPOLIS" and "FLORIANÓPOLIS". But if I make the same query with OData, it only brings "florianopolis". I'm using OData V4 – Tarsis F Feb 09 '22 at 17:34
  • Can you please edit the post to include the endpoint that is executing the query, so show the controller method and the URL that you are sending. (I know it seems obvious but there is something not right here) – Chris Schaller Feb 09 '22 at 22:00
  • There it goes: url -> api/cidades/get?$filter=contains(nome, 'Florianopolis') controler -> [HttpGet] [EnableQuery] public ActionResult> Get() { try { return Ok(_mapper.Map>(_db.Cidades)); } catch (System.Exception e) { return BadRequest(e.GetBaseException().Message); } }. It should bring 'Florianópolis' aswell, like entity framework. – Tarsis F Feb 10 '22 at 18:51
  • This is because you are using _AutoMapper_ and you havbe projected into an IEnumerable, which is usually redundant in OData as the Edm model is itself an ORM, given that EF is an ORM, you have 3 layers of mapping going on that the URL query needs to be translated into. – Chris Schaller Feb 10 '22 at 21:26
  • Thank you very much, you're right it's because AutoMapper it doesn't work the way I need. But now I have another problem, I need to converto the data to my DTO. How can I use AutoMapper in that case? – Tarsis F Feb 10 '22 at 22:17
  • I already found the problem removing the [EnableQuery] and using ODataQueryOptions did the job! Thank you again! – Tarsis F Feb 10 '22 at 22:45
  • Wait, [EnableQuery] is very crucial! You should still use ODataQueryOptions, but it is still an inefficient process. I would call that a hack, not a solution – Chris Schaller Feb 10 '22 at 22:54
  • I recommend to [add an answer](https://stackoverflow.com/help/self-answer) to describe the cause of the problem you found and how you resolved the problem. Then you can also accept your answer after 48 hours (if you believe it is the best answer at that time). – AndrewF Feb 19 '22 at 03:40

2 Answers2

1

If your OData model was mapped directly to EF models AND an IQueryable<T> expression was passed into OK() then the query is explicitly passed through to the database engine as SQL:

SELECT * FROM Cidades WHERE nome LIKE '%Florianopolis%'

When that occurs, the Collation settings in the database connection will determine the comparison matching logic.

When your database collation is case and accent insensitive, but your data is still filtered as if it was not, then this is an indication that an IEnumerable<T> has been passed into OK() and the comparison logic is being evaluated in C# which by default in insensitive to both case and accent. Unfortunately this means that it is very likely that the entire data table has been loaded into memory first so that the filter can be applied.

In your case the OData model is mapped to DTO expressions that are mapped to the EF models via AutoMapper and that is where the generated query can break down. By calling Map() you are loading ALL records from the EF table and leaving the $filter criteria to be applied by the EnableQueryAttribute

For OData query conventions to be applied automatically you must return an IQueryable<T> from your method, or atleast pass an IQueryable<T> into the OK() response handler. With AutoMapper, you can use the Queryable Extensions to satisfy the IQueryable<T> requirement:

Queryable Extensions
When using an ORM such as NHibernate or Entity Framework with AutoMapper’s standard mapper.Map functions, you may notice that the ORM will query all the fields of all the objects within a graph when AutoMapper is attempting to map the results to a destination type.

...

ProjectTo must be the last call in the chain. ORMs work with entities, not DTOs. So apply any filtering and sorting on entities and, as the last step, project to DTOs.

In OData the last requirement (about ProjectTo) is still problematic because the EnableQueryAttribute will append the query options to the IQueryable<T> response, which will still end up materializing the entire table into memory first (IEnumerable<T>) and then apply the filter, which is still incredibly inefficient. It is this behaviour that is generally observed when someone complains about poor performance from an OData implementation, it is not always AutoMapper, but usually the pattern that the data source is loaded into memory in its entirety and then filtered. Following the default guidance for AutoMapper will lead you in this direction.

Instead we need to use an additional package: AutoMapper.Extensions.ExpressionMapping that will give us access to the UseAsDataSource extension method.

UseAsDataSource
Mapping expressions to one another is a tedious and produces long ugly code.

UseAsDataSource().For<DTO>() makes this translation clean by not having to explicitly map expressions. It also calls ProjectTo<TDO>() for you as well, where applicable.

This changes your implementation to the following:

[HttpGet] 
[EnableQuery] 
public ActionResult<IQueryable<CidadeDTO>> Get() 
{ 
    return Ok(_db.Cidades.UseAsDataSource().For<CidadeDTO>()); 
} 

Don't fall into the trap of assuming that AutoMapper is necessary or best practice for an OData API implementation. If you are not using the unique features that AutoMapper provides then adding an additional abstraction layer can end up over-complicating your solution.

I'm not against AutoMapper, I use it a lot for Integrations, ETL, GraphQL and non-DDD style data schemas where the DTO models are significantly different to the underlying EF/data storage models. But it is a maintenance and performance overhead that a simple DDD data model and OData API based solution can easily do without.

  • Don't hire an excavator when a simple shovel will do the job.

AutoMapper is a convention based ORM that can be useful when you want to change the structure between implementation layers in your code, traditionally you might map Business Domain models that may represent aggregates or have flattened structures to highly normalised Database models.

OData is also a convention based ORM. It was designed to facilitate many of the same operations that AutoAmpper provides with the exception of Flattening and Unflattening models. These operations are deferred to the EF engine. The types exposed via OData mapping are DTOs

If your DTO models are the same relational structure as your EF models, then you would generally not use AutoMapper at all, the OData Edm mapping is optimised specifically to manage this type of workload and is designed to be and has been integrated directly into the serialization layer, making the Edm truely Data Transfer Objects that only exist over the wire and in the client.

Chris Schaller
  • 13,704
  • 3
  • 43
  • 81
  • Thank you for you answer, but visual studio complains about missing parameters in the call "return Ok(_db.Cidades.UseAsDataSource().For());". For now I'm using that: [HttpGet] public ActionResult> Get(ODataQueryOptions opts) { var queryResult = opts.ApplyTo(_db.Cidades); return Ok(_mapper.Map>(queryResult)); }. So I apply the filter before the return. Do you think that way it's still loading the entire table? – Tarsis F Feb 12 '22 at 13:17
  • Using `options.ApplyTo(_db.Pessoas);` should filter the data correctly, but I doubt that it would map to your DTO correctly, unless your DTO matched the DB implementation, at which point I start to question why you are using automapper at all. As for the missing parameters you will need to consult the automapper docs for instructions, I provided the links to the official docs but I do not know what version you are using. – Chris Schaller Feb 20 '22 at 13:52
0

This did the job

[HttpGet]
public ActionResult<IQueryable<PessoaDTO>> Get(ODataQueryOptions<Pessoa> options)
    {
        try
        {
            var queryResult = options.ApplyTo(_db.Pessoas);
            return Ok(queryResult);
        }
        catch (System.Exception e)
        {
            return BadRequest(e.GetBaseException().Message);
        }

    }
Tarsis F
  • 51
  • 3