Attribute based index hints change my results

Question

I have this query that hasn't changed since I first got it working:

ISearchResponse<Series> response = await IndexManager.GetClient()
    .SearchAsync<Series>(r => r
        .Filter(f => f.Term<Role>(t => t.ReleasableTo.First(), Role.Visitor))
        .SortDescending(ser => ser.EndDate)
        .Size(1));

My IndexManager.GetClient() is simply responsible for setting up my connection to ElasticSearch, and ensuring that the indexes are built properly. The rest of the code gets the most recent article series that is releasable to the general public.

Inside the IndexManager I set up explicit index mapping, and when I did that I got results from my query every time. The code looked like this:

client.Map<Series>(m => m.Dynamic(DynamicMappingOption.Allow)
    .DynamicTemplates(t => t
        .Add(a => a.Name("releasableTo").Match("*releasableTo").MatchMappingType("string").Mapping(map => map.String(s => s.Index(FieldIndexOption.NotAnalyzed))))
        .Add(a => a.Name("id").Match("*id").MatchMappingType("string").Mapping(map => map.String(s => s.Index(FieldIndexOption.NotAnalyzed))))
        .Add(a => a.Name("services").Match("*amPm").MatchMappingType("string").Mapping(map => map.String(s => s.Index(FieldIndexOption.NotAnalyzed)))
            .Match("*dayOfWeek").MatchMappingType("string").Mapping(map => map.String(s => s.Index(FieldIndexOption.NotAnalyzed))))
        .Add(a => a.Name("urls").Match("*Url").MatchMappingType("string").Mapping(map => map.String(s => s.Index(FieldIndexOption.NotAnalyzed))))
));

While all well and good, doing this for every type we stored wasn't really going to scale well. So I made a conscious decision to use the attributes and map it that way:

// In IndexManager
client.Map<T>(m => m.MapFromAttributes());

// In the type definition
class Series
{
    // ....

    [DataMember]
    [ElasticProperty(Index = FieldIndexOption.NotAnalyzed, Store = true)]
    public HashSet<Role> ReleasableTo { get; set; }

    // ....
}

As soon as I do this, I no longer get results. When I look at my indexes in Kibana, I see my 'releasableTo' field is not analyzed and it is indexed. However the query I wrote no longer works. If I remove the filter clause I get results, but I really need that to work.

What am I missing? How do I get my query to work again?

I don't even know how to begin fixing this... Anyone have input? — Berin Loritsch, Mar 23 '15 at 23:13
Not knowing the framework you're working with, do you have a way to inspect the generated queries as they're sent to ES, and a way to compare the mappings in each case? — Chris Heald, Mar 25 '15 at 04:02
I'm using the C# NEST API. Same query that used to work no longer does. Not sure how ES handles dynamic mapping vs. attribute based explicit mapping. I've used kibana and the ES API to look at the mappings, but nothing obvious. — Berin Loritsch, Mar 25 '15 at 13:02
You can `curl http://your_host:9200/your_index/_mapping?pretty` to get the full mapping from the index. I'd suggest comparing the mappings in each case to see if there are any discrepancies. Elasticsearch also has "dynamic mapping", where it will guess at a mapping for a field the first time it sees it, if you want to go the super lazy route. — Chris Heald, Mar 25 '15 at 19:04
Something isn't adding up here. I've got a number of attributes where the elastic property index options are respected, but the "releasableTo" index is not. So while that explains why the terms query is failing, it doesn't explain why this particular property is not respected and all the other ones are. I have properties with mixed case that are being set correctly and ones that are bound to a HashSet that are set correctly. Is "releasableTo" a reserved word in ES? — Berin Loritsch, Mar 25 '15 at 23:57
@ChrisHeald, thanks for the reminder. I took a much closer look at the results of that query than I did before. This rather critical field was the only thing not being mapped the way I needed it to. — Berin Loritsch, Mar 26 '15 at 00:33

score 3 · Accepted Answer · answered Mar 26 '15 at 00:31

It appears that the ElasticSearch attributes to provide indexing hints don't know what to do with enums.

The problem turned out to be the fact that the Role type was an enumeration. The client.Map<Series>(m => m.MapFromAttributes()) call skipped that property. At run time, it dynamically maps the property to a string.

// In the type definition
class Series
{
    // ....

    [DataMember]
    [ElasticProperty(Index = FieldIndexOption.NotAnalyzed, Store = true)]
    public HashSet<Role> ReleasableTo { get; set; }

    // ....
}

To get the field properly indexed I had to explicitly set it's type in the ElasticProperty attribute. Changing the code to this:

// In the type definition
class Series
{
    // ....

    [DataMember]
    [ElasticProperty(Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String, Store = true)]
    public HashSet<Role> ReleasableTo { get; set; }

    // ....
}

made my query work again. The moral of the story is that unless it's a primitive type, be explicit when setting the field type.

FYI https://github.com/elastic/elasticsearch-net/issues/901. Speaking with @greg-marzouka (core NEST developer). He is going to reopen the issue. — pickypg, Mar 26 '15 at 01:04
@pickypg, thank you. I weighed in over there as well. Hopefully it will see some attention. If not, I'll re-open it with my counter proposal :) — Berin Loritsch, Mar 26 '15 at 11:32
The fix for the above issue made it into [NEST 1.4.3](https://github.com/elastic/elasticsearch-net/releases/tag/1.4.3). — pickypg, Mar 28 '15 at 03:35

Attribute based index hints change my results

1 Answers1