I have a problem with sorting namely, sorting work but only for price field. When I try to sort by start_date, end_date, uid, cat title get the message about exceeding the limit:
Data too large, the date for [ "name of field here"] would be larger than the limit of [19798897459 / 18.4gb]]
I do not know why this is happening code looks correct sample query for elastica looks like this:
Mapping:
"auctions": {
"_all": { "enabled": false },
"properties": {
"cat": { "store": true, "type": "long" },
"curr": { "index": "not_analyzed", "store": true, "type": "string" },
"end_date": { "store": true, "type": "long" },
"price": { "store": true, "type": "long" },
"start_date": { "store": true, "type": "long" },
"tcat": { "store": true, "type": "long" },
"title": { "store": true, "type": "string" },
"uid": { "store": true, "type": "long" }
}
},
Request:
/search?uids=335,547&title=Karta&orderBy=uid&orderDir=asc
Method:
private NativeSearchQueryBuilder getSearchQuery(AuctionIndexSearchParams searchParams, Pageable pageable) {
final List<FilterBuilder> filters = Lists.newArrayList();
final NativeSearchQueryBuilder searchQuery = new NativeSearchQueryBuilder();
Optional.ofNullable(searchParams.getCategoryId()).ifPresent(v -> filters.add(boolFilter().must(termFilter("cat", v))));
Optional.ofNullable(searchParams.getCurrency()).ifPresent(v -> filters.add(boolFilter().must(termFilter("curr", v))));
Optional.ofNullable(searchParams.getTreeCategoryId()).ifPresent(v -> filters.add(boolFilter().must(termFilter("tcat", v))));
Optional.ofNullable(searchParams.getUid()).ifPresent(v -> filters.add(boolFilter().must(termFilter("uid", v))));
final BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder();
//access for many uids
if (searchParams.getUids() != null) {
if (searchParams.getItemId() != null || searchParams.getTitle() != null) {
Optional.ofNullable(searchParams.getUids().split(",")).ifPresent(v -> {
filters.add(boolFilter().must(termsFilter("uid", v)));
});
} else {
for (String user : searchParams.getUids().split(",")) {
boolQueryBuilder.should(queryStringQuery(user).field("uid"));
}
}
}
//access for many categories
if (searchParams.getCategories() != null) {
Optional.ofNullable(searchParams.getCategories().split(",")).ifPresent(v -> {
filters.add(boolFilter().must(termsFilter("cat", v)));
});
}
if (searchParams.getItemId() != null) {
boolQueryBuilder.must(queryStringQuery(searchParams.getItemId()).field("_id"));
}
if (Optional.ofNullable(searchParams.getTitle()).isPresent()) {
boolQueryBuilder.must(queryStringQuery(searchParams.getTitle()).analyzeWildcard(true).field("title"));
}
if (Optional.ofNullable(searchParams.getStartDateFrom()).isPresent()
|| Optional.ofNullable(searchParams.getStartDateTo()).isPresent()) {
filters.add(rangeFilter("start_date").from(searchParams.getStartDateFrom()).to(searchParams.getStartDateTo()));
}
if (Optional.ofNullable(searchParams.getEndDateFrom()).isPresent()
|| Optional.ofNullable(searchParams.getEndDateTo()).isPresent()) {
filters.add(rangeFilter("end_date").from(searchParams.getEndDateFrom()).to(searchParams.getEndDateTo()));
}
if (Optional.ofNullable(searchParams.getPriceFrom()).isPresent()
|| Optional.ofNullable(searchParams.getPriceTo()).isPresent()) {
filters.add(rangeFilter("price").from(searchParams.getPriceFrom()).to(searchParams.getPriceTo()));
}
searchQuery.withQuery(boolQueryBuilder);
FilterBuilder[] filterArr = new FilterBuilder[filters.size()];
filterArr = filters.toArray(filterArr);
searchQuery.withFilter(andFilter(filterArr));
if (searchParams.getOrderBy() != null && searchParams.getOrderDir() != null) {
if (searchParams.getOrderDir().toLowerCase().equals("asc")) {
searchQuery.withSort(SortBuilders.fieldSort(searchParams.getOrderBy()).order(SortOrder.ASC));
} else {
searchQuery.withSort(SortBuilders.fieldSort(searchParams.getOrderBy()).order(SortOrder.DESC));
}
}
if (pageable != null) {
searchQuery.withPageable(pageable);
}
System.out.println(searchQuery.build().getQuery());
System.out.println(searchQuery.build().getFilter());
System.out.println(searchQuery.build().getSort());
return searchQuery;
}
System.out.println(searchQuery.build().getQuery());
{
"bool": {
"must": {
"query_string": {
"query", "card"
"fields": [ "title"]
"analyze_wildcard": true
}
}
}
}
System.out.println (searchQuery.build().getFilter());
{
"and" {
"filters": [{
"bool": {
"must": {
"terms": {
"uid" [ "335", "547"]
}
}
}
}]
}
}
System.out.println(searchQuery.build().getSort());
null
Any ideas what might cause this exception?
I should add that I've tried these solutions:
But the effect was even worse, then no query did not work as quickly.
For any help I will be extremely grateful!
/_stats/fielddata?fields=*
{
"_shards": {
"total": 10,
"successful": 5,
"failed": 0
},
"_all": {
"primaries": {
"fielddata": {
"memory_size_in_bytes": 19466671904,
"evictions": 0,
"fields": {
"_id": {
"memory_size_in_bytes": 0
},
"cat": {
"memory_size_in_bytes": 0
},
"price": {
"memory_size_in_bytes": 3235221240
},
"title": {
"memory_size_in_bytes": 16231450664
}
}
}
},
"total": {
"fielddata": {
"memory_size_in_bytes": 19466671904,
"evictions": 0,
"fields": {
"_id": {
"memory_size_in_bytes": 0
},
"cat": {
"memory_size_in_bytes": 0
},
"price": {
"memory_size_in_bytes": 3235221240
},
"title": {
"memory_size_in_bytes": 16231450664
}
}
}
}
},
"indices": {
"allek": {
"primaries": {
"fielddata": {
"memory_size_in_bytes": 19466671904,
"evictions": 0,
"fields": {
"_id": {
"memory_size_in_bytes": 0
},
"cat": {
"memory_size_in_bytes": 0
},
"price": {
"memory_size_in_bytes": 3235221240
},
"title": {
"memory_size_in_bytes": 16231450664
}
}
}
},
"total": {
"fielddata": {
"memory_size_in_bytes": 19466671904,
"evictions": 0,
"fields": {
"_id": {
"memory_size_in_bytes": 0
},
"cat": {
"memory_size_in_bytes": 0
},
"price": {
"memory_size_in_bytes": 3235221240
},
"title": {
"memory_size_in_bytes": 16231450664
}
}
}
}
}
}
Edit:
I solved the problem as follows:
After discernment, it turned out that I'm using version 1.7. The documentation I found information that doc_values must be set in the mapping to true if you want to sort or aggregate. Fields strings need to add another field multifield.
So after a map change to something more or less like this:
{
"_all": {
"enabled": false
},
"properties": {
"cat": {
"store": true,
"type": "long",
"doc_values": true
},
"curr": {
"index": "not_analyzed",
"store": true,
"type": "string",
"doc_values": true
},
"end_date": {
"store": true,
"type": "long",
"doc_values": true
},
"price": {
"store": true,
"type": "long",
"doc_values": true
},
"start_date": {
"store": true,
"type": "long",
"doc_values": true
},
"tcat": {
"store": true,
"type": "long",
"doc_values": true
},
"title": {
"store": true,
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 256,
"doc_values": true
}
}
},
"uid": {
"store": true,
"type": "long",
"doc_values": true
}
}
}
Sorting work, but slowed down the whole sysytem search, I will say that much, although the documentation is from about 10-20%.
You should also remember to reindex data!
Thanks!
