UDFs or Probabilistic data structures in a roadmap

Question

We're interested here in working with high-cardinality indexes. (Which are known to be a problem for Elastic Search)

We already know from you that for

select count(distinct high_cardinality_field) from my_table

you already have some optimizations to count it. Will it be possible someday to write something like:

select count_via_hyperloglog(high_cardinality_field) from my_table

having count_via_hyperloglog as a UDF or something, as it is possible right now in ES via ES-plugins?

score 4 · Accepted Answer · answered Mar 24 '14 at 10:31

4

in crate this feature is on our backlog as an additional aggregation function which uses the hyperlog algorithm. we plan to do the naming derived from presto http://prestodb.io/docs/current/functions/aggregate.html. Your example will then probably look like:

select approx_distinct(high_cardinality_field) from my_table

However, a possible performance improvement for one specific field per table is to cluster your table based on the high cardinality field as described under https://crate.io/docs/current/sql/ddl.html#routing

answered Mar 24 '14 at 10:31

dobe

105
3

Thank you! What is a time feeling for this feature to be supported? – Vlad Vlaskin Mar 24 '14 at 14:18
Also generally, would it be possible to write UDFs? – Vlad Vlaskin Mar 25 '14 at 10:37
huh, it is hard to estimate when this feature will make it into a sprint, probably in a month probably – dobe Mar 26 '14 at 09:55
Currently we have no plugin interface. However contributions are welcome :) take a look at io.crate.operation.aggregation.impl.CountAggregation as an example. – dobe Mar 26 '14 at 09:58

knutwalker · Answer 2 · 2014-03-24T07:19:44.963

High cardinality counting with HyperLogLog is planned for 1.1.0, the documentation is already up: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html

Example:

{
    "aggs" : {
        "author_count" : {
            "cardinality" : {
                "field" : "author"
            }
        }
    }
}

As for something like UDF, you can use scripts, .e.g. by combining a filter aggregation with a script filter

{
    "aggs": {
        "in_stock_products": {
            "filter": {
                "script": {
                    "script": "doc['price'].value > minPrice"
                    "params": {
                        "minPrice": 5
                    }
                }
            },
            "aggs": {
                "avg_price": {
                    "avg": {
                        "field": "price"
                    }
                }
            }
        }
    }
}

Thanks that makes total sense. What is an estimated release time for ES 1.1? — Vlad Vlaskin, Mar 24 '14 at 14:18

UDFs or Probabilistic data structures in a roadmap

2 Answers2