-1

Let's say I'm ingesting real-time data and want the end users who are viewing the data in real time to be able to group the information in different ways and quickly get views that represent the new groupings.

So if the data was for example, all the transactions on an e-commerce web site, and the user was viewing a live grid(e.g. in a Kibana-like website, with live data delivered via websocket) of transactions and wanted to group by country and view the top 10 countries by notional price of their transactions, the UI would send that command down to the servers and the servers would do all the necessary calculations to feed the user the aggregated notionals of the top countries, and constantly update that stream as new data was processed.

I know there are frameworks (e.g. Flink,Storm, Kafka Streams) that let you define such calculations in code, but is there any framework that lets the user pick different aggregations and set those up in real-time?

torahmike
  • 420
  • 5
  • 10

2 Answers2

1

I'll answer for Flink:

Apache Zeppelin has a good integration with Flink. It lets users set up dynamic Flink queries. These are on-demand (user-defined), and off course you'll need a Flink cluster to attach to. These also refresh the Zeppelin UI.

I'm speculating, but I think that the backend will received Flink's updates, and the UI will fetch for new data at a fixed interval. While this is not done reactively, I believed this is as good as it can get currently.

Here is a Flink blog article on it: https://flink.apache.org/news/2020/06/15/flink-on-zeppelin-part1.html

BenoitParis
  • 3,166
  • 4
  • 29
  • 56
1

Flink can do this out of the box, using Flink SQL with the SQL client. You can interactively create dynamic, continuously updating queries that stream their results into Elasticsearch (for example).

This talk is a good intro that shows off what's possible. It includes a bunch of example queries, and uses Grafana on top of MySQL for dashboarding. You can do the same with Elasticsearch/Kibana, if you prefer.

David Anderson
  • 39,434
  • 4
  • 33
  • 60