0

I need to do some data analytics on data acquired from IOT sensors (CEP, outliers detection, etc.)

I searched over the internet to find which tool I can use according to our constraints:

  • All developers in my team are C# developers
  • The solution must be deployed on premise

I found Apache Storm to be a great solution based on what I read on it and we can develop spouts, bolts in c# from visual studio by installing the azure data lake tools.

I installed everything and started to develop a simple topology. The problem is that it seems that topologies can be deployed only to Azure HD Insights

My question : is it possible to deploy a c# topology to a storm cluster which is not in Azure? I want to submit to submit my topology to a Storm docker image.

Dovic
  • 1
  • 1

1 Answers1

0

It is definitely possible. I haven't been able to find the source for Azure data lake tools (maybe it isn't open source?), but the reference for how to implement Storm topologies in other languages is here http://storm.apache.org/releases/2.0.0-SNAPSHOT/Multilang-protocol.html.

There's an example of how to implement multilang topologies here https://github.com/apache/storm/blob/master/examples/storm-starter/src/jvm/org/apache/storm/starter/WordCountTopology.java, which implements one of its bolts in Python. You might note that there's a ShellBolt at the top of that class, which calls a python file. The python file contains the bolt implementation https://github.com/apache/storm/blob/master/examples/storm-starter/multilang/resources/splitsentence.py, which uses the Storm multilang implementation for Python here https://github.com/apache/storm/blob/master/storm-multilang/python/src/main/resources/resources/storm.py.

Sadly Storm only ships with wrappers for Python, Javascript and Ruby, but you could write your own (and contribute it back to the project if it turns out well)?

If you don't want to implement your own wrapper, a quick google turned up https://github.com/ziyunhx/storm-net-adapter, which I haven't tried, but it might be worth a shot?

You might also poke at Microsoft to get them to open source the data lake tools, or at least provide support for running on non-Azure clusters. It's very likely their tools contain a multilang wrapper for Storm.

Stig Rohde Døssing
  • 3,621
  • 2
  • 7
  • 7