Streaminsight user defined functions limitation

Question

What are the limitation of stream insight user defined functions?

Does the object need to be serializable?

Can it call external (remote) services?

If so these look to be very - very - very powerful!

Answer 1

Off the top of my head, a User Defined Function (UDF) is a static method call and operates on one event at a time. If you need something to work with more than one event at a time, you'll need to take a look at User Defined Operators (UDO) or User Defined Aggregates (UDAs). If you need to maintain state for any reason, you should be looking at UDOs or User Defined Stream Operators (UDSOs).

Remember that your payload classes only provide a schema to StreamInsight. So they don't need to be marked as serializable. Anything that gets serialized by StreamInsight will need to be marked serializable (i.e. configuration classes for adapters).

You can call out to external/remote services using the different UDFs, UDOs, UDAs, and UDSOs. However, these calls will be effectively blocking calls on one of the StreamInsight scheduler threads and this will increase latency. Event input and output should be done by the adapters only and the UDFs etc, should be used for processing the streams.