Stack Overflow
Stack Overflow

Questions tagged [streaming-analytics]

15 questions
3
votes
1 answer

How to write the query to extract all the field value along with condition satisfied values in KSQL

Created stream with the following field CREATE STREAM pageviews_original_string(view_time string, user_id varchar, pageid varchar) WITH (kafka_topic='pageviews',value_format='DELIMITED',KEY='pageid'); Changed the pageid into the uppercase along…
hepzi
  • 435
  • 4
  • 17
2
votes
2 answers

Is it possible to consume from an Azure EventHub topic using Apache Beam / Google Cloud Dataflow?

Problem We'd like to consume from an EventHub topic in Azure using a dataflow pipeline in Google Cloud. Question Is it known whether KafkaIO allows for consuming from EventHubs directly in an Apache Beam/Google Cloud dataflow job? (see this post)…
Jonny5
  • 1,390
  • 1
  • 15
  • 41
2
votes
0 answers

Rules Engine on Azure

I'm new to Azure and i'm trying to understand rules engine on streaming analytics. Could you please help me understand the following. 1) Implementation of a JSON format for complex rule definition 2) Implementation of a JSON format for time-series…
Nomad18
  • 131
  • 1
  • 3
  • 8
1
vote
0 answers

Pyspark selecting multiple ordered data streams into one RDD in a performant way

I am redesigning a real-time prediction pipeline over streaming IoT sensor data. The pipeline is ingesting sensor data samples, structured as (sensor_id, timestamp, sample_index, value) as they are created in the source system, saves them locally…
Eliaz
  • 31
  • 4
1
vote
2 answers

Stream Analytics - Processing JSON with no array name

IoT Edge v2 with the modbus module sends data to IoT Hub in the format of: [ { "DisplayName": "Voltage", "HwId": "", "Address": "400001", "Value": "200", "SourceTimestamp": "2019-01-03 23:40:24" }, { "DisplayName":…
KevinH
  • 33
  • 7
1
vote
1 answer

U-sql : How to process an Avro file with multiple JSON arrays with multiple objects?

I receive an Avro file in my Data Lake Store thru streaming analytics and an event hub using capture. The structure of the file looks like…
Matthijs Hoogwerf
  • 11
  • 5
0
votes
0 answers

Flink reading sorted events but sorting is not maintained after partitioning

Using Flink RichSourceFunction I am reading a file which has events in sorted order based on timestamp field. The file is very large in size, 500GB. I am reading this file sequentially using only one split (TimeStampedFileSplit) for the whole file…
user3388770
  • 119
  • 1
  • 11
0
votes
1 answer

Managed Identity Azure Role Assignment for Streaming Job

I have an event hub EH1 in "Sub1" and Streaming Job ASA1 in "Sub2". EH1 is the streaming input to ASA1. For added security, I am planning to use Managed Identity to access event hub from an Streaming Analytics…
Abhijit
  • 1
0
votes
2 answers

Data streaming API- High availability

In my architecture on AWS, I have a service running on an EC2 instance which calls Twitter streaming API for data ingestion i.e. ingestion of real-time tweets. I call this service TwitterClient. Twitter API uses a kindof long polling over HTTP…
theReluctantProgrammer
  • 73
  • 5
0
votes
1 answer

Detect end of event time session window (Apache Flink Java)

Assuming all events arrive on time and no lateness is allowed, how do I do some processing only when the session window has ended? I.e. the watermark has passed (lastEventInWindowTimestamp + inactivityGap). I couldn't find any API method that is…
343GuiltySpark
  • 123
  • 1
  • 2
  • 11
0
votes
1 answer

Accessing per-key state store in Apache Flink that changes dynamically

I have a stream of messages with different keys. For each key, I want to create an event time session window and do some processing on it only if: MIN_EVENTS number of events has been accumulated in the window (essentially a keyed state) For each…
343GuiltySpark
  • 123
  • 1
  • 2
  • 11
0
votes
1 answer

Read CSV file with 3 columns into Datastream. JAVA Apache Flink

I've been struggling for a while setting up a flink application that creates a Datastream> from a csv file. The columns in this file (columns ID, dateTime and Result) are all String but they should be converted…
user7432713
  • 197
  • 3
  • 17
0
votes
1 answer

Compare batches of average values with each other in WSO2 Stream Processor

I've written some code in Siddhi that logs/prints the average of a batch of the last 100 events. So the average for event 0-100, 101-200, etc. I now want to compare these averages with each other to find some kind of trend. In first place I just…
user7432713
  • 197
  • 3
  • 17
0
votes
1 answer

Converting string to double or float issue in WSO2 stream processor

I have a CSV file I want to load in WS02 stream processor for simulating events. I have a column named "Result". This is a string column in the CSV file. In the WSO2 stream processor I tried to convert the incoming Result string to double or float.…
user7432713
  • 197
  • 3
  • 17
0
votes
1 answer

Sample data - Streaming Analytics

I want to do small POC or project on HANA Streaming Analytics. Can anyone please suggest me where I can get sample data to proceed or begin with? Regards , Vinaya
Vinaya Chandra H G
  • 3
  • 4