Questions tagged [stream-processing]
272 questions
0
votes
1 answer
Flink or Spark for incremental data
I have no experience with either Flink or Spark, and I would like to use one of them for my use case. I'd like to present my use case and hopefully get some insight of whether this can be done with either, and if they can both do that, which one…
phoenix7360
- 2,807
- 6
- 30
- 41
0
votes
1 answer
What happens to messages that come to a server implements stream processing after the source reached its bound?
Im learning akka streams but obviously its relevant to any streaming framework :)
quoting akka documentation:
Reactive Streams is just to define a common mechanism of how to move
data across an asynchronous boundary without losses, buffering or
…
jack miao
- 1,398
- 1
- 16
- 32
0
votes
1 answer
How to merge old data to save space in Elasticsearch
I tried to find information about this, but I have not find what I was looking for.
I am storing metrics every minutes in an Elasticsearch database. My idea is that the frequency is important only in a short period.
For example, I want to have my…
ImbaBalboa
- 851
- 9
- 23
0
votes
0 answers
How to stress an AMD GPU to nearly 100% workload using minimal LOC in OpenCL?
Out of curiosity I want to stress my HD5750 GPU to nearly 100% workload using OpenCL, how would I do this with minimal effort/code? Apparently the HD5750 has 720 stream processors, so I presume I have to do some parallel computational loop on all…
JohnnyFromBF
- 9,873
- 10
- 45
- 59
0
votes
2 answers
Any of the collections or data structure is non blocking in java
In event processing a function puts values into a collection and another removes from the same collection. The items should be placed inside the collection in the order they received from the source (sockets) and read in the same way or else the…
user64287
- 53
- 1
- 8
0
votes
2 answers
Port existing php application in spark streaming
We have a huge existing application in php which
Accepts a log file
Initialises all the database, in-memory store resources
Processes every line
Creates a set of output files
Above process happens per input file.
Input files are written by a…
Shades88
- 7,934
- 22
- 88
- 130
0
votes
1 answer
Apache Apex minimum development environment
I am wondering what the minimum environment I need to develop and test my Apex apps?
Running in Eclipse on Windows, I generated Apex project using the Apex archtype and tried to run the default JUnit test that gets created but get the following…
brusli
- 79
- 9
0
votes
4 answers
How to write my own job in samza
Recently I am trying to do some stream processing work on Samza framework. I have deployed the hello-samza example successfully. However, when I try to write my own job, I have no idea where to start my work.
I have read this document, but I still…
zzx
- 9
- 3
0
votes
1 answer
Apache Samza aggregation rules for missing expected events in rolling time-period
My use-case is to identify entities from which expected events have not been received after X amount of time in real-time.
For example:
If we have received PaymentInitiated event at time T but didn't receive either of PaymentFailed / PaymentAborted…
Harish
- 7,589
- 10
- 36
- 47
0
votes
0 answers
How can I access to Android file system (e.g. navigate a dir) and stream data through the browser without using an app?
I'm trying to develop the client side application of a remote audio player. I developed the server side application but I'm lost on the android side.
I want to navigate all the files in the music directory, obtain a list stored in my android phone…
user2630928
- 1
- 1
0
votes
1 answer
Handling Exception in java Piped Streams
I have a xmlStream which I am converting to jsonStream using org.apache.wink.json4j.utils.XML. Here is the code
public void process(InputStream xmlStream) {
final BufferedInputStream bufferedXmlStream = new BufferedInputStream(xmlStream);
…
user1739372
- 193
- 1
- 5
0
votes
2 answers
How to append object to a json file with JSR 353 (Java API for JSON Processing)
Using JSR-353 (https://jsonp.java.net/index.html)
I would like to open a json file and append some object in the root array, eg :
[{"foo":"bar"}]
I would like with a code about like this :
try(JsonGenerator writer = Json.createGenerator(new…
Yann Le Moigne
- 612
- 6
- 16
0
votes
1 answer
WebGL one-to-many data processing
Is there any scheme using WebGL which allows to process one data record to an previously unknown number of records?
Using OpenGL for example, a geometry program can be used to multiply vertices depending on their attributes, and thus output data of…
dronus
- 10,774
- 8
- 54
- 80
-1
votes
2 answers
Is there any scalable data streaming framework that includes a UI that supports defining on-demand queries?
Let's say I'm ingesting real-time data and want the end users who are viewing the data in real time to be able to group the information in different ways and quickly get views that represent the new groupings.
So if the data was for example, all the…
torahmike
- 420
- 5
- 10
-1
votes
1 answer
Filtering from large dataset in a stream processing
I'm working on a streaming application, the input stream is a series of events. For each event, I need to filter out the valid subscribers from relational database.
eventId => (eventId, [subscriberIds])
Suppose the subscription table is bounded(or…
Daniel S. Hatten
- 41
- 3