Stack Overflow
Stack Overflow

Questions tagged [hazelcast-jet]

Hazelcast Jet is a distributed computing platform for low-latency stream and batch processing. It benefits from integration with Hazelcast IMDG providing fast, scalable Big Data processing that’s lightweight and embeddable.

Hazelcast Jet is a distributed computing platform for low-latency stream and batch processing. It benefits from integration with Hazelcast IMDG providing fast, scalable Big Data processing that’s lightweight and embeddable.

188 questions
0
votes
0 answers

Reading a large Kafka topic into a map with Hazelcast-jet

I have a Kafka topic that started at about 100GB that I tried to read in to a IMap with Hazelcast-jet. The machine has plenty of memory and I gave it 300 GB of heap. The topic was partitioned into 147 partitions, but when I run the code telling…
Marc Mason
  • 11
  • 3
0
votes
1 answer

Max Aggregation with Hazelcast-jet

I want to do a simple max across an entire dataset. I started with the Kafka example at: https://github.com/hazelcast/hazelcast-jet-code-samples/blob/0.7-maintenance/kafka/src/main/java/avro/KafkaAvroSource.java I just changed the pipeline…
Marc Mason
  • 11
  • 3
0
votes
2 answers

Is it possible to nest Hazelcast Jet pipelines such that an inner pipeline can compute results for an outer pipeline?

Consider the following scenario: We want to take a large distributed collection of objects, and for each object in the collection we want to kick of another computation which uses the current object and another large distributed collection to…
newlogic
  • 807
  • 8
  • 25
0
votes
1 answer

Hazelcast-Jet drainTo syntax issue

I am trying to aggregate using Jet, source and sink are a Kafka topic, requirement is to take GPB (google proto buf) messages from source and publish a GPB messages. Problem is I am able to publish Double but not a GPB message and it gives me…
Abhishek
  • 519
  • 1
  • 6
  • 24
0
votes
1 answer

Does hazelcast jet send/receive data from cluster

We have hosted a Hazelcast cluster on a server and a different app on a different server in the same region uses Hazelcast Jet client instance to aggregate the data from Kafka source using pipeline. In this setup, does Jet client instance send data…
Abhishek
  • 519
  • 1
  • 6
  • 24
0
votes
1 answer

Does hazelcast jet stream stores data in nodes along with aggregation

I am using hazelcast jet to aggreagte(sum) stream of data Source is kafka where i receive integer and jet stream simply adds each incoming number. I have few questions 1. When it receives each number along with a it saves the data in IMap, how can…
Abhishek
  • 519
  • 1
  • 6
  • 24
0
votes
1 answer

Aggregate continuous stream of number from a file using hazelcast jet

I am trying to sum continuous stream of numbers from a file using hazelcast jet pipe .drawFrom(Sources.fileWatcher))) .map(s->Integer.parseInt(s)) .addTimestamps() .window(WindowDefinition.sliding(10000,1000)) …
Abhishek
  • 519
  • 1
  • 6
  • 24
0
votes
1 answer

Is it possible that Sources.mapJournal() is slow for frequently updating of IMap?

I'm trying to emulate the stream drawing from Sources.mapJournal through IMap which receives data from IoT device. The processing of this stream is too slow and I'm getting the big accumulated outcome after 30-60 seconds. When I started to update…
Sergey Granev
  • 107
  • 8
0
votes
1 answer

How to make sure tuples with the same key are processed in order

I have made job with Hazelcast Jet that converts a stream of IoT measurements into a stream of alarms. So, whenever the humidity level of one sensor goes above a threshold, an alarm is raised. When it falls again below the threshold, the alarm is…
PeeWee2201
  • 1,464
  • 2
  • 15
  • 23
0
votes
2 answers

Full join between multiple streams coming from different sources

I am using hazelcast jet 0.6.1 for real time analysis. There are multiple streams (mostly from remote journal) coming from different sources. I would like to know, if full join supported between multiple streams. If yes, will you please point me…
Harshad Murtekar
  • 57
  • 3
0
votes
2 answers

DAG source return false on emitFromTraverser and processor wait for all element loaded by source before start processing

USECASE HazelcastJet version 0.6.1 Hazelcast version 3.10.2 Given this (simpified version) of a DAG VERTICES S1 Source that emits 5 items of type A (read from DB with partitioning) Local parallelism = 1 S2 Source that emits 150K items of type B…
Lubbo
  • 1,030
  • 1
  • 10
  • 18
0
votes
0 answers

Hazelcast jet running only on 1 node

I am new to Hazelcast Jet so was wondering if I am doing something wrong. I am testing tis locally. I start up 2 instances of Hazelcast Jet locally: Jet.newJetInstance(); This is just by running separate class that has public static void main…
sachin jain
  • 224
  • 1
  • 4
  • 16
0
votes
2 answers

Hazelcast Jet stream processing end window emission

I've stomped across an interesting observation trying to cross check results of aggregation for my stream processing. I've created a test case when pre-defined data set was fed into a journaled map and aggregation was supposed to populate 1 result…
Viktor Stolbin
  • 2,899
  • 4
  • 32
  • 53
0
votes
1 answer

Hazelcast Jet sliding window unit of measurement

Sorry for may be silly question but it is unclear from docs what is the unit of measurement for sliding window? Is it milliseconds, seconds or number of items in the stream? I've noticed the aggregation operation was producing empty results and I…
Viktor Stolbin
  • 2,899
  • 4
  • 32
  • 53
0
votes
1 answer

Spring Boot: Failed to instantiate [com.hazelcast.jet.JetInstance]

I'm running hazelcast-jet-spring artifact (not hazelcast-jet) v.0.6.1 on Spring Boot v.2.0.3.RELEASE and getting this error: Caused by: java.lang.IllegalArgumentException: DataSerializableFactory[-45] is already registered!…
Sergey Granev
  • 107
  • 8
1 2 3
…
12 13