Stack Overflow
Stack Overflow

Questions tagged [hazelcast-jet]

Hazelcast Jet is a distributed computing platform for low-latency stream and batch processing. It benefits from integration with Hazelcast IMDG providing fast, scalable Big Data processing that’s lightweight and embeddable.

Hazelcast Jet is a distributed computing platform for low-latency stream and batch processing. It benefits from integration with Hazelcast IMDG providing fast, scalable Big Data processing that’s lightweight and embeddable.

188 questions
2
votes
1 answer

Hazelcast Jet Kafka with not serializable event handler

I want to use hazelcast-jet-kafka in my app, because in my case the number of kafka partitions is limited. How I understand jet-kafka parallelism doesn't depend on kafka partitions, it would be nice to find explanations of how jet-kafka achieve…
Aliksz
  • 29
  • 5
2
votes
2 answers

Is it possible to inject dependencies into Hazelcast Jet pipeline stages?

For example, given a simple pipeline such as: Pipeline p = Pipeline.create(); p.readFrom(TestSources.items("the", "quick", "brown", "fox")) .map(mapFn) .writeTo(Sinks.logger()); I'd like mapFn to be something requiring a non-serialisable…
Chris Wiggins
  • 31
  • 2
2
votes
1 answer

How can I make Jet processors fault tolerant

I use the Hazelcast Jet core API to design new processors for my DAGs. Some of these processors might fail throwing exceptions that - if not handled somehow - will cause the entire job to fail and stop. I'm trying therefore to design a mechanism to…
Mirko Luchi
  • 321
  • 3
  • 4
2
votes
2 answers

File consumption in multinode hazelcast setup

I see examples where CSV files can be consumed using jet eg. BatchSource source = Sources.filesBuilder(sourceDir) .glob("*.csv") .build(path -> Files.lines(path).skip(1).map(SalesRecordLine::parse)); In a…
gurpal2000
  • 1,004
  • 1
  • 17
  • 35
2
votes
1 answer

Hazelcast Jet - fault tolerance in processing Redis streams

I have a 3 node Hazelcast Jet cluster deployed with few jobs that reads and write to Redis Streams. As of now the streams from Redis can either be read from a specific position or from the beggining or the newer ones. If a job got restarted for some…
vvra
  • 2,832
  • 5
  • 38
  • 82
2
votes
1 answer

hazelcast-jet deployment and data ingestion

I have a distributed system running on AWS EC2 instances. My cluster has around 2000 nodes. I want to introduce a stream processing model which can process metadata being periodically published by each node (cpu usage, memory usage, IO and etc..).…
Z.SP
  • 319
  • 1
  • 2
  • 12
2
votes
0 answers

Hazelcast-Jet client deployement threw java.lang.NoClassDefFoundError(org/apache/kafka/clients/consumer/ConsumerRecord)

I've started 3 instances of Hazelcast-Jet(jet-start.sh). Could see the cluster formation as 3 nodes. In one of the cluster nodes, I'm trying to run few pipeline's using the JetClient to Source from Kafka to IMap and source from IMap to Kafka. Here…
srikanth
  • 958
  • 16
  • 37
2
votes
1 answer

Hazelcast Jet internal optimization

I examine the Hazelcast Jet for my project needs, but I found the documentation really vague regarding the following topics: 1) When I perform a data join on two list streams...like for example: BatchStage trades =…
Tomas Kloucek
  • 251
  • 2
  • 12
2
votes
1 answer

Hazelcast Jet and Node.JS client serialization issue

I am having Jet 0.6 as a backup and some Node.JS process with hazelcast-nodejs-client 0.8.0 installed. I am trying to push to map an object from Node process that is exactly a reflection of similar one on the Jet side. However I don't understand…
Viktor Stolbin
  • 2,899
  • 4
  • 32
  • 53
2
votes
1 answer

Hazelcast Jet discard empty aggregation results

I have a sliding window and custom aggregation accumulator that might have empty results. What would be a proper way to discard such en 'empty' aggregation accumulators from getting into a sink? Pipeline pipeline = Pipeline.create(); …
Viktor Stolbin
  • 2,899
  • 4
  • 32
  • 53
2
votes
0 answers

Hazelcast Jet Pipelines API: processing data from more than one parent node

This question is about the Pipeline API in Hazelcast Jet 0.5.1 The pipeline I am trying to create has two infinite sources: one is a ticker (a custom source which sends one event every minute), the other is a Kafka Topic. It looks like…
Valentin Prévost
  • 51
  • 2
2
votes
1 answer

Understanding some concepts and Hazelcast Jet integrated with Kafka

I'm trying to map some concepts between Spark Structured Streaming and Hazelcast Jet, and understand another subjects as well. Q1 - In the Spark, each Kafka partition will become a partition inside spark, then they will be processed by individual…
Kleyson Rios
  • 2,597
  • 5
  • 40
  • 65
2
votes
1 answer

MapReduce single item from streaming source against finite items from map in Hazelcast Jet

Being new to Hazelcast Jet, I was trying to build a setup where single item from an infinite source (i.e. a Map Journal of user requests) is MapReduced against a (possibly changing and) huge Map of reference items. Specifically, for this example I…
sunside
  • 8,069
  • 9
  • 51
  • 74
2
votes
1 answer

How to get job at new node of hazelcast jet cluster

Can anybody say how new jet cluster instance should start job? Use case 1: start jet cluster by 3 node submit job to cluster all 3 nodes start job and process data Use case 2: start 4th node 4th node do nothing because it's no new submit job…
MGaidamak
  • 65
  • 6
2
votes
2 answers

How to force a Hazelcast (Jet) client to use a specific interface?

I'm just trying to test and improve performance with a client -> instance setup on a cluster. Interestingly it seems that the 1G network interface on the machine with the hazelcast (jet) instance is a major bottleneck. I already configured the…
Anders Bernard
  • 541
  • 1
  • 6
  • 19
1
2
3
…
12 13