Stack Overflow
Stack Overflow

Questions tagged [trident]

Abstraction on top of Apache Storm for doing realtime computation.

Trident is a high-level abstraction for doing realtime computing on top of Apache Storm. It allows you to seamlessly intermix high throughput (millions of messages per second), stateful stream processing with low latency distributed querying. If you're familiar with high level batch processing tools like Pig or Cascading, the concepts of Trident will be very familiar – Trident has joins, aggregations, grouping, functions, and filters. In addition to these, Trident adds primitives for doing stateful, incremental processing on top of any database or persistence store. Trident has consistent, exactly-once semantics, so it is easy to reason about Trident topologies. (source)

122 questions
2
votes
1 answer

storing intermediate data in storm topology

I am reading data from 2 kafka topics. Which can be described as: Topic1 data content: VehicleRegistrationNo, Timestamp, Location Topic2 data content: VehicleRegistrationNo, Timestamp, Speed I need to merge these 2 messages based on nearest…
Swati
  • 535
  • 11
  • 25
2
votes
1 answer

How to use storm Trident for batching tuples?

I was using storm previously and I need to more batching capabilities so I searched for batching in storm. And I found out Trident which do micro-batching in real-time. But somehow, I cannot figure out how Trident handle micro-batching (flow, batch…
yelo
  • 371
  • 1
  • 5
  • 17
2
votes
0 answers

How to persist batch ID when working with opaque state in Trident?

I'm getting this error when re-running Trident topology using local cluster: Current batch (1) is behind state's batch Previous transaction ID is stored in Cassandra along with a counter value, but it seems like transaction ID is always set to 1 at…
Michael Spector
  • 36,723
  • 6
  • 60
  • 88
2
votes
1 answer

Running Trident Topology in Storm TrackedTopology Unit Test

How can I run a JUnit test of a Trident Topology to allow tuples to flow through the topology while testing and verifying the output at each stage? I've tried running within Storm's Testing framework, but it's falling short of allowing verification…
brianghig
  • 197
  • 5
2
votes
1 answer

Persisting data from Apache Storm, any Framework?

We are using Kafka-storm in our project. In storm we will use multiple bolts for transformations. But before that, as part of POC, we want to persist data into DB. Which framework we should use? For BigData scenario which can be used? Is Trident…
Plymouth Rock
  • 472
  • 2
  • 6
  • 20
2
votes
1 answer

Compare Opaque or Transactional Kafka spout

What is the difference between OpaqueTridentKafkaSpout and TransactionalTridentKafkaSpout? When would you use one over the other?
Harish Shan
  • 122
  • 7
2
votes
1 answer

Storm supervisor not starting when adding 3 nodes

I am trying to test and Storm+Kafka+Trident job on a multi-node Storm cluster. When I run my job in machine 1, The job runs and records are processed When I run my job after adding a second worker, then also the job runs without any problems. The…
subbu
  • 65
  • 6
2
votes
1 answer

Inserting rows to cassandra using Storm Trident

I'm trying to insert a simple row to a table in Cassandra 2.0.5, Storm version 0.9.0.1. My test is as follows: I have a table consisting of an id (int) and sentence (text) colum. id is the primary key. My spout generates sentences and I add an ID…
Guy Wald
  • 599
  • 1
  • 10
  • 25
2
votes
1 answer

issue on implementing transactional topology in trident

My use case is to call a query to fetch records from db with different input parameters. After fetching records, do some processing, and then finally write it into a file. My input parameter values depend on the complete processing of the previous…
user2435082
  • 295
  • 5
  • 16
2
votes
1 answer

InvalidClassException local class incompatible when running storm topology

My storm application running a single topology is working just fine when running in local mode but when running on a distributed environment I get these exceptions in my worker logs. It's the same exception but it refers to 2 different classes. The…
forhas
  • 11,551
  • 21
  • 77
  • 111
2
votes
1 answer

Should i use Trident to compute the global mean of tuples in Storm?

I want to compute with Storm the mean from incoming tuples made of [int id,int value]. As you can see i can't partition the data by using a fields grouping. I need a topology architecture to distribute this computation and the only way im thinking…
simon
  • 21
  • 2
2
votes
1 answer

Getting data from one database and process and store it to another database using trident topology

I want to get data from one database through a spout and process the data and store it in another database using trident.I am new to storm and trident and i am not sure how to implement it.I got the data from the database in a spout(separate java…
Ezhil
  • 261
  • 2
  • 10
  • 31
1
vote
1 answer

Why does the locaton of my animated circles flicker?

With trident I created a (seemingly) simple animation. A number of circles are moving from bottom to top and back again with a sine interpolation: The animation itself seems to work, but there is one or two frames where all my spheres flicker all…
peq
  • 329
  • 1
  • 2
  • 9
1
vote
0 answers

rancher trident storage integration w/ rancherOS 1.55

i got two rancher cluster (deployed on vsphere). one with centos7 as base os and one with rancheros 1.55 (deployed via rancher ui w/ vsphere provider). on the centos7 based cluster trident integration installation w/; kubectl create ns…
Patrick Hermann
  • 43
  • 6
1
vote
0 answers

how to groupby tuples within a window in Storm trident?

I need to do a groupby on tuples which fall in a tumblingWindow (the groupby is based on two other fields plus their time window) in Storm trident and then apply an aggregation function on them. The following code aggregates all tuples in a…
A.B.
  • 51
  • 2
  • 10
1
2
3
…
8 9