0

For my used it create additional instance of bridge and two more kafka topic which cause latency issue for data flowing between so -> s

So - source
p - processor
s - sink

Use-case for creating data flow

1) for new record flow should so => s

2) for update records flow should be so => p => s

DSL:
demo1=so|p > :foo
demo2=:demo1.so > :foo
demo3=:foo > s

Issues:

1)here data will flow from so -> Input channel/queue ->bridge(internal used demo.so topic) -> foo topic ->output channel/queue ->s

here data is flowing from source to sink using 4 queue which can latency issue compare to data flow like source ->kafka ->sink using regular spring boot with kafka consumer/producer using 1 kafka topic**

This will create so and foo as addition queues which will consume my resources and hardware.

2)Also this create additional instance of bridge and in real time i can configure amazon web service to scale my application instances(so,p,s) but who and how scaling processor .bridge is occurred

This will create additional spring boot application for bridge named as demo2.bridge which will share my resources and hardware

if I am trying to connect to different stream flows then bridge make sense but with in single and simple stream flow having bridge does not sounds like real time production solution .

Also if I am developing complex flow which has N fan out then this will create N bridge instance and N+ 2 addition queue/topics consuming additional resources and impacting application performance.

Can you suggest any better to implement above use case without having additional queues and instance creation

lebelinoz
  • 4,890
  • 10
  • 33
  • 56
Rohan Surve
  • 1
  • The way you indented your text made it look like everything is code. I fixed it for you. – lebelinoz Oct 19 '17 at 00:11
  • Well I havent fixed it. But using regular kafka or messaging system I can A->B->C for update and A->C for new records .Which means at max only 3 queues and 3 spring boot application deployment and can use all hardware i provided for it. But for use case in scdf 4 application (additional one spring binder) and almost 6 kafka queues created due to which over all latency is increased and also resource sharing is increased additional application and queue – Rohan Surve Oct 20 '17 at 10:35

1 Answers1

0

Let's replay your example with a concrete use-case since the topology and its description in your post is a bit confusing.

Let's assume you have the following as your primary pipeline.

stream1 = http --server.port=9001 | filter > :fooTopic

In this case, http and filter apps will be connected with a unique topic (i.e., stream1.http) and whenever you have a named-channel in the downstream position, SCDF internally uses the bridge-processor to connect the output of filter app to a named destination. In total, for the above pipeline, you will have a 2 topics (i.e, stream1.http and fooTopic).

This is by design.

Now, let's say you'd like to TAP this primary pipeline and dump the "unfiltered" data to the same fooTopic.

stream2 = ":stream1.http > :fooTopic"

In this case, there will be no new topics created. A bridge-processor will be used to connect both the named-destination behind the scenes. This is by design as well.

if I am trying to connect to different stream flows then bridge make sense but with in single and simple stream flow having bridge does not sounds like real time production solution .

Just to reiterate, a bridge-processor is only used when you have a named-destination either at the source or sink position (or both). Please clean-slate your existing setup and retry with a simple example (also with meaningful apps/names).

Sabby Anandan
  • 5,636
  • 2
  • 12
  • 21