0

How to effectively implement subscription mechanism in G-Wan? Suppose, I want to make g-wan aggregate data from various tickers and farther process it. And, obviously, every feed provides the data in its unique format.

The straightforward way would be to create connections and subscribe to data in the init() function of the connection handler, then parse source info from the responses and dispatch data from the main() function to dedicated queues. But this approach doesn't seem to make any use of the effective task scheduling engine of G-Wan. So, may be a dedicated software would solve the problem faster?

Another approach would be creating dedicated servlets for every subscription. For that, in the main() func of the connection handler, I would need rewriting headers and including names of corresponding servlets. In this case I would employ the whole g-wan machinery. But doesn't the rewriting headers negate all performance advantage of g-wan?

Gil
  • 3,279
  • 1
  • 15
  • 25
Sergei
  • 11
  • 2

1 Answers1

0

G-WAN already provides a simple publisher/subscriber engine, see the Comet servlet example.

This works fine with slow (typically 1 update per second) feeds.

For real-time and BigData feeds, there's no alternative to using G-WAN protocol handlers (to bypass connection handler rewrites and to precisely define the desired latency).

That's what happened for this project distributing 150 million messages per second via 75,000 channels to 1.5 million subscribers.

We have also made a (now famous) demo for the ORACLE OpenWorld expo in SFO that processed 1.2 billion TPS (transactions per second) on one single server, by using G-WAN as a cache for the ORACLE noSQL database (a Java KV store).

So the limits are more a question of precise tuning than G-WAN's core engine limitations.

Gil
  • 3,279
  • 1
  • 15
  • 25
  • It could be interesting for the public to see the Java source code and reproduce the demo. Do you have any plan to open source it? – Nagi Feb 16 '15 at 15:06
  • Gil, thanks for your explanations.I see that I was not clear enough in my question. My concern was **not** to provide pushing data to subscribers, but to make g-wan subscribe to third party data feeds, i.e. to make it a client. To address my problem, g-wan will initiate kept-alive connections to feeds. So, the question becomes: would these connections go through protocol handler? In this case, PRT_ACCEPTED state would never be seen by the protocol handler. Also, do I understand it right: there can exist and work several protocol handlers at the same time? – Sergei Feb 17 '15 at 15:40
  • @Nagi, G-WAN is a freeware and G-WAN-based applications are generally custom works paid by a client. Not being paid by the taxpayer (universities, grants), we have to treat our IPR as a company asset. – Gil Feb 18 '15 at 15:36
  • @Sergei: there's also a PRT_CONNECTED state, and that's what you are interested in, and yes, a G-WAN instance can use as many protocol handlers as needed (one per listener). – Gil Feb 18 '15 at 15:39