0

The context : I want to break down a module onto sub modules. Each one with a responsability. After the breakdown, I would like to compose / orchestrate the services provided by each submodules to build aggregate objects.

My current solution : The composition used RxJava. First I load a list of ObjectA then inside a flatMap I call asynchronously (using defer and Schedulers.io) multiple services and finally inside the flatMap I use a zip operator to build a new Object.

objectsA.flatMap {
   Observable<A> oa = loadAAsync();
   Observable<B> ob = loadBAsync();
   Observable<C> oc = loadCAsync();
   return Observable.zip(oa, ob, oc, (a, b, c) -> { build() };
}

The methods loadXAsync call a service that load an object from the Databse.

My problem : This solution is currently slow. x5 times the original one (a single SQL request). I was a bit dissapointed. In fact the solution is slower each time I add a new loading part. With only A, quite good, B decreases performance, C etc etc. My first feeling is that the connection pool used (HiakriCP) cannot provide enough database connection, becoming a bottleneck. Currently I have a pool size of 10. The HikariWiki said that large enough.

My question : How do you handle this case ? is it normal for RxJava "experts" ? do you increase the pool size ? or do you limit the number of threads ?

Update 1 @divers : Perhaps I'm totally wrong about the connection pool. I will try to explain my thinking.

The calls are made asynchronously. Each new thread will pick a connection and make a request to the DB. The Schedulers.io uses a thread pool that will grow as needed. If I got a hundreds of ObjectA emitted. Each one will make 5 sql requests.

I think (need to check this one) that flatMap isn't blocking. So I will have a lot of threads and a lot of batches of 5 requests in parallel. ex: objectA#1 emitted -> 5 requests, objectA#2 emitted before the end of objectA#1 so 5 five requests in parallel.

My pool size is fixed and contains 10 connections. That's why I thought my problem comes from here.

*Update 2 * : Here is the SO Question / code that I tried on my project

*Update 3 * : Configuring my pool size to 50. When I monitor the Hikari MBean, I saw that during a short period of time, the

  • ActiveConnections = 50 (so all connections used)
  • IdleConnections = 0
  • ThreadsAwaitingConnection : the max value I saw was 210.

Please correct me :)

Community
  • 1
  • 1
Archange
  • 397
  • 5
  • 21
  • Not sure that I got you right, but if each loadXAsync call does request to db, then quite obvious that it would be much slower than to make 1 SQL request with joins.. How it related to connection pool size? – Divers Dec 01 '15 at 09:48
  • What are you doing exactly in `loadXAsync`? – Reut Sharabani Dec 01 '15 at 11:17
  • The small extra speed you get from RxJava non blocking style will be totally negated by the extra DB calls. Think of the amount of work needed to send and retrieve an SQL query - it's huge. RxJava (or any other async / non blocking framework) is never going to make up for that. – Will Dec 01 '15 at 12:28
  • Thanks for your opinion. It's currently a work in progress to evaluate this kind of services integration. In the examples of Netflix reactiveLab, I saw something pretty similar (personnalizedCatalog, catalogs, reviews, critics) all of that aggregated using RxJava and Hystrix. They also took a penalty (network call over http), and I thought their data comes from a database. How can they achieve that when I got unlucky retrieving data from a local storage without network calls. :'( – Archange Dec 01 '15 at 12:55
  • @Archange Actually rxjava is single threaded by default, so if you don't do something special in your `loadAAsync` it would be executed one by one. Check http://www.introtorx.com/content/v1.0.10621.0/15_SchedulingAndThreading.html for more info – Divers Dec 01 '15 at 15:49
  • @Divers yes I know. I didn't show it but loadXAsync use the same code the one from [Ben](http://stackoverflow.com/questions/26249030/rxjava-fetching-observables-in-parallel?rq=1). Each call is wrap inside a Observable.defer and .subscribeOn method is bound to Schedulers.io(). I think it is the rigth way to make the call asynchronous. – Archange Dec 01 '15 at 17:02
  • How did the original design work with one SQL request? Was it some type of complex join or where clause? Are you now doing the joining and filtering in the application? I'm trying to understand how the modules interacted originally. One approach would be for the modules to build parts of the query asynchronously and new module that executes the combined query. – John Scattergood Dec 23 '15 at 19:17

0 Answers0