1

I have the below use case .

Execute DB operations in async, after that is done send out a kafka event to another microservice so that it reads from DB. However as of now the kafka event is being sent even before the DB operation is complete. My code looks as below :

firstTask = dbOperation1(k)
secondTask = dbOperation2(t.getId, t, existing)
thirdTask = Task(doSomeDBUpdate).executeOn(io).asyncBoundary

Task.sequence(Seq(firstTask, secondTask, thirdTask, pushToKafkaTask))

Is there any way to ensure pushToKafkaTask surely happens after the first three task ?

Adding further code snippets to show what the firstTask , secondTask and pushToKafkaTask look like

val firstTask = dbOperation1(k)

def dbOperation1(k: objPUT)(jt: JdbcTemplate, io: Scheduler): Task[Int] = {
    val params = Array(user.userId, DateUtils.currentTimestamp, k.getId)
    
    Task(jt.update(tDao.tUpdate, params: _*)).executeOn(io).asyncBoundary
  }
  
  
val secondTask = dbOperation2(t.getId, t, existing)


def dbOperation2(id: String,input: objPUTGen, existing: objPUTGen = null,iStd: Boolean = true, 
                 isNSRefUpdate: Boolean = false,)(implicit user: UserDetails, jt: JdbcTemplate): Task[_] =
                            
    Task.sequence(Seq(dbOperation3(id, input),
                      if (iStd)  dbOperation4( id, input) else Task.unit, dbOperation5(id, input, existing, isNSRefUpdate) ))
                      

def dbOperation3(id: String, input: TemplateGeneric)(implicit user: UserDetails, jt: JdbcTemplate, io: Scheduler): Task[_] = {

    val sDel =
      s"""
         | delete from "tableName"
         | where ID = ?
       """.stripMargin
       
    Task(jt.update(sDel, id)).executeOn(io).asyncBoundary
  }

 def pushToKafkaTask(id: String, cl: String)
                 (user: UserDetails,  kafkaBase: KafkaBase = OKafkaBase): Task[Unit] = {
   
    val msg = MyCaseClass(id, cl)
    kafkaBase.pushToKafkaInternalV2(NonEmptyList.of(msg), id, topic)
    
  }
Sourabh
  • 119
  • 1
  • 8
  • 2
    `Task.sequence(Seq(firstTask, secondTask, thirdTask)) >> pushToKafkaTask` - `>>` is just an alias for `flatMap` the trick here is that if you want to ensure a _"happens after"_ relationship that is what `flatMap` / ``Monad` provides. – Luis Miguel Mejía Suárez Dec 04 '21 at 17:36
  • @LuisMiguelMejíaSuárez I tried the above solution. But sadly it did not work for me. Somehow the kafka message is still being sent before the previous tasks. – Sourabh Dec 06 '21 at 09:05
  • @LuisMiguelMejíaSuárez , if I were to try ```firstTask.flatmap(a => pushToKafkaTask)``` , then that ensures that pushToKafkaTask is executed only after the firstTask is complete. But I need that firstTask, secondTask and thirdTask to all be completed before execution of pushToKafkaTask . ```Task.sequence(Seq(firstTask, secondTask, thirdTask)) >> pushToKafkaTask``` does not work in that case – Sourabh Dec 06 '21 at 11:02
  • That doesn't make sense, you should have a bug in the definition of your tasks or something. Even you original solution must work according to the documentation: https://monix.io/api/current/monix/eval/Task$.html#sequence[A,M[X]%3C:Iterable[X]](in:M[monix.eval.Task[A]])(implicitbf:monix.execution.compat.BuildFrom[M[monix.eval.Task[A]],A,M[A]]):monix.eval.Task[M[A]] – Luis Miguel Mejía Suárez Dec 06 '21 at 13:37
  • 1
    Are you perhaps wrapping a `Future` somewhere? And in a way which doesn't wait for future to complete? Sth like `val task: Task[Unit] = Task { startFuture(); ()}`? – Mateusz Kubuszok Dec 06 '21 at 13:56
  • 1
    Show us `doSomeDBOperation` and friends. I am almost certain that it creates a `Future` or other easer async computation, which `Task.apply` is not "aware" of. – Mateusz Kubuszok Dec 07 '21 at 14:02
  • @MateuszKubuszok I have updated the question with further code snippets, to show what db operation looks like. You may be right about it creating async computation. New to using scala and monix here. Any elaborate explanation and even blogs/links to read and get closer to the solution will be really helpful. – Sourabh Dec 07 '21 at 16:33
  • 1
    Could you provide dbOperation4 and dbOperation5? And confirm that this is https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/jdbc/core/JdbcTemplate.html not some other JdbcTemplate? (BTW: Async connections to DB done this way might offer no benefit, there is no point using a thread pool (`Scheduler`) for DB operations if you don't use a separate connection per each connection - there are libraries doing this e.g. Doobie. Unless your `JdbcTemplate` uses sth like `ThreadLocal` internally all threads would use the same connection). – Mateusz Kubuszok Dec 07 '21 at 18:11
  • @MateuszKubuszok you are right about JdbcTemplate. It is the same one from Spring as you pointed out. You have asked me for dbOperation4 and dbOperation5, I have too many such functions. It will be difficult to paste all the dbOperation code here, but what I can tell you is that they all end up doing the same as dbOperation1 and dbOperation2 i.e they all have return type as `Task[_]` and they return something like `Task(jt.update(someParams)).executeOn(io).asyncBoundary` . `pushToKafkaTask` does not `executeOn(io).asyncBoundary` . Added it to the question. Could that be a problem ? – Sourabh Dec 07 '21 at 20:46
  • 1
    TBH I don't see here the what causes one Task to go into another one, while its operation didn't ended. If it was my code I would comment out all operations and start adding them until I would pinpoint which one causes this effect. Then I would take a look what it does. Otherwise there is too much noise to signal to tell where to look closer. – Mateusz Kubuszok Dec 08 '21 at 09:02

0 Answers0