1

Followup to Configuration of MessageChannelPartitionHandler for assortment of remote steps

Even though the first question was answered (I think well), I think I'm confused enough that I'm not able to ask the right questions. Please direct me if you can.

Here is a sketch of the architecture I am attempting to build. Today, we have a job that runs a step across the cluster that works. We want to extend the architecture to run n (unbounded and different) jobs with n (unbounded and different) remote steps across the cluster.

I am not confusing job executions and job instances with jobs. We already run multiple job instances across the cluster. We need to be able to run other processes that are scalable in hte same way as the one we have defined already.

enter image description here

The source data is all coming from database which are known to the steps. The partitioner is defining the range of data for the "where" clause in the source database and putting that in the stepContext. All of the actual work happens in the stepContext. The jobContext simply serves to spawn steps, wait for completion, and provide the control API.

There will be 0 to n jobs running concurrently, with 0 to n steps from however many jobs running on the slave VM's concurrently.

Does each master job (or step?) require its own request and reply channel, and by extension its own OutboundChannelAdapter? Or are the request and reply channels shared?

Does each master job (or step?) require its own aggregator? By implication this means each job (or step) will have its own partition handler (which may be supported by the existing codebase)

The StepLocator on the slave appears to require a single shared replyChannel across all steps, but it appears to me that the messageChannelpartitionHandler requires a separate reply channel per step.

What I think is unclear (but I can't tell since it's unclear) is how the single reply channel is picked up by the aggregatedReplyChannel and then returned to the correct step. Of course I could be so lost I'm asking the wrong questions.

Thank you in advance

pojo-guy
  • 966
  • 1
  • 12
  • 39
  • 1
    I tired to answer your questions. However, if you still need help on this (I see how much time and effort you have put into trying to make this work), I suggest that you prepare a minimal example that I can run locally (with docker compose for instance) and I will take time to look at it. Once that in place, I suggest to discuss this scenario in a zoom session if you want where I can answer all your questions and clarify all these confusions. I don't exclude the possibility of a bug in Spring Batch, but without a minimal example that reproduces the issue, I can't really validate that. – Mahmoud Ben Hassine Aug 23 '22 at 08:20
  • The biggest issue right now appears to be 100% between my headphones. Let me go through this again and see if I can make sense of it. – pojo-guy Aug 23 '22 at 12:33
  • @MahmoudBenHassine This was very helpful. I refactored and put all of the master jobs into the same xml file, and now the shared int:channel and related beans are seen by the loader. I have some initialization coding issues in my classes that need to be resolved, but the step beans are being loaded and (some of them) fired now. – pojo-guy Aug 23 '22 at 16:02
  • 1
    Typo in my previous comment: `I tired to ..`, I obviously meant `I tried to ..`. I will never get tired offering help and answering questions from our community members ;-) Anyway, glad to hear you are making some progress on this. Keep me posted. Thank you. – Mahmoud Ben Hassine Aug 23 '22 at 17:33
  • Making good progress. Artem is helping me look at refactoring the batch steps since there's an audit requirement that is not really amenable to the spring batch tasklet. (https://stackoverflow.com/questions/74182525/is-there-a-spring-batch-pattern-to-persist-the-unaltered-values-before-the-itemp/74183171) – pojo-guy Oct 24 '22 at 17:18

1 Answers1

1

Does each master job (or step?) require its own request and reply channel, and by extension its own OutboundChannelAdapter? Or are the request and reply channels shared?

No, there is no need for that. StepExecutionRequests are identified with a correlation Id that makes it possible to distinguish them.

Does each master job (or step?) require its own aggregator? By implication this means each job (or step) will have its own partition handler (which may be supported by the existing codebase)

That should not be the case, as requests are uniquely identified with a correlation ID (similar to the previous point).

The StepLocator on the slave appears to require a single shared replyChannel across all steps, but it appears to me that the messageChannelpartitionHandler requires a separate reply channel per step.

The messageChannelpartitionHandler should be step or job scoped, as mentioned in the Javadoc (see recommendation in the last note). As a side note, there was an issue with message crossing in a previous version due to the reply channel being instance based, but it was fixed here.

Mahmoud Ben Hassine
  • 28,519
  • 3
  • 32
  • 50