1

I am curious to learn, Which design approach you would take in following scenario?

I have a FTP server where my java spring app (Service 1) will go and fetch the file. I will then store that file on to a S3 bucket. this file may have up to a million records. There is going to be another micro service (Service 2) which would need to need read this file data.

  1. How would you design and expose these million records from service1 so that those would readable by Service 2?

  2. Which technology stack you would use on either side and why?

    • these both services are going to hosted on same servers(For now).

Can you suggest a quick and efficient solution?

Thanks

Muks
  • 141
  • 2
  • 11

1 Answers1

0

Here's my approach. I wouldn't even create 2 services provided your scope of work. The only service will use Spring Integration and Spring Batch.

  • Spring Integration part will get the file from FTP and copy it to S3 and hands over to Spring Batch for processing
  • After processing with Batch, save the records.

If the job is to run once in a while and don't have much records, I would even not use Spring Batch. I would just use Spring Integration and process the records in parallel. But if you know you'll have millions of records and tons of files, you should consider using Spring Batch for batch processing.

cosmos
  • 2,143
  • 2
  • 17
  • 27
  • I think this is ok. many a times it will make sense. But then again, by combining 2 functionalities together we are not making the micro service independent of any other process or app. So for independent behavior of a microservice i devided them in two. one will make the sourcing process and other will serve the purpose of service the data. that is why i would like to know how can this be achieved. – Muks May 18 '18 at 17:20
  • Thanks for reply @cosmos though. really appreciate it. – Muks May 18 '18 at 17:21
  • For the second service, you can just use Spring Batch as it's capable of reading file from s3. – cosmos May 18 '18 at 17:27
  • Sure, Thanks. Would love to explore more. – Muks May 18 '18 at 17:42