0

I have an object with some attributes and a very large list of another object. it looks like this:

public class Company {
   private String attr1;
   private String attr2;
   private long attr3;
   private List<Employee> employees; // very large

...
}


public class Employee {
   private long id;
   private String name;
   .....
   
}

I want to send an object Company (with a list of over 500.000 employees) in a Rest request from a microservice to another (to be saved in its database) using Spring's WebClient. What is the most optimal way to do that without risking a server crash?

Thank you

Cambiasso
  • 113
  • 1
  • 13

1 Answers1

0

First of all, I would ask if this will be a repetitive scenario, or it is more like an initial import kind of situation.

If it is the case of an initial setup, I will go for an approach based on spring batch, since this project is intended for heavy loads like this case.

If it is a repetitive scenario, I would not recommend to try a request with 500k records. I will be breaking several patterns.

My recommendation is to consider the batch approach.

Dharman
  • 30,962
  • 25
  • 85
  • 135
pablo lamberti
  • 31
  • 7
  • I would like to use the batch approach as well, be I don't know if it's possible to use it with my data structure. I mean I know how it works if I had only a list of objects, but now that I have one object that contains the list, it is a bit tricky – Cambiasso Nov 11 '20 at 14:10
  • The key factor will be where the data is stored: is it a database or a large file? Or is it something else? – pablo lamberti Nov 11 '20 at 14:18
  • the data will be stored in the database – Cambiasso Nov 11 '20 at 15:11
  • I mean where the source data is stored? the 500k employees that you need to process. If you need to move them from a source database to a a destination database, you could implement a batch job that reads from source in chunks and then writes to the destination database. – pablo lamberti Nov 11 '20 at 15:17
  • do you have a concrete example to implement it? – Cambiasso Nov 11 '20 at 15:38
  • I mean, a batch job will be limited at the sender microservice and has no influence on the receiver microservice, right? – Cambiasso Nov 12 '20 at 11:04
  • I think for that heavy amount of load, the best approach will be a job that can read directly from the source database and write directly to the destination database. If that is not the case, you will need to think in a rest uri that allow you to add a single customer to the company and call it in the batch job for each customer, in a managed and tunned way. – pablo lamberti Nov 13 '20 at 15:38