13

I have a following situation.
I have a REST client, that acts as a facade towards 3 other REST clients. (I'm programming in Java, using Spring Boot)

One of the responsibilites of the client includes doing CRUD operations on users.
Now, all 3 other systems which expose their own REST APIs have some sort of user management.

When I receive a request to, for example, create user, I have to create them on those 3 systems via their REST APIs and persist in my database.

Now, in best case scenario, I just call their APIs, insert the user in my DB and everything is great.

However, consider the scenario where the creation of users suceeds on only 1 external service. Do I retry the operation on all others? Do I try to delete the user on ones it suceeded?

What is the correct way to do this?

ioreskovic
  • 5,531
  • 5
  • 39
  • 70

3 Answers3

5

There isn't an easy way to do this. If any part of your "transaction" fails, there is no way that you can rollback or retry reliably to guarantee consistency on all systems. You would need a tight integration with all three (four systems) to use a distributed transaction system.

Samuel
  • 16,923
  • 6
  • 62
  • 75
3

One way to do it (assuming you can tolerate different states between your nodes):

  1. Let's say, your facade has persistent queue of incoming CRUD requests. Once new request req is in the queue, you start asking REST clients to perform it;
  2. Once all REST clients performed request and reported success, you can remove it from your queue, and make this change effective to global state of your system. E.g., if req was CREATE, then new user is visible to outer world, if req is update, then updates become visible to outer world and so on. This implies global state is what is stored in database of facade system;
  3. Now, what to do if your facade goes down while req is in progress (not all REST clients reported success)? Your facade, after restart, must take all pending requests from the (persistent) queue and push them to REST clients. This implies REST clients can detect if they already processed that particular request (and it just happened, facade hadn't processed reply before it went down). Usually, this is achieved by using unique request id, e.g., UUID.

Above process works if it is safe for your system to have request processed only by part of REST clients (that means data is accessible to outer world only via facade). If not, you need some system which supports distributed transactions (google for two-phase commit).

Victor Sorokin
  • 11,878
  • 2
  • 35
  • 51
1

You'll need to deal with it on a case-by-case basis. In the example you provided, you could try and delete, but that might also fail.

Once you have a failure you need to:

  1. Handle retry of creating the user
  2. Handle clients that might access a user even though only 1 of 3 was created

For retry, you can either have the initiator retry or queue the requests.

In both cases you'll probably want to design your api so that if you attempt to re-create a user that was already created, it will treat it as an update.

For example, only one of three succeeded.

The web page initiating the request returns an error. The user re-tries. This time you update the first and retry creating the 2nd and 3rd.

For clients looking up records and getting a partial user, you'll either need a 4th system of record that tracks which ones were created, or the clients themselves will need to see that only 1 of three was created. This might not even be an issue if your clients are always just looking at one of the three at a time.

Dave
  • 13,518
  • 7
  • 42
  • 51