-1

I have a problem with MongoDB cluster when it comes to use multi-document transactions.

I have a 5 mongodb replicaset servers divided into three data centers. Two of them are in the first datacenter, two in the second, and one in the third datacenter (arbiter). At a time one of the servers is primary and third others are slave replicas. I wrote an application in java using Spring Boot and I masivelly use multi-document transactions in mongo.

Everything works fine when all the db servers are up. But I when I wanted to test high-availability by eliminating one of the datacenters I encountered strange problems. My application started to hang on each transaction (I need to wait about a minute, and then I get timeout from database), but still works fine when transactions are not used :-(.

Below is the exception i get in my application:

2020-07-06 16:58:51.748 ERROR 6 --- [0.0-5555-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.data.mongodb.MongoTransactionException: Query failed with error code 251 and error message 'Encountered non-retryable error during query :: caused by :: Transaction 4 has been aborted.' on server mongos-dc2.fake.domain.com:27017; nested exception is com.mongodb.MongoQueryException: Query failed with error code 251 and error message 'Encountered non-retryable error during query :: caused by :: Transaction 4 has been aborted.' on server mongos-dc2.fake.domain.com27017]

What could be the reason of that? Could you tell me what should I do to fix this behaviour?

devstructor
  • 1,306
  • 10
  • 12
  • Diagnose what exactly is "hanging", add details to the question. – D. SM Jul 06 '20 at 16:24
  • My guess is you are not using transactions correctly, but your question still does not contain enough information to be answerable. – D. SM Jul 07 '20 at 11:21
  • Believe me, when you use spring boot there is really no space to mistake. You just declare TransactionManager and use @Transactional annotation on some method and there you go. As I said everything works fine during local development on local single db and also on destination server when all the servers are up. Problem begins when I disconnect two of the servers. I hoped that someone had a similar problem and I see no reason to downvote – devstructor Jul 07 '20 at 11:32
  • I downvoted because I do not believe the question is answerable as written. If there really was no space to mistake, you wouldn't be having this issue, would you? Reduce your code as much as possible then once it completely fits on one screen page add it to the question. – D. SM Jul 07 '20 at 11:38
  • Consult the server logs as well. – D. SM Jul 07 '20 at 11:40

1 Answers1

0

I figured that the guilty one is the the mongo arbiter.

https://docs.mongodb.com/manual/tutorial/add-replica-set-arbiter/

As I described, four of my mongo servers are oridinary replicas (two in first datacenter and two in the second datacenter), but the last one (in third datacenter) is the arbiter, which role is only to vote in case of election of new primary. I changed that node type from arbiter to normal replica and everything started to work as expected :/ That still doesn't anwser my question, because I have no idea why something that was supposed to work, the official mongo feature, just failed. But probably this is the question to Mongo team more than a topic to stackoverflow.

Nevertheless I hope that this will help someone

devstructor
  • 1,306
  • 10
  • 12