0

When watching a change stream on a collection, I can specify a resume Token (with resumeAfter) to get changes after that token in the opslog. What happens, if this token cannot be found in the opslog (for example, the client was disconnected too long and provides an old token)?

Especially, will the async mongodb java driver detect this situation and how?

For me, that question is important because i must detect, whether I lost the connection and have to pull all data again or can rely on the changestream to get all changes from the collection.

The documentation states, that

Change streams are resumable, as long as the oplog has enough history to locate the last operation that the application received.

but does not specify, what happens otherwise. I also find it rather difficult to read the driver source, to determine how a client can detect whether the last token could be found in the opslog.

Konrad Lötzsch
  • 467
  • 7
  • 15
  • There is no "specific to Java Driver" implementation here, and it's exactly what the documentation states. The Oplog is a "capped collection", which means once entries are gone then they are "gone". It's up to your code to respond itself if a resume token cannot be "resumed" then simply pick up from reading the last thing that you can. Which usually means the "last matching event" for your filter condition. In short, you're being "way too broad" here. Try to make your question "specific" about what it is "your application" needs to do here, and then you get advice. – Neil Lunn Jun 05 '18 at 07:06

1 Answers1

1

What happens if this token cannot be found in the oplog

As mentioned on the comment, this behaviour happens in the MongoDB server. When a client (i.e. application written using the Java driver) submits a resumeToken the server will validate the token. See also resume_token.cpp (v3.7.0)

Any errors found by the server will be thrown back to the client, the client will then raise an exception. In the case of MongoDB Java driver it would be com.mongodb.MongoCommandException.

If falling off of the OpLog is a concern, I would suggest to calculate the replica set oplog size accordingly in conjunction with the caching period of the resume token. For example. If the OpLog could store up to 24 hours of operations, the resume token may be cached every 8-12 hours.

Wan B.
  • 18,367
  • 4
  • 54
  • 71