Use Mongo Scala Driver Macros to read documents with additional nested fields

Question

Is it possible to use the mongo-scala-driver macros to read documents from a mongo db which contain additional fields to the fields specified in the case classes?

I expected the fields not specified in the case classes but in the document to be ignored on deserialisation. But an exception is thrown. Due to the schema less approach of mongo db it is impossible to specify the total set of fields in the case classes.

The Document I want to deserialize looks like this json:

{
    "_id": 6,
    "nestedOne": {
        "nestedOneOne": 123,
        "nestedOneTwo": 456
    },
    "nestedTwo": {
        "nestedTwoOne": 789
    }
}

I expected the following minimal example to work:

import org.bson.codecs.configuration.CodecRegistries.{fromProviders, fromRegistries}
import org.bson.codecs.configuration.CodecRegistry
import org.mongodb.scala.bson.codecs.DEFAULT_CODEC_REGISTRY
import org.mongodb.scala.bson.codecs.Macros._
import org.mongodb.scala.model.Filters
import org.mongodb.scala.MongoClient
import org.mongodb.scala.bson.collection.immutable.Document
import scala.concurrent.Await

import scala.concurrent.duration._

// prepare the case classes and register them
case class TestDocument(_id: Int, nestedOne: Option[NestedOne])
case class NestedOne(nestedOneOne: Int, nestedOneTwo: Int)
val testCodecRegistry: CodecRegistry = fromRegistries(
  fromProviders(classOf[TestDocument], classOf[NestedOne]),
  DEFAULT_CODEC_REGISTRY
)

// prepare the test data & insert them
val testDocument = Document(
  "_id" -> 6,
  "nestedOne" -> Document("nestedOneOne" -> 123, "nestedOneTwo" -> 456),
  "nestedTwo" -> Document("nestedTwoOne" -> 789)
)

val mongoClient = MongoClient("mongodb://localhost:1234")
val database = mongoClient.getDatabase("testdatabase")
val testCollection = database.getCollection("test_collection").withCodecRegistry(testCodecRegistry)
Await.result(testCollection.insertOne(testDocument).toFuture, 2.minutes)

// try to read and deserialize the document again
val _ = Await.result(testCollection.find[TestDocument](Filters.equal("_id", 6)).toFuture(), 2.minutes)

But the following Exception is thrown:

head of empty list
java.util.NoSuchElementException: head of empty list
    at scala.collection.immutable.Nil$.head(List.scala:428)
    at scala.collection.immutable.Nil$.head(List.scala:425)
    at org.mongodb.scala.bson.codecs.macrocodecs.MacroCodec.readDocument(MacroCodec.scala:204)
    at org.mongodb.scala.bson.codecs.macrocodecs.MacroCodec.readDocument$(MacroCodec.scala:193)
    at de.peterschrott.mongotest.MongoTest$$anon$1$TestDocumentMacroCodec$3.readDocument(MongoTest.scala:29)
    at org.mongodb.scala.bson.codecs.macrocodecs.MacroCodec.readValue(MacroCodec.scala:173)
    at org.mongodb.scala.bson.codecs.macrocodecs.MacroCodec.readValue$(MacroCodec.scala:169)
    at de.peterschrott.mongotest.MongoTest$$anon$1$TestDocumentMacroCodec$3.readValue(MongoTest.scala:29)
    at org.mongodb.scala.bson.codecs.macrocodecs.MacroCodec.decode(MacroCodec.scala:104)
    at org.mongodb.scala.bson.codecs.macrocodecs.MacroCodec.decode$(MacroCodec.scala:96)
    at de.peterschrott.mongotest.MongoTest$$anon$1$TestDocumentMacroCodec$3.decode(MongoTest.scala:29)
    at com.mongodb.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52)
    at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:53)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41)
    at org.bson.codecs.configuration.LazyCodec.decode(LazyCodec.java:47)
    at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:101)
    at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:56)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41)
    at com.mongodb.connection.ReplyMessage.<init>(ReplyMessage.java:57)
    at com.mongodb.connection.CommandProtocol.getResponseDocument(CommandProtocol.java:139)
    at com.mongodb.connection.CommandProtocol.access$000(CommandProtocol.java:51)
    at com.mongodb.connection.CommandProtocol$CommandResultCallback.callCallback(CommandProtocol.java:271)
    at com.mongodb.connection.ResponseCallback.onResult(ResponseCallback.java:48)
    at com.mongodb.connection.ResponseCallback.onResult(ResponseCallback.java:23)
    at com.mongodb.connection.DefaultConnectionPool$PooledConnection$2.onResult(DefaultConnectionPool.java:470)
    at com.mongodb.connection.DefaultConnectionPool$PooledConnection$2.onResult(DefaultConnectionPool.java:464)
    at com.mongodb.connection.UsageTrackingInternalConnection$3.onResult(UsageTrackingInternalConnection.java:119)
    at com.mongodb.connection.UsageTrackingInternalConnection$3.onResult(UsageTrackingInternalConnection.java:115)
    at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:49)
    at com.mongodb.connection.InternalStreamConnection.executeCallbackAndReceiveResponse(InternalStreamConnection.java:378)
    at com.mongodb.connection.InternalStreamConnection.access$1700(InternalStreamConnection.java:66)
    at com.mongodb.connection.InternalStreamConnection$ResponseBuffersCallback.onResult(InternalStreamConnection.java:420)
    at com.mongodb.connection.InternalStreamConnection$ResponseBuffersCallback.onResult(InternalStreamConnection.java:389)
    at com.mongodb.connection.InternalStreamConnection$ResponseHeaderCallback.onSuccess(InternalStreamConnection.java:562)
    at com.mongodb.connection.InternalStreamConnection$ResponseHeaderCallback.access$2200(InternalStreamConnection.java:517)
    at com.mongodb.connection.InternalStreamConnection$ResponseHeaderCallback$ResponseBodyCallback.onResult(InternalStreamConnection.java:584)
    at com.mongodb.connection.InternalStreamConnection$ResponseHeaderCallback$ResponseBodyCallback.onResult(InternalStreamConnection.java:568)
    at com.mongodb.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:447)
    at com.mongodb.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:444)
    at com.mongodb.connection.AsynchronousSocketChannelStream$BasicCompletionHandler.completed(AsynchronousSocketChannelStream.java:218)
    at com.mongodb.connection.AsynchronousSocketChannelStream$BasicCompletionHandler.completed(AsynchronousSocketChannelStream.java:201)
    at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
    at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
    at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
    at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
    at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
    at com.mongodb.connection.AsynchronousSocketChannelStream.readAsync(AsynchronousSocketChannelStream.java:125)
    at com.mongodb.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:444)
    at com.mongodb.connection.InternalStreamConnection.access$2000(InternalStreamConnection.java:66)
    at com.mongodb.connection.InternalStreamConnection$ResponseHeaderCallback.onResult(InternalStreamConnection.java:541)
    at com.mongodb.connection.InternalStreamConnection$ResponseHeaderCallback.onResult(InternalStreamConnection.java:517)
    at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:49)
    at com.mongodb.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:447)
    at com.mongodb.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:444)
    at com.mongodb.connection.AsynchronousSocketChannelStream$BasicCompletionHandler.completed(AsynchronousSocketChannelStream.java:218)
    at com.mongodb.connection.AsynchronousSocketChannelStream$BasicCompletionHandler.completed(AsynchronousSocketChannelStream.java:201)
    at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
    at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishRead(UnixAsynchronousSocketChannelImpl.java:430)
    at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:191)
    at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213)
    at sun.nio.ch.KQueuePort$EventHandlerTask.run(KQueuePort.java:301)
    at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Is there a way to tackle the addressed issue?

function fromRegistries, waitForResult and fromProviders is missing. Also, the variable database is missing. — Citrullin, Jun 19 '17 at 12:18
Your test document does not match the expected data structure represented in case classes. The error message could be improved and I'll look to adding that in the future. — Ross, Jun 26 '17 at 10:04

score 1 · Answer 1 · answered Jun 26 '17 at 10:22

1

The issue is due to differing datatypes:

case class TestDocument(_id: Long, nestedOne: Option[NestedOne])
case class NestedOne(nestedOneOne: Int, nestedOneTwo: Int)
val testCodecRegistry: CodecRegistry = fromRegistries(
  fromProviders(classOf[TestDocument], classOf[NestedOne]),
  DEFAULT_CODEC_REGISTRY
)

Has the following data shape:

Document("_id" -> 6L,
         "nestedOne" -> Document("nestedOneOne" -> 1, "nestedOneTwo" -> 2))

You are inserting this data shape:

Document(
  "_id" -> 6L,
  "nestedOne" -> List(Document("nestedOneOne" -> 123)),
  "nestedTwo" -> List(Document("nestedTwoOne" -> 789))
)

So the result is an error, I've added SCALA-319 to improve the error that is thrown in this case.

answered Jun 26 '17 at 10:22

Ross

17,861
2
55
73

I see the problem with my example. I actually made a mistake. Sorry for that one. I updated my question, rather the code, to be aligned with your suggestion. But I still get the Exception when the document has fields that are not specified in the case class. – peterschrott Jun 26 '17 at 11:28
2

That was fixed in [SCALA-307](https://jira.mongodb.org/browse/SCALA-307) which will be released as part of 2.2.0 - snapshots are available now. – Ross Jul 04 '17 at 08:47
@Ross the snapshot is not available on maven. Any hints about how to add dependency to this snapshot? I am not including the ObjectId field in my case classes, and the codec macro fails to work, with "head of empty list" exception. – Bhavin Doshi Nov 16 '17 at 21:07

Use Mongo Scala Driver Macros to read documents with additional nested fields

1 Answers1