Execute MongoTemplate.aggregate without row retrival

Question

I'm using the Spring Mongo driver to execute a large mongo aggregation statement that will run for a period of time. The output stage of this aggregation writes the output of the aggregation into a new collection. At no point do I need to retrieve the results of this aggregation in-memory.

When I run this in Spring boot, the JVM is running out of memory doing row retrieval, although I'm not using or storing any of the results.

Is there a way to skip row retrieval using MongoTemplate.aggregate?

Ex:

mongoTemplate.aggregate(Aggregation.newAggregation(
   Aggregation.sort(new Sort(new Sort.Order(Sort.Direction.DESC, "createdOn"))),

   Aggregation.group("accountId")
      .first("bal").as("bal")
      .first("timestamp").as("effectiveTimestamp"),

   Aggregation.project("_id", "effectiveTimestamp")
                        .andExpression("trunc(bal * 10000 + 0.5) / 100").as("bal"),

   aggregationOperationContext -> new Document("$addFields", new Document("history",Arrays.asList(historyObj))),

   // Write results out to a new collection - Do not store in memory
   Aggregation.out("newBalance")
).withOptions(Aggregation.newAggregationOptions().allowDiskUse(true).build()),
   "account", Object.class
);

>Is there a way to skip row retrieval using MongoTemplate.aggregate? - I fear, not (when you ask that strict). ..and the problem is maybe not with `Aggregation.out`, but `Arrays.asList()` ...? 1. how large is historyObj? 2. have a stacktrace? ...increasing memory is of course "always worth" — xerx593, Feb 25 '20 at 20:12
`Arrays.asList(historyObj)` will always be a 1 element array that's used to build this query — Max Friederichs, Feb 25 '20 at 20:16

score 4 · Answer 1 · edited Jun 06 '21 at 15:03

Use AggregationOption - skipOutput() . This will not return a result in case of aggregation pipeline contains $out/$merge operation.

mongoTemplate.aggregate(aggregation.withOptions(newAggregationOptions().skipOutput().allowDiskUse(true).build()), "collectionNme", EntityClass.class);

If you are using MongoDriver without framework.

MongoClient client = MongoClients.create("mongodb://localhost:27017");
 MongoDatabase database = client.getDatabase("my-collection");
 MongoCollection<Document> model = database.getCollection(collectionName);
 AggregateIterable<Document> aggregateResult = model.aggregate(bsonListOfAggregationPipeline);
 
 // instead iterating over call toCollection() to skipResult
 aggregateIterable.toCollection();

References:

Max Friederichs · Accepted Answer · 2020-02-28T16:45:26.290

0

I was able to resolve this by using

MongoTempalte.aggregateStream(...).withOptions(Aggregation.newAggregationOptions().cursorBatchSize(0).build)

edited Feb 28 '20 at 16:45

answered Feb 26 '20 at 20:11

Max Friederichs

569
3
13

Execute MongoTemplate.aggregate without row retrival

2 Answers2

Linked