0

I want to perform a query that returns some data from my MongoDB server, but when the amount of data becomes big I get an "EOF" error from the c.Find().All() query.

Basically I have:

activeData := []DataEntry{}
activeDataQuery := bson.M{"data.active": true}
err := sigdb.Find(activeDataQuery).All(&activeData)

Which works fine for a small test with about 50,000 items, but when I try my full dataset, which is more than one million items, it returns "EOF", eventhough there is data there to be queried.

What could be causing this? I am running both the Go program nad the MongoDB server on my laptop running Ubuntu 14.04 using Go 1.3.

Edit: Upon further trials, I am also getting: "write tcp 127.0.0.1:27017: broken pipe" from the same query.

Lars
  • 1,006
  • 1
  • 9
  • 29

1 Answers1

6

The All method will load all matching data into memory, which is a very bad way to handle large data sets. With luck you'll get such timeouts before the method is finished, and in the worst case the machine will crash out of memory.

On any sort of non-trivial data set, use normal iteration instead, with the Iter and Next methods.

Gustavo Niemeyer
  • 22,007
  • 5
  • 57
  • 46
  • This worked, but will this also happen if the query itself is very large? I would like to use $nin to make a query that excludes a lot of data. The problem is that if I do this, the $nin field has hundreds of thousands of items in it, causing the same crash. – Lars Mar 08 '15 at 12:53
  • A query with hundreds of thousands of items in an exclusion list is also very bad, on any database. I would consider what the problem at hand is, consider what data structures are at play, what changes, how often, and then organize a more realistic solution. – Gustavo Niemeyer Mar 09 '15 at 16:43
  • Actually, even with a few rows error like this happens: https://github.com/go-mgo/mgo/issues/473 – Inanc Gumus Aug 01 '17 at 14:54