Why are my mgo queries running slower than they do in php

Question

I've swapped an endpoint from our PHP 7 app to a new Go service. The service takes a geographic bounding box and returns properties from a mongo database. The problem is that it's currently taking 4-5 times as long as the old PHP service took to do the same thing. ~90% of the time is spent in the GetProps function below.

var session *mgo.Session

func connectToDB() *mgo.Session {
    dialInfo := &mgo.DialInfo{
        Addrs:    []string{"xxx1.mongodb.net:27017", "xxx2.mongodb.net:27017", "xxx3.mongodb.net:27017"},
        Database: "admin",
        Username: "me",
        Password: "xxx",
        DialServer: func(addr *mgo.ServerAddr) (net.Conn, error) {
            return tls.Dial("tcp", addr.String(), &tls.Config{})
        },
        Timeout: time.Second * 10,
    }
    session, err := mgo.DialWithInfo(dialInfo)
    if err != nil {
        log.Panic(err)
    }
    session.SetMode(mgo.Monotonic, true)
    return session
}

func GetProps(propRequest Request) []Property {
    results := make([]Property, 0)
    sessionCopy := session.Copy()
    defer sessionCopy.Close()
    props := sessionCopy.DB("mapov").C("properties")
    props.Find(bson.M{
        "geo": bson.M{
            "$geoWithin": bson.M{
                "$geometry": bson.M{
                    "type":        "Polygon",
                    "coordinates": propRequest.BoundingPoly,
                },
            },
        },
    }).Sort("-rank").Limit(propRequest.CpPropsRequired).All(&results)
    return results
}

func init() {
    session = connectToDB()
}

The PHP 7 service does pretty much the same thing -

$collection = $mapovdb->properties; 
$query = ['geo' => [
  '$geoWithin' => [
    '$geometry' => [
      'type' => 'Polygon',
      'coordinates' => $boundingPoly
    ]
  ]
]];

$cursor = $collection->find( $query, $queryOptions); // $queryOptions includes the matching sort and limit

But it's way quicker (I ran the two services next to each other for 12 hours randomising the traffic).

I tried changing my property struct so it just takes a single field, but that didn't seem to affect the performance.

type Property struct {
    Name         string    `bson:"name" json:"name"`
}

What am I doing wrong? Surely I should be able to match the performance of the php7 driver?

UPDATE

I've swapped out the builtin http library for fasthttp. This seems to have made everything faster. I haven't had time to work out why yet (but will come back here when I do). My current theory is that the builtin http library creates a new goroutine for each new tcp connection not each new http connection and this is causing my db queries to queue - either because the load balancer is reusing tcp connections, or because the client is reusing them (http/2?).

Is it actually the MongoDB calls that are slower? Or request handling overall? Have you profiled it using `pprof` to see where the hotspots are? — Adrian, Mar 30 '18 at 19:09
Thanks @Adrian - I’ll have a look at pprof and see if I can get more info. (Currently I’m just running a timing function - I start the clock before the find call and stop it immediately afterwards) — Aidan Ewen, Mar 30 '18 at 19:16
I do notice that the calls are not identical. In Go you're doing an additional sort & limit which you do not do in the PHP code. — Adrian, Mar 30 '18 at 19:23
Sorry the php does the sort and limit too. I’ll update my question — Aidan Ewen, Mar 30 '18 at 19:24
Also note that your Go code is also retrieving all results and decoding the BSON into objects, which is not included in the shown PHP code, which ends at the point where it receives a cursor in order to begin retrieving results. — Adrian, Mar 30 '18 at 19:25
The two services return the same data in the same format. The Golang service spends 90% of the time in the find function. And it’s taken an average of 512ms over a 12 hour period. The php service took an average of 123ms over that 12 hour period. — Aidan Ewen, Mar 30 '18 at 19:32
I don't see anything offhand to cause a performance issue, but there's not a lot to go on. I would definitely profile it. I know from experience that mgo adds very little overhead to queries. — Adrian, Mar 30 '18 at 19:42
How large is the limit in practice? Does it help at all if you allocate the results slice with the respective capacity? — Peter, Mar 30 '18 at 20:41
@AidanEwen what is your GOMAXPROCS value? Some of the design decisions on mgo were made with scalability and concurrency in mind. It performs well if you allow it to use more threads. please, have a read about similar issue here https://groups.google.com/forum/#!topic/mgo-users/wkEE7LNpWpc — vitr, Mar 31 '18 at 02:26

Why are my mgo queries running slower than they do in php

0 Answers0