1

I am running a cron whose code is written in Golang,and i am using mongoDb(version -4.4) as database, the cron runs after every 5 minutes of interval, i have been noticing that when i start the cron within a week the cpu usage hikes from 0 to 60% or even more and i have to restart the cron service so that it shouldn't effect the services of cron later on, which means that there's an memory leak, resources are not being released properly, i'm not able to figure out what might be the reason, i have been using the single db connection for the whole cron let's say a single cron is being for 2000 merchants. I'm running my cron with Worker pool using Buffered Channels
Below is the code which i am using for running my cron using worker pool

func RunCronForDbNames(cronType, subsType string) {
    cronStatus := GetCronStatusConditions(cronType)
    if cronStatus {
        dbNames, _ := controller.GetDbNamesForCron(subsType, cronType)
        if len(dbNames) == 0 {
            return
        }
        client, ctx := ConnectDb("ip")
        defer client.Disconnect(ctx)
        cronDataChannel := make(chan view.CronChannelData, len(dbNames))
        var wg sync.WaitGroup
        startTime := time.Now().Unix()
        var cronChData view.CronChannelData
        cronChData.Database.Client = client
        cronChData.Database.Ctx = ctx
        cronChData.StartTime = startTime
        for _, dbName := range dbNames {
            isContinue := db.CheckGlobalCronStatus(cronType, dbNames)
            if !isContinue {
                continue
            }
            wg.Add(1)
            contextKeys := make(map[string]interface{})
            contextKeys["db_name"] = dbName
            contextKeys["role"] = ""
            var c gin.Context
            c.Keys = make(map[string]any)
            c.Keys = contextKeys
            cronChData.C = c
            cronChData.Database.MainDatabase = dbName
            cronDataChannel <- cronChData
        }
        close(cronDataChannel)
        for w := 1; w <= 10; w++ {
            go Worker(w, cronDataChannel, &wg, cronType)
        }
        wg.Wait()
    }
    return
}

My worker is running with 10 workers at a time

func Worker(id int, jobs <-chan view.CronChannelData, wg *sync.WaitGroup, cronType string) {
    switch cronType {
    case config.CompleteBookingCron:
        for job := range jobs {
            controller.CompleteBookingsCron(job, wg)
        }
    }
    return
}

In CompleteBookingsCron, we mark the bookings as completed and send mails and Sms to our customers based on their settings. For sending Emails and Sms, go-routines are being used.

Can someone help me find out what could be the reason for the increasing of cpu usage, what things should i follow in order to getting the resources freed up properly which should not increase the Cpu usage?

rohit
  • 89
  • 5
  • Where are you subtracting from your WaitGroup, wg.Wait() would block for ever here – aksh02 Jun 26 '23 at 06:50
  • @aksh02 i have been doing that in CompleteBookingsCron function where the main part is being processed. – rohit Jun 26 '23 at 06:57
  • Use [pprof](https://pkg.go.dev/net/http/pprof) to look at running and blocked goroutines. Are you sure your goroutines finish in five minutes or less? – Peter Jun 26 '23 at 07:23
  • @Peter, there might be some merchants whose goroutines take longer than 5 minutes, For those merchants, we are not processing the cron in the next batch. – rohit Jun 26 '23 at 08:27
  • You're not closing`cronDataChannel`. The purpose of the `WaitGroup` is to be tied to the lifetime of the goroutine, it doesn't make sense to close it in `CompleteBookingsCron` if you don't ever return from the goroutine. which is why you almost always should defer the close immediately in the goroutine function. You are probably leaking goroutines (which you can check with the pprof http interface or just create a full stack trace). – JimB Jun 26 '23 at 13:21

0 Answers0