Keep retrying a function in Golang

Question

I am trying to make a functionality which would work in the following manner:

1. As soon as the service function is called, it uses the Fetch function to get records from a service (which come in the form of byte array), JSON unmarshal the byte array, populate the struct and then send the struct to a DB function to save to database.
2. Now, since this needs to be a continuous job, I have added two if conditions such that, if the records received are of length 0, then we use the retry function to retry pulling the records, else we just write to the database.

I have been trying to debug the retry function for a while now, but it is just not working, and basically stops after the first retry (even though I specify the attempts as 100). What can I do to make sure, it keeps retrying pulling the records ?

The code is as Follows:

// RETRY FUNCTION
func retry(attempts int, sleep time.Duration, f func() error) (err error) {
for i := 0; ; i++ {
    err = f()
    if err == nil {
        return
    }

    if i >= (attempts - 1) {
        break
    }

    time.Sleep(sleep)
    sleep *= 2

    log.Println("retrying after error:", err)
}
return fmt.Errorf("after %d attempts, last error: %s", attempts, err) }


//Save Data function 

type Records struct {
Messages [][]byte
}

func (s *Service) SaveData(records Records, lastSentPlace uint) error {

//lastSentPlace is sent as 0 to begin with.
for i := lastSentPlace; i <= records.Place-1; i++ {

    var msg Records
    msg.Unmarshal(records.Messages[i])

    order := MyStruct{
        Fruit:    msg.Fruit,
        Burger:   msg.Burger,
        Fries:    msg.Fries,
     }

    err := s.db.UpdateOrder(context.TODO(), nil , order)
    if err != nil {
        logging.Error("Error occured...")
    }
    
}return nil}



//Service function (This runs as a batch, which is why we need retrying)

func (s *Service) MyServiceFunction(ctx context.Context, place uint, length uint) (err error) {

var lastSentPlace = place

records, err := s.Poll(context.Background(), place, length)
if err != nil {
    logging.Info(err)
}

// if no records found then retry.
if len(records.Messages) == 0 {
    
    err = retry(100, 2*time.Minute, func() (err error) {
        records, err := s.Poll(context.Background(), place, length)
        
        // if data received, write to DB
        if len(records.Messages) != 0 {
            err = s.SaveData(records, lastSentPlace)
        }
        return
    })
    // if data is not received, or if err is not null, retry
    if err != nil || len(records.Messages) == 0 {
        log.Println(err)
        return
    }
// if data received on first try, then no need to retry, write to db 
} else if len(records.Messages) >0 {
    err = s.SaveData(records, lastSentPlace)
    if err != nil {
        return err
    }
}

return nil }

I think, the issue is with the way I am trying to implement the retry function, I have been trying to debug this for a while, but being new to the language, I am really stuck. What I wanted to do was, implement a backoff if no records are found. Any help is greatly appreciated.

Thanks !!!

Answer 1

I make a simpler retry.

• Use simpler logic for loop to ensure correctness.
• We sleep before executing a retry, so use i > 0 as the condition for the sleeping.

Here's the code:

func retry(attempts int, sleep time.Duration, f func() error) (err error) {
    for i := 0; i < attempts; i++ {
        if i > 0 {
            log.Println("retrying after error:", err)
            time.Sleep(sleep)
            sleep *= 2
        }
        err = f()
        if err == nil {
            return nil
        }
    }
    return fmt.Errorf("after %d attempts, last error: %s", attempts, err)
}

Answer 2

I know this is an old question, but came across it when searching for retries and used it as the base of a solution.

This version can accept a func with 2 return values and uses generics in golang 1.18 to make that possible. I tried it in 1.17, but couldn't figure out a way to make the method generic.

This could be extended to any number of return values of any type. I have used any here, but that could be limited to a list of types.

func retry[T any](attempts int, sleep int, f func() (T, error)) (result T, err error) {
    for i := 0; i < attempts; i++ {
        if i > 0 {
            log.Println("retrying after error:", err)
            time.Sleep(time.Duration(sleep) * time.Second)
            sleep *= 2
        }
        result, err = f()
        if err == nil {
            return result, nil
        }
    }
    return result, fmt.Errorf("after %d attempts, last error: %s", attempts, err)
}

Usage example:

var config Configuration

something, err := retry(config.RetryAttempts, config.RetrySleep, func() (Something, error) { return GetSomething(config.Parameter) })

func GetSomething(parameter string) (something Something, err error) {
    // Do something flakey here that might need a retry...
    return something, error
}

Hope that helps someone with the same use case as me.

Answer 3

The function you are calling is using a context. So it is important that you handle that context.

If you don't know what a context is and how to use it, I would recomend that post: https://blog.golang.org/context

Your retry function should also handle the context. Just to get you on the track I give you a simple implementation.

func retryMyServiceFunction(ctx context.Context, place uint, length uint, sleep time.Duration) {
    for {
        select {
        case ctx.Done():
            return
        default:
            err := MyServiceFunction(ctx, place, length)
            if err != nil {
                log.Println("handle error here!", err)
                time.Sleep(sleep)
            } else {
                return
            }
        }
    }
}

I don't like the sleep part. So you should analyse the returned error. Also you have to think about timeouts. When you let your service sleep to long there could be a timeout.

Answer 4

There is a library for the retry mechanism. https://github.com/avast/retry-go

url := "http://example.com"
var body []byte

err := retry.Do(
    func() error {
        resp, err := http.Get(url)
        if err != nil {
            return err
        }
        defer resp.Body.Close()
        body, err = ioutil.ReadAll(resp.Body)
        if err != nil {
            return err
        }

        return nil
    },
)

fmt.Println(body)

Answer 5

In the GoPlayground in the comments of the accepted answer, there are some things I would consider adding. Using continue and break in the for loop would make the loop even simpler by not using the if i > 0 { statement. Furthermore I would use early return in all the functions to directly return on an error. And last I would consistently use errors to check if a function failed or not, checking the validity of a value should be inside the executed function itself.

This would be my little attempt:

package main

import (
    "errors"
    "fmt"
    "log"
    "time"
)

func main() {
    var complicatedFunctionPassing bool = false
    var attempts int = 5

    // if complicatedFunctionPassing is true retry just makes one try
    // if complicatedFunctionPassing is false retry makes ... attempts
    err := retry(attempts, time.Second, func() (err error) {
        if !complicatedFunctionPassing {
            return errors.New("somthing went wrong in the important function")
        }
        log.Println("Complicated function passed")
        return nil
    })
    if err != nil {
        log.Printf("failed after %d attempts with error: %s", attempts, err.Error())
    }
}

func retry(attempts int, sleep time.Duration, f func() error) (err error) {
    for i := 0; i < attempts; i++ {
        fmt.Println("This is attempt number", i+1)
        // calling the important function
        err = f()
        if err != nil {
            log.Printf("error occured after attempt number %d: %s", i+1, err.Error())
            log.Println("sleeping for: ", sleep.String())
            time.Sleep(sleep)
            sleep *= 2
            continue
        }
        break
    }
    return err
}

You can try it out here: https://go.dev/play/p/Ag8ObCb980U