golang's hystrix library "circuit open" without "timeout" error

Question

we use hystrix for our golang application, here we are getting hystrix: circuit open error even though there is no hystrix: timeout

hystrix configuration:

hystrix.ConfigureCommand(cmd, hystrix.CommandConfig{
        Timeout:               timeout,
        MaxConcurrentRequests: 1000,
        ErrorPercentThreshold: 5,
        SleepWindow:           15 * 60 * 1000, //15 Minutes
    })

in the hystrix fallback section,we log the error messages information. we can clearly see that we have got only hystrix: circuit open error without any other error

and sometimes it behaves very randomly, in below image we can see there is no correlation between hystrix: timeout and hystrix: circuit open

sudo/sample hystrix code:

func crawl(vendor string, req *http.Request, timeout int) (result []byte) {

    hystrix.Do(vendor, func() error {

        resp, err := httpClient.Do(req)
        if err != nil {
            log.Errorln("Error sending post request: ", err)
        } else {
            defer resp.Body.Close()
        }
        respBody, errResp := ioutil.ReadAll(resp.Body)
        if errResp != nil {
            log.WithFields(log.Fields{"RequestID": requestID}).Errorln("Error reading parser response", errResp, resp.Status)
        }

        if resp.StatusCode == 200 {
            result = respBody
        } else {
            log.Errorln(" SERVER SIDE ERROR", resp.StatusCode, obj)
        }

        return nil
    }, func(err error) error {
        logApiTimeouts(vendor, err)
        log.WithFields(log.Fields{"RequestID": requestID}).Errorln("Hystrix Error:", err, obj)
        return nil
    })
}

has anybody came across this error and how to fix this?

score 1 · Answer 1 · answered Aug 26 '20 at 02:01

Might be because of the requestVolumeThreshold which by default is set to 20.

Per documentation from the hystrix golang client,

// DefaultVolumeThreshold is the minimum number of requests needed before a circuit can be tripped due to health
DefaultVolumeThreshold = 20

You can set it with,

// Settings returns the hystrix command config
func (c Config) Settings() hystrix.CommandConfig {
   return hystrix.CommandConfig{
      Timeout:                8000,
      RequestVolumeThreshold: 1000, // FIXME: all calls must be paginated
   }
}

Adjusting it to a certain number greater than the default fixed my error.

golang's hystrix library "circuit open" without "timeout" error

1 Answers1