0

We are calling an API within ActiveBatch with the following PowerShell.

$refreshResponse = Invoke-RestMethod -Uri $psApiUri -Headers $authHeaders -Method 'GET'

When this is used to call the dev environment version of the endpoint it works fine. However in production after five minutes we are seeing a second, duplicate call happening against the API. It happens every time in production after five minutes, there is nothing in the output that indicates the second call happened.

The only difference spotted so far between the two environments is that Prod is load balanced. What could potentially cause this to happen in only one of our environments.

It might not be related but also in prod only, after 10 minutes the call fails with the following error. Many of the runs are 8-9 minutes so we don't get the failure, but still the double call.

Invoke-RestMethod : The underlying connection was closed: An unexpected error occurred on a receive.

MrTCS
  • 177
  • 2
  • 8
  • What makes you think that its PowerShell/`Invoke-RestMethod` making the second call? Do they stop appearing in your logs after 5 minutes if you run your script from the interactive prompt and then exit the process immediately? – Mathias R. Jessen Oct 05 '22 at 14:45
  • 2
    One thing you can be pretty sure of is that a single call to `Invoke-RestMethod` isn't doing this. That does have a retry mechanism (`-RetryIntervalSec`, `-MaximumRetryCount`) but only if you ask for it, and way shorter than 5 minutes. That leaves the possibility of this line being called more than once when you're not expecting it (for example, the entire script might be retried by the caller), and problems elsewhere in the stack. – Jeroen Mostert Oct 05 '22 at 14:45
  • We are basing it off of the logging from the API, shows the same account for both calls, and this account is only used here. We thought at first it could be ActiveBatch running it twice but there is no additional logging data to show that and not sure why it would only happen in one environment then. When the team manually kicked it off from swagger there is no double run. We can only run once a day, so tomorrow I'm running it interactively from the ActiveBatch server to see if the same thing happens to eliminate ActiveBatch as a cause. – MrTCS Oct 05 '22 at 15:12
  • @MathiasR.Jessen I manually ran the script this morning from just a cmd window and at exactly five minutes in there was a duplicate call. We can see from some network monitoring that the second call is indeed coming from the server I executed the script on. We are going to try again tomorrow morning but with me closing out the window as soon as I execute the Invoke-RestMethod call. – MrTCS Oct 06 '22 at 12:04
  • We were able to reproduce this issue against our staging environment, which is also load balanced. We tested making the invoke call and then immediately closing the cmd window, and no duplicate calls then. – MrTCS Oct 06 '22 at 14:28

0 Answers0