2

I occasionally get the above error when making requests with the python requests library to Qualtrics APIs.

In a nutshell, I have a Google Cloud Function on Google Cloud that will trigger when a csv file is placed on a specific Cloud Storage Bucket. The function will create a Qualtrics distribution list on Qualtrics, upload the contacts and then download the distribution links. Every day, three files are uploaded on Cloud Storage, each for a different survey, and so three Google Cloud instances will be started.

My gripes with the issues are:

  • it doesn't happen regularly, in fact the workflow correctly worked for the past year
  • it doesn't seem to be tied to the processed files: when the function crashes and I manually restart it by reuploading the same csv into the bucket, it will work smoothly

The problem started around when we added the third daily csv to the process, and tend to happen when two files are being processed at the same time. For these reasons my suspects are:

  • I'm getting rate limited by Qualtrics (but I would expect a more clear message from Qualtrics)
  • The requests get in some way "crossed up" when two files are processed. I'm not sure if requests.request implicitly opens a session with the APIs. In that case the problem could be generated by multiple sessions being open at the same time from two csv being processed at the same time

As I said, the error seem to happen without a pattern, and it has happened on any part of the code where I'm doing a request to the APIs, so I'm not sure if sharing extensive code is helpful, but in general the requests are performed in a pretty standard way:

requests.request("POST", requestUrl, data=requestPayload, headers=headers)

requests.request("GET", requestUrl, headers=headers) etc

i.e.: I'm not using any particular custom option

Mich R
  • 111
  • 5
  • Hey there, from your question I understand you're using a Cloud Function to interact with Qualtrics' API, specifically to make distribution lists based on three files which contain contacts that are then uploaded. Please let me know if there's something I've misunderstood. As far as Cloud Functions goes, it will act as any other client when it comes to egress messages (it's different inside G.Cloud since it's internal), so I would like to know how you're interacting with Qualtrics, as though the problem may not be with Qualtrics, knowing this will help find the root cause. – fabc Oct 07 '21 at 13:25
  • Yes, you're correct about the architecture. During a run, the cloud function will perform various operations (creating the contact list, parsing the csv, etc) and then it will cycle through the csvs and upload the contacts on qualtrics one by one. This phase is where i usually get the error, which will happen seemingly randomly for single contacts. The upload is done through a POST request on `https://fra1.qualtrics.com/API/v3/mailinglists/{mailinglistid}/contacts` The request includes a json payload containing the customer info. – Mich R Oct 12 '21 at 09:48

1 Answers1

1

In the end I kind of resolved the issue with a workaround:

  • I separated the processing of the three csv so that there is no overlap in processing time between two files
  • implemented a retry policy in the POST request

Since then, separating processing time for the files reduced substantially the number of errors (from one or more each day to around 1 error a week), and even when they happen the retry policy circumvents the error at the first retry. I realize this may not be the ideal solution, so I'm open to alternatives if someone comes up with something better (or even more insights on the root problem).

Mich R
  • 111
  • 5
  • Pardon me, but I'm curious: how did you separate the 3 csv processes? Completely separating them or separating them only when sending the post (in different threads rather than instances of a function)? – fabc Oct 14 '21 at 14:06
  • I completely separated them by processing each file at a different time. Basically the Cloud Function invocation that processes file 2 starts way after file 1 is done, and file 3 is processed way after file 2 is done. This because a couple of times the error also happened on other parts of the process (e.g.: when creating the contact list) and also because I admit I'm not that well versed in multi-threading :) – Mich R Oct 15 '21 at 08:08