1

I'm writing an API that consists of several microservices. I have the code in a private Gitlab repo. I have a custom CI/CD pipeline configured to run a couple of different steps automatically on every commit to master (e.g. build, test, deploy to a dev environment). Deploying to prod is manual.

I have written some unit tests around this code, which naturally test only small units of the code. These, of course, are run with every commit, because if they fail, that means something in the code has broken.

I also have regression tests which we run after deploying. One of these is actually a bash script that uses curl to hit my production endpoint with certain parameters and checks to make sure that I'm getting 200 responses. I have parameterized this script so I can easily point it at my dev environment (instead of prod).

I use this regression test (and others like it) to check that my already-deployed service is functioning properly. And I run it right after deploying as a final, double-check to confirm that everything is working. But I want to automate that.

My question is where does this fit in a CI/CD workflow? It wouldn't make sense to run this kind of regression test on a commit, because that commit is not necessarily coupled with a deploy. And because there are any number of reasons why the service might be down that are unrelated to whatever code changes went into the most recent commit. In other words, the pipeline should not fail because of external circumstances.

Are there any best practices for running and automating regressions tests?

soapergem
  • 9,263
  • 18
  • 96
  • 152
  • 1
    By the way: curling an endpoint to check if it responds with 200 isn’t a regression test but merely a smoke test. Regression tests are ones that check the behavior: unit tests, integration tests, E2E tests. – Mike Doe Sep 13 '19 at 17:59
  • @emix fair enough! That was just one of them anyway. I have others that actually evaluate the data and more complex interactions between the services – soapergem Sep 13 '19 at 18:01

1 Answers1

1

Great question. There are a couple of interesting points here.

  1. When to run the regression tests (as they exist today) in your CI / CD environment.

The obvious answer to this is to run as a post deploy step. Using the same approach you are currently using to limit the deploy step to the master branch only you can limit this post deploy step to the master branch only.

If you add more details about your environment. For example the CI / CD system that you are using and your current configuration I would be very happy to provide more concrete details on how to achieve this.

  1. It wouldn't make sense to run this kind of regression test on a commit

An interesting approach that I have seen a couple of times. Is using a cloud service (AWS / GCloud etc.) to spin up an environment on each CI run. This means that the full pipeline can be run for every commit. While it takes more resources, it means that you can find issues prior to merging to master. Of course up to you whether the ROI adds up in your environment.

James Wilson
  • 1,541
  • 7
  • 20