If I test a Pull Request build, do I need to run the same tests after merging?

Question

I'm using Travis CI that has Pull Request builds and Branch builds. I'm sure this is common to other CI services.

If I have a develop branch and a feature/A branch then when I open a pull request from feature/A targeting develop, the pull request build runs my unit tests.

Assuming my unit tests pass, I merge the pull request and a branch build triggers because of the new commit made to develop. This branch build builds my container and deploys it to a develop environment.

Should I be running the same suite of unit tests on my branch build that I did during the pull request build or is it safe to assume that because the pull request tests passed, the branch build would too? Would running these tests be wasting cycles?

Answer 1

This is actually a good question and not uncommon at all, in my experience is a good idea to run the tests again, unless you are the only one making pull requests to that branch, let's say your tests runs correctly but someone made a pull request right before you, and your feature-branch is not up to date with that code, is possible that some of the code merged into the develop-branch affect some of the flows you already tested, and that may cause a fail/misbehavior on your code. One way to avoid the push of these misbehaviors to a live environment, is to run the tests again right after accepting a pull request, this will add some cycles to your pipeline but I think it could mitigate hotfixes/issues.

Answer 2

In fact it end up with a matter of trade-off...

There are a lot of variables that make that in the end that's you that will decide where to put the limit (and later perhaps push it further).

You should understand that you won't be able to find and fix all the bugs and problems with your CI workflow so you will have to find the good way to have the expected quality.

It depends on:

• what level of quality you expect in your master branch and so the value you give to an early feedback.
• what it will cost you to achieve that.

If you use GitHub, the service simulate a merge of your PR with the target branch and that's what you normally build. So, from this point of view, you don't need to rebuild and test your target branch after the merge.

But there are some corner cases...

If another PR is merged after the last commit pushed in your PR, the PR is not re-built (GitHub just verify if that's still mergeable).

Here you have two solutions:

1. You set as mandatory to sync (through a rebase or sync merge) the PR before merging it => No need to rebuild the target branch.
2. No mandatory => You need to build the target branch. That will be your "integration build".

From a quality point of view, it's better to build the target branch because it is doing a clean build and there are sometimes corner cases that you don't catch in the PR build. If you need to avoid these corner cases because you must have a very high quality product because, for example, lives or high amount of money, or whatever the reason depends on the output the build, you should build it again.

So as you see, it all depend of your CI workflow, quality expected and cost to put it in place.

Side note: Most of the time it's still useful to build the target branch to create clean artifacts (so the build that you do in PR is different than the one of the target branch). So it still a good practice...