How to test input that should fail for a custom GitHub action in a workflow?

Question

I have a custom GitHub action that encapsulates a linter using a Dockerfile. On push I want to verify that the linter works correctly, that is it should succeed on correct input and fail on incorrect input:

.github/workflows/test-action.yml

name: Test Action
 
on:
  workflow_dispatch:
  push:
    branches:
      - master
 
jobs:
  test-correct:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Lint correct input
        uses: ./
        with:
          file: should-succeed.ex
  test-incorrect:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Lint incorrect input
        uses: ./
        with:
          file: should-fail.ex

However on GitHub under e.g. https://github.com/myorg/myrepo/actions/runs/123456789 this will of course now color the successful job as green and the unsuccessful job as red. How can I tell GitHub to reverse the color so that a fail results in success and success results in failure?

Edit: I tried the following but it does not work, because then if: failure() will not trigger:

[...]
      - name: Lint incorrect input
        uses: ./
        continue-on-error: true
        with:
          file: should-fail.ex
      - if: failure()
        run: true
      - if: success()
        run: false

On the other hand if I remove the continue-on-error: true line, then it will not work either, because the whole job will be counted as failed even if I return true.

Answer 1

jobs:
  test-correct:
  [...]
  test-incorrect:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: This should fail
        id: lint-incorrect
        uses: ./
        continue-on-error: true
        with:
          file: should-fail.ex
      - name: Invert success and failure
        run: if [[ ${{ steps.lint-incorrect.outcome }} == "failure" ]]; then exit 0; else exit 1; fi

Using continue-on-error: true in combination with the outcome property, which does not change with continue-on-error, seems to do the trick. The only thing missing now is that I only want report exit code 2 (validation error) as success while keeping exit code 1 (runtime error) as error but this does not seem to be achievable with this method.

Answer 2

There are different levels of testing: system, integration, system.

Your approach - it's running a system test on GitHub. The obvious downside is that you can't test your action locally and you have a long feedback loop commiting and pushing every change to the repo, bloating it with wrong attempts.

I described my approach in the article. The main idea is:

• You can use unit tests with familiar tools according the technology you use to test separate units of your action
• For the thing you try to do in the question (testing the whole action) you can use https://github.com/cardinalby/github-action-ts-run-api TS library to run tests both locally and on CI (using your workflow). The library emulates GitHub runner and has API to pass different inputs, analyse outputs and exit code of your action without pushing it to the repo.