Travis CI : How to allow failures with a customized environment variable?

Question

Following this answer, I wrote this Travis configuration file :

language: php

php:
    - 5.3
    - 5.4
    - 5.5
    - 5.6
    - 7
    - hhvm
    - nightly

branches:
    only:
        - master
        - /^\d+\.\d+\.\d+$/

matrix:
    fast_finish: true
    include:
        - php: 5.3
          env: deps="low"
        - php: 5.5
          env: SYMFONY_VERSION=2.3.*
        - php: 5.5
          env: SYMFONY_VERSION=2.4.*
        - php: 5.5
          env: SYMFONY_VERSION=2.5.*
        - php: 5.5
          env: SYMFONY_VERSION=2.6.*
        - php: 5.5
          env: SYMFONY_VERSION=2.7.*
        - php: 5.5
          env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail
    allow_failures:
        - php: nightly
        - env: TEST_GROUP=canFail

before_script:
    - composer self-update
    - if [ "$SYMFONY_VERSION" != "" ]; then composer require --dev --no-update symfony/symfony=$SYMFONY_VERSION; fi
    - if [ "$deps" = "low" ]; then composer update --prefer-lowest; fi
    - if [ "$deps" != "low" ]; then composer update --prefer-source; fi

script: phpunit

But Travis CI counts only the php nightly version as an "allowed to fail" version. Am I using the environment variables in a wrong way ?

UPDATE

Just a precision, I know that I can directly write the environment like that:

matrix:
    include:
        - php: 5.5
          env: SYMFONY_VERSION=2.8.*@dev
    allow_failures:
        - env: SYMFONY_VERSION=2.8.*@dev

but still I don't get why the other way doesn't work.

It doesn't work the other way because Travis CI doesn't extrapolate the information. Therefore you have to exactly match the env for each php version the matrix builds. (I think they have a feature request, but it's very low priority for them) — Walt Sorensen, Nov 18 '16 at 05:07

Walt Sorensen · Accepted Answer · 2016-11-14T05:43:53.043

What you specify in allow_failures: is your allowed failures

"You can define rows that are allowed to fail in the build matrix. Allowed failures are items in your build matrix that are allowed to fail without causing the entire build to fail. This lets you add in experimental and preparatory builds to test against versions or configurations that you are not ready to officially support."

Unfortunately, I believe the way the matrix reads your first set of code is as php nightly version as the "allowed to fail" version with the environment as part of nightly.

Because of how Travis allows failures it must be an exact match, you can not just specify env: as an allowed failure you have to specify for each php version with the env: you want to allow as a failure for example

allow_failures:
  - php: 5.3
    env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail
  - php: 5.4
    env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail
  - php: 5.5
    env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail
  - php: 5.6
    env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail
  - php: 7.0
    env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail
  - php: hhvm
    env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail
  - php: nightly # Allow all tests to fail for nightly

score 1 · Answer 2 · answered Oct 05 '16 at 08:33

According to this issue, the php and env keys must match perfectly. env can be either a single value or an array, but in both case it must be a perfect match. So if you want your build:

- php: 5.5
  env: SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail

to be allowed to fail, you have to either give the whole env key SYMFONY_VERSION=2.8.*@dev TEST_GROUP=canFail and the whole env key and the PHP version (if you had the same env key for different PHP versions).

Travis CI : How to allow failures with a customized environment variable?

2 Answers2