I would like to have TravisCI or some other GitHub application automatically modify pull requests based on a script that I provide. How can I do this? (Or maybe there is a better solution than what I've come up with.)
Reason:
I'm trying to add integration testing to a GitHub project so that it will automatically run the project and compare the output against known good outputs. If there is a difference, it should report an error.
However, my tests run on multiple platforms and use different versions of libraries. Different combinations of platform and library cause different outputs, all valid. For example, an old version of a library might make a different optimization than the newer one so the outputs are different but both valid.
Because of the difficulty of generating output for every possible combination of libraries, I thought that TravisCI could generate them and add them to the pull request on the PR author's behalf.
Or a totally different solution to system testing would be fine, too. Thanks!
A more concrete example:
The project that I'm working on is pcb2gcode. It converts circuit board layouts (gerbers) to instructions for a CNC machine (g-codes) to mill and drill a board. Depending on the version of some libraries, like boost and gerbv, the output might be slightly different but still valid.
I want to have a directory of test gerbers that are automatically run and compared against expected outputs, which are committed into github already. If the actual output matches the expected output, the test passes.
But the expected output depends on the libraries. One solution is to have expected outputs for different combinations of libraries. Then the comparison is done against the correct set of expected outputs.
Generating those expected outputs is difficult because developers would need to run the code with all combinations of libraries. TravisCI is running the tested and generating outputs anyway so it might as well also write those new expected outputs into the PR. Then all tests will pass and the expected output can be inspected along with the rest of the PR during code review.
Or some other better method of system testing!
