0

We have an application that is 3 months into development worked on by 2 developers. The website is relatively small and only 1/3rd completed, but the tests are reasonably comprehensive.

Running Protractor on our dev machines takes 60 minutes when running in serial, and under 40 minutes when running sharded with 2 concurrent sessions.

Running Protractor against BrowserStack with 6 compatibilities over 2 concurrent sessions takes under 4 hours.

Extrapolating these test times for the completed product (i.e. 4 hours + 200%) implies a total test time of around 12 hours. That exceeds the 120 min limit on travis-ci.com and puts us into the $999/mo tier on Browserstack. This also means that, at best, any change (including bug fixes) will take 12 hours to appear live.

Do other people encounter these same problems, and if so, how do they solve them?

Lawrence Wagerfield
  • 6,471
  • 5
  • 42
  • 84

2 Answers2

0

In my opinion either you should change your testing strategy and start releasing new features/bugs just by running sanity suite or you can pick most very few most important platforms to run entire test to release. I know this is not an ideal way to release features but at least it will give you pretty much confidence to go ahead.

Second option use multiple jobs in CI tool to run tests on different platforms and if there is any financial limitation then I'd say lets have a local CI tool setup. Probably you can go with Jenkins.

Priyanshu
  • 3,040
  • 3
  • 27
  • 34
  • In your suggested new release strategy, would you still have the entire suite run at some point? If so, when / how often? – Lawrence Wagerfield Mar 11 '15 at 17:51
  • I'd say go with sanity option only when there is something on high priority to release and you want to test that feature on all platforms. Ideally sanity suite should not take more than 10% of entire suite execution time. I'd always go with option to run entire test only on important platforms and when it comes to run entire test on all platform that should happen every night. We configured it in our CI tool to run test everyday 6 PM which finishes next morning by 6-7 AM so we get complete report when we reach to office. – Priyanshu Mar 11 '15 at 17:58
0

We have just finished such kind of CI implementation (TFS + Jenkins). We also use Browserstack and we have about 150 e2e tests for out last project. It also takes quite big amount of time to execute these tests. So what we did we decided to execute these tests not after each commit, but on daily basis as nightly job. In the morning it is possible to review results and looks if smth goes wrong/why build was failed. We use the cheapest plan and we have 4 different config files (chrome/safari/firefox/iOS), each config file has 4 browser/platform settings. Actually a little tricky stuff: even if you have cheapest browserstack plan you are able to execute 4 tests in parallel without any problems.

Btw we used Git + Bitbucket + Babmoo and it was much more user friendly than TFS + Jenkins

Sergey Teplyakov
  • 603
  • 8
  • 18
  • This is Mukul from BrowserStack. We have recently released a [Jenkins Plugin](https://wiki.jenkins-ci.org/display/JENKINS/BrowserStack+Plugin) that allows you to integrate your Jenkins CI server with BrowserStack easily. Hope this helps! – mukulg Jul 19 '16 at 11:25