I am working on test suite and it's now passing most of the time but not always! Here is a sample of 5 very basic tests (index related tests). In those tests, the two lasts ones (assert url tests) are randomly passing (sometimes both fails, sometimes both succeed, sometime one works and the other fails). I can't find any pattern in the way they pass/fail to re-produce the working conditions, even in debug mode. My first though was a race condition, but it seems like it's not, since the problem persists even with wait() timers. So I though, maybe my implementation of the test/understanding of casper is not correct.
Feel free to give me feedback on the implementation of this test, or any potential improvement allowing the last two tests to pass 100% of the time. Current MMVC framework is backbone and tests are compiled before running, I can provide à compiled (js) version of the test if needed.
Test:
{t} = require '../helpers'
{baseUrl} = require '../config'
mouse = require('mouse').create(casper)
casper.options.loadImages = false
casper.options.verbose = true
casper.options.logLevel = 'debug'
casper.test.begin 'Home page loads correctly', 3, (test) ->
casper.start "#{baseUrl}page/log-out"
casper.thenOpen baseUrl, ->
test.assertHttpStatus 200, 'Status code is 200'
casper.then ->
test.assertTitle t('siteName'), "Title is '#{t 'siteName'}'"
casper.then ->
test.assertTextExists t('home:featureWhatIsBitcoinTitle'), 'Page body contains intro text'
casper.run ->
test.done()
casper.test.begin 'Home page links to sign up page', 1, (test) ->
casper.start "#{baseUrl}page/log-out"
casper.thenOpen baseUrl, ->
@evaluate -> document.querySelector('#signup').click()
casper.then ->
test.assertUrlMatch /\/page\/sign-up/, 'Url matches sign up page'
casper.run ->
test.done()
casper.test.begin 'Home page links to log in page', 1, (test) ->
casper.start "#{baseUrl}page/log-out"
casper.thenOpen baseUrl, ->
@evaluate -> document.querySelector('a[href="/page/log-in"]').click()
casper.then ->
test.assertUrlMatch /\/page\/log-in/, 'Url matches log in page'
casper.run ->
test.done()
Stack trace:
casperjs test tests/integration/compiled/tests/home.js
Test file: tests/integration/compiled/tests/home.js
# Home page loads correctly
[info] [phantom] Starting...
[info] [phantom] Running suite: 6 steps
PASS Home page loads correctly (3 tests)
[debug] [phantom] opening url: http://something.dev/page/log-out, HTTP GET
[debug] [phantom] Navigation requested: url=http://something.dev/page/log-out, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "http://something.dev/page/log-out"
[debug] [phantom] Navigation requested: url=https://platform.twitter.com/widgets/follow_button.html?screen_name=something&show_screen_name=false, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] Navigation requested: url=https://www.facebook.com/plugins/like.php?href=https%3A%2F%2Fwww.facebook.com%2Fsomething&width&layout=button_count&action=like&show_faces=false&share=false&height=35&appId=000000000000, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] Navigation requested: url=https://www.google.com/recaptcha/api/fallback?k=LOLILOLILOLILOLILOLILOLILOLILOL&hl=en&v=r000000000&t=0&ff=true, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] Successfully injected Casper client-side utilities
[debug] [phantom] start page is loaded
[debug] [phantom] opening url: http://something.dev/, HTTP GET
[debug] [phantom] Navigation requested: url=http://something.dev/, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "http://something.dev/"
[debug] [phantom] Successfully injected Casper client-side utilities
[info] [phantom] Step anonymous 4/6 http://something.dev/ (HTTP 200)
PASS Status code is 200
[info] [phantom] Step anonymous 4/6: done in 3149ms.
[info] [phantom] Step anonymous 5/6 http://something.dev/ (HTTP 200)
PASS Title is 'NVM | SOMETHING'
[info] [phantom] Step anonymous 5/6: done in 3469ms.
[debug] [phantom] Navigation requested: url=https://platform.twitter.com/widgets/follow_button.html?screen_name=something&show_screen_name=false, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] Navigation requested: url=https://www.facebook.com/plugins/like.php?href=https%3A%2F%2Fwww.facebook.com%2Fsomething&width&layout=button_count&action=like&show_faces=false&share=false&height=35&appId=000000000000, type=Other, willNavigate=true, isMainFrame=false
[info] [phantom] Step anonymous 6/6 http://something.dev/ (HTTP 200)
PASS Page body contains intro text
[info] [phantom] Step anonymous 6/6: done in 3666ms.
[debug] [phantom] Navigation requested: url=about:blank, type=Other, willNavigate=true, isMainFrame=false
[info] [phantom] Done 6 steps in 3685ms
# Home page links to sign up page
[info] [phantom] Starting...
[info] [phantom] Running suite: 5 steps
PASS Home page links to sign up page (1 test)
[debug] [phantom] opening url: http://something.dev/page/log-out, HTTP GET
[debug] [phantom] Navigation requested: url=http://something.dev/page/log-out, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "http://something.dev/page/log-out"
[debug] [phantom] Successfully injected Casper client-side utilities
[debug] [phantom] start page is loaded
[debug] [phantom] opening url: http://something.dev/, HTTP GET
[debug] [phantom] Navigation requested: url=http://something.dev/, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] Navigation requested: url=https://platform.twitter.com/widgets/follow_button.html?screen_name=something&show_screen_name=false, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] Navigation requested: url=https://www.facebook.com/plugins/like.php?href=https%3A%2F%2Fwww.facebook.com%2Fsomething&width&layout=button_count&action=like&show_faces=false&share=false&height=35&appId=000000000000, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] url changed to "http://something.dev/"
[debug] [phantom] Successfully injected Casper client-side utilities
[info] [phantom] Step anonymous 4/5 http://something.dev/ (HTTP 200)
[info] [phantom] Step anonymous 4/5: done in 524ms.
[info] [phantom] Step anonymous 5/5 http://something.dev/ (HTTP 200)
FAIL Url matches sign up page
# type: assertUrlMatch
# file: tests/integration/compiled/tests/home.js
# subject: false
# currentUrl: "http://something.dev/"
# pattern: "/\\/page\\/sign-up/"
[info] [phantom] Step anonymous 5/5: done in 651ms.
[debug] [phantom] Navigation requested: url=https://platform.twitter.com/widgets/follow_button.html?screen_name=something&show_screen_name=false, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] Navigation requested: url=https://www.facebook.com/plugins/like.php?href=https%3A%2F%2Fwww.facebook.com%2Fsomething&width&layout=button_count&action=like&show_faces=false&share=false&height=35&appId=000000000000, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] Navigation requested: url=about:blank, type=Other, willNavigate=true, isMainFrame=false
[info] [phantom] Done 5 steps in 866ms
# Home page links to log in page
[info] [phantom] Starting...
[info] [phantom] Running suite: 5 steps
PASS Home page links to log in page (1 test)
[debug] [phantom] opening url: http://something.dev/page/log-out, HTTP GET
[debug] [phantom] Navigation requested: url=http://something.dev/page/log-out, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "http://something.dev/page/log-out"
[debug] [phantom] Successfully injected Casper client-side utilities
[debug] [phantom] start page is loaded
[debug] [phantom] opening url: http://something.dev/, HTTP GET
[debug] [phantom] Navigation requested: url=http://something.dev/, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] Navigation requested: url=https://platform.twitter.com/widgets/follow_button.html?screen_name=something&show_screen_name=false, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] Navigation requested: url=https://www.facebook.com/plugins/like.php?href=https%3A%2F%2Fwww.facebook.com%2Fsomething&width&layout=button_count&action=like&show_faces=false&share=false&height=35&appId=000000000000, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] url changed to "http://something.dev/"
[debug] [phantom] Successfully injected Casper client-side utilities
[info] [phantom] Step anonymous 4/5 http://something.dev/ (HTTP 200)
[debug] [phantom] url changed to "http://something.dev/page/log-in"
[info] [phantom] Step anonymous 4/5: done in 717ms.
[debug] [phantom] Navigation requested: url=https://platform.twitter.com/widgets/follow_button.html?screen_name=something&show_screen_name=false, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] Navigation requested: url=https://www.facebook.com/plugins/like.php?href=https%3A%2F%2Fwww.facebook.com%2Fsomething&width&layout=button_count&action=like&show_faces=false&share=false&height=35&appId=000000000000, type=Other, willNavigate=true, isMainFrame=false
[info] [phantom] Step anonymous 5/5 http://something.dev/page/log-in (HTTP 200)
PASS Url matches log in page
[info] [phantom] Step anonymous 5/5: done in 761ms.
[debug] [phantom] Navigation requested: url=https://www.google.com/recaptcha/api/fallback?k=LOLILOLILOLILOLILOLILOLILOL=en&v=r000000000&t=0&ff=true, type=Other, willNavigate=true, isMainFrame=false
[debug] [phantom] Navigation requested: url=about:blank, type=Other, willNavigate=true, isMainFrame=false
[info] [phantom] Done 5 steps in 949ms
FAIL 5 tests executed in 5.344s, 4 passed, 1 failed, 0 dubious, 0 skipped.
Details for the 1 failed test:
In tests/integration/compiled/tests/home.js
Home page links to sign up page
assertUrlMatch: Url matches sign up page
[debug] [phantom] Navigation requested: url=about:blank, type=Other, willNavigate=true, isMainFrame=true
[debug] [phantom] url changed to "about:blank"
