Selenium tests started to fail after test suite increased. Usually, tests are failing after reaching 10 passed tests, after that, all the rest tests are failed with no info that retry was triggered. All these tests are passing successfully locally, and I tried to mix them and create different order batches and still, 1st ten tests are passed in CI and the rest are failed. These failures are not happening every code push, it happens about once every 3 code pushes.
Tried to mix tests, run them locally many times to make sure there is no failures. I have changed the scope for webdriver instance due to memory issues in dockerized environment and now it runs each test in new environment with no data from previously run tests.
Also when these failures just started appearing I haven't seen it in logs, it was only 10 passed tests and message console log output exceeded timeout, so I added a parameter to circleci/config.yml
no_output_timeout: 20m
which started to show failures of the rest tests.
platform linux -- Python 3.6.4, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /usr/local/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.6.4', 'Platform': 'Linux-4.4.0-96-generic-x86_64-with', 'Packages': {'pytest': '4.3.1', 'py': '1.8.0', 'pluggy': '0.9.0'}, 'Plugins': {'reportportal': '1.0.4', 'parallel': '0.0.9', 'metadata': '1.8.0', 'html': '1.20.0', 'flaky': '3.5.3'}}
rootdir: /home/robot/functional/retail, inifile:
plugins: reportportal-1.0.4, parallel-0.0.9, metadata-1.8.0, html-1.20.0, flaky-3.5.3
collecting ...
collected 36 items
tests/ui_workflows/test_ecom_publish.py::TestEcomPublish::test_create_new_item_and_publish_on_ecom[firefox] SKIPPED
tests/ui_workflows/test_retail_create.py::TestRetailCreate::test_create_new_item[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
PASSED
tests/ui_workflows/test_retail_create.py::TestRetailCreate::test_create_new_customer_type[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
PASSED
tests/ui_workflows/test_retail_create.py::TestRetailCreate::test_create_new_sales_tax[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
PASSED
tests/ui_workflows/test_retail_import.py::TestRetailCreate::test_import_items_from_csv[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
PASSED
tests/ui_workflows/test_retail_import.py::TestRetailCreate::test_import_items_from_xlsx[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
PASSED
tests/ui_workflows/test_retail_import.py::TestRetailCreate::test_import_300_items_from_xlsx[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
PASSED
tests/ui_workflows/test_retail_item.py::TestRetailItem::test_create_item_with_all_attributes[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
PASSED
tests/ui_workflows/test_retail_item.py::TestRetailItem::test_create_non_taxable_item_and_make_sale[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
PASSED
tests/ui_workflows/test_retail_item.py::TestRetailItem::test_create_non_discountable_item_and_make_sale[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
PASSED
tests/ui_workflows/test_retail_item.py::TestRetailItem::test_create_non_discount_non_tax_item_and_make_sale[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
PASSED
tests/ui_workflows/test_retail_item.py::TestRetailItem::test_create_item_and_quick_edit_all_attr[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
ERROR
tests/ui_workflows/test_retail_item_matrix.py::TestRetailMatrix::test_create_color_size_attributes_matrix[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
ERROR
tests/ui_workflows/test_retail_item_matrix.py::TestRetailMatrix::test_add_4th_attr_color_size_attr_matrix[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
ERROR
tests/ui_workflows/test_retail_item_matrix.py::TestRetailMatrix::test_create_color_attributes_matrix[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
ERROR
tests/ui_workflows/test_retail_item_matrix.py::TestRetailMatrix::test_add_4th_color_attribute_matrix[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
ERROR
tests/ui_workflows/test_retail_item_matrix.py::TestRetailMatrix::test_create_size_attributes_matrix[firefox] Using Selenium grid at http://selenium-grid:4444/wd/hub
ERROR
==================================== ERRORS ====================================
_ ERROR at setup of TestRetailItem.test_create_item_and_quick_edit_all_attr[firefox] _
> lambda: ihook(item=item, **kwds),
when=when,
)
/usr/local/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py:306:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/ui_workflows/conftest.py:89: in driver
driver = webdriver.Remote(command_executor=selenium_grid_endpoint, desired_capabilities=capabilities)
/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py:156: in __init__
self.start_session(capabilities, browser_profile)
/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py:251: in start_session
response = self.execute(Command.NEW_SESSION, parameters)
/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py:320: in execute
self.error_handler.check_response(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x7f30b61d3d68>
response = {'status': 500, 'value': '{\n "sessionId": null,\n "status": 13,\n "value": {\n "class": "org.openqa.grid.common...\tat org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:112)\\n\\t... 38 more\\n"\n }\n}'}
def check_response(self, response):
"""
Checks that a JSON response from the WebDriver does not have an error.
:Args:
- response - The JSON response from the WebDriver server as a dictionary
object.
:Raises: If the response contains an error message.
"""
status = response.get('status', None)
if status is None or status == ErrorCode.SUCCESS:
return
value = None
message = response.get("message", "")
screen = response.get("screen", "")
stacktrace = None
if isinstance(status, int):
value_json = response.get('value', None)
if value_json and isinstance(value_json, basestring):
import json
try:
value = json.loads(value_json)
if len(value.keys()) == 1:
value = value['value']
status = value.get('error', None)
if status is None:
status = value["status"]
message = value["value"]
if not isinstance(message, basestring):
value = message
message = message.get('message')
else:
message = value.get('message', None)
except ValueError:
pass
exception_class = ErrorInResponseException
if status in ErrorCode.NO_SUCH_ELEMENT:
exception_class = NoSuchElementException
elif status in ErrorCode.NO_SUCH_FRAME:
exception_class = NoSuchFrameException
elif status in ErrorCode.NO_SUCH_WINDOW:
exception_class = NoSuchWindowException
elif status in ErrorCode.STALE_ELEMENT_REFERENCE:
exception_class = StaleElementReferenceException
elif status in ErrorCode.ELEMENT_NOT_VISIBLE:
exception_class = ElementNotVisibleException
elif status in ErrorCode.INVALID_ELEMENT_STATE:
exception_class = InvalidElementStateException
elif status in ErrorCode.INVALID_SELECTOR \
or status in ErrorCode.INVALID_XPATH_SELECTOR \
or status in ErrorCode.INVALID_XPATH_SELECTOR_RETURN_TYPER:
exception_class = InvalidSelectorException
elif status in ErrorCode.ELEMENT_IS_NOT_SELECTABLE:
exception_class = ElementNotSelectableException
elif status in ErrorCode.ELEMENT_NOT_INTERACTABLE:
exception_class = ElementNotInteractableException
elif status in ErrorCode.INVALID_COOKIE_DOMAIN:
exception_class = InvalidCookieDomainException
elif status in ErrorCode.UNABLE_TO_SET_COOKIE:
exception_class = UnableToSetCookieException
elif status in ErrorCode.TIMEOUT:
exception_class = TimeoutException
elif status in ErrorCode.SCRIPT_TIMEOUT:
exception_class = TimeoutException
elif status in ErrorCode.UNKNOWN_ERROR:
exception_class = WebDriverException
elif status in ErrorCode.UNEXPECTED_ALERT_OPEN:
exception_class = UnexpectedAlertPresentException
elif status in ErrorCode.NO_ALERT_OPEN:
exception_class = NoAlertPresentException
elif status in ErrorCode.IME_NOT_AVAILABLE:
exception_class = ImeNotAvailableException
elif status in ErrorCode.IME_ENGINE_ACTIVATION_FAILED:
exception_class = ImeActivationFailedException
elif status in ErrorCode.MOVE_TARGET_OUT_OF_BOUNDS:
exception_class = MoveTargetOutOfBoundsException
elif status in ErrorCode.JAVASCRIPT_ERROR:
exception_class = JavascriptException
elif status in ErrorCode.SESSION_NOT_CREATED:
exception_class = SessionNotCreatedException
elif status in ErrorCode.INVALID_ARGUMENT:
exception_class = InvalidArgumentException
elif status in ErrorCode.NO_SUCH_COOKIE:
exception_class = NoSuchCookieException
elif status in ErrorCode.UNABLE_TO_CAPTURE_SCREEN:
exception_class = ScreenshotException
elif status in ErrorCode.ELEMENT_CLICK_INTERCEPTED:
exception_class = ElementClickInterceptedException
elif status in ErrorCode.INSECURE_CERTIFICATE:
exception_class = InsecureCertificateException
elif status in ErrorCode.INVALID_COORDINATES:
exception_class = InvalidCoordinatesException
elif status in ErrorCode.INVALID_SESSION_ID:
exception_class = InvalidSessionIdException
elif status in ErrorCode.UNKNOWN_METHOD:
exception_class = UnknownMethodException
else:
exception_class = WebDriverException
if value == '' or value is None:
value = response['value']
if isinstance(value, basestring):
if exception_class == ErrorInResponseException:
raise exception_class(response, value)
raise exception_class(value)
if message == "" and 'message' in value:
message = value['message']
screen = None
if 'screen' in value:
screen = value['screen']
stacktrace = None
if 'stackTrace' in value and value['stackTrace']:
stacktrace = []
try:
for frame in value['stackTrace']:
line = self._value_or_default(frame, 'lineNumber', '')
file = self._value_or_default(frame, 'fileName', '<anonymous>')
if line:
file = "%s:%s" % (file, line)
meth = self._value_or_default(frame, 'methodName', '<anonymous>')
if 'className' in frame:
meth = "%s.%s" % (frame['className'], meth)
msg = " at %s (%s)"
msg = msg % (meth, file)
stacktrace.append(msg)
except TypeError:
pass
if exception_class == ErrorInResponseException:
raise exception_class(response, message)
elif exception_class == UnexpectedAlertPresentException:
alert_text = None
if 'data' in value:
alert_text = value['data'].get('text')
elif 'alert' in value:
alert_text = value['alert'].get('text')
raise exception_class(message, screen, stacktrace, alert_text)
> raise exception_class(message, screen, stacktrace)
E selenium.common.exceptions.WebDriverException: Message: Error forwarding the new session Request timed out waiting for a node to become available.
E Stacktrace:
E at org.openqa.grid.web.servlet.handler.RequestHandler.process (RequestHandler.java:118)
E at org.openqa.grid.web.servlet.DriverServlet.process (DriverServlet.java:85)
E at org.openqa.grid.web.servlet.DriverServlet.doPost (DriverServlet.java:69)
E at javax.servlet.http.HttpServlet.service (HttpServlet.java:707)
E at javax.servlet.http.HttpServlet.service (HttpServlet.java:790)
E at org.seleniumhq.jetty9.servlet.ServletHolder.handle (ServletHolder.java:865)
E at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1655)
E at io.prometheus.client.filter.MetricsFilter.doFilter (MetricsFilter.java:170)
E at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter (ServletHandler.java:1642)
E at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle (ServletHandler.java:533)
E at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:146)
E at org.seleniumhq.jetty9.security.SecurityHandler.handle (SecurityHandler.java:548)
E at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)
E at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:257)
E at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle (SessionHandler.java:1595)
E at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:255)
E at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle (ContextHandler.java:1340)
E at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:203)
E at org.seleniumhq.jetty9.servlet.ServletHandler.doScope (ServletHandler.java:473)
E at org.seleniumhq.jetty9.server.session.SessionHandler.doScope (SessionHandler.java:1564)
E at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:201)
E at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope (ContextHandler.java:1242)
E at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:144)
E at org.seleniumhq.jetty9.server.handler.StatisticsHandler.handle (StatisticsHandler.java:174)
E at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)
E at org.seleniumhq.jetty9.server.Server.handle (Server.java:503)
E at org.seleniumhq.jetty9.server.HttpChannel.handle (HttpChannel.java:364)
E at org.seleniumhq.jetty9.server.HttpConnection.onFillable (HttpConnection.java:260)
E at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:305)
E at org.seleniumhq.jetty9.io.FillInterest.fillable (FillInterest.java:103)
E at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run (ChannelEndPoint.java:118)
E at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.runTask (EatWhatYouKill.java:333)
E at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:310)
E at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.tryProduce (EatWhatYouKill.java:168)
E at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:126)
E at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:366)
E at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:765)
E at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:683)
E at java.lang.Thread.run (Thread.java:748)
/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py:242: WebDriverException
