XPath trouble with Selenium

Question

I guess it is one of those eternal questions, but I need some assistance with an XPath-expression. The HTML searched with Selenium looks like this:

<div class="container">
  <div class"row">
    <div class="col-md-6 col-md-offset-3 jumbotron">
      <div class="text-center">
        <h1>Start a new To-Do list</h1>
        <form method="POST" action="/lists/new">
          <input name="item_text" id="id_new_item"
            class="form-control input-lg"
            placeholder="Enter a to-do item" />
          <input type="hidden" name="csrfmiddlewaretoken" value="***********">
          <div class="form-group has-error">
            <span class="help-block">You can&#39;t have an empty list item</span>
          </div>    
        </form>
      </div>
    </div>
  </div>
</div>

The search expression in Python looks like this:

self.wait_for(lambda: self.assertEqual(
    self.browser.find_element_by_xpath(
        "//span[contains(text(), 'You can&#39;t have an empty list item')]"
        )
    )
)

This is run within a test, and it cannot locate the text even though it is obviously there. The ttaceback from the test is:

ERROR: test_cannot_add_empty_list_items (functional_tests.test_list_item_validation.ItemValidationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/eric/Git/TDD/functional_tests/test_list_item_validation.py", line 15, in test_cannot_add_empty_list_items
    self.wait_for(lambda: self.assertEqual(
  File "/home/eric/Git/TDD/functional_tests/base.py", line 40, in wait_for
    raise e
  File "/home/eric/Git/TDD/functional_tests/base.py", line 37, in wait_for
    return fn()
  File "/home/eric/Git/TDD/functional_tests/test_list_item_validation.py", line 17, in <lambda>
    "//span[contains(text(), 'You can&#39;t have an empty list item')]"
  File "/home/eric/Git/TDD/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 394, in find_element_by_xpath
    return self.find_element(by=By.XPATH, value=xpath)
  File "/home/eric/Git/TDD/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
    'value': value})['value']
  File "/home/eric/Git/TDD/venv/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/eric/Git/TDD/venv/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: //span[contains(text(), 'You can&#39;t have an empty list item')]


----------------------------------------------------------------------
Ran 4 tests in 34.851s

FAILED (errors=1)

EDIT: The assertion should be assertTrue instead of assertEqual, as I am not comparing the result to anything.

As a first step in debugging, simplify your xpath expression to look for a span element with `class='help-block'`, or even simpler, any span element. — John Gordon, Jan 31 '19 at 18:29

Tomalak · Accepted Answer · 2019-01-31T19:29:10.623

There is no ' in your HTML document. There is a '.

The ' only informs the HTML parser to insert a single quote into the document tree at this position, it does not actually end up as something you could search for.

You can do this:

self.wait_for(lambda: self.assertEqual(
    self.browser.find_element_by_xpath(
        '//span[contains(text(), "You can\'t have an empty list item")]'
        )
    )
)

but this only works as long as the quotes are exactly this way. When your search text contains a double quote, the above breaks and you must escape things the other way around. That's feasible as long as the search text is predefined.

As long as the resulting XPath is valid, you're good to go. In this case, the above results in this perfectly valid XPath expression:

//span[contains(text(), "You can't have an empty list item")]

But if the search text is variable (e.g. user-defined) then things get hairy. Python knows string escape sequences, you can always use \" or \' to get a quote into a string. XPath knows no such thing.

Assume the search text is given as You can't have an "empty" list item. This is easy to generate with Python, but it won't work:

//span[contains(text(), "You can't have an "empty" list item")]
-------------------------------------------^ breaks here

and this XPath won't work either:

//span[contains(text(), 'You can't have an "empty" list item')]
--------------------------------^ breaks here

and neither will this one, because XPath has no escape sequences:

//span[contains(text(), 'You can\'t have an "empty" list item')]
---------------------------------^ breaks here

What you can do in XPath to work around this is concatenate differently quoted strings. This:

//span[contains(text(), concat('You can', "'" ,'t have an "empty" list item'))]

is perfectly valid and will search for the text You can't have an "empty" list item.

And what you can do in Python is create this very structure:

split the search string at '
join the parts with ', "'", '
prepend concat(', append ')
insert into the XPath expression

The following would allow a string search that can never throw a run-time error because of malformed XPath:

search_text = 'You can\'t have an "empty" list item'

concat_expr = "', \"'\", '".join(search_text.split("'"))
concat_expr = "concat('" + concat_expr + "')"

xpath = "//span[contains(text(), %s)]" % concat_expr

xpath, as a Python string literal (what you would see when you print it to the console):

'//span[contains(text(), concat(\'You can\', "\'", \'t have an "empty" list item\'))]'

The way the XPath engine gets to see it (i.e. the actual string in memory):

//span[contains(text(), concat('You can', "'", 't have an "empty" list item'))]

The lxml library allows XPath variables, which is a lot more elegant than that, but I doubt that Selenium's find_elements_by_xpath supports them.

score 0 · Answer 2 · answered Jan 31 '19 at 20:02

@Tomalak answer gives us a great insight on text() of xpath. However, as you are using find_element_by_xpath() you will be at ease clubbing up the class attribute and you can use the following xpath based solution:

self.wait_for(lambda: self.assertEqual(
    self.browser.find_element_by_xpath(
    "//span[@class='help-block' and contains(., 'have an empty list item')]"
    )
  )
)

XPath trouble with Selenium

2 Answers2