1

I am trying to get the hidden elements in twitter login page. I followed a procedure which simply gets the hidden elements in that page. But the problem is when i try to get value of those elements, i am getting key error. the code is:

    import requests, lxml.html
from bs4 import BeautifulSoup
s = requests.session()
login = s.get('https://twitter.com/login')
login_html = lxml.html.fromstring(login.text)
hidden_inputs = login_html.xpath(r'//form//input[@type="hidden"]')
form = {x.attrib["name"]: x.attrib["value"] for x in hidden_inputs}
print(form)

I am getting error at x.attrib['value']. How to rectify this?

Akhil Reddy
  • 371
  • 1
  • 6
  • 26
  • `x.attrib.get('value')` or use a contition if you don't want `None` values. – t.m.adam Jan 25 '18 at 16:48
  • `print([x.attrib.keys() for x in hidden_inputs])`, before the line that throws the exception, to see what keys you can specify. Or to make it error proof `x.attrib.get("value", "not exist!!!")`. – CristiFati Jan 25 '18 at 16:49
  • The hidden fields do not have any value element which is why you are getting the error. – Subhrajyoti Das Jan 25 '18 at 16:54
  • @CristiFati That is _not_ error-proofing. That is error-obscuring. – erip Jan 25 '18 at 17:02

3 Answers3

1

Here is an example of (some) the objects you will get:

<InputElement 1a62c5ef778 name='ui_metrics' type='hidden'>

There is no "value" key.

If you print this:

for x in hidden_inputs:
     print(x.attrib)

Then you will be able to see which tags have values:

{'type': 'hidden', 'name': 'authenticity_token', 'value': '7fca6a14828cd425dad8437cc291687fc2ff1f96'}

So you will have to explicitly check for the ones that do have values

Matt Botha
  • 26
  • 1
  • 4
1

enter image description here

I use google devtools check twitter login page and get this image. The last 2 inputs either do not have value or it is not key value pair, so I guess that's why you got the error.

Haifeng Zhang
  • 30,077
  • 19
  • 81
  • 125
0

This error indicates that a member x of hidden_inputs is not a dictionary including the key "value". You should print out hidden_inputs to see its elements, and make sure they are dictionaries that include the key "value".

v2v1
  • 640
  • 8
  • 18