0

My code in Python3:

import requests

URL = "http://xxx.xxx.com/querytext"

try:
    text = urllib.parse.quote_plus(text)
    r = requests.get(URL, params={"text": text}, auth=("guest", "1234"))
    r_json = r.json()
except Exception:
    return None

tag_results = r_json["data"]["result"]

My 'text' may contain some special characters so I want to do url encoding, as shown above. However, for one testing example, if I don't use this line:

text = urllib.parse.quote_plus(text)

I can get the expected result. Otherwise, I can't. So i suspect my way of using url encoding is wrong. What's wrong exactly?

One example of the text could be '#Thisisgreat#, yes!'

marlon
  • 6,029
  • 8
  • 42
  • 76

2 Answers2

1

Requests does the encoding for you, so you don't need to do it manually. So you should remove the line text = urllib.parse.quote_plus(text).

Jack Taylor
  • 5,588
  • 19
  • 35
  • In the example, 'You can see that the URL has been correctly encoded by printing the URL:' there is no special characters. So I am not sure whether it needs additional encoding for special characters. – marlon Sep 15 '21 at 04:19
  • The encoding that Requests does includes encoding special characters, so there is no need to pre-encode strings that contain special characters. – Jack Taylor Sep 16 '21 at 06:32
0

As stated in the request docs (https://docs.python-requests.org/en/latest/):

Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, thanks to urllib3.

So you do not need to parse the text manually. Doing so may "double parse" it.

gficher
  • 18
  • 5
  • So you mean in my code above, i need to remove 'text = urllib.parse.quote_plus(text)'? – marlon Sep 15 '21 at 04:02
  • 'or to form-encode your POST data', Is the requests POST or GET data? – marlon Sep 15 '21 at 04:03
  • I tried your code using requestbin.com without the parsing and it worked out. Are you having any issues not having it parsed manually? – gficher Sep 15 '21 at 04:07
  • You should look at 'There’s no need to manually add query strings to your URLs'. The params are the queries. – gficher Sep 15 '21 at 04:08
  • My http service didn't return expected results, so I am suspecting to add urlencoding. 'There’s no need to manually add query strings to your URLs', yes, but the question is whether the 'text' field in the params map needs to encoded or not? – marlon Sep 15 '21 at 04:15
  • May you try using "quote" instead of "quote_plus"? – gficher Sep 15 '21 at 04:21
  • There is no 'quote' in python3? – marlon Sep 15 '21 at 04:23
  • https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote – sh.seo Sep 15 '21 at 04:29