How to apply proxy with authentication using http.client for API

Question

I'm attempting to use Bing's Entity search API, however I need to go through a proxy in order to establish a connection.

My company has giving me a proxy in the following format to use: 'http://username:password@webproxy.subdomain.website.com:8080'

I've tried using HTTPConnection.set_tunnel(host, port=None, headers=None) with no avail. Does anyone have any idea how to run the following Python query below using the proxy I was provided?

import http.client, urllib.parse
import json


proxy = 'http://username:pwd@webproxy.subdomain.website.com:8080' 
subscriptionKey = 'MY_KEY_HERE'
host = 'api.cognitive.microsoft.com'
path = '/bing/v7.0/entities'
mkt = 'en-US'
query = 'italian restaurants near me'

params = '?mkt=' + mkt + '&q=' + urllib.parse.quote (query)

def get_suggestions ():
    headers = {'Ocp-Apim-Subscription-Key': subscriptionKey}
    conn = http.client.HTTPSConnection(host)
    conn.request("GET", path + params, None, headers)
    response = conn.getresponse()
    return response.read()

result = get_suggestions ()
print (json.dumps(json.loads(result), indent=4))

As a side note, I was able to run the following sucessfully with the proxy.

nltk.set_proxy('http://username:pwd@webproxy.subdomain.website.com:8080')
nltk.download('wordnet')

score 1 · Accepted Answer · edited Aug 31 '21 at 13:41

I ended up using the requests package, they make it very simple to channel your request through a proxy, sample code below for reference:

import requests

proxies = {
    'http':  f"http://username:pwd@webproxy.subdomain.website.com:8080",
    'https': f"https://username:pwd@webproxy.subdomain.website.com:8080"
}
url = 'https://api.cognitive.microsoft.com/bing/v7.0/entities/'
# query string parameters

params = 'mkt=' + 'en-US' + '&q=' + urllib.parse.quote (query)

# custom headers
headers = {'Ocp-Apim-Subscription-Key': subscriptionKey}

#start session
session = requests.Session()
#persist proxy across request
session.proxies = proxies

# make GET request
r = session.get(url, params=params, headers=headers)

score 0 · Answer 2 · answered Oct 11 '20 at 23:31

If you use urlib2 or requests, the easiest way to configure a proxy is setting the HTTP_PROXY or the HTTPS_PROXY environment variable:

export HTTPS_PROXY=https://user:pass@host:port

This way the library handles the proxy configuration for you, and if your code runs inside a container you can pass that information at runtime.

How to apply proxy with authentication using http.client for API

2 Answers2