Dealing with HTTPS CONNECT requests

Question

I'm writing a really simple web proxy through python and right now I'm working on dealing with HTTPS CONNECT requests so I can open HTTPS websites. I'm trying to set up an SSL tunnel but my code is just not quite right. I think I'm close but if someone could take a look and push me in the right direction that would be great. My current understanding of what I'm supposed to do is

• Recognize that the request is a CONNECT request
• Send a message back to the browser as I have defined in the variable connect_req in my code
• That's about it

Here's my code:

def ProxyThread(conn, client_addr):
    request = conn.recv(MAX_BUFFER)
    #print request
    # Parsing
    method, webserver, port = ParseReq(request)
    print 'Request = ' + method + ' ' + webserver + ':' + str(port) + '\n'

    try:
        serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        serverSocket.connect((webserver, port))

        if method == 'CONNECT':
            connect_req = 'HTTP/1.1 200 Connection established\r\n'
            connect_req += 'Proxy-agent: localhost\r\n\r\n'

            conn.send(connect_req.encode())

        serverSocket.send(connect_req)

        while 1:
            data = serverSocket.recv(MAX_BUFFER)

        # while there is data to receive from server
            if len(data) > 0:
                conn.send(data)

            else:
                break

        serverSocket.close()
        conn.close()

    except socket.error, (message):
        print message

        if conn:
            conn.close()

        if serverSocket:
            serverSocket.close()
        return

Edit 1: Updated code to start a thread when I get a HTTPS req

def ProxyThread(conn, client_addr):
    request = conn.recv(MAX_BUFFER)
    method, webserver, port = ParseReq(request)

    #Handle index out of range exception - Throw out the request
    if method is None or webserver is None or port is -1:
        return

    print 'Request = ' + method + ' ' + webserver + ':' + str(port) + ' START\n'

    serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    try:
        if method == 'CONNECT':
            connect_req = 'HTTP/1.0 200 Connection established\r\n'
            connect_req += 'Proxy-agent: ProxyServer/1.0\r\n'
            connect_req += '\r\n'
            print connect_req
            conn.send(connect_req)
            thread = threading.Thread(target=HTTPSProxyThread, args=(conn, serverSocket))
            thread.start()

        serverSocket.connect((webserver, port))
        serverSocket.send(request)

        while 1:
            data = serverSocket.recv(MAX_BUFFER)

        # while there is data to receive from server
            if len(data) > 0:
                conn.send(data)

            else:
                break

        print 'Request = ' + method + ' ' + webserver + ':' + str(port) + ' FINISH\n'
        serverSocket.close()

        conn.close()


def HTTPSProxyThread(conn, serverSocket):
    while 1:
        request = conn.recv(MAX_BUFFER)
        print request
        method, webserver, port = ParseReq(request)
        serverSocket.connect((webserver, port))
        serverSocket.send(request)

        while 1:
            data = serverSocket.recv(MAX_BUFFER)

        # while there is data to receive from server
            if len(data) > 0:
                conn.send(data)

            else:
                break

Answer 1

A lot of people seem to be building their own web proxies in Python, or Node.js these days.

As someone who has spent the past 22 years making a web proxy, I wonder why people do it to themselves, especially where there is free product available on all the main platforms where somebody has already dealt with the issues such as (some of these you will have to deal with later)

• tunneling (CONNECT)
• chunking
• HTTP authentication
• Funky non-compliant behaviour from a surprising number of servers and clients.
• performance
• scalability
• logging
• caching
• policy framework and enforcement
• etc

Whilst it's a fun way to pass the time for a while, the more of these proxies there are out there, the more broken the web becomes overall if the naive implementations are used for more general traffic. If you're just using this for your own specific deployment requirement, the ignore this comment.

I guess the point I'm trying to make is that making a well-behaved (let alone performant) web proxy is non-trivial.