How to crawl ASP.NET web applications

Question

I'm writing an application which needs to communicate with a ASP.NET website that doesn't have an API.

I'm using Python 3.5.2 with Requests 2.18.4 to achieve my purpose.

The problem is the site is using _dopostback() so I achieved my goal using Selenium but it was slow and it opened a browser for every request that I wanted to make.

First I analyzed the POST data sent to the website for login using BurpSuite. The contained some hidden fields from the webpage HTML and the username, password, captcha and a radio button.

A successful login attempt will return 302 HTTP response code and redirect the browser to the profile page.

I extract the hidden fields and get the captcha image and enter it manually myself and POST the data to the website but it returns 200 response telling an error occurred.

This is the Code:

import re
import requests
from urllib.request import urlretrieve
from collections import OrderedDict
from bs4 import BeautifulSoup as BS
from PIL import Image



def get_and_show_captcha(session):
    soup = BS(current_response.text, "html.parser")
    image_url = soup.find("img", {"id": 'RadCaptcha1_CaptchaImageUP'})["src"]
    Image.open(urlretrieve(base_url  + image_url)[0]).show()



base_url = 'http://example.com/'
section_login = 'login.aspx'

proxies = {
  'http': 'http://127.0.0.1:8080',
}

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0',
    'DNT': '1',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Upgrade-Insecure-Requests': '1',
}


session = requests.Session()
session.headers.update(headers)
r = session.get('http://example.com/login.aspx', proxies=proxies)


login_post_dict = OrderedDict()
login_post_dict['__EVENTTARGET'] = ''
login_post_dict['__EVENTARGUMENT'] = ''
login_post_dict['__VIEWSTATE'] = ''
login_post_dict['__VIEWSTATEGENERATOR'] = ''
login_post_dict['__PREVIOUSPAGE'] = ''
login_post_dict['__EVENTVALIDATION'] = ''
login_post_dict['txtUserName'] = 'USERNAME'
login_post_dict['txtPassword'] = 'PASSWORD'
get_and_show_captcha(session=session)
capthca = input('CAPTCHA: ')
login_post_dict['rcTextBox1'] = capthca
login_post_dict['RadCaptcha1_ClientState'] = ''
login_post_dict['SOMERADIO'] = 'RADIOBUTTON'
login_post_dict['btn_enter'] = 'ENTER'
login_post_dict['txtUserName2'] = ''
login_post_dict['txt_email'] = ''


soup = BS(r.text, 'html.parser')

capture_id = re.compile("id=\"(\w*)\"")
capture_value = re.compile("value=\"(.*)\"")

for html_input in soup.find_all('input', attrs={'type': 'hidden'}):
    for hidden_id in capture_id.findall(str(html_input)):
        if hidden_id != r"RadCaptcha1_ClientState":
            login_post_dict[hidden_id] = ''
            for hidden_id_value in capture_value.findall(str(html_input)):
                if hidden_id_value is '':
                    continue
                login_post_dict[hidden_id] = hidden_id_value





session.headers.update({'Referer': base_url + section_login})
print(login_post_dict)
session.post(base_url + section_login, data=login_post_dict, proxies=proxies)

I send the script data through BurpSuit in order to see whats being sent exactly.

Any solution?