I could manage to figure out the solution, so I'm publishing it here. What I'm going to publish here is not the exact solution for my problem rather a general approach to understand and check how authentication can be tracked. I used this technique to trace the login process in my case.
In my case ServiceNow uses a cookie based authentication and passes information back and forth among 4 pages. The first page generates an ID called NSC, and passes it towards to the second page as a cookie to generate another ID called SMSESSION ID, which is then passed to a third page together with NSC ID in the cookie to generate the final JSESSION ID. Finally the process passes over all previously generated 3 IDs to the login page in a cookie to validate the session.
I used Google Developer Tools to figure this out. What I would recommend you to do is the following.
1.) Go to the login page in Google Chrome that you would like to pass and wait until the site loads. Do not log in yet.
2.) Open Developer Tools (Right click, Inspect elements menu option). If you are familiar with other browser's dev capabilities that's also fine.
3.) Go to Application tab of Dev Tools and click on Clear storage at the left side menu bar. This will clear all data which is stored for this page. You can do the same in Setting menu of Chrome by clearing the cookies and other data. This is required to clear all historical steps which happened already on the page to not to make any confusion.
4.) Once it's done go to the Network tab of Dev Tools and click on the Clear menu option (next to the Record button). This will clear the Network log history.
5.) As a next step tick the Preserve log checkbox on the Network tab. This will allow us to keep track of every steps even in case of any redirection. If you don't tick this option you will loose all data once your login page redirects you to somewhere else, because it clears the Network log.
6.) Now as we removed all historical data and set up everything, we can start the investigation. Log in to the page with your user id and password and keep Developer Tools open, so you can see all network requests. Wait until the login process finishes and start to examine your network log entries one by one.
7.) You will see some GET and POST requests. This is your login process flow. Open the first one by double clicking on it. It will show you information organized in sections just like (General, Response Headers, Request Headers, Query Params, Form Data etc.). This is the information exchange which happens between the web server and the client (your machine). You need to simulate the same with you script.This means whatever you see in the Request Headers section, you need to pass exactly the same with your script. This way you will receive the very same Response Headers and you can grab all the information from there which is required to move forward.
Let me show you and example.
In my first POST request I can see the following in the Network log:
General
Request URL:https://mysnserver.net/siteminderagent/forms/dssologinprod.fcc?TYPE=33554433&REALMOID=06-0cffd45f-7ca7-106f-bbab-84fb3af10000&GUID=&SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=-SM-28THtkr3KQi%2fJmb193GjY0nVjpKo6ULc%2fJNV5hRyjzC17qWZfgyVPkR%2f7EAWoDVu3Gd3y3kTm3N2p0B8KVp0Hixjin0ZsDZ3&TARGET=-SM-%2f
Request Method:POST
Status Code:302 Found
Remote Address:1.1.1.196:443
Response Headers
Cache-Control:no-store
Connection:Keep-Alive
Content-Length:1541
Content-Type:text/html; charset=iso-8859-1
Date:Wed, 21 Sep 2016 19:11:46 GMT
Keep-Alive:timeout=5, max=496
Location:https://anothersite.com/SmMakeCookie.ccc?SMSESSION=-SM-w0Gp2DpiPEG&PERSIST=0&TARGET=-SM-https%3a%2f%2fservicemanagement%2net%2f
Set-Cookie:SMSESSION=w0Gp2DpiPEGPrLepzXds9qUTVER/Xl75WO36n37IxRpLaE6dwQPwN2+iaNn4rQZODb+65k2Gy9fggnKU04I7rSU6; path=/; domain=.mysnserver.net; secure
Set-Cookie:SMIDENTITY=EoIkGNtD3Y+FBWumdJuml3J78o61Qtc07b73XmqEeze; path=/; domain=.mysnserver.net; secure
Set-Cookie:NSC_1.1.1.196-443-C72169=ffffffffaaa3746145525d5f4f58455e445a4a4253a5;expires=Wed, 21-Sep-2016 21:11:47 GMT;path=/;secure;httponly
Set-Cookie:SMTRYNO=; expires=Fri, 25 Mar 2016 19:11:46 GMT; path=/; domain=.mysnserver.net; secure
Request Headers
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate, br
Accept-Language:en-US,en;q=0.8
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:238
Content-Type:application/x-www-form-urlencoded
Host:mysnserver.net
Origin:https://mysnserver.net
Referer:https://anothersite.net/forms/dssologinprod.fcc?TYPE=33554433&REALMOID=06-0cffd45f-7ca7-106f-bbab-84fb3af10000&GUID=&SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=-SM-28THtkr3KQi%2fJmb193GjY0nVjpKo6ULc%2fJNV5hRyjzC17qWZfgyVPkR%2f7EAWoDVu3Gd3y3kTm3N2p0B8KVp0Hixjin0ZsDZ3&TARGET=-SM-%2f
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36
Query String Parameters
TYPE:33554433
REALMOID:06-0cffd45f-7ca7-106f-bbab-84fb3af10000
GUID:
SMAUTHREASON:0
METHOD:GET
SMAGENTNAME:-SM-28THtkr3KQi/Jmb193GjY0nVjpKo6ULc/JNV5hRyjzC17qWZfgyVPkR/7EAWoDVu3Gd3y3kTm3N2p0B8KVp0Hixjin0ZsDZ3
TARGET:-SM-/
Form Data
SMENC:ISO-8859-1
SMLOCALE:US-EN
target:/
smquerydata:
smauthreason:0
smagentname:28THtkr3KQi/Jmb193GjY0nVjpKo6ULc/JNV5hRyjzC17qWZfgyVPkR/7EAWoDVu3Gd3y3kTm3N2p0B8KVp0Hixjin0ZsDZ3
postpreservationdata:
USER:my_userid
PASSWORD:my_password
Whatever you can see in the Request Headers section, that needs to be passed to the first URL to get the Response Headers information. If you see in the Response Headers I have received several IDs which was given by the server. This means that I need to prepare my first request in Python to pass the very same information that I have in the Request Header. Like this:
auth_url1 = 'https://mysnserver.net/siteminderagent/forms/dssologinprod.fcc?TYPE=33554433&REALMOID=06-0cffd45f-7ca7-106f-bbab-84fb3af10000&GUID=&SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=-SM-28THtkr3KQi%2fJmb193GjY0nVjpKo6ULc%2fJNV5hRyjzC17qWZfgyVPkR%2f7EAWoDVu3Gd3y3kTm3N2p0B8KVp0Hixjin0ZsDZ3&TARGET=-SM-%2f'
# Initiating session
s = requests.session()
request_header_1 = {
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate, br',
'Accept-Language':'en-US,en;q=0.8',
'Cache-Control':'max-age=0',
'Connection':'keep-alive',
'Content-Length':'238',
'Content-Type':'application/x-www-form-urlencoded',
'Host':'mysnserver.net',
'Origin':'https://mysnserver.net',
'Referer':'https:///anothersite.net/forms/dssologinprod.fcc?TYPE=33554433&REALMOID=06-0cffd45f-7ca7-106f-bbab-84fb3af10000&GUID=&SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=-SM-28THtkr3KQi%2fJmb193GjY0nVjpKo6ULc%2fJNV5hRyjzC17qWZfgyVPkR%2f7EAWoDVu3Gd3y3kTm3N2p0B8KVp0Hixjin0ZsDZ3&TARGET=-SM-%2f',
'Upgrade-Insecure-Requests':'1',
'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36'
}
form_data_1 = {
'SMENC':'ISO-8859-1',
'SMLOCALE':'US-EN',
'target':'/',
'smquerydata':'',
'smauthreason':'0',
'smagentname':'28THtkr3KQi/Jmb193GjY0nVjpKo6ULc/JNV5hRyjzC17qWZfgyVPkR/7EAWoDVu3Gd3y3kTm3N2p0B8KVp0Hixjin0ZsDZ3',
'postpreservationdata':'',
'USER':'my_userid', #<----- Put your user ID here
'PASSWORD':'my_password' #<----- Put your password here
}
r = s.post(auth_url1, headers=request_header_1, data=form_data_1, verify=False, allow_redirects=False)
# Get NSC ID from the response header which needs to be passed over in the 3rd request
nsc_id = r.cookies.keys()[2] + "=" + r.cookies.values()[2]
That's it. You need to follow the very same process if you have more redirection until you pass the last page and your session authenticates. After this you can use the cookie information that you collected to authenticate all of your upcoming requests. As you can see I have initiated a session with s = requests.session() command, that I can use to submit all of my requests without passing my user id and pw for all requests. Take care when you need to send a GET and when you need to send a POST request. You can see this in the General information section of the header.
One more important note. Use allow_redirects=False in your requests if you have redirection on your site. In this way you can make sure that your request is not got redirected to other sites an you get back the proper Response Headers information.
