Selenium, getting headers for a call a page makes (I can see the header in dev tools)

Question

I am scraping a site, which makes a call to a REST api with a bearer token.

If I open the request in Chrome developer tools, I can right-click the request in question and see the headers, including the authorization header with the bearer token.

How can I access this header value in Selenium, so I can get the bearer token value?

Note: It is a publicly available bearer token on a publicly available website, where a cinema website calls a provider to get their showtimes. It seems to expire every few hours.

PS. I am scraping movie showtimes for a local aggregator site, nothing nefarious (in my opinion). I could just scrape from the generated webpage, but consuming a REST-api is just a lot nicer and less error-prone.

score 0 · Answer 1 · answered Dec 25 '22 at 18:32

I think like an option in case I am assuming you're using the same url, user, pass to get into the system, you can use any option in your code to make a GET (RestAssured in Java, for example). Then just parse a JSON response and get your Bearer.

In RestAssured it may be smth like that:

RestAssured.baseURI = "url";
RequestSpecification h = RestAssured.given();
Response res = h.get("/lalala");
ResponseBody b = res.getBody(); //get response
//convert response body to string
String responseBody = b.asString();
//JSON Representation from Response Body
JsonPath jsnPath = res.jsonPath();
String s = jsnPath.get("Bearer");

It is actually a public website, so I am not using any username/password, so this would not work. But my question was not very clear on that, so I understand your suggestion. :) — Kjensen, Dec 26 '22 at 12:15

score 0 · Answer 2 · answered Dec 28 '22 at 14:48

I hacked my way to a solution, with the help of this post: Selenium 4 C# Chrome DevTools

Basically... :

start a session
wire up chromedriver devtools
listen for all XHR requests

check the header for each request, until I find one with a bearer token.

 private string GetBearerToken()
 {
     string uri = "https://somesite.com";

     var options = new ChromeOptions();
     options.AddArgument("--headless");
     options.AddArgument("--blink-settings=imagesEnabled=false");

     var driver = new ChromeDriver(options);

     string bearerToken = string.Empty;

     IDevTools devTools = driver as IDevTools;

     DevToolsSession session = devTools.GetDevToolsSession();
     FetchAdapter fetchAdapter = session.GetVersionSpecificDomains<OpenQA.Selenium.DevTools.V108.DevToolsSessionDomains>().Fetch;

     var enableCommandSettings = new OpenQA.Selenium.DevTools.V108.Fetch.EnableCommandSettings();

     var requestPattern = new OpenQA.Selenium.DevTools.V108.Fetch.RequestPattern();
     requestPattern.RequestStage = RequestStage.Request;
     requestPattern.ResourceType = ResourceType.XHR;

     enableCommandSettings.Patterns = new OpenQA.Selenium.DevTools.V108.Fetch.RequestPattern[] { requestPattern };

     fetchAdapter.Enable(enableCommandSettings);

     EventHandler<OpenQA.Selenium.DevTools.V108.Fetch.RequestPausedEventArgs> requestIntercepted = (sender, e) =>
     {
         if (string.IsNullOrEmpty(bearerToken) && e.Request.Headers.Any(z => z.Key == "authorization"))
         {
             //Console.WriteLine(e.Request.Url);
             string value = e.Request.Headers["authorization"];
             if (value.StartsWith("Bearer"))
             {
                 bearerToken = value.Substring(("Bearer ".Length));
             }
         }

         fetchAdapter.ContinueRequest(new OpenQA.Selenium.DevTools.V108.Fetch.ContinueRequestCommandSettings()
         {
             RequestId = e.RequestId
         });
     };

     fetchAdapter.RequestPaused += requestIntercepted;

     driver.Navigate().GoToUrl(uri);

     //could definetely swap this for some more intelligent waiting, but for my usecase I am fine with just waiting for a fixed amount of time. 
     Thread.Sleep(5000);

     driver.Close();

     driver.Dispose();

     driver = null;

     return bearerToken;
 }

Selenium, getting headers for a call a page makes (I can see the header in dev tools)

2 Answers2