I am scraping a site, which makes a call to a REST api with a bearer token.

If I open the request in Chrome developer tools, I can right-click the request in question and see the headers, including the authorization header with the bearer token.

How can I access this header value in Selenium, so I can get the bearer token value?

Note: It is a publicly available bearer token on a publicly available website, where a cinema website calls a provider to get their showtimes. It seems to expire every few hours.

PS. I am scraping movie showtimes for a local aggregator site, nothing nefarious (in my opinion). I could just scrape from the generated webpage, but consuming a REST-api is just a lot nicer and less error-prone.

CodePudding user response：

I think like an option in case I am assuming you're using the same url, user, pass to get into the system, you can use any option in your code to make a GET (RestAssured in Java, for example). Then just parse a JSON response and get your Bearer.

In RestAssured it may be smth like that:

RestAssured.baseURI = "url";
RequestSpecification h = RestAssured.given();
Response res = h.get("/lalala");
ResponseBody b = res.getBody(); //get response
//convert response body to string
String responseBody = b.asString();
//JSON Representation from Response Body
JsonPath jsnPath = res.jsonPath();
String s = jsnPath.get("Bearer");
  

CodePudding user response：

I hacked my way to a solution, with the help of this post: Selenium 4 C# Chrome DevTools

Basically... :

1. start a session

2. wire up chromedriver devtools

3. listen for all XHR requests

4. check the header for each request, until I find one with a bearer token.

    private string GetBearerToken()
    {
        string uri = "https://somesite.com";
   
        var options = new ChromeOptions();
        options.AddArgument("--headless");
        options.AddArgument("--blink-settings=imagesEnabled=false");
   
        var driver = new ChromeDriver(options);
   
        string bearerToken = string.Empty;
   
        IDevTools devTools = driver as IDevTools;
   
        DevToolsSession session = devTools.GetDevToolsSession();
        FetchAdapter fetchAdapter = session.GetVersionSpecificDomains<OpenQA.Selenium.DevTools.V108.DevToolsSessionDomains>().Fetch;
   
        var enableCommandSettings = new OpenQA.Selenium.DevTools.V108.Fetch.EnableCommandSettings();
   
        var requestPattern = new OpenQA.Selenium.DevTools.V108.Fetch.RequestPattern();
        requestPattern.RequestStage = RequestStage.Request;
        requestPattern.ResourceType = ResourceType.XHR;
   
        enableCommandSettings.Patterns = new OpenQA.Selenium.DevTools.V108.Fetch.RequestPattern[] { requestPattern };
   
        fetchAdapter.Enable(enableCommandSettings);
   
        EventHandler<OpenQA.Selenium.DevTools.V108.Fetch.RequestPausedEventArgs> requestIntercepted = (sender, e) =>
        {
            if (string.IsNullOrEmpty(bearerToken) && e.Request.Headers.Any(z => z.Key == "authorization"))
            {
                //Console.WriteLine(e.Request.Url);
                string value = e.Request.Headers["authorization"];
                if (value.StartsWith("Bearer"))
                {
                    bearerToken = value.Substring(("Bearer ".Length));
                }
            }
   
            fetchAdapter.ContinueRequest(new OpenQA.Selenium.DevTools.V108.Fetch.ContinueRequestCommandSettings()
            {
                RequestId = e.RequestId
            });
        };
   
        fetchAdapter.RequestPaused  = requestIntercepted;
   
        driver.Navigate().GoToUrl(uri);
   
        //could definetely swap this for some more intelligent waiting, but for my usecase I am fine with just waiting for a fixed amount of time. 
        Thread.Sleep(5000);
   
        driver.Close();
   
        driver.Dispose();
   
        driver = null;
   
        return bearerToken;
    }