How to trigger a Javascript-generated HTML to show on the page with Python requests?-CodePudding

I'm using the requests module from Python to get a webpage that requires login to access.

I know how to have requests enter the login details on a page, but I am having trouble accessing the form fields on this particular page because they are generated by Javascript after clicking a button.

Here's the website in question: https://pennmedialab.getconnect2.com/signin.aspx

When first navigating to the link, the page show two options "Sign in with your 'PennKey'" and "Sign in with your local connect2 account". I need to click on the latter.

Here is the HTML for the latter button, which has an "href" attribute that triggers some Javascript code which in turn displays the username and password fields:

<a id="ctl00_MainContent_signin_showLocalLk" href="javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&quot;ctl00$MainContent$signin$showLocalLk&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, true))">Sign in with your local connect2 account</a>

Here is my Python code:

import requests
payload = {
    'ctl00$MainContent$signin$usernameField': 'my_username',
    'ctl00$MainContent$signin$passwordField': 'my_password'
}

with requests.Session() as s:
    p = s.post('https://pennmedialab.getconnect2.com/SignIn.aspx', data=payload)
    r = s.get('https://pennmedialab.getconnect2.com/Lease/Checkin.aspx?id=10750')

Currently, the post() method cannot log in because it can't find the fields with the name 'ctl00$MainContent$signin$usernameField' and 'ctl00$MainContent$signin$passwordField'.

Is there a way to combine the link "https://pennmedialab.getconnect2.com/SignIn.aspx" with the href attribute

javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions(&quot;ctl00$MainContent$signin$showLocalLk&quot;, &quot;&quot;, true, &quot;&quot;, &quot;&quot;, false, true))

so that the page that requests is posting to actually contains the username and password fields?

I've tried:

https://pennmedialab.getconnect2.com/SignIn.aspx/javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions("ctl00$MainContent$signin$showLocalLk", "", true, "", "", false, true)) and

https://pennmedialab.getconnect2.com/SignIn.aspx?javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions("ctl00$MainContent$signin$showLocalLk", "", true, "", "", false, true)) but with no luck.

CodePudding user response：

Executing javascript only with python isn't possible. You can use Selenium (for example with Chrome) instead.

It doesn't use protection like cloudfare, but in case of getting detected, try using Selenium-Profiles or undetected chromedriver.

If you really need to log-in using python requests, you'll need to analyse the requests to the website using the network tab in the browser developer-tools and then recreate them using python. That can be hard and might need maintenance regulary since the website might change.