How to extract elements with app-root and scripts with BeautifulSoup?-CodePudding

I'm quite new to webscraping and I've encountered a weird problem that I fail to resolve. I want to scrape general information from a EU site (EU Website for scraping). I want to extract information regarding the opening and closing date and so on with BeautifulSoup.

The problem is that when I compare the HTML code through InspectElement I'm able to display span-tags and div-tags.

Is there away to circumvent the "app-root" and "script-elements" and convert them to regular html code from which I can parse the information?

#URL
eu_url = ""
#Opening connection grabbing the page
uClient = uReq(eu_url)

#Read page and save as a variable
page = uClient.read()

#Close connection
uClient.close()

#HTML Parsing with BeautifulSoup
page_soup = soup(page, "html.parser").prettify()

This outputs in:

<body>
<app-root>
  <div id="loader-wrapper">
   <div id="loader">
   </div>
  </div>
 </app-root>
 <!-- Sitemap Generator -->
 <!-- <script type="text/javascript">
    var _0xaea9 = ["sitemapgenerator:: ", "innerHTML", "body", "document", "stringify", "*", "postMessage", "setTimeout"]; window[_0xaea9[7]](function () { parent[_0xaea9[6]](_0xaea9[0]   JSON[_0xaea9[4]](window[_0xaea9[3]][_0xaea9[2]][_0xaea9[1]]), _0xaea9[5]) }, 3000);
</script> -->
 <noscript>
  <!-- to the list of topics and faqs -->
  <a href="/info/funding-tenders/opportunities/data/topic-list.html">
   Topic List
  </a>
 </noscript>
 <script src="runtime.626faac5e81de98cbf62.js" type="text/javascript">
 </script>
 <script src="polyfills.7351ea247f50946be3ed.js" type="text/javascript">
 </script>
 <script src="scripts.89d4984a80cecb9fb6d4.js" type="text/javascript">
 </script>
 <script src="main.4ceb2dd3c58718a9975b.js" type="text/javascript">
 </script>
</body>

The script should open the entire code and look something like this: Inspect Element Code

Does anyone have a solution for this?

CodePudding user response：

There's an API you can query to get the data you need.

Try this:

import requests

api_url = "https://ec.europa.eu/info/funding-tenders/opportunities/data/topicDetails/erasmus-edu-2022-net-edu-migrants.json?lang=en"

response = requests.get(api_url).json()["TopicDetails"]["actions"][0]
print(f"{response['plannedOpeningDate']} - {response['deadlineDates'][0]}")

Output:

07 June 2022 - 12 October 2022