can't get sub-layer content within many div layer by using beautifulsoap-CodePudding

I want to access contents inside div with class name "ar-faculty-section-content" in https://www.fed.cuhk.edu.hk/cri/faculty/prof-yin-hong-biao/. I tried to use below method to get to the target content. But it doesn't work.

 profile = requests.get("https://www.fed.cuhk.edu.hk/cri/faculty/dr-sze-man-man-paul/")
 x = BeautifulSoup(profile.text,"html.parser")
 x.find_all("h5", { "class" : "ar-faculty-section-content" })

The result is as following

[<div  style="font-weight: 400 !important">

]

How can I get the entire content in that div section such as h5 li......?

CodePudding user response：

Here's how:

import requests
from bs4 import BeautifulSoup

url = "https://www.fed.cuhk.edu.hk/cri/faculty/prof-yin-hong-biao/"

source_html = requests.get(url).text
soup = BeautifulSoup(source_html, 'lxml')
h5 = soup.select_one(".ar-faculty-section-content h5").getText()
li_elements = [li.getText() for li in soup.select(".ar-faculty-section-content li")]

print(h5)
print("\n".join(li_elements))

Output:

Introduction
Yin, H., & Huang, S. (2021). Applying structural equation modelling to research on teaching and teacher education: Looking back and forward. Teaching and Teacher Education. DOI: 10.1016/j.tate.2021.103438
Yin, H., & Shi, L. (2021). Which type of interpersonal interaction better facilitates college student learning and development in China: Face-to-face or online? ECNU Review of Education. DOI: 10.1177/20965311211010818

and a lot more ...