Web scraping of multiple tags using python-CodePudding

Hi I am working with python for web scraping. Here is my code:

from bs4 import BeautifulSoup
import requests

page = requests.get(
    'https://www.indeed.com/viewjob?jk=78fc5cc6a9d2aaa3&q=developer&l=Hammond, LA&tk=1g3udv32opki1801&from=web&advn=2300444857198541&adid=371529140&ad=-6NYlbfkN0C3HlOxE-u7vDWDmHVgHclVijSpnbvDTTioTnwCLVe0OEwH_1p9qQb-3snK62Gml60thtHyOlr-diC2sIty8supkOLuy2apQt4gi355WXBpDDHQbuCkuMyYIfjito5_MzRa3sg8VkVKd5pvUD9rUt1RWPXpPzu2chM4oyLuN4riMCIsCh8gpIyWcPu7RV4Xt1Zp8PdeRuChYB95XZ0TM5bOYVexvf3lCdm4d3RG2TNPX5iZvX0mlZBhUQ2kufKY6TKI_2UZvTMgDAYwVjtFnB0qxEJi9aMmmp2GHECMAyifjTOAZkTUQnyIjUK_mFI7R7siYE6sIQSqPTt0pfEfvT4U-dfQpsmzdA1D0ZYdO-igFhm2rrEIwalOqCYEFwd3_cTBVkXzQBiiVA==&pub=4a1b367933fd867b19b072952f68dceb&vjs=3').text
soup = BeautifulSoup(page, 'lxml')
jobs = soup.find(
    'div', class_='jobsearch-JobComponent-description icl-u-xs-mt--md')
    
    job_desc = jobs.find('p').text.replace('', '')
    print(f"job description:{job_desc}")

In above code I have used BeautifulSoup.Also I was able to get the Job description. But My probelm is I am getting only one line sentence because inside the div tag of job discription there are more p tag and I was able to print only the first tag. As I have attached in picture How can i get all the job description div as paragraph ? [sample image page][1] Also I tried to use for loop:

job_desc = jobs.find_all('p')
for desc in job_desc:
    job_de = desc.find('p')
    print(f"job description:{job_de}")

As response I was able to get

job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None
job description:None

here is the link I am trying to scripe: https://www.indeed.com/viewjob?jk=78fc5cc6a9d2aaa3&q=developer&l=Hammond, LA&tk=1g3udv32opki1801&from=web&advn=2300444857198541&adid=371529140&ad=-6NYlbfkN0C3HlOxE-u7vDWDmHVgHclVijSpnbvDTTioTnwCLVe0OEwH_1p9qQb-3snK62Gml60thtHyOlr-diC2sIty8supkOLuy2apQt4gi355WXBpDDHQbuCkuMyYIfjito5_MzRa3sg8VkVKd5pvUD9rUt1RWPXpPzu2chM4oyLuN4riMCIsCh8gpIyWcPu7RV4Xt1Zp8PdeRuChYB95XZ0TM5bOYVexvf3lCdm4d3RG2TNPX5iZvX0mlZBhUQ2kufKY6TKI_2UZvTMgDAYwVjtFnB0qxEJi9aMmmp2GHECMAyifjTOAZkTUQnyIjUK_mFI7R7siYE6sIQSqPTt0pfEfvT4U-dfQpsmzdA1D0ZYdO-igFhm2rrEIwalOqCYEFwd3_cTBVkXzQBiiVA==&pub=4a1b367933fd867b19b072952f68dceb&vjs=3 [1]: https://i.stack.imgur.com/Dsm8w.png

CodePudding user response：

Try this:

from bs4 import BeautifulSoup
import requests


soup = BeautifulSoup(page, 'lxml')
job_desc = soup.find(
    'div', 
    class_='jobsearch-JobComponent-description icl-u-xs-mt--md'
).get_text()
    
print(f"job description:{job_desc}")

If you only want the human-readable text inside a document or tag, you can use the get_text() method. It returns all the text in a document or beneath a tag, as a single Unicode string:

A link to the get_text() in the beautiful soup docs: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#get-text