I want to get all article link from "https://www.cnnindonesia.com/search?query=covid" Here is my code:

links = []
base_url = requests.get(f"https://www.cnnindonesia.com/search?query=covid")
soup = bs(base_url.text, 'html.parser')
cont = soup.find_all('div', class_='container')

for l in cont:
    l_cont = l.find_all('div', class_='l_content')
    for bf in l_cont:
        bf_cont = bf.find_all('div', class_='box feed')
        for lm in bf_cont:
            lm_cont = lm.find('div', class_='list media_rows middle')
            for article in lm_cont.find_all('article'):
                a_cont = article.find('a', href=True)
                if url:
                    link = a['href']
                    links.append(link)

and result is as follows:

links
[]

CodePudding user response：

Each article has this structure:

<article class="col_4">
   <a href="https://www.cnnindonesia.com/...">
       <span>...</span>
       <h2 class="title">...</h2>
   </a>
</article>

Simpler to iterate over the article elements then look for a elements.

Try:

from bs4 import BeautifulSoup
import requests

links = []
response = requests.get(f"https://www.cnnindonesia.com/search?query=covid")
soup = BeautifulSoup(response.text, 'html.parser')
for article in soup.find_all('article'):
    url = article.find('a', href=True)
    if url:
        link = url['href']
        print(link)
        links.append(link)
print(links)

Output:

https://www.cnnindonesia.com/nasional/...pola-sawah-di-laut-natuna-utara
...
['https://www.cnnindonesia.com/nasional/...pola-sawah-di-laut-natuna-utara', ...
'https://www.cnnindonesia.com/gaya-hidup/...ikut-penerbangan-gravitasi-nol']

CodePudding user response：

Sorry, can't add comments not enough reputation.

I think this line: for url in lm_row_cont.find_all('a'):

Should have the a tag as '<a>'

Or you could use regex (skipping the above) to match the relevant items after grabbing a div.