Need to extract all h2 text from some links and I tried it by using BeautifulSoup, but it didn't worked.

I also want to output them to CSV

# -*- coding: utf-8 -*-

from bs4 import BeautifulSoup
import requests
import csv

r01 = requests.get("https://www.seikatsu110.jp/library/vermin/vr_termite/23274/") 
r02 = requests.get("https://yuko-navi.com/termite-control-subsidies")


soup_content01 = BeautifulSoup(r01.content, "html.parser")
soup_content02 = BeautifulSoup(r02.content, "html.parser")

alltxt01 = soup_content01.get_text()
alltxt02 = soup_content02.get_text()

with open('h2.csv', 'w ',newline='',encoding='utf-8') as f:
    n = 0

    for subheading01 in soup_content01.find_all('h2'):
        sh01 = subheading01.get_text()

        writer = csv.writer(f, lineterminator='\n')
        writer.writerow([n, sh01])
        n  = 1

    for subheading02 in soup_content02.find_all('h2'):
        sh02 = subheading02.get_text()

        writer = csv.writer(f, lineterminator='\n')
        writer.writerow([n, sh01, sh02])
        n  = 1
pass

expected csv output is as below:

Please somebody tell me what's wrong with this code.

CodePudding user response：

This is one way to reach your goal as stated:

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd
from itertools import zip_longest

headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}

r01 = requests.get("https://www.seikatsu110.jp/library/vermin/vr_termite/23274/", headers=headers) 
r02 = requests.get("https://yuko-navi.com/termite-control-subsidies", headers=headers)

first_url_headings = [x.get_text(strip=True) for x in bs(r01.text, 'html.parser').select('h2')]
second_url_headings = [x.get_text(strip=True) for x in bs(r02.text, 'html.parser').select('h2')]

df_list = list(zip_longest(first_url_headings, second_url_headings))
df = pd.DataFrame(df_list, columns = ['First site', 'Second site'])
df.to_csv('termites_stuffs.csv')
print(df)

Result in terminal (also saved as a csv file):

Documentation for BeautifulSoup: https://beautiful-soup-4.readthedocs.io/en/latest/index.html

Also for Requests: https://requests.readthedocs.io/en/latest/

And for pandas: https://pandas.pydata.org/pandas-docs/stable/index.html

CodePudding user response：

Just in addation to approach of @Barry the Platipus with itertools, that is great. - pandas is also my favorite and there is an alternative way with native dict comprehension.

Iterate your urls and create a dict that holds number or url as key and a list of heading texts as value. These could be easily transformed to a DataFrame and exported to CSV:

d = {}
for e,url in enumerate(urls,1):
    soup = BeautifulSoup(requests.get(url).content)
    d[f'sh{e}'] = [h.get_text() for h in soup.find_all('h2')]

pd.DataFrame(dict([ (k,pd.Series(v)) for k,v in d.items()]))#.to_csv('yourfile.csv', index = False)

Example

from bs4 import BeautifulSoup
import requests
import pandas as pd

urls = ['https://www.seikatsu110.jp/library/vermin/vr_termite/23274/','https://yuko-navi.com/termite-control-subsidies']
d = {}
for e,url in enumerate(urls,1):
    soup = BeautifulSoup(requests.get(url).content)
    d[f'sh{e}'] = [h.get_text() for h in soup.find_all('h2')]

pd.DataFrame(dict([ (k,pd.Series(v)) for k,v in d.items()]))#.to_csv('yourfile.csv', index = False)

Output