Writing webscrapped data into CSV-CodePudding

With the below code I am able to scrape product infromation from two websites. My goal is to write the scraped data into a CSV where column A is used for the class "label" and column B is used for the class "value"

Can anyone help me achieve the desired outcome?

from bs4 import BeautifulSoup
import requests
import pandas as pd

url_list = ["https://21shares.com/product/abtc", "https://21shares.com/product/aeth/"]


for link in url_list:
    r = requests.get(link)
    r.encoding = 'uft-8'
    html_content = r.text
    soup = BeautifulSoup(html_content, "lxml")
    datas = soup.find('div', {'class':'product-sidebar-container'})

    for data in datas:
        soup.findAll("span", {"class": "Label", "Value": True})
        print(data.getText(separator=("\n")))

CodePudding user response：

Initialize an empty list and as you iterate through datas, create a dictionary to append to the list. Once you have that list, it's easy to convert to a dataframe and write to csv with pandas. Follow the example below which is your code modified.

from bs4 import BeautifulSoup
import requests
import pandas as pd

url_list = ["https://21shares.com/product/abtc", "https://21shares.com/product/aeth/"]

rows = []
for link in url_list:
    r = requests.get(link)
    r.encoding = 'uft-8'
    html_content = r.text
    soup = BeautifulSoup(html_content, "lxml")
    datas = soup.find('div', {'class':'product-sidebar-container'})

    for data in datas:
        for each in soup.findAll("span", {"class": "label"}):
            label = each.text
            value = each.find_next('span', {'class':'value'}).text
            
            row = {'label':label, 'value':value}
            rows.append(row)


df = pd.DataFrame(rows)
df.to_csv('file.csv', index=False)

CodePudding user response：

You have a typo, should be 'utf-8'.