How can I save this scrapped data into a data frame?-CodePudding

Hey this is my code that i used to scrap some data from website for practise. Can you help me set it into a data frame and save it?

url = "https://aedownload.com/download-magazine-promo-for-element-3d-free-videohive/"
page = requests.get(url)
soup = BeautifulSoup(page.content, "html.parser")

title = soup.find(class_="blog-title").text.strip()
project_details = soup.find( class_="project-details").text
link_wp = soup.find (class_="wp-video-shortcode").text
link_infopage = soup.find(class_="infopage112").text
project_description = soup.find(class_= "Project-discription").text
print(title)
print(project_details)
print(link_wp)
print(link_infopage)
print(project_description)

CodePudding user response：

Create an empty dictionary and append items to dict1 and use pandas to create dataframe

dict1={}
dict1['title'] = soup.find(class_="blog-title").text.strip()
dict1['project_details'] = soup.find( class_="project-details").text
dict1['link_wp'] = soup.find (class_="wp-video-shortcode").text
dict1['link_infopage'] = soup.find(class_="infopage112").text
dict1['project_description'] = soup.find(class_= "Project-discription").text


import pandas as pd
df = pd.DataFrame()
df = df.append(dict1, ignore_index=True)

Output:

       title         project_details    link_wp    link_infopage    project_description
0   Download Magazine Promo for Element 3D – FREE ...   \nMagazine Promo for Element 3D 23030644 Video...   https://previews.customer.envatousercontent.co...   Buy it  \nFree Download\n\n\n\n\n\n\nRelated Templates...

CodePudding user response：

To create new DataFrame from the data you can try:

import requests
import pandas as pd
from bs4 import BeautifulSoup

url = "https://aedownload.com/download-magazine-promo-for-element-3d-free-videohive/"
page = requests.get(url)
soup = BeautifulSoup(page.content, "html.parser")

title = soup.find(class_="blog-title").text.strip()
project_details = soup.find(class_="project-details").text
link_wp = soup.find(class_="wp-video-shortcode").text
link_infopage = soup.find(class_="infopage112").text
project_description = soup.find(class_="Project-discription").text


df = pd.DataFrame(
    {
        "title": [title],
        "project_details": [project_details],
        "link_wp": [link_wp],
        "link_infopage": [link_infopage],
        "project_description": [project_description],
    }
)
df.to_csv("data.csv", index=False)

Saves data.csv (screenshot from LibreOffice):