CSV Writer stops writing data after 21 rows-CodePudding

I have this huge list of products and prices and I'm trying to export it onto a CSV file for easier readability, but for some reason, it only writes in 21 lines of the data and then stops.

This is the current code:

for x in range(1, 20):
    url = (f'website.com/page{x}')
    page = requests.get(url)

    soup = bs(page.content, 'html.parser')
    articles = soup.find_all('article', class_='product-grid-item')

    with open('website.csv', 'w', encoding='utf8', newline='') as f:
        
        thewriter = writer(f)
        header = ['Title', 'Price']
        thewriter.writerow(header)

        for article in articles:
            title = article.find('h3', class_='product-item-title').text.strip()
            price = article.find('div', class_='product-item-price').text.strip()
            
            products = [title, price]
            thewriter.writerow(products)

The list of products comes out to be over 200 products but it only prints the last 21. Am I doing something wrong I'm new to web scraping and exporting data into csv?

CodePudding user response：

This is because at the top it says

for x in range(1, 20):

This will make it start from 1 and loop until 20. If you want it to loop more times increase that number.

To learn more about the range function in python I recommend this

Syntax:

range(start, stop, step)

Parameter Values:

Start: An integer number specifying at which position to start. Default is 0. (optional)

Stop: An integer number specifying at which position to stop (not included). (Required)

Step: An integer number specifying the incrementation. Default is 1 (Optional)

CodePudding user response：

This line of code was not working, only writing the last list:

with open('website.csv', 'w', encoding='utf8', newline='') as f:

I changed the 'write' call to an 'append' call and now it writes all 300 products into the cvs file:

 with open('website.csv', 'a', encoding='utf8', newline='') as f: