I can't figure out why the .find_all() isn't returning all the tags I search for.

Basically I'm trying to scrape this site and get a list of all the "tr" tags (mainly interested in the ones in the tag) but it only returns the first "tr" tag that is nested in the tag, and I'm assuming it never makes it to the . End goal: I just need to extract the wine names from each row in that table body.

def web_parser1():
  import requests
  from bs4 import BeautifulSoup
  import numpy as np
  import pandas as pd

  response_obj_1 = requests.get("https://www.globalwinescore.com/ranking/?idx=Wine&dFR[country][0]=Usa")
  soup = BeautifulSoup(response_obj_1.text, "html.parser")
  trtags = soup.find_all("tr")
  print(trtags)

CodePudding user response：

Actually, the url is dynamic, bs4 can't mimic javascript but data is also generating api calls json response as post method from where easily can grab data using requests module only.

Script:

import requests
import json
body= {"requests":[{"indexName":"Wine","params":"query=&hitsPerPage=50&maxValuesPerFacet=1000&page=0&facets=["country","regions","color","type","appellation","vintage","classification","labels","score"]&tagFilters=&facetFilters=[["country:Usa"]]"},{"indexName":"Wine","params":"query=&hitsPerPage=1&maxValuesPerFacet=1000&page=0&attributesToRetrieve=[]&attributesToHighlight=[]&attributesToSnippet=[]&tagFilters=&analytics=false&facets=country"}]}
headers= {
    'content-type': 'application/json',
   }

api_url = "https://heioa8c3j2-1.algolianet.com/1/indexes/*/queries?x-algolia-agent=Algolia for vanilla JavaScript (lite) 3.24.5;instantsearch.js 2.2.3;JS Helper 2.23.0&x-algolia-application-id=HEIOA8C3J2&x-algolia-api-key=02525019312db516b514d0a95aefb1c1"


jsonData = requests.post(api_url, data=json.dumps(body), headers=headers).json()
#print(jsonData)
for item in jsonData['results'][0]['hits']:
    name=item['wine']
    print(name)

Output:

Ridge Vineyards, Monte Bello, Santa Cruz Mountains
Kapcsandy Family Winery, State Lane Vineyard Grand-Vin Cabernet Sauvignon, Napa Valley
Abreu Vineyard, Thorevilos Cabernet Sauvignon, Napa Valley
Continuum, Proprietary Red, Oakville
Ridge Vineyards, Monte Bello, Santa Cruz Mountains
Abreu Vineyard, Madrona Ranch Cabernet Sauvignon, Napa Valley
Abreu Vineyard, Thorevilos Cabernet Sauvignon, Napa Valley
Lokoya Winery, Mount Veeder Cabernet Sauvignon, Napa Valley
Abreu Vineyard, Thorevilos Cabernet Sauvignon, Napa Valley
Screaming Eagle, Cabernet Sauvignon, Napa Valley
Verite, La Joie, Sonoma County
Screaming Eagle, Cabernet Sauvignon, Napa Valley
Harlan Estate, Napa Valley
Ridge Vineyards, Monte Bello, Santa Cruz Mountains
Dana Estates, Lotus Vineyard Cabernet Sauvignon, Napa Valley
Verite, La Muse, Sonoma County
Abreu Vineyard, Madrona Ranch Cabernet Sauvignon, Napa Valley
Kongsgaard, The Judge Chardonnay, White, Napa Valley
Screaming Eagle, Cabernet Sauvignon, Napa Valley
Dunn Vineyards, Trailer Vineyard Cabernet Sauvignon, Howell Mountain
Harlan Estate, Napa Valley
Abreu Vineyard, Madrona Ranch Cabernet Sauvignon, Napa Valley
Schrader Cellars, Old Sparky Beckstoffer To Kalon Vineyard Cabernet Sauvignon, Napa Valley    
Dominus Estate, Christian Moueix, Napa Valley
Harlan Estate, Napa Valley
Hundred Acre, Few And Far Between Cabernet Sauvignon, Napa Valley
Abreu Vineyard, Madrona Ranch Cabernet Sauvignon, Napa Valley
Harlan Estate, Napa Valley
Abreu Vineyard, Madrona Ranch Cabernet Sauvignon, Napa Valley
Colgin Cellars, Tychson Hill Vineyard Cabernet Sauvignon, Napa Valley
Harlan Estate, Napa Valley
Screaming Eagle, Cabernet Sauvignon, Napa Valley
Venge Vineyards, Bone Ash Vineyard Cabernet Sauvignon, Napa Valley
Peter Michael Winery, Au Paradis Cabernet Sauvignon, Oakville
Colgin Cellars, Tychson Hill Vineyard Cabernet Sauvignon, Napa Valley
Harlan Estate, Napa Valley
Colgin Cellars, Ix Estate Red, Napa Valley
Schrader Cellars, Beckstoffer To Kalon Vineyard Cabernet Sauvignon, Napa Valley
Schrader Cellars, T6 Beckstoffer To Kalon Vineyard Cabernet Sauvignon, Napa Valley
Futo Estate, 5500 Sld - Cabernet Sauvignon, Stags Leap District
Hundred Acre, Fortification Port, Napa Valley
Chappellet, Pritchard Hill Cabernet Sauvignon, Napa Valley
Colgin Cellars, Cariad Red, Napa Valley
Screaming Eagle, Cabernet Sauvignon, Napa Valley
Schrader Cellars, T6 Beckstoffer To Kalon Vineyard Cabernet Sauvignon, Napa Valley
Cayuse Vineyards, En Chamberlin Vineyard Syrah, Walla Walla Valley
Harlan Estate, Napa Valley
Robert Mondavi Winery, To Kalon Vineyard Reserve Cabernet Sauvignon, Oakville
Schrader Cellars, Ccs Beckstoffer To Kalon Vineyard Cabernet Sauvignon, Napa Valley
Screaming Eagle, Cabernet Sauvignon, Napa Valley