getting the relevant tag for scraping address from website-CodePudding

I am trying to scrape locations of Walmart in the State of Missouri using the link below:

https://www.walmart.com/store/finder?location=Missouri&distance=50

library(rvest)
library(xml2)
library(tidyverse)

url <- read_html("https://www.walmart.com/store/finder?location=Missouri&distance=50")

I used SelectorGadget to check what is in the NearbyStores and use it to extract store address.

Trying extracting the city first but I get nothing

url %>% html_elements(".city")
{xml_nodeset (0)}

Then I tried to extract address and store type but still get nothing.

url %>% html_elements(".result-element-address")
{xml_nodeset (0)}
  
url %>% html_elements(".result-element-store-type")
{xml_nodeset (0)}

I am trying to create a data frame with name of the city, and address

CodePudding user response：

The tag you are looking for foes not exist in the document you are requesting. It is built dynamically by javascript code after the page loads. Fortunately the actual data does exist on the page, in the form of a json string inside one of the script tags. This requires a bit of parsing, but contains all the info you need:

library(rvest)
library(xml2)
library(tidyverse)

url <- read_html("https://www.walmart.com/store/finder?location=Missouri&distance=50")
stores <- html_element(url, xpath = "//script[@id='storeFinder']") %>% 
  html_text() %>%
  jsonlite::parse_json()
  
do.call(rbind, lapply(stores$storeFinder$storeFinderCarousel$stores, 
       function(x) as.data.frame(x$address)))
#>    postalCode                 address           city state country
#> 1       65401        500 S Bishop Ave          Rolla    MO      US
#> 2       65584   185 Saint Robert Blvd   Saint Robert    MO      US
#> 3       65453            100 Ozark Dr           Cuba    MO      US
#> 4       65560       1101 W Highway 32          Salem    MO      US
#> 5       65066         1888 Highway 28     Owensville    MO      US
#> 6       63080       350 Park Ridge Rd       Sullivan    MO      US
#> 7       65101      401 Supercenter Dr Jefferson City    MO      US
#> 8       65065         4252 Highway 54    Osage Beach    MO      US
#> 9       65483 1433 S Sam Houston Blvd        Houston    MO      US
#> 10      65109   724 Stadium West Blvd Jefferson City    MO      US
#> 11      65026      1802 S Business 54          Eldon    MO      US
#> 12      65020             94 Cecil St      Camdenton    MO      US
#> 13      65536    1800 S Jefferson Ave        Lebanon    MO      US