How do I extract certain html nodes using rvest?-CodePudding

I'm new to web-scraping so I may not be doing all the proper checks here. I'm attempting to scrape information from a url, however I'm not able to extract the nodes I need. See sample code below. In this example, I want to get the product name (Madone SLR 9 eTap Gen) which appears to be stored in the buying-zone__title class.

library(tidyverse)
library(rvest

url <- "https://www.trekbikes.com//us/en_US/bikes/road-bikes/performance-road-bikes/madone/madone-slr/madone-slr-9-etap-gen-7/p/37420"

read_html(url) %>% 
    html_nodes(".buying-zone__title") %>% 
    html_text()

When I run the code above, I get {xml_nodeset (0)}. How can I fix this? I would also like to scrape the year, price, available colors and specs from that page. Any help will be appreciated.

CodePudding user response：

There is a lot of dynamic content on that page which can be reviewed by disabling JS running in browser or comparing rendered page against page source.

You can view page source with Ctrl U, then Ctrl F to search for where the product name exists within the non-rendered content.

The title info you want is present in lots of places and there are numerous way to obtain. I will offer an "as on tin" option as the code gives clear indications as to what is being selected for.

I've updated the syntax and reduced the volume of imported external dependencies.

library(magrittr)
library(rvest)

url <- "https://www.trekbikes.com//us/en_US/bikes/road-bikes/performance-road-bikes/madone/madone-slr/madone-slr-9-etap-gen-7/p/37420"           
page <- read_html(url)    
name <- page %>% html_element("[product-name]") %>% html_attr("product-name")

specs <- page %>% html_elements('[]') %>% html_table()
price <- page %>% html_element('#gtm-product-display-price') %>% html_attr('value') %>% as.numeric()