I got web-scraping error with HTTP error 403-CodePudding

I try to web scraping. I have error "HTTP error 403."

library(rvest)
library(dplyr)

link <- "https://www.hepsiburada.com/lg-70nano756pa-70-177-ekran-uydu-alicili-4k-ultra-hd-smart-led-tv-p-HBCV00000YCM48"
page <- read_html(link)

My Output and error messages:

link <- "https://www.hepsiburada.com/lg-70nano756pa-70-177-ekran-uydu-alicili-4k-ultra-hd-smart-led-tv-p-HBCV00000YCM48"
> page <- read_html(link)
Error in open.connection(x, "rb") : HTTP error 403.
> 
> price = page %>% html_nodes(".productPrice") %>% html_text()
Error in UseMethod("xml_find_all") : 
  no applicable method for 'xml_find_all' applied to an object of class "function"
> seller = page %>% html_nodes(".merchantName") %>% html_text()
Error in UseMethod("xml_find_all") : 
  no applicable method for 'xml_find_all' applied to an object of class "function"

price = page %>% html_nodes(".productPrice") %>% html_text()
seller = page %>% html_nodes(".merchantName") %>% html_text()

CodePudding user response：

If httr2 and rvest fails, you can always rely on RSelenium to websrape. Here is an example where I scrape the title and the price of the product.

library(tidyverse)
library(rvest)
library(RSelenium)
library(netstat)

rD <- rsDriver(browser = "firefox", port = free_port())
remDr <- rD[["client"]]

remDr$navigate(
  "https://www.hepsiburada.com/lg-70nano756pa-70-177-ekran-uydu-alicili-4k-ultra-hd-smart-led-tv-p-HBCV00000YCM48"
)

html <- remDr$getPageSource()[[1]] %>%
  read_html

tibble(
  price = html %>%  
    html_element("#offering-price span") %>%  
    html_text2(),
  title = html %>%  
    html_element("#product-name") %>%  
    html_text2()
)

# A tibble: 1 x 2
  price  title                                                         
  <chr>  <chr>                                                         
1 17.941 "LG 70NANO756PA 70\" 177 Ekran Uydu Alicili 4K Ultra HD Smart~