web-scrapping returning a 403 error after passing different headers-CodePudding

I am trying to scrape a website using a package in R.

When I run the following:

library(idealisto) #https://github.com/hmeleiro/idealisto
get_city("https://www.idealista.com/alquiler-viviendas/madrid-madrid/", "sale")

I get:

Error in read_html.response(.) : Forbidden (HTTP 403).

Looking into more details of the function get_city() I find that the problem is with the following part of the code:

desktop_agents <- c("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36", 
                    "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36", 
                    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36", 
                    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14", 
                    "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36", 
                    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36", 
                    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36", 
                    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36", 
                    "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36", 
                    "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0")

url = "https://www.idealista.com/en/venta-viviendas/madrid-provincia/"
x <- GET(url, add_headers(`user-agent` = desktop_agents[sample(1:10, 1)]))

Which returns the following output:

Response [https://www.idealista.com/en/venta-viviendas/madrid-provincia/]
Date: 2022-04-04 18:52 Status: 403 Content-Type: application/json;charset=utf-8 Size: 360 B

However, I should be getting a Status: 200. I try to pass some headers manually but I still get the same Status error:

headers = c(
  'accept' = 'text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
  'accept-encoding' = 'gzip, deflate, br',
  'accept-language' = 'es-ES,es;q=0.9,en;q=0.8',
  'cache-control' = 'max-age=0',
  'referer' = 'https://www.idealista.com/en/',
  'sec-fetch-mode' = 'navigate',
  'sec-fetch-site' = 'same-origin',
  'sec-fetch-user' = '?1',
  'upgrade-insecure-requests' =  '1',
  'user-agent' = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36'
)

url = "https://www.idealista.com/en/venta-viviendas/madrid-provincia/"
x <- GET(url, add_headers(headers))

Any idea how I can get around this Status error?

CodePudding user response：

Your syntax for add_headers is wrong. You can't pass a named vector - you have to pass the arguments directly to the function:

library(httr)

headers <- add_headers(
  'accept' = 'text/html,application/xhtml xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
  'accept-encoding' = 'gzip, deflate, br',
  'accept-language' = 'es-ES,es;q=0.9,en;q=0.8',
  'cache-control' = 'max-age=0',
  'referer' = 'https://www.idealista.com/en/',
  'sec-fetch-mode' = 'navigate',
  'sec-fetch-site' = 'same-origin',
  'sec-fetch-user' = '?1',
  'upgrade-insecure-requests' =  '1',
  'user-agent' = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36'
)

url = "https://www.idealista.com/en/venta-viviendas/madrid-provincia/"

GET(url, headers)
#> Response [https://www.idealista.com/en/venta-viviendas/madrid-provincia/]
#>   Date: 2022-04-04 19:10
#>   Status: 200
#>   Content-Type: text/html; charset=UTF-8
#>   Size: 263 kB
#> <!DOCTYPE html>
#> <html lang="en" env="es" username="" data-userauth="false" >
#> <head>
#> <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
#> <title>Property for sale in Madrid province, Spain: houses and flats &#8212; ...
#> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
#> <meta name="description" content="37,980 houses and flats for sale in Madrid,...
#> <meta name="author" content="idealista.com">
#> <meta http-equiv="cleartype" content="on">
#> <meta name="pragma" content="no-cache"/>
#> ...

^{Created on 2022-04-04 by the reprex package (v2.0.1)}