Web scraping data that is not displayed on a webpage using rvest-CodePudding

This is a follow-up to a previous question: Scraping data using R and placing results in a data frame

I'm trying to scrape reviews from Glassdoor including the sub-ratings (work-life balance, culture and values, etc). The sub-ratings are in a drop down menu and are displayed as a number of stars (1-5). Dave2e posted a very helpful solution to my previous question, but I've found that some companies' review pages are formatted differently so that the solution doesn't work. An example of a company where it doesn't work is below.

library(stringr)
library(httr)  
library(xml2)  
library(rvest) 
library(purrr) 
library(tidyverse)
library(lubridate)

Subratings <- data.frame()
url <- "https://www.glassdoor.com/Reviews/Fresenius-Medical-Care-North-America-Reviews-"
settings_url <- ".htm?filter.iso3Language=eng"

for (x in 1:3) {
     pg_reviews <- read_html(GET(paste(url, "E10445", "_P", x, settings_url, sep = "")))
     
     #the ratings are stored in a data structure in a script
     #find all the scripts and then search
     scripts<-pg_reviews %>% html_elements(xpath='//script')
     
     #search the scripts for the ratings
     ratingsScript <- which(grepl("ratingCareerOpportunities", scripts))
     #filter the script down to just the data.  This is JSON like haven't figured out the beginning or end
     data1 <-scripts[ratingsScript] %>% html_text2() %>% str_extract("\"urlParams\":. \\}\\}\\}\\}") 
     
     
     #extract the ratings
     WorkLifeBalance  <- str_extract_all(data1, '(?<="ratingWorkLifeBalance":)\\d') %>% unlist() %>% as.integer()
     CultureAndValues <- str_extract_all(data1, '(?<="ratingCultureAndValues":)\\d') %>% unlist() %>% as.integer()
     DiversityAndInclusion        <- str_extract_all(data1, '(?<="ratingDiversityAndInclusion":)\\d') %>% unlist() %>% as.integer()
     SeniorLeadership <- str_extract_all(data1, '(?<="ratingSeniorLeadership":)\\d') %>% unlist() %>% as.integer()
     CareerOpportunities <- str_extract_all(data1, '(?<="ratingCareerOpportunities":)\\d') %>% unlist() %>% as.integer()
     CompensationAndBenefits<- str_extract_all(data1, '(?<="ratingCompensationAndBenefits":)\\d') %>% unlist() %>% as.integer()
     
     #Combine columns
     combine <- cbind(WorkLifeBalance,CultureAndValues,DiversityAndInclusion,SeniorLeadership,
                      CareerOpportunities,CompensationAndBenefits)
     
     Subratings <- rbind(Subratings,combine)     
}

CodePudding user response：

It looks like this page has one less closing parenthesis, try: str_extract("\"urlParams\":. \\}\\}\\}").
This should work on the previous pages also.

After much searching, the employee reviews are stored in the string starting with "reviews": and ending with }]}.
By adding a leading { to this turns the reviews into valid JSON and thus a simple conversion.

library(stringr) 
library(httr)
library(xml2)
library(rvest) 
library(dplyr)

Subratings <- data.frame() 
url <- "https://www.glassdoor.com/Reviews/Fresenius-Medical-Care-North-America-Reviews-" 
settings_url <- ".htm?filter.iso3Language=eng"

dfs <- lapply(1:3, function(x) { 
   pg_reviews <- read_html(GET(paste(url, "E10445", "_P", x, settings_url, sep = "")))

   #the ratings are stored in a data structure in a script
   #find all the scripts and then search
   scripts<-pg_reviews %>% html_elements(xpath='//script')

   #search the scripts for the ratings
   ratingsScript <- which(grepl("ratingCareerOpportunities", scripts))

   #Extract text for the reviews from the script.  This is almost valid JSON format
   reviews <-scripts[ratingsScript] %>% html_text2() %>% 
                          str_extract("\"reviews\":. ?\\}\\]\\}") 
  # char <- nchar(reviews)  #debugging status

   #add a leading { to make valid JSON and convert
   answer <-jsonlite::fromJSON(paste("{", reviews))
   answer
})

bind_rows(dfs)