Scrape data from graph generated with Highcharts-CodePudding

Trying to scrape the price history of several games from a website.

Highcharts.js is used to generate a graph with two series from historical data. A sample page is https://gg.deals/game/snowrunner/.

I can access the data with JavaScript using:

Highcharts.charts[0].series[0].data

and

Highcharts.charts[0].series[1].data

However, I would like to know if there is another method I could use to get the data without having to parse JavaScript code.

CodePudding user response：

One Option is to use the api that provides the data for the highcharts.

Note: There are different sources data-without-keyshops-url and data-with-keyshops-url

Step#1 - Generate the dataUrl from the product page

soup = bs(requests.get('https://gg.deals/game/snowrunner/').text, 'lxml')

dataUrl = 'https://gg.deals' soup.select_one('#historical-chart-container')['data-without-keyshops-url']

Step#2 - Request the json data

r  = requests.get(dataUrl, headers={'X-Requested-With': 'XMLHttpRequest'})
r.json()

Example

import requests,json
from bs4 import BeautifulSoup as bs

url = 'https://gg.deals/game/snowrunner/'
r = requests.get(url)
soup = bs(r.text, 'lxml')

dataUrl = 'https://gg.deals' soup.select_one('#historical-chart-container')['data-without-keyshops-url']
r  = requests.get(dataUrl, headers={'X-Requested-With': 'XMLHttpRequest'})
r.json()['chartData']['deals']

Output

[{'x': 1581956320000,
  'y': 39.99,
  'shop': 'Epic Games Store',
  'name': '17 Feb 2020 16:18 - 22 May 2020 15:38'},
 {'x': 1590161908000,
  'y': 29.99,
  'shop': 'Epic Games Store',
  'name': '22 May 2020 15:38 - 11 Jun 2020 17:11'},
 {'x': 1591895501000,
  'y': 39.99,
  'shop': 'Epic Games Store',
  'name': '11 Jun 2020 17:11 - 16 Jun 2020 13:18'},
 {'x': 1592313517000,
  'y': 31.99,
  'shop': 'Epic Games Store',
  'name': '16 Jun 2020 13:18 - 30 Jun 2020 13:20'},
 {'x': 1593523255000,
  'y': 39.99,
  'shop': 'Epic Games Store',
  'name': '30 Jun 2020 13:20 - 23 Jul 2020 16:08'},
 {'x': 1595520520000,
  'y': 31.99,
  'shop': 'Epic Games Store',
  'name': '23 Jul 2020 16:08 - 6 Aug 2020 15:03'},
 {'x': 1596726196000,
  'y': 39.99,
  'shop': 'Epic Games Store',
  'name': '6 Aug 2020 15:03 - 24 Sep 2020 16:19'},...]