I am stuck at the following, I want to check the below site in an automate way if there are new packages/shares available (new boxes with a buy option essentially):
https://staking.pocketfives.com/staking/market-place
Given You don't need to log in, it should be okay to scrape afaik.
So far I have read a dozen questions and tried all the methods in the title, however I was unable to make it work. When I use Beautifulsoup or requests_html it just gives me back only the title and one other line of text, not the text in the boxes that I need. The closest I got is this:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import pyperclip
from webdriver_manager.chrome import ChromeDriverManager
import time
import random
link='https://staking.pocketfives.com/staking/market-place'
CD=ChromeDriverManager().install()
driver = webdriver.Chrome(CD)
driver.get(link)
element=driver.find_element_by_tag_name('body')
time.sleep(random.uniform(6, 12))
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(random.uniform(3, 6))
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(random.uniform(3, 6))
element.send_keys(Keys.CONTROL,'a')
element.send_keys(Keys.CONTROL,'c')
driver.quit()
alltext=pyperclip.paste()
print(alltext)
I then check every so many minutes in a loop and compare the text from iteration x 1 to iteration x to see if anything has changed.
However this has one big problem, even when scrolling through the website, this does not copy all text! Just the last part of the site.
This is obviously not the most elegant way either, so I am very open to other solutions. I tried making it work with other packages, I just can't seem to run the dynamic scripts on the site correctly, even using render etc. I went into the HTML code but I have trouble understanding it and it consists of a lot of scripts.
Help is much appreciated!
CodePudding user response：
I'm not really sure what you are after, but they do have an api. Does this get you what you want?
import requests
import pandas as pd
url = 'https://api2.pocketfives.com/FrontOfficeStake/GetStakeListRequest'
payload= {
'currentPage': '1',
'pageSize': '9999'}
jsonData = requests.post(url, json=payload).json()
df = pd.DataFrame(jsonData['stakeList'])
Output:
print(df.to_string())
stakeId sellerId sellerName sellerImageUrl venueId venueName venueImageUrl tournamentId startTime title buyIn guarantee markup offeredPercent offeredPrice thresholdPercent thresholdPrice purchaseCapPercent purchaseCapPrice soldPercent soldPrice availablePercent availablePrice note hasPassword myTransactionSummary replyCount
0 108 481 RobinPoker //cdn.pocketfives.com/v72/monthly_2021_10/6A79114C-C8C4-4EB1-8476-D1E9E01B1D83.thumb.jpeg.f629f2b49f04b8c9d2c7c272e04cd973.jpeg 1 WSOP https://cdn.pocketfives.com/staking/common/venue/wsop.png 1234 2021-11-01T18:00:00 Event #61: Deepstack Championship No-Limit Hold'em 600.0 0.0 1.3 50.0 390.00 0.0 0.0 5.00 39.00 50.000 390.00 0.000 0.0 <p><span style="background-color: rgb(246, 248, 249);">*You are purchasing action for FIRST BULLET ONLY (if player get knocked out and re-enters, you DO NOT have action.)</span></p> False None 0
1 39 480 Pamsi //cdn.pocketfives.com/v72/monthly_2021_09/pam_gg.thumb.jpg.ce2c1f50b5bc3c805fa4545ab9a3227d.jpg 1 WSOP https://cdn.pocketfives.com/staking/common/venue/wsop.png 1238 2021-11-03T18:00:00 Event #65: MINI Main Event No-Limit Hold’em (freezeout) 1000.0 0.0 1.1 15.0 165.00 0.0 0.0 1.00 11.00 15.000 165.00 0.000 0.0 <p>Pamela Balzano</p> False None 0
2 42 23 JonathanLittle //cdn.pocketfives.com/v72/monthly_2021_09/1004.thumb.JPG.10a21649c3a3cfef8d0ebc5d6f133ec9.JPG 1 WSOP https://cdn.pocketfives.com/staking/common/venue/wsop.png 1240 2021-11-04T18:00:00 Event #67A: MAIN EVENT No-Limit Hold'em World Championship - Day 1A 10000.0 0.0 1.0 10.0 1000.00 0.0 0.0 0.10 10.00 10.000 1000.00 0.000 0.0 None False None 0
3 109 481 RobinPoker //cdn.pocketfives.com/v72/monthly_2021_10/6A79114C-C8C4-4EB1-8476-D1E9E01B1D83.thumb.jpeg.f629f2b49f04b8c9d2c7c272e04cd973.jpeg 1 WSOP https://cdn.pocketfives.com/staking/common/venue/wsop.png 1244 2021-11-08T20:00:00 Event #68A: LITTLE ONE FOR ONE DROP No-Limit Hold’em 1111.0 0.0 1.2 50.0 666.60 0.0 0.0 5.00 66.66 50.000 666.60 0.000 0.0 <p><span style="background-color: rgb(246, 248, 249);">*You are purchasing action for FIRST BULLET ONLY (if player get knocked out and re-enters, you DO NOT have action.)</span></p> False None 0
4 40 480 Pamsi //cdn.pocketfives.com/v72/monthly_2021_09/pam_gg.thumb.jpg.ce2c1f50b5bc3c805fa4545ab9a3227d.jpg 1 WSOP https://cdn.pocketfives.com/staking/common/venue/wsop.png 1244 2021-11-08T20:00:00 Event #68A: LITTLE ONE FOR ONE DROP No-Limit Hold’em 1111.0 0.0 1.1 15.0 183.32 0.0 0.0 1.00 12.22 15.000 183.32 0.000 0.0 <p>Pamela Balzano</p> False None 0
5 41 480 Pamsi //cdn.pocketfives.com/v72/monthly_2021_09/pam_gg.thumb.jpg.ce2c1f50b5bc3c805fa4545ab9a3227d.jpg 1 WSOP https://cdn.pocketfives.com/staking/common/venue/wsop.png 1248 2021-11-11T20:00:00 Event #70A: CRAZY EIGHTS No-Limit Hold'em 8-Handed 888.0 0.0 1.1 15.0 146.52 0.0 0.0 1.00 9.77 15.000 146.52 0.000 0.0 <p>Pamela Balzano</p> False None 0
6 59 23 JonathanLittle //cdn.pocketfives.com/v72/monthly_2021_09/1004.thumb.JPG.10a21649c3a3cfef8d0ebc5d6f133ec9.JPG 1 WSOP https://cdn.pocketfives.com/staking/common/venue/wsop.png 1257 2021-11-15T23:00:00 Event #76: Super Turbo Bounty No-Limit Hold'em (freezeout) 10000.0 0.0 1.0 10.0 1000.00 0.0 0.0 0.10 10.00 10.000 1000.00 0.000 0.0 <p>*You are purchasing action for FIRST BULLET ONLY (if player gets knocked out and re-enters, you DO NOT have action.)</p><p><br></p><blockquote><br></blockquote><p><br></p> False None 0
7 34 56 Daniel Negreanu //cdn.pocketfives.com/v72/monthly_2021_09/239657270_ScreenShot2021-09-10at10_04_11AM.thumb.png.fdefefe1ee4ccfa9148e72b34c59bdf0.png 1 WSOP https://cdn.pocketfives.com/staking/common/venue/wsop.png 1263 2021-11-18T23:00:00 Event #82: Super High Roller No-Limit Hold'em 250000.0 0.0 1.0 25.0 62500.00 0.0 0.0 0.20 500.00 25.000 62500.00 0.000 0.0 <p>*You are purchasing action for FIRST BULLET ONLY (if player get knocked out and re-enters, you DO NOT have action.</p><p><br></p><p>Are you guys as ready for the WSOP as I am? This is my first public offering for a piece of me in some WSOP action! Ive opened up 25% of of my action at NO MARKUP! Lets make some money and have some fun!</p> False None 0
8 43 23 JonathanLittle //cdn.pocketfives.com/v72/monthly_2021_09/1004.thumb.JPG.10a21649c3a3cfef8d0ebc5d6f133ec9.JPG 1 WSOP https://cdn.pocketfives.com/staking/common/venue/wsop.png 1267 2021-11-20T23:00:00 Event #85: HIGH ROLLER No-Limit Hold’em 50000.0 0.0 1.0 20.0 10000.00 0.0 0.0 0.20 100.00 15.700 7850.00 4.300 2150.0 None False None 0
9 52 23 JonathanLittle //cdn.pocketfives.com/v72/monthly_2021_09/1004.thumb.JPG.10a21649c3a3cfef8d0ebc5d6f133ec9.JPG 1 WSOP https://cdn.pocketfives.com/staking/common/venue/wsop.png 1269 2021-11-21T23:00:00 Event #87: High Roller No-Limit Hold’em 100000.0 0.0 1.0 70.0 70000.00 69.0 69000.0 0.00 0.00 64.852 64852.00 5.148 5148.0 <p>Please be aware that if this package does not sell out in its entirety, all investors will get a full refund and the package is canceled.</p> False None 0
10 36 56 Daniel Negreanu //cdn.pocketfives.com/v72/monthly_2021_09/239657270_ScreenShot2021-09-10at10_04_11AM.thumb.png.fdefefe1ee4ccfa9148e72b34c59bdf0.png 1 WSOP https://cdn.pocketfives.com/staking/common/venue/wsop.png 1269 2021-11-21T23:00:00 Event #87: High Roller No-Limit Hold’em 100000.0 0.0 1.0 25.0 25000.00 0.0 0.0 0.25 250.00 25.000 25000.00 0.000 0.0 <p><span style="background-color: rgb(246, 248, 249);">*You are purchasing action for FIRST BULLET ONLY (if player get knocked out and re-enters, you DO NOT have action.</span></p> False None 0