Home > Software engineering >  python request to get html page is missing the article element contents
python request to get html page is missing the article element contents

Time:11-12

im trying to record data from enter image description here

Now how do we replicate this in a scraper?

We need a websocket client and send those green messages to start receiving the data you want. For example websocket-client package we can do:

from websocket import create_connection
ws = create_connection("wss://bitcoin.clarkmoody.com/dashboard/ws")

# replicate the green messages
ws.send("""{"op":"c","ch":"","pl":{"c":"4de43be4236035c5","s":"9f6e08f07c263998"}}""")
ws.send("""{"op":"sub","ch":"mod"}""")
ws.send("""{"op":"sub","ch":"sta"}""")
ws.send("""{"op":"sub","ch":"sys"}""")
ws.send("""{"op":"sub","ch":"upd"}""")

# then you can start receiving the data

while True:
    print(ws.recv())

Now it's up to you to figure out the rest of reverse-engineering. For the initial messages it seems to be some sort of subscription (op: sub, ch:upd probably means operation subscribe channel UPD). Either way the above script should output this as first response message then continue spewing price adjustments:

{"op":"dat","ch":"mod","pl":[{"rows":[{"slug":"p-row","cells":[{"type":"label","slug":"p-label","quiet":true,"label":"Price"},{"type":"price","slug":"p","def":"Market price of Bitcoin","sep":true,"unit":"$","prefix":true,"places":2}]},{"slug":"sd-row","cells":[{"type":"label","slug":"sd-label","quiet":true,"label":"Sats per Dollar"},{"type":"integer","slug":"sd","def":"Value of one US Dollar, expressed in Satoshis","sep":true,"places":0}]},{"slug":"c-row","cells":[{"type":"label","slug":"c-label","quiet":true,"label":"Market Capitalization"},{"type":"price","slug":"c","def":"Product of market price times total mined supply","sep":true,"unit":"$","prefix":true,"places":2}]}],"slug":"markets","name":"Markets","order":10,"help":"Bitcoin spot price and futures information","feature":false,"headless":false},{"rows":[{"slug":"links-row","noInfo":true,"cells":<..TOO LONG FOR SO..>
  • Related