Scraping pre-market table on Barchart with Python-CodePudding

Apologies if this is a bit website specific (barchart.com). I used the guidance provided here for properly connecting and scraping barchart.com for Futures data. However, after hours of trying, I am at a loss as to how to pull off this same trick for their pre-market data table: Barchart_Premarket_Site.

Anyone know the trick to get the pre-market data?

Here is the basic connection, for which i get a 403:

import requests

geturl=r'https://www.barchart.com/stocks/pre-market-trading/volume-advances?orderBy=preMarketVolume&orderDir=desc'

s=requests.Session()
r=s.get(geturl)

#j=r.json()
print(r)`

CodePudding user response：

All that was required was to add more headers to the request. You can find your own headers using chrome > developer tools; and then just find the api request for the table and slam in a few of the headers associated with that request.

import requests


request_url = "https://www.barchart.com/proxies/core-api/v1/quotes/get?lists=stocks.us.premarket.volume_advances&orderDir=desc&fields=symbol,symbolName,preMarketLastPrice,preMarketPriceChange,preMarketPercentChange,preMarketVolume,preMarketAverage5dVolume,preMarketPreviousLast,preMarketPreviousChange,preMarketPreviousPercentChange,preMarketTradeTime,nextEarningsDate,nextEarningsDate,timeCode,symbolCode,symbolType,hasOptions&orderBy=preMarketVolume&meta=field.shortName,field.type,field.description,lists.lastUpdate&hasOptions=true&page=1&limit=100&raw=1"

headers = {
    'accept': 'application/json',
    'cookie': '_gcl_au=1.1.685644914.1670446600; _fbp=fb.1.1670446600221.1987872306; _pbjs_userid_consent_data=3524755945110770; _pubcid=e7cf9178-59bc-4a82-b6c4-a2708ed78b8d; _admrla=2.2-1e3aed0d7d9d2975-a678aeef-7671-11ed-803e-d12e87d011f0; _lr_env_src_ats=false; _cc_id=6c9e21e7f9c269f8501e2616f9e68632; __browsiUID=c0174d21-a0ab-4dfe-8978-29ae08f44964; __qca=P0-531499020-1670446603686; __gads=ID=220b766bf87e15f9-22fa0316ded8001f:T=1670446598:S=ALNI_MaEWcBqESsJKLF0AwoIVvrKjpjZ_g; panoramaId_expiry=1673549551401; panoramaId=9aa5615403becfbc8adf14a3024816d53938b8cdbea6c8f5cabb60112755d70c; udmsrc={}; _pk_id.1.73a4=1aee00a1c66e897b.1672997455.; _ccm_inf=1; bcPremierAdsListScreen=true; _hjSessionUser_2563157=eyJpZCI6ImI2MTM5NTQ4LWUxYzMtNTU2NS04MmM3LTk4ODQ5MWNjY2YxZCIsImNyZWF0ZWQiOjE2NzMwMzQ3OTY0NDAsImV4aXN0aW5nIjp0cnVlfQ==; bcFreeUserPageView=0; _gid=GA1.2.449489725.1673276404; _ga_4HQ9CY2XKK=GS1.1.1673303248.3.0.1673303248.0.0.0; _ga=GA1.2.606341620.1670446600; __aaxsc=2; aasd=5|1673314072749; webinar131WebinarClosed=true; _lr_geo_location_state=NC; _lr_geo_location=US; udm_edge_floater_fcap=[1673397095403,1673392312561,1673078162569,1673076955809,1673075752582,1673066137343,1673056514808,1673051706099,1673042087115,1673037276340,1672960427551,1672952009965,1672947201101]; pbjs-unifiedid={"TDID":"19345091-e7fd-4323-baeb-4627c879c6ba","TDID_LOOKUP":"TRUE","TDID_CREATED_AT":"2022-12-05T19:48:10"}; __gpi=UID=000008c6d06e1e0d:T=1670446598:RT=1673433090:S=ALNI_MZS6mLx8CJg9iN6kzx4JeDFHPOMjg; market=eyJpdiI6InJvcVNudkprUjQ1bE0yWWQrSTlYY1E9PSIsInZhbHVlIjoieUpabHpmSnJGSkIxc0o1enpyb1dLdENBSWp4UE5NYUZwUFg3OGs0TGJSL0dQWUNpTDU0a2hZbklOQTFNd09OVSIsIm1hYyI6IjBjMjJkNDExZjRhOTc2M2QwYWU3NGUyNmVlZTgyMzY2NWM2MjQyOTY2MjY2YmUxODI2Y2RkY2FlNzI3MjNkOTIifQ==; _lr_retry_request=true; __browsiSessionID=c02dadca-6355-415f-aa80-926cccd94759&true&false&DEFAULT&us&desktop-4.11.12&false; IC_ViewCounter_www.barchart.com=2; cto_bundle=dxDlRl90VldIJTJGa0VaRzRIS0xnQmdQOXVWVlhybWJ3NDluY29PelBnM0prMkFxZkxyZWh4dkZNZG9LcyUyRjY1VWlIMWRldkRVRlJ5QW05dHlsQU1xN2VmbzlJOXZFSTNlcFRxUkRxYiUyRlp6Z3hhUHpBekdReU5idVV0WnkxVll0eGp5TyUyQlVzJTJCVDVoRkpWWlZ4R0hOSUl2YTVJVDhBJTNEJTNE; cto_bidid=51ixCl92dkhqbmVmdnlTZHVYS25nWTk2eDVMUnVRNjhEMUhxa3FlcmFzRHVNSERUQkd5cFZrM0QyQyUyRkVNNkV6S0ZHOUZPcTBTR2lBUjA5QUc5YU1ucW9GMFZBWHB4aU9sMlo3WHAlMkJYWjZmJTJGWkpsWSUzRA; _awl=2.1673451629.5-df997ba8dc13bee936d8d14a9771e587-6763652d75732d6561737431-0; laravel_token=eyJpdiI6IjR2YStGblAxWlZoZzllcEtPUUFLNlE9PSIsInZhbHVlIjoiY3E2bHdQWFkyT1FFUHFka2NMMVoyREFvQlZwWXlxc3F0SlRuZnIyTHJsSWtNVFA0K1czcDloWFF2d0lVZys3azZyelkrWks5SWxuRW05MGlqV1I4QmViMU9KKzArVXJOTWNVK2hqZVRocVNHM3NZa1dNeStQbnNyYVBtcjlUeTZzT2lpV2t1ek1UOE1wSUFudmg0NzFTQ3VPeDJiYk16bGNBTzVqVHBCcFRZdTFsZjBVREVyUEhLeThjZm9wSGIzQ2NDVE0ya0xOQWx1VGx0aUlEUE9yakU4Q3RicWFmNDdkYjJSWHlsSWYwajlSUkozVmQ4OVNGNzZEeWhtUExtcXB6VnNrY2NsUzRFQnJyMlhiejFtc0l3U2p5SW5BbFFDZTN0dk9EUWNOR2hVYUdMbmhFUFZVT24xOFFGVkM3L2giLCJtYWMiOiIxYzM5Yzk1ZWNjNjM0NzdjMmM4YTJkZDg0ZmY5MWQwNWUzOTlhNTAwNjg2MTNmNTNlYzY4M2MzYWQ3MDA4MThlIn0=; XSRF-TOKEN=eyJpdiI6Ik1PMGEvOGFkZ1p1ekpNcXIvZWZtcHc9PSIsInZhbHVlIjoiMVZYQ3NCV1hjcWREdG5uSDVqYXZVVy91U29USys1dkJJeFNZZG9QVGNRNDhmMTJIeitVV2NUV0xSUC9ZTThVM3FDQWZBcVdNclhqSkx4MG1NTGhadUNlNXRLMEdUc3RDcEVwNnJVYU9FNTBub2NKRWxlekxBZmZEVXNhZUlwWnoiLCJtYWMiOiIxYTI0N2E2OGMxMzRhNmFiYTliMzBlYTdjYWZlNzUwMDRlY2Q5YjI2YzY4OGZlMWIxYmM0YTE3YzZkMTdhMGM3In0=; laravel_session=eyJpdiI6InJIcmMxRWVacmtGc2tENS9zYUFFOVE9PSIsInZhbHVlIjoibG1vQWh1d1dmaUNBZTV4dGdJbWhTVEoyMWVrblBFNTBycTBPai9ad2llcHRkd0hHUTI4ZS8rUFNFVm5LNEcvd1RXY1RwOHdVZHplNU92Vk9xUHZjYmMrUC9Cc3hJUkJNWE54OVR1UHFaTExpM1BRcWRSWEJ5Q3gvVVNzajdHZUoiLCJtYWMiOiI5NDVkOGU4NGM5Y2MwMThmMTgwMzQyOWQ1Yzc5MzU5ZGU2ZjkwMWRjYzBjZWJiZDFhMTQzODMzZmE2NWExMGQ3In0=',
    'referer': 'https://www.barchart.com/stocks/pre-market-trading/volume-advances?orderBy=preMarketVolume&orderDir=desc',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
    'x-xsrf-token': 'eyJpdiI6Im1LQVRpVEJONzZwMDRVQnhYK0I5SWc9PSIsInZhbHVlIjoiMkRIMnJBb1VDQmRscjNlajF1dVR2eWxRbGNJTGZCNWxMaWk3N0EzQWlyOWk0cXJBK2oyUVJ1N282R2VOVWh6WlhJcXdZdFplZmRqaFhPa203bi9HeFBxckJKeUVzVDRETHI5OHlxNDZnOEF5WVV5NXdNSWJiWk95UlFHRXQwN2siLCJtYWMiOiI1NTkyZjk2M2FlNTE0NDI0ODQ3YmE4ZjIyZDY1MzM2MTA3ZTY4NDA5NzA5YzViMjhiN2UwYTFhNTM1Y2ZkMjk5In0='
}

r = requests.get(request_url,headers=headers)