I want to scrape data from https://www.nasdaqtrader.com/trader.aspx?id=TradeHalts. I tried different approaches, like this, this, and this.

I could scrap static pages, but still don't understand the aspx format very well. I am copying here what I took from the first reference link:

import urllib
from bs4 import BeautifulSoup

headers = {
    'Accept':'text/html,application/xhtml xml,application/xml;q=0.9,*/*;q=0.8',
    'Origin': 'http://www.indiapost.gov.in',
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.17 (KHTML, like Gecko)  Chrome/24.0.1312.57 Safari/537.17',
    'Content-Type': 'application/x-www-form-urlencoded',
    'Referer': 'http://www.nitt.edu/prm/nitreg/ShowRes.aspx',
    'Accept-Encoding': 'gzip,deflate,sdch',
    'Accept-Language': 'en-US,en;q=0.8',
    'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
}

class MyOpener(urllib.request.FancyURLopener):
    version = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17'

myopener = MyOpener()
url = 'https://www.nasdaqtrader.com/Trader.aspx?id=TradeHalts'
# first HTTP request without form data
f = myopener.open(url)
soup = BeautifulSoup(f)
# parse and retrieve two vital form values
viewstate = soup.findAll("input", {"type": "hidden", "name": "__VIEWSTATE"})
eventvalidation = soup.findAll("input", {"type": "hidden", "name": "__EVENTVALIDATION"})

formData = (
     ('__EVENTVALIDATION', eventvalidation),
     ('__VIEWSTATE', viewstate),
     ('__VIEWSTATEENCRYPTED', ''),
)

encodedFields = urllib.parse.urlencode(formData)
# second HTTP request with form data
f = myopener.open(url, encodedFields)

# We use BeautifulSoup
soup = BeautifulSoup(f)

print(soup.content)

I cannot find the table information in the content. What am I missing?

Thanks for your help

CodePudding user response：

To get the data as pandas DataFrame you can use next example:

import requests
import pandas as pd
from io import StringIO


url = "https://www.nasdaqtrader.com/RPCHandler.axd"

headers = {
    "Referer": "https://www.nasdaqtrader.com/trader.aspx?id=TradeHalts",
}

payload = {
    "id": 2,
    "method": "BL_TradeHalt.GetTradeHalts",
    "params": "[]",
    "version": "1.1",
}

data = requests.post(url, json=payload, headers=headers).json()
data = StringIO(data["result"])

df = pd.read_html(data)[0]
print(df.head(10).to_markdown(index=False))

Prints: