Extract single data point from multiple, webscraping-CodePudding

I am trying to extract stock symbols (3rd column) from the table in below screener:

https://chartink.com/screener/2-short-trend and pass them on to a dataframe.

Due to my limited knowledge, I have hit a wall and can not move past it.

My code is:

from requests_html import HTMLSession
session = HTMLSession()
response = session.get('https://chartink.com/screener/2-short-trend')
response.html.render()

for result in response.html.xpath('//*[@id="DataTables_Table_0"]/tbody/tr/td/a[1]'):
    print(f'{result.text}\n')

Output:

Mahindra & Mahindra Limited

M&M

P&F

Apollo Tyres Limited

APOLLOTYRE

P&F

....

I just need stock symbols: M&M, APOLLOTYRE etc., and passed into a dataframe.

Can someone pls guide.

CodePudding user response：

Bit of a quick fix, but you could use a counter assuming that the relevant output is the second result for every company. Something like the below:

from requests_html import HTMLSession
import pandas as pd

session = HTMLSession()
response = session.get('https://chartink.com/screener/2-short-trend')
response.html.render()

i = 1
symbols = []
for result in response.html.xpath('//*[@id="DataTables_Table_0"]/tbody/tr/td/a[1]'):
    print(f'{result.text}\n')
    if i == 2:
        symbols.append(result.text)
        i -= 2
    else:
        i  = 1

df = pd.DataFrame({"Symbol": symbols})

I structured i to trigger appending the result to a symbols list at the position where the symbol is iterated over and then a dataframe is created using the output. Using that code gave me a dataframe with the 5 symbols from your link.