Try to pass a variable inside a table reference for pd.read_html command. Extract of the code given below. Is there a workaround to assign the number dynamically?
Here want the 6th table in the webpage. There are multiple tables on the webpage numbered 0 to 15 need to assign the table number to a variable.
import cloudscraper
import pandas as pd
url = 'sebi.gov.in/sebiweb/other/OtherAction.do?doPmr=yes'
yr = 2022
mth = 1
z=5
payload = { 'pmrId': 'INP000000043@@INP000000043@@ASK INVESTMENT MANAGERS PRIVATE LIMITED', 'year': yr, 'month': mth}
scraper = cloudscraper.CloudScraper() res = scraper.get(url, params=payload).text
df1 = pd.read_html(res)[z]
The error I get is : list indices must be integers or slices, not set
If I do df1 = pd.read_html(res)[5] it works.
Thanks in advance..
CodePudding user response：
I'm not sure why you are getting the error of z being a set. You might want to add a print statement of print(z) right before to clearly see what's happening. Otherwise, there are some other problems with the code.
'sebi.gov.in/sebiweb/other/OtherAction.do?doPmr=yes'isn't a valid url schema. you need to have thehttps://- This request is with a post, not a get. The params parameter here will not be used since it is a post.
Look at the edit's below to see what you needed to fix:
import cloudscraper
import pandas as pd
url = 'https://sebi.gov.in/sebiweb/other/OtherAction.do?doPmr=yes'
yr = 2022
mth = 1
z=5
payload = { 'pmrId': 'INP000000043@@INP000000043@@ASK INVESTMENT MANAGERS PRIVATE LIMITED', 'year': yr, 'month': mth}
scraper = cloudscraper.CloudScraper()
res = scraper.post(url, data=payload).text
df1 = pd.read_html(res)[z]
Output:
print(df1)
Investment Approach ... Assets under Management (AUM) as on last day of the month (Amount in INR crores)
Investment Approach ... Total
Unnamed: 0_level_2 ... Total
0 ASK Conviction Portfolio ... 187.47
1 ASK Domestic Resurgence Portfolio ... 1022.58
2 ASK Domestic Resurgence Portfolio STP ... 9.39
3 ASK Emerging Opportunities Portfolio ... 704.17
4 ASK Financial Opportunities Portfolio ... 1009.73
5 ASK Financial Opportunities Portfolio STP ... 7.00
6 ASK Growth Portfolio ... 2163.87
7 ASK Growth Portfolio STP ... 5.91
8 ASK High Conviction Portfolio ... 7.74
9 ASK Hybrid Portfolio ... 56.35
10 ASK India A Plus Portfolio ... 2.36
11 ASK India Select Portfolio ... 3848.58
12 ASK India Select Portfolio STP ... 14.31
13 ASK India Vision Portfolio ... 313.33
14 ASK India Vision Portfolio STP ... 7.18
15 ASK Indian Entrepreneur Portfolio ... 17441.59
16 ASK Indian Entrepreneur Portfolio STP ... 168.12
17 ASK Life Portfolio ... 149.47
18 ASK Liquid Strategy ... 335.53
19 ASK Specialised Portfolio ... 493.01
20 ASK Strategic Portfolio ... 82.57
21 Real Estate Special Opportunities Portfolio ... 166.58
22 Total ... 28196.85
[23 rows x 13 columns]