Relatively new to python and selenium. Trying to run the below code in python using selenium library. The drop down does not get activated on using the css selector. Also im not sure if im using the right syntax for selecting the date.
Any help will be appreciated. Thanks in advance.!!
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
s=Service("C:\\python\\chromedriver.exe")
driver = webdriver.Chrome(service=s)
driver.maximize_window()
wait=WebDriverWait(driver,20)
driver.get('http://www.amfiindia.com/research-information/other-data/mf-scheme-performance-details')
time.sleep(2)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,"#form-fund-details > div > div:nth-child(1) > div:nth-child(1) > div > div > button > span.bs-caret > span"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH,"//*[@class='end-type']/a[.='Open-ended']"))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,"##form-fund-details > div > div:nth-child(1) > div:nth-child(2) > div > div > button > span.bs-caret > span"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH,"//*[@class='primary-category']/a[.='Debt']"))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,"#form-fund-details > div > div:nth-child(1) > div:nth-child(3) > div > div > button > span.bs-caret > span"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH,"//*[@class='category']/a[.='Medium Duration']"))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,"#form-fund-details > div > div:nth-child(1) > div.col-sm-3.pdl0 > div > div > button > span.bs-caret > span"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH,"//*[@class='amc']/a[.='All']"))).click()
datefield = driver.find_element_by_id('nav-date')
datefield.click()
datefield.send_keys("09-Mar-2022")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'.btn btn-primary amfi-btn'))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,"#download-report-excel > img"))).click()
driver.quit()
CodePudding user response：
I am not clear what the problem is. I am assuming from your description that you are finding an issue choosing the date from the date picker? If so, the date picker is in an Iframe. We need to switch to it first. Also we need to clear the content of the input field before we send the new data. Quick and dirty code below. Hope this helps. This is in Java. From the syntax, you can easily substitue for eq Python.
driver.get("https://www.amfiindia.com/research-information/other-data/mf-scheme-performance-details");
driver.manage().window().maximize();
WebDriverWait wait = new WebDriverWait(driver, 20);
wait.until(ExpectedConditions.frameToBeAvailableAndSwitchToIt(By.xpath("//iframe[@src=' https://www.valueresearchonline.com/amfi/fund-performance']")));
wait.until(ExpectedConditions.elementToBeClickable(By.cssSelector("input#nav-date")));
driver.findElement(By.cssSelector("input#nav-date")).clear();
driver.findElement(By.cssSelector("input#nav-date")).sendKeys("07-Mar-2022");
CodePudding user response：
I would just skip using Selenium here. The only issue is that it is checked for CAPTCHA with cloudflare. However, there is a nifty package to handle that:
first do pip install cloudscraper in your environment.
Code:
import pandas as pd
import cloudscraper
nav_date = '09-Mar-2022'
url = 'https://www.valueresearchonline.com/amfi/fund-performance-data/'
payload = {
'end-type': '1',
'primary-category': 'SDT',
'category': 'SDT_MD',
'amc': 'ALL',
'nav-date': nav_date}
scraper = cloudscraper.CloudScraper()
res = scraper.get(url, params=payload).text
df = pd.read_html(res)[0]
df.to_csv(f'fund-performance-{nav_date}.csv')
Output:
print(df.to_string())
Scheme Benchmark Riskometer Latest NAV (₹) Previous NAV (₹) 1-Year Return (%)3-Year Return (%)5-Year Return (%)10-Year Return (%)Returns Since Launch (%) Daily AUM (Cr.)
Scheme Benchmark Scheme Benchmark Regular Direct Regular Direct Benchmark Unnamed: 9_level_1 Daily AUM (Cr.)
0 Aditya Birla Sun Life Medium Term Fund CRISIL Medium Term Debt Index Very High Moderate 26.2581 27.8779 9.20 9.84 6.38 NaN 1543.26
1 Axis Strategic Bond Fund NIFTY Medium Duration Index Moderately High Moderate 22.1686 23.8259 5.56 6.31 6.19 NaN 1836.37
2 BNP Paribas Medium Term Fund CRISIL Medium Term Debt Index Moderate Moderate 15.3021 16.2018 3.81 4.39 6.38 NaN 19.34
3 DSP Bond Fund CRISIL Medium Term Debt Index Moderate Moderate 65.8516 69.1150 4.41 4.80 6.38 NaN 356.60
4 HDFC Medium Term Debt Fund NIFTY Medium Duration Index Moderately High Moderate 45.4989 48.2688 5.73 6.53 6.19 NaN 4094.24
5 ICICI Prudential Medium Term Bond Fund CRISIL Medium Term Debt Index Moderately High Moderate 35.5755 38.2344 5.61 6.35 6.38 NaN 6886.97
6 IDFC Bond Fund Medium Term Plan NIFTY Medium Duration Index Moderate Moderate 37.8636 40.3042 4.57 5.30 6.19 NaN 2564.22
7 Kotak Medium Term Fund NIFTY Medium Duration Index Moderate Moderate 18.2767 19.6585 6.03 7.12 6.19 NaN 2975.84
8 L&T Resurgent India Bond Fund NIFTY Medium Duration Index Moderate Moderate 16.4484 17.5125 4.81 5.76 6.19 NaN 1037.95
9 Nippon India Strategic Debt Fund CRISIL Medium Term Debt Index High Moderate 12.5607 13.3599 18.24 18.89 6.38 NaN 230.34
10 SBI Magnum Medium Duration Fund NIFTY Medium Duration Index Moderately High Moderate 40.9884 43.5373 4.87 5.44 6.19 NaN 10426.94
11 Sundaram Medium Term Bond Fund NIFTY Medium Duration Index Moderate Moderate 58.8925 63.8394 2.87 3.97 6.19 NaN 59.83
12 Tata Medium Term Fund CRISIL Medium Term Debt Index Moderate Moderate 29.5529 32.3416 5.57 6.55 6.38 NaN 83.34
13 Union Medium Duration Fund CRISIL Medium Term Debt Index Moderate Moderate 10.6048 10.6718 4.82 5.21 6.38 NaN 219.25
14 UTI Medium Term Fund CRISIL Medium Term Debt Index Moderate Moderate 14.9461 15.6951 7.24 8.01 6.38 NaN 61.28