I just wanted to scrape some data from https://www.naukrigulf.com/ but it gave me None value-CodePudding

from bs4 import BeautifulSoup as bs
import requests as r

src = r.get('https://www.naukrigulf.com/construction-jobs?industryType=10&locale=en').content
soup=bs(src,'lxml')

first_item = soup.find('div',{'class':'srp-listing'})

print(first_item)

if it gave you None value please tell me

CodePudding user response：

The page loads data from external URL, so beautifulsoup doesn't see it. You can use this snippet to load data to pandas DataFrame:

import json
import requests
import pandas as pd

api_url = "https://www.naukrigulf.com/spapi/jobapi/search"

params = {
    "ClusterInd": "10",
    "Keywords": "construction",
    "Limit": "30",
    "Offset": "0",
}

headers = {"appId": "205", "clientId": "desktop", "systemId": "1112"}

data = requests.get(api_url, params=params, headers=headers).json()

df = pd.DataFrame([j["Job"] for j in data["Jobs"]])
print(df.head(5).to_markdown(index=False))

Prints:

Designation	Location	jobInfo	Experience	Company	JobId	JdURL	LatestPostedDate	Vacancies	IsTopEmployer	IsFeaturedEmployer	IsPremium	IsWebJob	IsQuickWebJob	IsApplied	IsTopEmployerLite	Whitelistedkeywords	keywords	IsFormBasedApply	jobRedirection	shortlisted	expiringSoon	isArchived	recruiterActive	isRecruiterActive
Construction Manager	Saudi Arabia - Saudi Arabia	We are hiring for the position of Construction Manager Project Location: Dammam Remarks: Aramco project experience preferred	{'Min': '20', 'Max': '25'}	{'Name': 'China Railway Construction Corporation Limited Sa', 'Id': '205049'}	190722000045	https://www.naukrigulf.com/construction-manager-jobs-in-saudi-arabia-in-china-railway-construction-corporation-limited-sa-20-to-25-years-n-cd-205049-jid-190722000045	1658213448	1	false	false	false	false	false	false	false	construction manager,construction,civil,construction management,civil construction	Construction Manager, Construction, Civil, Construction Management, Civil Construction	false	false	False	False	False	True	True
Design and Construction Interface/Engineering Manager	Saudi Arabia - Saudi Arabia	Project Location: Dammam Remarks: Aramco project experience preferred	{'Min': '15', 'Max': '20'}	{'Name': 'China Railway Construction Corporation Limited Sa', 'Id': '205049'}	190722000083	https://www.naukrigulf.com/engineering-manager-jobs-in-saudi-arabia-in-china-railway-construction-corporation-limited-sa-15-to-20-years-n-cd-205049-jid-190722000083	1658216985	1	false	false	false	false	false	false	false	design,engineering manager,design engineering,engineering management,engineering	Design, Construction Interface, Engineering Manager, Design Engineering, Design Construction, Construction Interface, Engineering Management, Engineering	false	false	False	False	False	True	True
General Manager/Construction Manager(Arabic Speaker)	Abu Dhabi - United Arab Emirates	Drive and spearhead change management, when required to develop cost estimates for the performance of extra work and changed conditions;Prepare trade contracts and bid packages in conjunction with the Procurement and Contract Section, when required, as well as provide feedback for evaluation purposes;Must be Native Arabic Speaker	{'Min': '15', 'Max': '25'}	{'Name': 'Confidential Company', 'Id': '200495'}	210722000035	https://www.naukrigulf.com/general-manager-constructio-jobs-in-abu-dhabi-uae-in-confidential-15-to-25-years-n-cd-200495-jid-210722000035	1658390954	1	false	false	false	false	false	false	false	construction management,project manager,civil construction,project management,infrastructure	construction management, project manager, Civil Construction, Project Management, Infrastructure	false	false	False	False	False	True	True
Construction Manager	Dubai - United Arab Emirates	-At least 15 years of experience in construction project management and from the Main Contractor background;Negotiate contracts with vendors and present formal documentation for approval when required Ensure that contracts are fit for purpose, cost-effective and incorporate appropriate Service Level Agreements Provide overall site management, coordination, planning, specification of business proposals and coordination of subcontractors Study job specifications to determine appropriate co...	{'Min': '10', 'Max': '15'}	{'Name': 'Confidential Company', 'Id': '89635'}	200722000081	https://www.naukrigulf.com/construction-manager-jobs-in-dubai-uae-in-confidential-10-to-15-years-n-cd-89635-jid-200722000081	1658311596	1	false	false	false	false	false	false	false	construction management,construction,project,civil,site	Construction Management, Construction, Project, Civil, Site	false	false	False	False	False	True	True
QC Manager With Underpass Construction Project Experience	Al Madina Al Munawarah , Riyadh , Jubail - Saudi Arabia	Experience in underground tunnel construction is preferred;Minimum 10 years working experience in engineering construction as a QC Manager; 2;Proficient in English speaking and writing	{'Min': '10', 'Max': '16'}	{'Name': 'Branch of China Railway 18th Bureau Group Co Ltd', 'Id': '192341'}	190722000003	https://www.naukrigulf.com/project-manager-constructio-jobs-in-al-madina-al-munawarah-saudi-arabia-in-branch-of-china-railway-18th-bureau-group-co-ltd-10-to-16-years-n-cd-192341-jid-190722000003	1658175153	1	false	false	false	false	false	false	false		QC Manager With Underpass Construction Project Experience	false	false	False	False	False	False	False

CodePudding user response：

import cloudscraper

scraper = cloudscraper.create_scraper()

url = 'https://www.naukrigulf.com/spapi/jobapi/search?ClusterInd=10&Experience=&Keywords=construction&KeywordsAr=&Limit=30&Location=&LocationAr=&Offset=30&SortPreference=relevance&breadcrumb=1&clusterSelected=1&geoIpCityName=Tokyo&geoIpCountryName=Japan&locationId=87&nationality=&nationalityLabel=&pageNo=2&seo=1&showBellyFilters=true&srchId='

headers = {
    'appId' : '205',
    'systemId': '1112'
}

r = scraper.get(url, headers=headers)
print(r.json())