Home > database >  Can't scrape information going back to the number of days I specify
Can't scrape information going back to the number of days I specify

Time:10-08

I want to grab the posting date and title from this webpage using a python script that I've already created. The script is working errorlessly.

Now I want to add some logic to the script so that it will only fetch stuff up to the day when I enter a number of days into variable days_to_go_back. For example, if I enter 2 right next to days_to_go_back, the script should scrape content up to October 6.

import requests
from bs4 import BeautifulSoup

link = 'https://atlanta.craigslist.org/search/lab'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}

days_to_go_back = 2

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link)
    soup = BeautifulSoup(res.text,"lxml")
    for item in soup.select("[id^='search-results'] li[class*='result']"):
        date = ' '.join(item.select_one("time.result-date").text.split())
        title = item.select_one("h3.result-heading > a").get_text(strip=True)
        print(date,title)

Current output:

Oct 7 General labor crew member needed
Oct 7 General Labor Demo
Oct 7 LAWN CARE & AERATION SPECIALISTS (ROSWELL, GA)
Oct 7 LAWN CARE & AERATION SPECIALISTS (ROSWELL, GA)
Oct 7 Craftsman Wanted
Oct 7 Home Cleaners Wanted! Make over $145  for 1bd, $165 for 2bd cleans.
Oct 7 Sign Company In Search of Installers and Fabricators
Oct 7 ***PILE DRIVERS-DRILLERS-TRAINEES***
Oct 7 Warehouse Package Handler
Oct 7 Workers Needed for Local Fair (Cash Paid Daily)
Oct 7 Warehouse Team Member - Multiple Shifts Available
Oct 7 Construction
Oct 7 Lyft Driver (part-time / full-time)
Oct 7 Lyft Driver (part-time / full-time)
Oct 7 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 7 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 7 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 7 Amazon Warehouse Assistant - Immediate Hire
Oct 7 Amazon Picker - Overnight Shifts Available
Oct 7 Warehouse Team Member - With Sign-On Bonus!
Oct 7 Amazon Warehouse Assistant
Oct 7 Need a Crew of 4 
Oct 6 Helper and Construction
Oct 6 Truck Driver/El Camionero
Oct 6 Workers Needed for Local Fair (Cash Paid Daily)
Oct 6 Roofing Crews Needed! ($2,000 Sign On Bonus - Start Today!)
Oct 6 Available Immediately General Labor/Driver with Truck or Van Positions
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 Automotive Technicians Needed
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 Automotive Technicians Needed
Oct 6 Automotive Technicians Needed
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 6 $20-25/hr. SIMPLE VALET TRASH COLLECTOR-Camp Creek/East Point 30331
Oct 6 Asphalt workers labors
Oct 6 CDL-A Company Drivers - $5K SIGN-ON BONUS!            
  • Related