I want to get data as json format. Right now I am getting data as dictionary which little bit messy for me. Here is my code:

my_dict = {"job_title":[],"time_posted":[],"number_of_proposal":[],"page_link":[]};
for page_num in range(1, 12):
    time.sleep(3)
    url = (
        f'my_url').format(page_num)
    print(url)
    headers = requests.utils.default_headers()
    print(headers)
    headers.update(
        {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0', })
    print(headers)
    r = requests.get(url, headers=headers).text
    soup = BeautifulSoup(r, 'lxml')

    box = soup.select('.item__top_container⤍ListItem⤚3pRrO')
    for i in box:
        job_title = i.select('.item__title⤍ListItem⤚2FRMT')[0].text.lower()
        job_title = job_title.replace('opportunity', ' opportunity').replace(
            'urgent', ' urgent').strip()
        print(job_title)
        time_posted = i.select('time')[0].text.lower()
        remove_month_year = ["month", "year"]
        print(time_posted)
        proposal = i.select(
            '.item__info⤍ListItem⤚1ci50 li:nth-child(3)')[0].text.replace('Proposals', '').strip()
        keywords = ['scrap', 'data mining']
        if(any(key_words in job_title for key_words in keywords)):
            if(not any(remove_m_y in time_posted for remove_m_y in remove_month_year)):
                   my_dict["job_title"].append(job_title)
                   my_dict["time_posted"].append(time_posted)
                   my_dict["number_of_proposal"].append(proposal)
                   my_dict["page_link"].append(url)

my dictionary data look like this:

{'job_title': ['web scraping of product reviews', 'yell web scraping in python', 'google business scraping',],'time_posted': ['6 days ago', '9 days ago', '3 days ago'], 'page_link': ['url1','url2','url3']}

My expected result will be look like this:

{"job_title":"web scraping of product reviews","time_posted":"6 days ago","page_link":"url1"},{"job_title":"yell web scraping in python","time_posted":"9 days ago","page_link":"url2"}

CodePudding user response：

You can change the structure with the following code:

my_list = []

for i in range(len(my_dict["job_title"])):
    my_list.append({
        "job_title": my_dict["job_title"][i],
        "time_posted": my_dict["time_posted"][i],
        "number_of_proposal": my_dict["number_of_proposal"][i],
        "page_link": my_dict["page_link"][i]
    })

Even better would be to directly create the list in the first loop like you need it in the end.

my_list = []
for i in box:
    job_title = i.select('.item__title⤍ListItem⤚2FRMT')[0].text.lower()
    job_title = job_title.replace('opportunity', ' opportunity').replace(
        'urgent', ' urgent').strip()
    print(job_title)
    time_posted = i.select('time')[0].text.lower()
    remove_month_year = ["month", "year"]
    print(time_posted)
    proposal = i.select(
        '.item__info⤍ListItem⤚1ci50 li:nth-child(3)')[0].text.replace('Proposals', '').strip()
    keywords = ['scrap', 'data mining']
    if(any(key_words in job_title for key_words in keywords)):
        if(not any(remove_m_y in time_posted for remove_m_y in remove_month_year)):
            my_list.append({
                "job_title": job_title,
                "time_posted": time_posted,
                "number_of_proposal": number_of_proposal,
                "page_link": page_link
            })

CodePudding user response：

I think you are defining your data structure wrong. From your expected result I understand that you want: {"job_title": "title 1", "time_posted":"6 days ago" ... }, {"job_title": "title2"...}

So, a list of dictionaries. Now you have a dictionary with list type values.

You have two options:

1.- Process your dictionary to adquire your desired structure

final_list = []

for _ in range(len(my_dict["job_title"])):
    item_dict = {}
    for key in my_dict:
        item_dict[key] = my_dict[key].pop(0)
    final_list.append(item_dict)

print(final_list) 
# [{'job_title': 'web scraping of product reviews', 'time_posted': '6 days ago', 'page_link': 'url1'}, {'job_title': 'yell web scraping in python', 'time_posted': '9 days ago', 'page_link': 'url2'}, {'job_title': 'google business scraping', 'time_posted': '3 days ago', 'page_link': 'url3'}]

2.- Same as user jugi mentioned, It's the best option. He has answered while I was writing this, so I'll post this anyways because my option 1 is slightly different.

CodePudding user response：

You can create a dictionary for each entry with a comprehension:

# Just using x because it's shorter. This does not create a copy
x = my_dict
x = [{'job_title': x['job_title'][i], 'time_posted': x['time_posted'][i],
      'page_link': x['page_link'][i]} for i in range(len(x['page_link']))]

>>> x
[{'job_title': 'web scraping of product reviews',
  'page_link': 'url1',
  'time_posted': '6 days ago'},
 {'job_title': 'yell web scraping in python',
  'page_link': 'url2',
  'time_posted': '9 days ago'},
 {'job_title': 'google business scraping',
  'page_link': 'url3',
  'time_posted': '3 days ago'}]