A total BeautifulSoup newbie here. I need to scrape the publication tab's content from a certain URL (listed in the code sample below). I need to scrape the publications and split them into 'authors', 'title', and 'journal', which I can then convert to pandas DataFrame. I tried scraping the content using the code below:

page_url = 'https://www.vet.upenn.edu/research/centers-laboratories/research-laboratory/research-laboratory/anguera-laboratory'

def get_soup(url):
    r = requests.get('http://localhost:8050/render.html', params={'url': url, 'wait': 3})
    soup = BeautifulSoup(r.text, 'html.parser')

url_soup = get_soup(page_url)

publis = url_soup.find_all('div',{'class':'tab-content pad'})

for item in publis:
    publi = {
        'items': item.find('div',{'id':'tabs-publications'}).text.split('\n\n')
    }

publication_df = pd.DataFrame(publi)
new_publication_df = publication_df['items'].str.split(':', 1, expand=True)
new_publication_df[[1,2]] = new_publication_df[1].str.split(']',expand=True)

This returns what I need, but it is really prone to typos (e.g. there is one publication that uses 'J' rather than ']'). Is there any way that BeautifulSoup can automatically separate the text into three columns?

CodePudding user response：

Think you should use available structure instead of splitting the strings.

Select all <em> that holds the title, its previous_sibling, that holds the authors and also its next_sibling, that holds the journal.

Example

import requests
from bs4 import BeautifulSoup


r = requests.get('https://www.vet.upenn.edu/research/centers-laboratories/research-laboratory/research-laboratory/anguera-laboratory')
soup = BeautifulSoup(r.text)
data = []
for e in soup.select('#tabs-publications em'):
    data.append({
        'author':e.previous.get_text(strip=True)[:-1],
        'title':e.get_text(strip=True),
        'journal':e.next_sibling.get_text(strip=True)
    })

data

Output data

[{'author': 'Hantsoo Liisa, Kornfield Sara, Anguera Montserrat C, Epperson C Neill',
  'title': 'Inflammation: A Proposed Intermediary Between\xa0Maternal Stress and Offspring Neuropsychiatric Risk. [PMID30314641]',
  'journal': 'Biological psychiatry 85(2): 97-106, Jan 2019.'},
 {'author': 'Sierra Isabel, Anguera Montserrat C',
  'title': 'Enjoy the silence: X-chromosome inactivation diversity in somatic cells.[PMID31108425]',
  'journal': 'Current opinion in genetics & development 55: 26-31, May 2019.'},
 {'author': 'Syrett Camille M, Anguera Montserrat C',
  'title': 'When the balance is broken: X-linked gene dosage from two X chromosomes and female-biased autoimmunity. [PMID31125996]',
  'journal': 'Journal of leukocyte biology May 2019.'},
 {'author': 'Kotzin Jonathan J, Iseka Fany, Wright Jasmine, Basavappa Megha G, Clark Megan L, Ali Mohammed-Alkhatim, Abdel-Hakeem Mohamed S, Robertson Tanner F, Mowel Walter K, Joannas Leonel, Neal Vanessa D, Spencer Sean P, Syrett Camille M, Anguera Montserrat C, Williams Adam, Wherry E John, Henao-Mejia Jorge',
  'title': 'The long noncoding RNA regulates CD8 T cells in response to viral infection.[PMID31138702]',
  'journal': 'Proceedings of the National Academy of Sciences of the United States of America 116(24): 11916-11925, Jun 2019.'},
 {'author': 'Syrett Camille M, Paneru Bam, Sandoval-Heglund Donavon, Wang Jianle, Banerjee Sarmistha, Sindhava Vishal, Behrens Edward M, Atchison Michael, Anguera Montserrat C',
  'title': 'Altered X-chromosome inactivation in T cells may promote sex-biased autoimmune diseases. [PMID30944248',
  'journal': 'JCI insight 4(7), Apr 2019.'},...]

Output DataFrame

pd.DataFrame(data)

...