How to extract hashtags from Tweets in pandas data frame?-CodePudding

I have a dataset of tweets with several variable (columns) and I want to extract all the hashtags from a tweet (text) and place the result in a new column (hashtags). Below is what I am trying:

import pandas as pd
data = pd.read_csv("Sample.csv", lineterminator='\n')

def hashtags(string):
    Hash = data.text.str.findall(r'#.*?(?=\s|$)')
    return Hash
data['hashtags'] = data['text'].apply(lambda x: hashtags(x))

However, when I run the function hashtags, my notebook is just stuck (not finishing execution or giving any error). My file only have around 10k rows.

Also, if this code run successfully, I am expecting to get something like this:

[#asd, #fer, #gtr]

But I want the resultant column should have only name of the hashtags like [asd, fer, gtr]. Please suggest what change I should do in the code.

I tried to look for solution in previous asked questions but most of them used regular expression and I am looking for a solution using pandas.

Thanks in Advance.

CodePudding user response：

I downloaded some sample twitter data in a .csv from here, https://twitter-sentiment-csv.herokuapp.com/. I've used a slice of the first 10 rows for this example.

def find_tags(row_string):
    # use a list comprehension to find list items that start with #
    tags = [x for x in row_string if x.startswith('#')]
    
    return tags

df = pd.DataFrame({'sentiment': {0: 'neutral',
  1: 'neutral',
  2: 'neutral',
  3: 'neutral',
  4: 'neutral',
  5: 'neutral',
  6: 'neutral',
  7: 'positive',
  8: 'neutral',
  9: 'neutral'},
 'text': {0: 'RT @fakeTakeDump: TRAMS STELARA BICYCLE PINOCHLE JUMBO INDEX SEPTAVALENT TYPEWRITER HOMEBREWING AND ANTI-LOCK HULLO KITTY IN FORTUNE COOKIE…',
  1: 'RT @fauzanzain: Hi warga twitter, sekarang aku lagi cari career coach nih yang punya latar belakang tech recruiter / mid to senior digital…',
  2: 'RT @fakeTakeDump: WOODWORKING THE FORUM SHOPS LIKENESS SPECTROHELIOSCOPE CHEEMS FLAVONOIDS ROCKET IS NEITHER SUGAR DADDY CANNED TUNA HANDMA…',
  3: 'WOODWORKING THE FORUM SHOPS LIKENESS SPECTROHELIOSCOPE CHEEMS FLAVONOIDS ROCKET IS NEITHER SUGAR DADDY CANNED TUNA…',
  4: 'RT @KirkDBorne: Recap of 60 days of #DataScience and #MachineLearning — days 1 through 60: by @NainaChaturved8 \\n———…',
  5: 'Recap of 60 days of #DataScience and #MachineLearning — days 1 through 60:  by… ',
  6: 'RT @IBAConservative: @dax_christensen The truth is out! They can’t hold it back. \\n#CrimesAgainstHumanity \\n#TrudeauTyranny \\n#TrudeauMustResi…',
  7: "RT @drmwarner: As per these children's health organizations, keeping masks on in schools 2wks post March break would have made much more se…",
  8: 'RT @cryptotommy88: TL;DR\\n✅ Collective analytics business \\n✅ Draw power from data science &amp; crowd-sourced knowledge\\n✅ 1st product PFPscore:…',
  9: 'RT @cryptotommy88: TL;DR\\n✅ Collective analytics business \\n✅ Draw power from data science &amp; crowd-sourced knowledge\\n✅ 1st product PFPscore:…'},
 'user': {0: 'BotDuran',
  1: 'ezash',
  2: 'BlkHwk0ps',
  3: 'fakeTakeDump',
  4: 'RobotProud',
  5: 'KirkDBorne',
  6: 'cloudcnworld',
  7: 'NeuroTeck',
  8: 'BIGwinCutiejoy8',
  9: 'luckbigw1n'}})

df['split'] = df['text'].str.split(' ')

df['tags'] = df['split'].apply(lambda row : find_tags(row))
# replace # as requested in OP, replace for new lines and \ as needed.
df['tags'] = df['tags'].apply(lambda x : str(x).replace('#', '').replace('\\n', ',').replace('\\', '').replace("'", ""))

Output df['tags']:

0                                []
1                                []
2                                []
3                                []
4    [DataScience, MachineLearning]
5    [DataScience, MachineLearning]
6                                []
7                                []
8                                []
9                                []
Name: tags, dtype: object