pd.to_datetime errors = 'ignore' strange behavior-CodePudding

I have a dataframe called 'result' that contains a column called "Action". This column contains lots of strings, some of which are dates. I'm trying to convert the strings that contain dates into datetimes, ignoring the rows that don't contain dates because they will error out.

Normally I would use pd.to_datetime(column =, format =, errors = 'ignore') but in this case, that doesn't convert anything.

HOWEVER, when I change errors = 'coerce', it DOES convert the dates, but of course converts everything else to NaN. I want to use ignore because there is still valuable data in the other rows.

result["Action"] = pd.to_datetime(result["Action"], format = '%A %B %d %Y', errors = 'ignore')

result["Action"] = pd.to_datetime(result["Action"], format = '%A %B %d %Y', errors = 'coerce')

CodePudding user response：

If errors is set to ignore, then invalid parsing will return the input. So in your case the input is result["Action"](The entire column).

The solution to this problem is to apply pd.to_datetime rowwise with errors='ignore'. By doing so you will get the same row back if the row does not follow the format.

>>> import pandas as pd
>>>
>>> df = pd.DataFrame({'Action': ['Tuesday November 30 2021', 'Appointment time clicked']})
>>> df
                     Action
0  Tuesday November 30 2021
1  Appointment time clicked
>>>
>>> def custom(action):
...     date_time = pd.to_datetime(action, format='%A %B %d %Y', errors='ignore')
...     return date_time
...
>>> df.Action = df.Action.apply(custom)
>>> df
                     Action
0       2021-11-30 00:00:00
1  Appointment time clicked

CodePudding user response：

One work around I can think of is to re-parse the DF if needed

for d in result['Action']:
  try:
    res = pd.to_datetime(d, format = '%A %B %d %Y')
    print(res)
  except:
    res = pd.to_datetime(d, format = '%A %B %d %Y', errors='ignore')
    print(res)