With this sample dataframe:
>>> d = pd.DataFrame({'si': ['1', '2', 'NA'], 's': ['a', 'b', 'c']})
>>> d.dtypes
#
si object
s object
dtype: object
My first attempt was to use astype and the 'Int64' NA aware int type, but I got a
traceback
>>> d.si.astype('Int64')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-144-ed289e0c95aa> in <module>
----> 1 d.si.astype('Int64')
...
then I try the to_numeric
method:
pandas to_numeric
integer downcast cast floats
In [112]: d.loc[:, 'ii'] = pd.to_numeric(d.si, errors='coerce', downcast='integer')
In [113]: d.dtypes
Out[113]:
si object
s object
ii float64
dtype: object
In [114]: d
Out[114]:
si s ii
0 1 a 1.0
1 2 b 2.0
2 NA c NA
In the above I expect to have ii
column with integers and integer nan
Documentation say:
downcast : {'integer', 'signed', 'unsigned', 'float'}, default None
If not None, and if the data has been successfully cast to a
numerical dtype (or if the data was numeric to begin with),
downcast that resulting data to the smallest numerical dtype
possible according to the following rules:
- 'integer' or 'signed': smallest signed int dtype (min.: np.int8)
- 'unsigned': smallest unsigned int dtype (min.: np.uint8)
- 'float': smallest float dtype (min.: np.float32)
CodePudding user response:
Unfortunately, pandas
is still adapting/transitioning to fully supporting integer NaN
. For that, you have to explicitly convert it to Int64
after your pd.to_numeric
operation.
No need to downcast.
# Can also use `'Int64' as dtype below.
>>> pd.to_numeric(df['col'], errors='coerce').astype(pd.Int64Dtype())
# or
>>> pd.to_numeric(df['col'], errors='coerce').astype('Int64')
0 1
1 2
2 3
3 <NA>
Name: col, dtype: Int64
CodePudding user response:
You have errors='coerce'
set, and the documentation for that option says (emphasis mine):
errors
: {'ignore'
,'raise'
,'coerce'
}, default'raise'
- If
'raise'
, then invalid parsing will raise an exception.- If
'coerce'
, then invalid parsing will be set as NaN.- If
'ignore'
, then invalid parsing will return the input.
Since your si
column contains NaN
s, you can't convert it to an integer column because NaN
is a float, and therefore all other values in the column are upcasted to the float64
dtype.