Pandas: find complex numbers in a dataframe-CodePudding

I have a large dataframe test. Calling a custom function, it raises this error:

TypeError: ("'<' not supported between instances of 'complex' and 'float'", 'occurred at index ir_mt_1y')

How can I find the complex numbers in test?

My attempt is something like

pd.api.types.is_complex_dtype(test)

which simply gives

False

Edit for additional info:

test['ir_mt_1y'].head().to_dict()

{0: 0.11141100978777804,
 1: 2.775837591360907,
 2: 3.7201836925385474,
 3: -1.8665959833285382,
 4: 3.8716614787099735}

and

test['ir_mt_1y'].dtypes

dtype('O')

and

test['ir_mt_1y'].astype(float)

TypeError: can't convert complex to float

CodePudding user response：

x1 = [3,5,67,1,5,5]
x2 = ['f', 'f', 'g','f','j','k']
x3 = [(92.289 151.96j),(-118.52-342.43j),(-7.1735-76.487j),(-458.09-1565.6j),(-118.41-340.58j),(-118.52-342.43j)]
x4 = [2,2,2,1,1,1]
df = pd.DataFrame(data = [x1, x2, x3, x4]).T
for value in df.columns.values:
    df[value].apply(type)

Output:

0    <class 'int'>
1    <class 'int'>
2    <class 'int'>
3    <class 'int'>
4    <class 'int'>
5    <class 'int'>
Name: 0, dtype: object
0    <class 'complex'>
1    <class 'complex'>
2    <class 'complex'>
3    <class 'complex'>
4    <class 'complex'>
5    <class 'complex'>
Name: 1, dtype: object
0    <class 'str'>
1    <class 'str'>
2    <class 'str'>
3    <class 'str'>
4    <class 'str'>
5    <class 'str'>
Name: 2, dtype: object
0    <class 'int'>
1    <class 'int'>
2    <class 'int'>
3    <class 'int'>
4    <class 'int'>
5    <class 'int'>
Name: 3, dtype: object

Maybe you can use this and the following comparison:

>>> type((92.289 151.96j)) == complex
True

CodePudding user response：

df_find_complex = pd.DataFrame(index=test.index)
for col in test.columns:
    df_find_complex[col] = [type(x) == complex for x in test[col]]

Now I can indirectly find the complex numbers in test from df_find_complex.

Since my final objective (out of the question) is to replace complex numbers with NaN, I can do kind of:

test1 = pd.DataFrame(np.where(df_find_complex, np.nan, test), columns=test.columns)

and call my function on test1.