Home > Net >  Pandas Reading txt file as int instad of hex (str) (SOLVED)
Pandas Reading txt file as int instad of hex (str) (SOLVED)

Time:06-25

I am currently trying to use Pandas to sort through Can-Bus data however when I try to make a DataFrame with 2 similar text files, I get two completely different DataFrames.

When I make a data frame with the txt file "CANDUMP With Codes.txt" I get the following DataFrame:

 0   ID  B0  B1  B2  B3   B4   B5   B6   B7
0      1.656033e 09  0D0  4E  FD  00  00   00   00   FF   FF
1      1.656033e 09  0D1  00  00  00  00  NaN  NaN  NaN  NaN
2      1.656033e 09  0D2  00  00  FF  FF   00   00   00   00
3      1.656033e 09  0D3  00  06  C0  0F   48   00   72  NaN
4      1.656033e 09  0D4  00  00  00  00   00   00   00   00
5      1.656033e 09  018  02  91  70  00   00   00   00   23
6      1.656033e 09  140  00  0F  89  42   00   00   04   01
7      1.656033e 09  141  84  26  5C  27   87   82   A7   00
8      1.656033e 09  142  84  26  00  00   00   00   00   00
9      1.656033e 09  144  C0  00  51  0A   05   A4   A0   00
10     1.656033e 09  156  00  00  00  00   00   00   00   00
11     1.656033e 09  018  02  91  70  02   00   00   00   25
12     1.656033e 09  140  00  00  88  42   00   00   04   01
13     1.656033e 09  141  84  26  5C  27   87   82   A7   00
14     1.656033e 09  142  84  26  00  00   00   00   00   00
15     1.656033e 09  360  B9  00  7C  82   28   80   01   00
16     1.656033e 09  361  00  29  00  D9   04   70   74   75
17     1.656033e 09  152  F0  E4  00  00   00   02   08   00
18     1.656033e 09  0D0  4E  FD  00  00   00   00   00   FF
19     1.656033e 09  0D1  00  00  00  00  NaN  NaN  NaN  NaN
20     1.656033e 09  0D2  00  00  FF  FF   00   00   00   00
21     1.656033e 09  0D3  00  06  C0  0F   48   00   73  NaN
22     1.656033e 09  0D4  00  00  00  00   00   00   00   00
23     1.656033e 09  018  02  91  70  04   00   00   00   27
24     1.656033e 09  140  00  01  88  42   00   00   04   01
25     1.656033e 09  141  84  26  5C  27   87   82   A7   00
26     1.656033e 09  142  84  26  00  00   00   00   00   00
27     1.656033e 09  144  C0  00  51  0A   05   A4   A0   00
28     1.656033e 09  156  00  00  00  00   00   00   00   00
29     1.656033e 09  6E1  A1  80  00  00   00   60   00   00
30     1.656033e 09  018  02  91  70  06   00   00   00   29
31     1.656033e 09  140  00  02  88  42   00   00   04   01
32     1.656033e 09  141  84  26  5C  27   87   82   A7   00
33     1.656033e 09  142  84  26  00  00   00   00   00   00
34     1.656033e 09  152  F0  F4  00  00   00   02   08   00
35     1.656033e 09  0D0  4E  FD  00  00   00   00   FF   FF
36     1.656033e 09  0D1  00  00  00  00  NaN  NaN  NaN  NaN
37     1.656033e 09  0D2  00  00  FF  FF   00   00   00   00
38     1.656033e 09  0D3  00  06  C0  0F   48   00   74  NaN
39     1.656033e 09  0D4  00  00  00  00   00   00   00   00
40     1.656033e 09  4C3  03  00  00  00   00   00   00   00
41     1.656033e 09  018  02  91  70  08   00   00   00   2B
42     1.656033e 09  140  00  03  88  42   00   00   04   01
43     1.656033e 09  141  84  26  5C  27   87   82   A7   00
44     1.656033e 09  142  84  26  00  00   00   00   00   00
45     1.656033e 09  144  C0  00  51  0A   05   A4   A0   00
46     1.656033e 09  156  00  00  00  00   00   00   00   00
47     1.656033e 09  370  00  00  00  00   00   05   00   00
48     1.656033e 09  018  02  91  70  0A   00   00   00   2D
49     1.656033e 09  140  00  04  8B  42   00   00   04   01
50     1.656033e 09  282  88  94  FA  60   61   03   30   00
51     1.656033e 09  152  F0  04  00  00   00   02   08   00
52     1.656033e 09  0D0  4E  FD  00  00   00   00   FF   FF
53     1.656033e 09  0D1  00  00  00  00  NaN  NaN  NaN  NaN
54     1.656033e 09  0D2  00  00  FF  FF   00   00   00   00
55     1.656033e 09  0D3  00  06  C0  0F   48   00   75  NaN
56     1.656033e 09  0D4  00  00  00  00   00   00   00   00
57     1.656033e 09  018  02  91  70  0C   00   00   00   2F
58     1.656033e 09  140  00  05  8B  42   00   00   04   01
59     1.656033e 09  141  84  26  5C  27   88   82   A7   00
60     1.656033e 09  142  84  26  00  00   00   00   00   00
61     1.656033e 09  144  C0  00  51  0A   05   A4   A0   00
62     1.656033e 09  360  BA  00  7C  82   28   80   01   00
63     1.656033e 09  361  00  29  00  D9   05   E0   7C   75
64     1.656033e 09  156  00  00  00  00   00   00   00   00
65     1.656033e 09  440  42  02  00  00   00   00   00   00
66     1.656033e 09  372  00  70  00  00   00   00   00   00
67     1.656033e 09  018  02  91  70  0E   00   00   00   31
68     1.656033e 09  140  00  06  8B  42   00   00   04   01
69     1.656033e 09  141  84  26  5C  27   88   82   A7   00
70     1.656033e 09  142  84  26  00  00   00   00   00   00
71     1.656033e 09  4C1  01  00  00  00   00   00   00   00
72     1.656033e 09  152  F0  14  00  00   00   02   08   00
73     1.656033e 09  0D0  4E  FD  00  00   00   00   00   FF
74     1.656033e 09  0D1  00  00  00  00  NaN  NaN  NaN  NaN
75     1.656033e 09  0D2  00  00  FF  FF   00   00   00   00
76     1.656033e 09  0D3  00  06  C0  0F   48   00   76  NaN
77     1.656033e 09  0D4  00  00  00  00   00   00   00   00
78     1.656033e 09  018  02  91  70  00   00   00   00   23
79     1.656033e 09  140  00  07  8B  42   00   00   04   01
80     1.656033e 09  141  84  26  5C  27   88   82   A7   00
81     1.656033e 09  142  84  26  00  00   00   00   00   00
82     1.656033e 09  144  C0  00  51  0A   05   A4   A0   00
83     1.656033e 09  156  00  00  00  00   00   00   00   00
84     1.656033e 09  018  02  91  70  02   00   00   00   25
85     1.656033e 09  140  00  08  8B  42   00   00   04   01
86     1.656033e 09  141  84  26  5C  27   88   82   A7   00
87     1.656033e 09  142  84  26  00  00   00   00   00   00
88     1.656033e 09  370  00  00  00  00   00   06   00   00
89     1.656033e 09  152  F0  24  00  00   00   02   08   00
90     1.656033e 09  0D0  4E  FD  00  00   00   00   FF   FF
91     1.656033e 09  0D1  00  00  00  00  NaN  NaN  NaN  NaN
92     1.656033e 09  0D2  00  00  FF  FF   00   00   00   00
93     1.656033e 09  0D3  00  06  C0  0F   48   00   77  NaN
94     1.656033e 09  0D4  00  00  00  00   00   00   00   00
95     1.656033e 09  018  02  91  70  04   00   00   00   27
96     1.656033e 09  140  00  09  89  42   00   00   04   01
97     1.656033e 09  141  84  26  5C  27   88   82   A7   00
98     1.656033e 09  142  84  26  00  00   00   00   00   00
99     1.656033e 09  144  C0  00  51  0A   05   A4   A0   00
100    1.656033e 09  156  00  00  00  00   00   00   00   00

When I make a DataFrame with the txt file "ID 00d0.txt" I get the following DataFrame:

0   ID  B0  B1  B2  B3  B4  B5  B6  B7
0     1.656033e 09  0d0  A0   0   0   0   0   0  ff   1
1     1.656033e 09  0d0  30   0   0   0   0   0  ff   1
2     1.656033e 09  0d0  30   0   0   0   0   0  ff   1
3     1.656033e 09  0d0  30   0   0   0   0   0  ff   1
4     1.656033e 09  0d0  30   0   0   0   0   0  ff   1
5     1.656033e 09  0d0  30   0   0   0   0   0  ff   1
6     1.656033e 09  0d0  30   0   0   0   0   0  ff   1
7     1.656033e 09  0d0  30   0   0   0   0   0  ff   1
8     1.656033e 09  0d0  30   0   0   0   0   0  ff   1
9     1.656033e 09  0d0  30   0   0   0   0   0  ff   1
10    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
11    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
12    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
13    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
14    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
15    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
16    1.656033e 09  0d0  30   0   1   0   0   0  00   1
17    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
18    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
19    1.656033e 09  0d0  30   0   1   0   0   0  00   1
20    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
21    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
22    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
23    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
24    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
25    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
26    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
27    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
28    1.656033e 09  0d0  30   0   1   0   0   0  ff   2
29    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
30    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
31    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
32    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
33    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
34    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
35    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
36    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
37    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
38    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
39    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
40    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
41    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
42    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
43    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
44    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
45    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
46    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
47    1.656033e 09  0d0  30   0   0   0   0   0  00   2
48    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
49    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
50    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
51    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
52    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
53    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
54    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
55    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
56    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
57    1.656033e 09  0d0  30   0   0   0   0   0  00   1
58    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
59    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
60    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
61    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
62    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
63    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
64    1.656033e 09  0d0  30   0   1   0   0   0  ff   1
65    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
66    1.656033e 09  0d0  30   0   0   0   0   0  00   1
67    1.656033e 09  0d0  30   0   0   0   0   0  ff   1
68    1.656033e 09  0d0  2f   0   0   0   0   0  ff   1
69    1.656033e 09  0d0  2f   0   0   0   0   0  ff   1
70    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
71    1.656033e 09  0d0  2f   0   0   0   0   0  ff   1
72    1.656033e 09  0d0  2f   0   0   0   0   0  ff   2
73    1.656033e 09  0d0  2f   0   0   0   0   0  ff   1
74    1.656033e 09  0d0  2f   0   1   0   0   0  fe   1
75    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
76    1.656033e 09  0d0  2f   0   1   0   0   0  00   1
77    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
78    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
79    1.656033e 09  0d0  2f   0   0   0   0   0  ff   1
80    1.656033e 09  0d0  2f   0   0   0   0   0  ff   1
81    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
82    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
83    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
84    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
85    1.656033e 09  0d0  2f   0   1   0   0   0  00   2
86    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
87    1.656033e 09  0d0  2f   0   1   0   0   0  fe   1
88    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
89    1.656033e 09  0d0  2f   0   0   0   0   0  00   1
90    1.656033e 09  0d0  2f   0   0   0   0   0  ff   1
91    1.656033e 09  0d0  2f   0   0   0   0   0  ff   1
92    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
93    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
94    1.656033e 09  0d0  2f   0   0   0   0   0  ff   2
95    1.656033e 09  0d0  2f   0   0   0   0   0  ff   1
96    1.656033e 09  0d0  2f   0   0   0   0   0  ff   1
97    1.656033e 09  0d0  2f   0   0   0   0   0  fe   1
98    1.656033e 09  0d0  2f   0   0   0   0   0  ff   1
99    1.656033e 09  0d0  2f   0   1   0   0   0  ff   1
100   1.656033e 09  0d0  2f   0   1   0   0   0  ff   1

When I try to add some values together from ID 00d0 I get errors of mixing str with int values however with the "CANDUMP With Codes" file this does not happen. The problem is that I need ID 00d0 to be stored using hex (str) like the CANDUMP file.

Down below is the sample code I used to get the DataFrames as well as the content of the text files (I dont know if you can add txt files to a post so I put it in code below). If anyone has any idea why this is happening and has a potential solution, I am all for it.

Thank you all for your time.

Code I am using:

import pandas as pd
import time

startTime = time.time()

pd.set_option('display.max_rows', None)

df = pd.read_table("ID 00d0.txt", header=None, delim_whitespace=True)
df.columns = ['0','ID',"B0","B1","B2","B3","B4","B5","B6","B7"]

df2 = pd.read_table("CANDUMP With Codes.txt", header=None, delim_whitespace=True)
df2.columns = ['0','ID',"B0","B1","B2","B3","B4","B5","B6","B7"]

print(df2)


executionTime = round((time.time() - startTime),10)
print('Execution time in seconds: '   str(executionTime))

CANDUMP With Codes.txt:

 1656033437.571007 0D0   4E FD 00 00 00 00 FF FF
 1656033437.571007 0D1   00 00 00 00
 1656033437.571007 0D2   00 00 FF FF 00 00 00 00
 1656033437.571007 0D3   00 06 C0 0F 48 00 72
 1656033437.571007 0D4   00 00 00 00 00 00 00 00
 1656033437.571007 018   02 91 70 00 00 00 00 23
 1656033437.571007 140   00 0F 89 42 00 00 04 01
 1656033437.571007 141   84 26 5C 27 87 82 A7 00
 1656033437.571007 142   84 26 00 00 00 00 00 00
 1656033437.571007 144   C0 00 51 0A 05 A4 A0 00
 1656033437.571007 156   00 00 00 00 00 00 00 00
 1656033437.571007 018   02 91 70 02 00 00 00 25
 1656033437.571007 140   00 00 88 42 00 00 04 01
 1656033437.571007 141   84 26 5C 27 87 82 A7 00
 1656033437.571007 142   84 26 00 00 00 00 00 00
 1656033437.571007 360   B9 00 7C 82 28 80 01 00
 1656033437.571007 361   00 29 00 D9 04 70 74 75
 1656033437.571007 152   F0 E4 00 00 00 02 08 00
 1656033437.571007 0D0   4E FD 00 00 00 00 00 FF
 1656033437.571007 0D1   00 00 00 00
 1656033437.571007 0D2   00 00 FF FF 00 00 00 00
 1656033437.571007 0D3   00 06 C0 0F 48 00 73
 1656033437.571007 0D4   00 00 00 00 00 00 00 00
 1656033437.571007 018   02 91 70 04 00 00 00 27
 1656033437.571007 140   00 01 88 42 00 00 04 01
 1656033437.571007 141   84 26 5C 27 87 82 A7 00
 1656033437.571007 142   84 26 00 00 00 00 00 00
 1656033437.571007 144   C0 00 51 0A 05 A4 A0 00
 1656033437.571007 156   00 00 00 00 00 00 00 00
 1656033437.571007 6E1   A1 80 00 00 00 60 00 00
 1656033437.571007 018   02 91 70 06 00 00 00 29
 1656033437.571007 140   00 02 88 42 00 00 04 01
 1656033437.571007 141   84 26 5C 27 87 82 A7 00
 1656033437.571007 142   84 26 00 00 00 00 00 00
 1656033437.571007 152   F0 F4 00 00 00 02 08 00
 1656033437.571007 0D0   4E FD 00 00 00 00 FF FF
 1656033437.571007 0D1   00 00 00 00
 1656033437.571007 0D2   00 00 FF FF 00 00 00 00
 1656033437.571007 0D3   00 06 C0 0F 48 00 74
 1656033437.571007 0D4   00 00 00 00 00 00 00 00
 1656033437.571007 4C3   03 00 00 00 00 00 00 00
 1656033437.571007 018   02 91 70 08 00 00 00 2B
 1656033437.571007 140   00 03 88 42 00 00 04 01
 1656033437.571007 141   84 26 5C 27 87 82 A7 00
 1656033437.571007 142   84 26 00 00 00 00 00 00
 1656033437.571007 144   C0 00 51 0A 05 A4 A0 00
 1656033437.571007 156   00 00 00 00 00 00 00 00
 1656033437.571007 370   00 00 00 00 00 05 00 00
 1656033437.571007 018   02 91 70 0A 00 00 00 2D
 1656033437.571007 140   00 04 8B 42 00 00 04 01
 1656033437.571007 282   88 94 FA 60 61 03 30 00
 1656033437.571007 152   F0 04 00 00 00 02 08 00
 1656033437.571007 0D0   4E FD 00 00 00 00 FF FF
 1656033437.571007 0D1   00 00 00 00
 1656033437.571007 0D2   00 00 FF FF 00 00 00 00
 1656033437.571007 0D3   00 06 C0 0F 48 00 75
 1656033437.571007 0D4   00 00 00 00 00 00 00 00
 1656033437.571007 018   02 91 70 0C 00 00 00 2F
 1656033437.571007 140   00 05 8B 42 00 00 04 01
 1656033437.571007 141   84 26 5C 27 88 82 A7 00
 1656033437.571007 142   84 26 00 00 00 00 00 00
 1656033437.571007 144   C0 00 51 0A 05 A4 A0 00
 1656033437.571007 360   BA 00 7C 82 28 80 01 00
 1656033437.571007 361   00 29 00 D9 05 E0 7C 75
 1656033437.571007 156   00 00 00 00 00 00 00 00
 1656033437.571007 440   42 02 00 00 00 00 00 00
 1656033437.571007 372   00 70 00 00 00 00 00 00
 1656033437.571007 018   02 91 70 0E 00 00 00 31
 1656033437.571007 140   00 06 8B 42 00 00 04 01
 1656033437.571007 141   84 26 5C 27 88 82 A7 00
 1656033437.571007 142   84 26 00 00 00 00 00 00
 1656033437.571007 4C1   01 00 00 00 00 00 00 00
 1656033437.571007 152   F0 14 00 00 00 02 08 00
 1656033437.571007 0D0   4E FD 00 00 00 00 00 FF
 1656033437.571007 0D1   00 00 00 00
 1656033437.571007 0D2   00 00 FF FF 00 00 00 00
 1656033437.571007 0D3   00 06 C0 0F 48 00 76
 1656033437.571007 0D4   00 00 00 00 00 00 00 00
 1656033437.571007 018   02 91 70 00 00 00 00 23
 1656033437.571007 140   00 07 8B 42 00 00 04 01
 1656033437.571007 141   84 26 5C 27 88 82 A7 00
 1656033437.571007 142   84 26 00 00 00 00 00 00
 1656033437.571007 144   C0 00 51 0A 05 A4 A0 00
 1656033437.571007 156   00 00 00 00 00 00 00 00
 1656033437.571007 018   02 91 70 02 00 00 00 25
 1656033437.571007 140   00 08 8B 42 00 00 04 01
 1656033437.571007 141   84 26 5C 27 88 82 A7 00
 1656033437.571007 142   84 26 00 00 00 00 00 00
 1656033437.571007 370   00 00 00 00 00 06 00 00
 1656033437.571007 152   F0 24 00 00 00 02 08 00
 1656033437.571007 0D0   4E FD 00 00 00 00 FF FF
 1656033437.571007 0D1   00 00 00 00
 1656033437.571007 0D2   00 00 FF FF 00 00 00 00
 1656033437.571007 0D3   00 06 C0 0F 48 00 77
 1656033437.571007 0D4   00 00 00 00 00 00 00 00
 1656033437.571007 018   02 91 70 04 00 00 00 27
 1656033437.571007 140   00 09 89 42 00 00 04 01
 1656033437.571007 141   84 26 5C 27 88 82 A7 00
 1656033437.571007 142   84 26 00 00 00 00 00 00
 1656033437.571007 144   C0 00 51 0A 05 A4 A0 00
 1656033437.571007 156   00 00 00 00 00 00 00 00

ID 00d0.txt:

1656033437.571007 00d0 A0 00 00 00 00 00 ff 01
1656033437.590747 00d0 30 00 00 00 00 00 ff 01
1656033437.610978 00d0 30 00 00 00 00 00 ff 01
1656033437.630766 00d0 30 00 00 00 00 00 ff 01
1656033437.650933 00d0 30 00 00 00 00 00 ff 01
1656033437.670918 00d0 30 00 00 00 00 00 ff 01
1656033437.690742 00d0 30 00 00 00 00 00 ff 01
1656033437.710797 00d0 30 00 00 00 00 00 ff 01
1656033437.730938 00d0 30 00 00 00 00 00 ff 01
1656033437.750732 00d0 30 00 00 00 00 00 ff 01
1656033437.770774 00d0 30 00 00 00 00 00 ff 01
1656033437.790819 00d0 30 00 00 00 00 00 ff 01
1656033437.810714 00d0 30 00 00 00 00 00 ff 01
1656033437.830694 00d0 30 00 00 00 00 00 ff 01
1656033437.850853 00d0 30 00 00 00 00 00 ff 01
1656033437.870689 00d0 30 00 00 00 00 00 ff 01
1656033437.890834 00d0 30 00 01 00 00 00 00 01
1656033437.910732 00d0 30 00 01 00 00 00 ff 01
1656033437.930933 00d0 30 00 01 00 00 00 ff 01
1656033437.950721 00d0 30 00 01 00 00 00 00 01
1656033437.970946 00d0 30 00 01 00 00 00 ff 01
1656033437.990844 00d0 30 00 01 00 00 00 ff 01
1656033438.010802 00d0 30 00 01 00 00 00 ff 01
1656033438.030679 00d0 30 00 01 00 00 00 ff 01
1656033438.050847 00d0 30 00 01 00 00 00 ff 01
1656033438.070932 00d0 30 00 01 00 00 00 ff 01
1656033438.090849 00d0 30 00 01 00 00 00 ff 01
1656033438.110787 00d0 30 00 01 00 00 00 ff 01
1656033438.130893 00d0 30 00 01 00 00 00 ff 02
1656033438.151006 00d0 30 00 01 00 00 00 ff 01
1656033438.170744 00d0 30 00 01 00 00 00 ff 01
1656033438.190754 00d0 30 00 01 00 00 00 ff 01
1656033438.210979 00d0 30 00 01 00 00 00 ff 01
1656033438.230791 00d0 30 00 00 00 00 00 ff 01
1656033438.250794 00d0 30 00 00 00 00 00 ff 01
1656033438.270932 00d0 30 00 00 00 00 00 ff 01
1656033438.290712 00d0 30 00 00 00 00 00 ff 01
1656033438.310696 00d0 30 00 00 00 00 00 ff 01
1656033438.330892 00d0 30 00 00 00 00 00 ff 01
1656033438.35081 00d0 30 00 00 00 00 00 ff 01
1656033438.370926 00d0 30 00 00 00 00 00 ff 01
1656033438.390713 00d0 30 00 00 00 00 00 ff 01
1656033438.410782 00d0 30 00 00 00 00 00 ff 01
1656033438.430785 00d0 30 00 00 00 00 00 ff 01
1656033438.450719 00d0 30 00 01 00 00 00 ff 01
1656033438.470731 00d0 30 00 01 00 00 00 ff 01
1656033438.490977 00d0 30 00 00 00 00 00 ff 01
1656033438.510828 00d0 30 00 00 00 00 00 00 02
1656033438.530749 00d0 30 00 00 00 00 00 ff 01
1656033438.55073 00d0 30 00 00 00 00 00 ff 01
1656033438.570944 00d0 30 00 00 00 00 00 ff 01
1656033438.590847 00d0 30 00 00 00 00 00 ff 01
1656033438.610798 00d0 30 00 00 00 00 00 ff 01
1656033438.630972 00d0 30 00 00 00 00 00 ff 01
1656033438.650738 00d0 30 00 00 00 00 00 ff 01
1656033438.670883 00d0 30 00 01 00 00 00 ff 01
1656033438.690741 00d0 30 00 00 00 00 00 ff 01
1656033438.710854 00d0 30 00 00 00 00 00 00 01
1656033438.730805 00d0 30 00 00 00 00 00 ff 01
1656033438.750711 00d0 30 00 00 00 00 00 ff 01
1656033438.770848 00d0 30 00 00 00 00 00 ff 01
1656033438.790796 00d0 30 00 00 00 00 00 ff 01
1656033438.81073 00d0 30 00 00 00 00 00 ff 01
1656033438.830729 00d0 30 00 01 00 00 00 ff 01
1656033438.850798 00d0 30 00 01 00 00 00 ff 01
1656033438.870746 00d0 30 00 00 00 00 00 ff 01
1656033438.890738 00d0 30 00 00 00 00 00 00 01
1656033438.910826 00d0 30 00 00 00 00 00 ff 01
1656033438.930735 00d0 2f 00 00 00 00 00 ff 01
1656033438.950811 00d0 2f 00 00 00 00 00 ff 01
1656033438.970722 00d0 2f 00 01 00 00 00 ff 01
1656033438.990992 00d0 2f 00 00 00 00 00 ff 01
1656033439.010762 00d0 2f 00 00 00 00 00 ff 02
1656033439.03074 00d0 2f 00 00 00 00 00 ff 01
1656033439.050974 00d0 2f 00 01 00 00 00 fe 01
1656033439.070785 00d0 2f 00 01 00 00 00 ff 01
1656033439.090808 00d0 2f 00 01 00 00 00 00 01
1656033439.110784 00d0 2f 00 01 00 00 00 ff 01
1656033439.130923 00d0 2f 00 01 00 00 00 ff 01
1656033439.150792 00d0 2f 00 00 00 00 00 ff 01
1656033439.170964 00d0 2f 00 00 00 00 00 ff 01
1656033439.190788 00d0 2f 00 01 00 00 00 ff 01
1656033439.210718 00d0 2f 00 01 00 00 00 ff 01
1656033439.230836 00d0 2f 00 01 00 00 00 ff 01
1656033439.250775 00d0 2f 00 01 00 00 00 ff 01
1656033439.270921 00d0 2f 00 01 00 00 00 00 02
1656033439.290832 00d0 2f 00 01 00 00 00 ff 01
1656033439.310932 00d0 2f 00 01 00 00 00 fe 01
1656033439.330775 00d0 2f 00 01 00 00 00 ff 01
1656033439.350807 00d0 2f 00 00 00 00 00 00 01
1656033439.37097 00d0 2f 00 00 00 00 00 ff 01
1656033439.390729 00d0 2f 00 00 00 00 00 ff 01
1656033439.411024 00d0 2f 00 01 00 00 00 ff 01
1656033439.430715 00d0 2f 00 01 00 00 00 ff 01
1656033439.450694 00d0 2f 00 00 00 00 00 ff 02
1656033439.47096 00d0 2f 00 00 00 00 00 ff 01
1656033439.490835 00d0 2f 00 00 00 00 00 ff 01
1656033439.51084 00d0 2f 00 00 00 00 00 fe 01
1656033439.530714 00d0 2f 00 00 00 00 00 ff 01
1656033439.550823 00d0 2f 00 01 00 00 00 ff 01
1656033439.570697 00d0 2f 00 01 00 00 00 ff 01

CodePudding user response:

IIUC, use dtype=str as parameter of read_table to prevent Pandas infer the datatype:

df = pd.read_table('ID 00d0.txt', header=None, delim_whitespace=True, dtype=str, 
                   names=['0','ID',"B0","B1","B2","B3","B4","B5","B6","B7"])

Output:

>>> df
                     0    ID  B0  B1  B2  B3  B4  B5  B6  B7
0    1656033437.571007  00d0  A0  00  00  00  00  00  ff  01
1    1656033437.590747  00d0  30  00  00  00  00  00  ff  01
2    1656033437.610978  00d0  30  00  00  00  00  00  ff  01
3    1656033437.630766  00d0  30  00  00  00  00  00  ff  01
4    1656033437.650933  00d0  30  00  00  00  00  00  ff  01
..                 ...   ...  ..  ..  ..  ..  ..  ..  ..  ..
96   1656033439.490835  00d0  2f  00  00  00  00  00  ff  01
97    1656033439.51084  00d0  2f  00  00  00  00  00  fe  01
98   1656033439.530714  00d0  2f  00  00  00  00  00  ff  01
99   1656033439.550823  00d0  2f  00  01  00  00  00  ff  01
100  1656033439.570697  00d0  2f  00  01  00  00  00  ff  01

[101 rows x 10 columns]

You can convert column '0' to float by chaining .astype({'0': float}).

CodePudding user response:

Why does this happen?

If you don't tell pandas in which format the data is, it will guess this for you. Since ID 00d0.txt contains only 00 in some columns, the type is ambigous and pandas chose to parse it as an int. You can see all datatypes for each dataframe by calling

df.info()

and looking at the Dtypes column.

What is the fix?

An easy fix would be to read in everything as a string. This is done by adding the keyword argument dtype=str to your read_table call like so:

df = pd.read_table("file.txt", header=None, delim_whitespace=True, dtype=str)

This way you can merge and combine dataframes as you please. Later, however, you may want to do some calculations on the columns, you could then convert them using an arbitrary function. For example to covert the hex strings into integers:

df["B0"] = df["B0"].apply(lambda x: int(x, 16))

Hope that helps :)

  • Related