how to grep unique ip addresses from file-CodePudding

I have a text file that contain some details. I want to grep the unique ip addresses and print all the tuples of that matching ip address.

Input text file contains details as follows:

Jan 11 11:11:11 2020-12-01 11: 11:11: {ABC-ABCDE-AB-123}ABCDEFGHI_ABCDEFG_ABCD: application:none, ab0.567 123.45.67.890:56789 [111.111.111.111:12345] ->  117.140.11.11:123 (TCP)
Jan 11 11:11:11 2020-12-01 11: 11:11: {ABC-ABCDE-AB-123}ABCDEFGHI_ABCDEFG_ABCD: application:none, ab0.567 123.45.67.890:12345 [111.111.111.111:12345] ->  110.140.11.11:123 (TCP)
Jan 11 11:11:11 2020-12-01 11: 11:11: {ABC-ABCDE-AB-123}ABCDEFGHI_ABCDEFG_ABCD: application:none, ab0.567 123.45.67.890:56789 [111.111.111.111:12345] ->  117.140.11.11:123 (TCP)
Jan 11 11:11:11 2020-12-01 11: 11:11: {ABC-ABCDE-AB-123}ABCDEFGHI_ABCDEFG_ABCD: application:none, ab0.567 123.75.88.860:98765 [111.111.111.111:12345] ->  117.140.79.118:123(UDP)
Jan 11 11:11:11 2020-12-01 11: 11:11: {ABC-ABCDE-AB-123}ABCDEFGHI_ABCDEFG_ABCD: application:none, ab0.567 123.75.88.860:98765 [111.111.111.111:12345] ->  117.140.79.118:123(UDP)

the output i need:

Jan 11 11:11:11 2020-12-01 11: 11:11: {ABC-ABCDE-AB-123}ABCDEFGHI_ABCDEFG_ABCD: application:none, ab0.567 123.45.67.890:12345 [111.111.111.111:12345] ->  110.140.11.11:123 (TCP)
Jan 11 11:11:11 2020-12-01 11: 11:11: {ABC-ABCDE-AB-123}ABCDEFGHI_ABCDEFG_ABCD: application:none, ab0.567 123.75.88.860:98765 [111.111.111.111:12345] ->  117.140.79.118:123(UDP)

I need the whole row and the ip must be unique

i don't care about other columns i just want all the row where ip adddress must be unique. this position ip most be unique in all output rows 123.75.88.860:98765 [111.111.111.111:12345] -> 117.140.79.118:123(UDP)

CodePudding user response：

a quick one, just check on the col#17:

$ awk -F' |:' '!a[$17]  ' yourFile                      
Jan 11 11:11:11 2020-12-01 11: 11:11: {ABC-ABCDE-AB-123}ABCDEFGHI_ABCDEFG_ABCD: application:none, ab0.567 123.45.67.890:56789 [111.111.111.111:12345] ->  117.140.11.11:123 (TCP)
Jan 11 11:11:11 2020-12-01 11: 11:11: {ABC-ABCDE-AB-123}ABCDEFGHI_ABCDEFG_ABCD: application:none, ab0.567 123.75.88.860:98765 [111.111.111.111:12345] ->  117.140.79.118:123(UDP)

CodePudding user response：

I'm still not sure I fully understand your problem statement, but I don't think what you are asking can be done with the grep tool alone.

Perhaps this perl command will do what you need? (assuming input file is named input)

perl -ne 'BEGIN { my %seen } if (/(\d \.\d \.\d \.\d )/ && ! $seen{$1}) { $seen{$1} = 1 ; print "$_" }' input

This should output the first line containing each unique IP address (ignoring the port number) in that first IP field. It works by pattern-matching the first substring in the line that looks like an IP address, and consulting a hash called seen to determine if this is the first line providing that IP address. If so, the line is output.

If this is still not what you need, please try to further clarify the question.