Spark more than how to deal with data sets of similar leftjoin operation-CodePudding

I have two large data sets, one is the IP address list, the other is network prefix list,
For example: ip_list (192.168.0.1, 192.168.0.2 192.168.1.9, 172.16.1.100... ), prefix_list (192.168.0.0/16, 172.16.1.0/24... )
Need to list all the right to the left of the IP and the prefix matching, it is concluded that all the prefix contains IP number,
For example, for the final result should be above examples (192.168.0.0/16, 3), (172.16.1.0/24, 1), namely for IP list (192.168.0.1 192.168.0.2, 192.168.1.9, 172.16.1.100) has three IP belongs to the 192.168.0.0/16 this segment, one IP belongs to 172.16.1.0/24 this segment,
Excuse me, ladies and gentlemen, this in the spark how to implement?

CodePudding user response:

Using the spark SQL