Es according to the total amount is about 1 billion, to the last month of data (about 50 million), using sparksql to load, and then deal with related business, unusually slow to load, thank the friends have made similar optimization sharing, attach loading code:
Val vehpassDataFrame=sparkSession. SqlContext. Read. The format (" org. Elasticsearch. Spark. SQL "). The options (options). The load (" alias_veh_pass/doc ")
VehpassDataFrame. Select (" HPHM HPZL ", ""," JGSJ ", "gctp1", "GCBH", "lhy_syxz"). The createTempView (" alias_veh_pass ")