Spark groupByKey算子学习使用详解

groupByKey():

    分组统计

示例代码：

def my_groupByKey():
    data = ['hello spark', 'hello world', 'hello world']
    rdd = sc.parallelize(data)
    map_rdd = rdd.flatMap(lambda line:line.split(' ')).map(lambda x:(x,1))
    group_by_rdd = map_rdd.groupByKey() 

 print(group_by_rdd.collect())
    result_rdd = group_by_rdd.map(lambda x:{x[0]:list(x[1])}) 

 print(result_rdd.collect())

输出结果：

('hello', <pyspark.resultiterable.ResultIterable object at 0x1061b45f8>), ('spark', <pyspark.resultiterable.ResultIterable object at 0x1061b4630>), ('world', <pyspark.resultiterable.ResultIterable object at 0x1061b4898>)]

[{'hello': [1, 1, 1]}, {'spark': [1]}, {'world': [1, 1]}]