当前位置：首页 - Spark

python 从mysql读取数据后，如何转为spark rdd?

作者:高景洋　日期:2021-01-29 18:27:52　浏览次数:1242

直接上代码：

		from pyspark import SparkContext,SparkConf
	

			conf = SparkConf()
		
			sc = SparkContext(conf=conf)

			list_url_group_data = ListUrlDA().select_list_url_count_group_by_websiteid(list_schedule_website_id) #从mysql读出来的数据 类型 List
		
			list_url_rdd = sc.parallelize(list_url_group_data) # 将List转换为rdd
		
			spark = SparkSession.builder.master("local").appName("SparkMysql").getOrCreate()
		
			schema = StructType([
		
			# true代表不为空
		
			StructField("WebsiteID", StringType(), True),
		
			StructField("count", StringType(), True)
		
			])
		
			df_list_url = spark.createDataFrame(list_url_rdd, schema=schema) # 将rdd转换为dataFrame
		
			df_list_url.show()
		
			calculate_product_count(list_filter_websiteids,df_list_url) # 业务逻辑处理方法
		
			spark.stop()

执行结果:

29-01-2021 17:47:34 CST hbase_scan INFO - +---------+------+
29-01-2021 17:47:34 CST hbase_scan INFO - |WebsiteID| count|
29-01-2021 17:47:34 CST hbase_scan INFO - +---------+------+
29-01-2021 17:47:34 CST hbase_scan INFO - |        1|215923|
29-01-2021 17:47:34 CST hbase_scan INFO - |       17|  7843|
29-01-2021 17:47:34 CST hbase_scan INFO - |       23|  1563|
29-01-2021 17:47:34 CST hbase_scan INFO - |       24|  1720|
29-01-2021 17:47:34 CST hbase_scan INFO - |       71|334890|
29-01-2021 17:47:34 CST hbase_scan INFO - |       94|  6782|
29-01-2021 17:47:34 CST hbase_scan INFO - |      103|     9|
29-01-2021 17:47:34 CST hbase_scan INFO - |      108|   319|
29-01-2021 17:47:34 CST hbase_scan INFO - |      167|  8352|
29-01-2021 17:47:34 CST hbase_scan INFO - |      168|  5417|
29-01-2021 17:47:34 CST hbase_scan INFO - |      171|  8598|
29-01-2021 17:47:34 CST hbase_scan INFO - |      221|    43|
29-01-2021 17:47:34 CST hbase_scan INFO - |      237|  1128|
29-01-2021 17:47:34 CST hbase_scan INFO - |      238|    11|
29-01-2021 17:47:34 CST hbase_scan INFO - |      242|   111|
29-01-2021 17:47:34 CST hbase_scan INFO - |      243|   922|
29-01-2021 17:47:34 CST hbase_scan INFO - |      251|   445|
29-01-2021 17:47:34 CST hbase_scan INFO - |      253|   372|
29-01-2021 17:47:34 CST hbase_scan INFO - |      279|  1739|
29-01-2021 17:47:34 CST hbase_scan INFO - |      282|    59|
29-01-2021 17:47:34 CST hbase_scan INFO - +---------+------+
29-01-2021 17:47:34 CST hbase_scan INFO - only showing top 20 rows

本文永久性链接：
<a href="http://r4.com.cn/art173.aspx">python 从mysql读取数据后，如何转为spark rdd?</a>

当前header:Host: r4.com.cn X-Host1: r4.com.cn X-Host2: r4.com.cn X-Host3: 127.0.0.1:8080 X-Forwarded-For: 18.119.104.238 X-Real-Ip: 18.119.104.238 X-Domain: r4.com.cn X-Request: GET /art173.aspx HTTP/1.1 X-Request-Uri: /art173.aspx Connection: close Accept: */* User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)