当前位置:首页 - Spark

pyspark日期处理及比较

作者:高景洋 日期:2021-01-28 11:38:26 浏览次数:930

业务需求

1、通过pyspark将hbase中的数据拉出

2、通过pyspark按UpdatedDate 、EnteredDate、DeletedDate ,根据WebsiteID字段汇总数量


小白方法:

1、spark拉出hbase数据

2、rdd1 = hbase_result_rdd.map('对日期字段进行处理')

     df1 = rdd_to_df(rdd1) # 将rdd转换为dataframe

     df2 = df1.filter(df1['UpdatedDate']>datetime.datetime.today().date) # 理想中的样子

     现实中的样子:各种日期类型转换问题报错,如:数据中的字段值为None \ 2021-01-22 18:47:48 \ 2021-01-22T18:47:48

大白方法:

1、因此,我们转换思路,查一下,spark有没有内置的日期处理函数。因为之前已经用过内置的随机函数rand(),内置的md5函数md5(),内置的添加列函数lit()

2、果然,pyspark存在内置的日期处理函数:

     to_timestamp :将列转换为日期

     current_date : 获取当前日期

3、最后,come on ~ ,来看下最终代码~

     


from pyspark.sql.functions import rand,md5,lit,to_timestamp,current_date


print('更新时间')
df_update_date = list_filter_websiteids.filter(list_filter_websiteids['UpdatedDate']!='--').select('WebsiteID',to_timestamp(list_filter_websiteids['UpdatedDate']).alias('UpdatedDate'))
df_update_date.show()
df_update_date_1 = df_update_date.filter(df_update_date['UpdatedDate']>current_date()).groupby('WebsiteID').count()
df_update_date_1.show()


执行结果:

28-01-2021 11:05:54 CST hbase_scan INFO - 更新时间
28-01-2021 11:05:54 CST hbase_scan INFO - +---------+-------------------+
28-01-2021 11:05:54 CST hbase_scan INFO - |WebsiteID|        UpdatedDate|
28-01-2021 11:05:54 CST hbase_scan INFO - +---------+-------------------+
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-28 08:26:46|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-28 08:59:03|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-28 08:45:54|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - |        1|2021-01-22 18:47:48|
28-01-2021 11:05:54 CST hbase_scan INFO - +---------+-------------------+
28-01-2021 11:05:54 CST hbase_scan INFO - only showing top 20 rows
28-01-2021 11:05:54 CST hbase_scan INFO - 
28-01-2021 11:05:55 CST hbase_scan INFO - 
28-01-2021 11:05:58 CST hbase_scan INFO - [Stage 24:=============================>                            (1 + 1) / 2]
28-01-2021 11:05:58 CST hbase_scan INFO -                                                                                 
28-01-2021 11:05:59 CST hbase_scan INFO - +---------+-----+
28-01-2021 11:05:59 CST hbase_scan INFO - |WebsiteID|count|
28-01-2021 11:05:59 CST hbase_scan INFO - +---------+-----+
28-01-2021 11:05:59 CST hbase_scan INFO - |      108|    1|
28-01-2021 11:05:59 CST hbase_scan INFO - |      521|   55|
28-01-2021 11:05:59 CST hbase_scan INFO - |       71|  762|
28-01-2021 11:05:59 CST hbase_scan INFO - |       23|   29|
28-01-2021 11:05:59 CST hbase_scan INFO - |       70|   16|
28-01-2021 11:05:59 CST hbase_scan INFO - |      167|   17|
28-01-2021 11:05:59 CST hbase_scan INFO - |        1| 1220|
28-01-2021 11:05:59 CST hbase_scan INFO - |      235|  138|
28-01-2021 11:05:59 CST hbase_scan INFO - |      292|   28|
28-01-2021 11:05:59 CST hbase_scan INFO - |      171|    3|
28-01-2021 11:05:59 CST hbase_scan INFO - |       58|  263|
28-01-2021 11:05:59 CST hbase_scan INFO - |      243|    1|
28-01-2021 11:05:59 CST hbase_scan INFO - |      168|   16|
28-01-2021 11:05:59 CST hbase_scan INFO - |       24|  549|
28-01-2021 11:05:59 CST hbase_scan INFO - |       83|  540|
28-01-2021 11:05:59 CST hbase_scan INFO - |      492|    5|
28-01-2021 11:05:59 CST hbase_scan INFO - |      251|    1|
28-01-2021 11:05:59 CST hbase_scan INFO - |      279|    4|
28-01-2021 11:05:59 CST hbase_scan INFO - |       91|  565|
28-01-2021 11:05:59 CST hbase_scan INFO - |       94|  244|
28-01-2021 11:05:59 CST hbase_scan INFO - +---------+-----+
28-01-2021 11:05:59 CST hbase_scan INFO - only showing top 20 rows


本文永久性链接:
<a href="http://r4.com.cn/art172.aspx">pyspark日期处理及比较</a>
apple手机报价