当前位置:首页 - Spark

pyspark连hbase报org.apache.spark.examples.pythonconverters.ImmutableBytesWritableToStringConverter

作者:高景洋 日期:2020-10-28 13:25:08 浏览次数:1458

pyspark连接hbase报org.apache.spark.examples.pythonconverters.ImmutableBytesWritableToStringConverter


1、在Spark 2.0版本上缺少相关把hbase的数据转换python可读取的jar包

2、下载jar包spark-examples_2.11-1.6.0-typesafe-001.jar
   下载地址:https://mvnrepository.com/artifact/org.apache.spark/spark-examples_2.11/1.6.0-typesafe-001
3、解压文件,并将解压后的文件移至 /usr/local/spark3.0.1/examples/jars 目录下
4、执行 cd /usr/local/spark3.0.1/conf 目录
5、打开 spark-env.sh 文件(如果该文件不存在,则先执行 cp spark-env.sh.template spark-env.sh 复制一个)
   执行 vi spark-env.sh
6、添加项 export SPARK_DIST_CLASSPATH=$(hadoop classpath):/usr/local/hbase2.2.2/lib/*:/usr/local/spark3.0.1/examples/jars/*
   将 /usr/local/spark3.0.1/examples/jars/* 追加至 SPARK_DIST_CLASSPATH 最后
7、程序运行正常

本文永久性链接:
<a href="http://r4.com.cn/art160.aspx">pyspark连hbase报org.apache.spark.examples.pythonconverters.ImmutableBytesWritableToStringConverter</a>
当前header:Host: r4.com.cn X-Host1: r4.com.cn X-Host2: r4.com.cn X-Host3: 127.0.0.1:8080 X-Forwarded-For: 34.203.242.200 X-Real-Ip: 34.203.242.200 X-Domain: r4.com.cn X-Request: GET /art160.aspx HTTP/1.1 X-Request-Uri: /art160.aspx Connection: close Accept: */* User-Agent: claudebot