pyspark连hbase报org.apache.spark.examples.pythonconverters.ImmutableBytesWritableToStringConverter
作者:高景洋 日期:2020-10-28 13:25:08 浏览次数:1782
pyspark连接hbase报org.apache.spark.examples.pythonconverters.ImmutableBytesWritableToStringConverter
1、在Spark 2.0版本上缺少相关把hbase的数据转换python可读取的jar包
2、下载jar包spark-examples_2.11-1.6.0-typesafe-001.jar下载地址：https://mvnrepository.com/artifact/org.apache.spark/spark-examples_2.11/1.6.0-typesafe-001
3、解压文件，并将解压后的文件移至 /usr/local/spark3.0.1/examples/jars 目录下
4、执行 cd /usr/local/spark3.0.1/conf 目录
5、打开 spark-env.sh 文件（如果该文件不存在，则先执行 cp spark-env.sh.template spark-env.sh 复制一个）
执行 vi spark-env.sh
6、添加项 export SPARK_DIST_CLASSPATH=$(hadoop classpath):/usr/local/hbase2.2.2/lib/*:/usr/local/spark3.0.1/examples/jars/*
将 /usr/local/spark3.0.1/examples/jars/* 追加至 SPARK_DIST_CLASSPATH 最后
7、程序运行正常
本文永久性链接：
<a href="http://r4.com.cn/art160.aspx">pyspark连hbase报org.apache.spark.examples.pythonconverters.ImmutableBytesWritableToStringConverter</a>
<a href="http://r4.com.cn/art160.aspx">pyspark连hbase报org.apache.spark.examples.pythonconverters.ImmutableBytesWritableToStringConverter</a>
当前header:Host: r4.com.cn
X-Host1: r4.com.cn
X-Host2: r4.com.cn
X-Host3: 127.0.0.1:8080
X-Forwarded-For: 3.145.42.140
X-Real-Ip: 3.145.42.140
X-Domain: r4.com.cn
X-Request: GET /art160.aspx HTTP/1.1
X-Request-Uri: /art160.aspx
Connection: close
Accept: */*
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Accept-Encoding: gzip, br, zstd, deflate