arrow_back Spark 3.0.1: Connect to HBase 2.4.1

Raymond Raymond
Raymond
Stats
articleArticles 551
codeCode 3
imageDiagrams 51
descriptionNotebooks 0
chat_bubble_outlineThreads 8
commentComments 278
loyaltyKontext Points 6098
account_circle Profile
access_time 2 years ago link more_vert
#1540 Re: Spark 3.0.1: Connect to HBase 2.4.1

Are all those jars included in the current directory where you initiated the spark-shell?

You can manually put them into \jars directory in your Spark installation. 

format_quote

person Pavan Kumar access_time 2 years ago
Re: Spark 3.0.1: Connect to HBase 2.4.1

I think there is no issue with the build. But I'm unable to connect to Hbase from Spark. I'm using a docker environment where Zookeeper, HDFS, Spark, and HBase run in different containers in the same network.

Here are the jars I'm using.

spark-shell --jars hbase-spark-protocol-shaded-1.0.0.7.2.12.0-291.jar,htrace-core4-4.2.0-incubating.jar,hbase-shaded-protobuf-3.5.1.jar,protobuf-java-2.5.0.jar,hbase-protocol-2.4.8.jar,hbase-shaded-miscellaneous-3.5.1.jar,hbase-mapreduce-2.4.8.jar,hbase-server-2.4.8.jar,hbase-client-2.4.8.jar,hbase-common-2.4.8.jar,hbase-spark-1.0.1-SNAPSHOT.jar,hadoop-common-2.8.5.jar --files hbase-site.xml

I have almost all the required jars but still seeing below error. I tried my best to debug the isue but didn't find a way to get rid of this. Please advise me how to resolve this or redirect me if there is any detailed documentation about prerequisites.

java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/shaded/protobuf/generated/MasterProtos$MasterService$BlockingInterface

  at java.lang.ClassLoader.defineClass1(Native Method)

  at java.lang.ClassLoader.defineClass(ClassLoader.java:757)

  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

reply Reply
recommendMore from Kontext