arrow_back Spark 3.0.1: Connect to HBase 2.4.1
person Pavan Kumar access_time 3 years ago
Thanks for pointing that @Raymond. My Hadoop, Spark, Scala, and Hbase versions are 3.2.1, 3.1.1,2.12, and 2.4.7 respectively.
Maven build:
mvn -Dspark.version=3.1.1 -Dscala.version=2.12.10 -Dscala.binary.version=2.12 -Dhbase.version=2.4.7 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1 -DskipTests -Dcheckstyle.skip -U clean package
I have upgraded Maven and the issue is resolved. But seeing a compilation error as below.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project hbase-kafka-proxy: Compilation failure
[ERROR] /home/ec2-user/git/spark-hbase/hbase-connectors/kafka/hbase-kafka-proxy/src/main/java/org/apache/hadoop/hbase/kafka/KafkaTableForBridge.java:[53,8] org.apache.hadoop.hbase.kafka.KafkaTableForBridge is not abstract and does not override abstract method getRegionLocator() in org.apache.hadoop.hbase.client.Table
I would be so grateful if you could help me with what I need to learn to resolve such issues.
Thank you so much for your help.
Thanks for pointing that @Raymond. My Hadoop, Spark, Scala, and Hbase versions are 3.2.1, 3.1.1,2.12, and 2.4.7 respectively.
Maven build:
mvn -Dspark.version=3.1.1 -Dscala.version=2.12.10 -Dscala.binary.version=2.12 -Dhbase.version=2.4.7 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1 -DskipTests -Dcheckstyle.skip -U clean package
I have upgraded Maven and the issue is resolved. But seeing a compilation error as below.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project hbase-kafka-proxy: Compilation failure
[ERROR] /home/ec2-user/git/spark-hbase/hbase-connectors/kafka/hbase-kafka-proxy/src/main/java/org/apache/hadoop/hbase/kafka/KafkaTableForBridge.java:[53,8] org.apache.hadoop.hbase.kafka.KafkaTableForBridge is not abstract and does not override abstract method getRegionLocator() in org.apache.hadoop.hbase.client.Table
I would be so grateful if you could help me with what I need to learn to resolve such issues.
Thank you so much for your help.
person Raymond access_time 3 years ago
Hi Pravan,
Is your Maven version is Apache Maven 3.6.0?
If you are using Spark 3.0.1 with HBase 2.4.1, you can directly try the one I built:
Hi Pravan,
Is your Maven version is Apache Maven 3.6.0?
If you are using Spark 3.0.1 with HBase 2.4.1, you can directly try the one I built:
person Pavan Kumar access_time 3 years ago
Hi @Raymond,
Thanks for this informative article. I followed the steps mentioned in this. But seeing the below error while building the project. It would be great if could you help me resolve the issue.
Thanks in advance.
Hi @Raymond,
Thanks for this informative article. I followed the steps mentioned in this. But seeing the below error while building the project. It would be great if could you help me resolve the issue.
Thanks in advance.
Hi Pavan,
The issue you encountered is the same one I mentioned in the article due to incompatible version of the HBase and connector code.
So please try the following command:
The built package should still work with HBase 2.4.7.
Regards,
Raymond