arrow_back Spark 3.0.1: Connect to HBase 2.4.1

comment Comments
Raymond Raymond #1534 access_time 3 years ago more_vert

Hi Pavan,

The issue you encountered is the same one I mentioned in the article due to incompatible version of the HBase and connector code. 

For HBase version, I have to use 2.2.4 as the latest hbase-connector code was based on that version. 

So please try the following command:

mvn -Dspark.version=3.1.1 -Dscala.version=2.12.10 -Dscala.binary.version=2.12 -Dhbase.version=2.2.4 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1 -DskipTests -Dcheckstyle.skip -U clean package

The built package should still work with HBase 2.4.7.


Regards,

Raymond

format_quote

person Pavan Kumar access_time 3 years ago

Thanks for pointing that @Raymond. My Hadoop, Spark, Scala, and Hbase versions are 3.2.1, 3.1.1,2.12, and 2.4.7 respectively.

Maven build:

mvn -Dspark.version=3.1.1 -Dscala.version=2.12.10 -Dscala.binary.version=2.12 -Dhbase.version=2.4.7 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1 -DskipTests -Dcheckstyle.skip -U clean package

I have upgraded Maven and the issue is resolved. But seeing a compilation error as below.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project hbase-kafka-proxy: Compilation failure

[ERROR] /home/ec2-user/git/spark-hbase/hbase-connectors/kafka/hbase-kafka-proxy/src/main/java/org/apache/hadoop/hbase/kafka/KafkaTableForBridge.java:[53,8] org.apache.hadoop.hbase.kafka.KafkaTableForBridge is not abstract and does not override abstract method getRegionLocator() in org.apache.hadoop.hbase.client.Table

I would be so grateful if you could help me with what I need to learn to resolve such issues.

Thank you so much for your help.

PK Pavan Kumar Yerravelly #1533 access_time 3 years ago more_vert

Thanks for pointing that @Raymond. My Hadoop, Spark, Scala, and Hbase versions are 3.2.1, 3.1.1,2.12, and 2.4.7 respectively.

Maven build:

mvn -Dspark.version=3.1.1 -Dscala.version=2.12.10 -Dscala.binary.version=2.12 -Dhbase.version=2.4.7 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1 -DskipTests -Dcheckstyle.skip -U clean package

I have upgraded Maven and the issue is resolved. But seeing a compilation error as below.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project hbase-kafka-proxy: Compilation failure

[ERROR] /home/ec2-user/git/spark-hbase/hbase-connectors/kafka/hbase-kafka-proxy/src/main/java/org/apache/hadoop/hbase/kafka/KafkaTableForBridge.java:[53,8] org.apache.hadoop.hbase.kafka.KafkaTableForBridge is not abstract and does not override abstract method getRegionLocator() in org.apache.hadoop.hbase.client.Table

I would be so grateful if you could help me with what I need to learn to resolve such issues.

Thank you so much for your help.

format_quote

person Raymond access_time 3 years ago

Hi Pravan,

Is your Maven version is Apache Maven 3.6.0? 

If you are using Spark 3.0.1 with HBase 2.4.1, you can directly try the one I built:

1.0.1 HBase Connectors for Spark 3.0.1 

Raymond Raymond #1532 access_time 3 years ago more_vert

Hi Pravan,

Is your Maven version is Apache Maven 3.6.0? 

If you are using Spark 3.0.1 with HBase 2.4.1, you can directly try the one I built:

1.0.1 HBase Connectors for Spark 3.0.1 

format_quote

person Pavan Kumar access_time 3 years ago

Hi @Raymond,

Thanks for this informative article. I followed the steps mentioned in this. But seeing the below error while building the project. It would be great if could you help me resolve the issue.

Thanks in advance.



PK Pavan Kumar Yerravelly #1531 access_time 3 years ago more_vert

Hi @Raymond,

Thanks for this informative article. I followed the steps mentioned in this. But seeing the below error while building the project. It would be great if could you help me resolve the issue.

Thanks in advance.



Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts