Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

Raymond Raymond visibility 11,132 event 2019-05-18 access_time 2 years ago language English

Previously, I demonstrated how to configured Apache Hive 3.0.0 on Windows 10.

On this page, I’m going to show you how to install the latest version Apache Hive 3.1.1 on Windows 10 using Windows Subsystem for Linux (WSL) Ubuntu distro.

infoYou can follow these instructions to install Apache Hive 3.1.2 too on WSL or any UNIX-alike systems incl. Debian, Ubuntu, openSUSE, Red Hat, MacOS, etc. 
warning Alert - Apache Hive is impacted by Log4j vulnerabilities; refer to page Apache Log4j Security Vulnerabilities to find out the fixes.

Prerequisites

Follow either of the following pages to install WSL in a system or non-system drive on your Windows 10.

Please also install Hadoop 3.2.0 on your WSL following the second page.

Now let’s start to install Apache Hive 3.1.1 in WSL.

Download binary package

Select a package from the download page:

https://hive.apache.org/downloads.html

For me, the recommended location is: http://www.strategylions.com.au/mirror/hive/hive-3.1.1/apache-hive-3.1.1-bin.tar.gz

In WSL bash terminal, run the following command to download the package:

wget http://www.strategylions.com.au/mirror/hive/hive-3.1.1/apache-hive-3.1.1-bin.tar.gz

Unzip binary package

If you have configured Hadoop 3.2.0 successfully, there should be one hadoop folder existing in your home folder already:

$ ls -lt
total 611896
drwxrwxrwx 1 tangr tangr      4096 May 16 00:32 dfs
drwxrwxrwx 1 tangr tangr      4096 May 15 23:48 hadoop
-rw-rw-rw- 1 tangr tangr  345625475 Jan 22 02:15 hadoop-3.2.0.tar.gz
-rw-rw-rw- 1 tangr tangr 280944629 Nov  1  2018 apache-hive-3.1.1-bin.tar.gz

Now unzip Hive package using the following command:

tar -xvzf apache-hive-3.1.1-bin.tar.gz -C ~/hadoop

In the hadoop folder there are now two subfolders:

$ ls ~/hadoop
apache-hive-3.1.1-bin  hadoop-3.2.0

Setup environment variables

In the prerequisites sections, we’ve already configured some environment variables like the following:

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export HADOOP_HOME=/home/tangr/hadoop/hadoop-3.2.0
export PATH=$PATH:$HADOOP_HOME/bin

*Note: your user name can be different.

Let’s run the following command to add Hive required environment variables into .bashrc file too:

vi ~/.bashrc

Add the following lines to the end of the file:

export HIVE_HOME=/home/tangr/hadoop/apache-hive-3.1.1-bin
export PATH=$HIVE_HOME/bin:$PATH

Change the highlighted user name to your own one.

Run the following command to source the variables:

source ~/.bashrc

Verify the environment variables:

echo $HIVE_HOME
/home/tangr/hadoop/apache-hive-3.1.1-bin

Setup Hive HDFS folders

Start your Hadoop services (if you have not done that) by running the following command:

$HADOOP_HOME/sbin/start-all.sh

In WSL, you may need to restart you ssh services if ssh doesn’t work:

localhost: ssh: connect to host localhost port 22: Connection refused

To restart the services, run the following command:

sudo service ssh restart

Run the following command (jps) to make sure all the services are running successfully.

$ jps
2306 NameNode
2786 SecondaryNameNode
3235 NodeManager
3577 Jps
2491 DataNode
3039 ResourceManager

As you can see, all the services are running successfully in my WSL.

Now let’s setup the HDFS folders for Hive.

Run the following commands:

hadoop fs -mkdir /tmp 
hadoop fs -mkdir -p /user/hive/warehouse
hadoop fs -chmod g+w /tmp
hadoop fs -chmod g+w /user/hive/warehouse

Configure Hive metastore

Now we need to run schematool to setup metastore for Hive. The command syntax looks like the following:

$HIVE_HOME/bin/schematool -dbType <db type> -initSchema

For argument dbType, it can be any of the following values:

derby|mysql|postgres|oracle|mssql

By default, Apache Derby will be used. However it is a standalone database and can only be used for one connection concurrently.

So now you have two options:

  • Option 1 (highly-recommended): Initialize using a remote database. For my scenario, I will use a SQL Server database as remote store. For more details, please follow this page to setup a remote database as datastore: Configure a SQL Server Database as Remote Hive Metastore.
  • Option 2: Initialize using Derby by running the following command:
$HIVE_HOME/bin/schematool -dbType derby -initSchema

Configure Hive API authentication

Add the following section to $HIVE_HOME/conf/hive-site.xml file:

<property>
    <name>hive.metastore.event.db.notification.api.auth</name>
     <value>false</value>
     <description>
       Should metastore do authorization against database notification related APIs such as get_next_notification.
       If set to true, then only the superusers in proxy settings have the permission
     </description>
   </property>

And then update Hadoop core-site.xml configuration file to add the following configurations:

<property>
      <name>hadoop.proxyuser.tangr.hosts</name>
      <value>*</value> </property>
<property>
      <name>hadoop.proxyuser.tangr.groups</name>
      <value>*</value> </property>

Replace the highlighted user name to your own user name.

Now all the configurations are done.

Start HiveServer2 service

Run the command below to start the HiveServer2 service:

$HIVE_HOME/bin/hive --service metastore &
$HIVE_HOME/bin/hive --service hiveserver2 &

Wait until you can open HiveServer2 Web UI:  http://localhost:10002/.

Practices

You can follow section ‘DDL practices’ in my previous post to test your Hive data warehouse.

Apache Hive 3.0.0 Installation on Windows 10 Step by Step Guide

I’ll continue to publish a number of other posts about installing latest Hadoop ecosystem tools/frameworks in WSL. You can follow this website by subscribing RSS.

More from Kontext
info Last modified by Administrator 2 years ago copyright This page is subject to Site terms.
comment Comments
G Guy access_time 3 years ago more_vert
#409 Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

thank you :)


the walk through helped me alot.

now waiting for impala guide !


format_quote

person Raymond access_time 3 years ago
Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

Thanks for summarizing this, Guy.

For the Hive 3.1.2 installation guide on this site, I've already incorporated it.

Cheers, Raymond

Raymond Raymond access_time 3 years ago more_vert
#408 Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

I have not got time to create the Impala guide yet but I've created one for HBase just in case you are interested.

Install HBase in WSL - Pseudo-Distributed Mode


format_quote

person Guy access_time 3 years ago
Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

yes please ! to both.

i will add the steps and fixes i did.

and would very much be happy with impala walk through.

Thank you

Guy

Raymond Raymond access_time 3 years ago more_vert
#407 Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

Thanks for summarizing this, Guy.

For the Hive 3.1.2 installation guide on this site, I've already incorporated it.

Cheers, Raymond

format_quote

person Guy access_time 3 years ago
Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

some missing steps:


just before running the initSchema, need to do 3 things:

1. need to copy the conf file:

cp $HIVE_HOME/conf/hive-default.xml.template $HIVE_HOME/conf/hive-site.xml


2. need to remove a typo from line 3215 of the file: &#8;

nano or vim /home/guy/hadoop/apache-hive-3.1.2-bin/conf/hive-site.xml and remore the letters in the middle of the line ( Ensures commands with OVERWRITE (such as INSERT OVERWRITE) acquire Exclusive locks for <<&#8;>> transactional tables.  This ensures that inserts (w/o overwrite) running concurrently)

3. also need to remove  guava-19.0.jar from   apache-hive-3.1.2/lib and copy the current one from the folder hadoop-3.3.0/share/hadoop/common/lib/


after that you can run the init schema for the derbyDB

G Guy access_time 3 years ago more_vert
#406 Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

some missing steps:


just before running the initSchema, need to do 3 things:

1. need to copy the conf file:

cp $HIVE_HOME/conf/hive-default.xml.template $HIVE_HOME/conf/hive-site.xml


2. need to remove a typo from line 3215 of the file: &#8;

nano or vim /home/guy/hadoop/apache-hive-3.1.2-bin/conf/hive-site.xml and remore the letters in the middle of the line ( Ensures commands with OVERWRITE (such as INSERT OVERWRITE) acquire Exclusive locks for <<&#8;>> transactional tables.  This ensures that inserts (w/o overwrite) running concurrently)

3. also need to remove  guava-19.0.jar from   apache-hive-3.1.2/lib and copy the current one from the folder hadoop-3.3.0/share/hadoop/common/lib/


after that you can run the init schema for the derbyDB

G Guy access_time 3 years ago more_vert
#405 Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

yes please ! to both.

i will add the steps and fixes i did.

and would very much be happy with impala walk through.

Thank you

Guy

format_quote

person Raymond access_time 3 years ago
Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

I'm glad it worked. I will try to publish one installation guide for Impala when I get time. If you are interested, you can also try and publish your steps on Kontext too. 

Cheers,

Raymond

Raymond Raymond access_time 3 years ago more_vert
#404 Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

I'm glad it worked. I will try to publish one installation guide for Impala when I get time. If you are interested, you can also try and publish your steps on Kontext too. 

Cheers,

Raymond

format_quote

person Guy access_time 3 years ago
Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

hello

yes, after fixing some stuff .. mentioned in my previous message, i am able now to connect.

the problems ( 5 of them ) where consisting on missing information.


thank you


can i also ask if you can add a tutorial of how to implement impala on windows WSL ? 

:)

G Guy access_time 3 years ago more_vert
#403 Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

hello

yes, after fixing some stuff .. mentioned in my previous message, i am able now to connect.

the problems ( 5 of them ) where consisting on missing information.


thank you


can i also ask if you can add a tutorial of how to implement impala on windows WSL ? 

:)

format_quote

person Raymond access_time 3 years ago
Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

Hi Guy,

Can you confirm whether you can connect to HiveServer2 or not?

It takes a little bit time for the web UI service to get started. 

Can you run the following command to see if two RunJar process are running (One is for metastore and another for HiverServer2)?

jps -mlv

-Raymond 

Raymond Raymond access_time 3 years ago more_vert
#402 Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

Hi Guy,

Can you confirm whether you can connect to HiveServer2 or not?

It takes a little bit time for the web UI service to get started. 

Can you run the following command to see if two RunJar process are running (One is for metastore and another for HiverServer2)?

jps -mlv

-Raymond 

format_quote

person Guy access_time 3 years ago
Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

one more thing, it appears that there are two bugs in the apache-hive-3.1.2

1. there is a typo in line 3215 of file hive-site.xml ( needed to be deleted )

2. the guava jar should be the same version as the hadoop one.

after fixing those 2 things.  ( and copying the default xml file to hive-site.xml ) i was able to run the :

$HIVE_HOME/bin/schematool -dbType derby -initSchema

script successfully

so now we are back to why i cannot log into the web page:

http://localhost:10002/

as two services are running

$HIVE_HOME/bin/hive --service metastore &
$HIVE_HOME/bin/hive --service hiveserver2 &


and no error in the log files.


thank you

G Guy access_time 3 years ago more_vert
#401 Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

one more thing, it appears that there are two bugs in the apache-hive-3.1.2

1. there is a typo in line 3215 of file hive-site.xml ( needed to be deleted )

2. the guava jar should be the same version as the hadoop one.

after fixing those 2 things.  ( and copying the default xml file to hive-site.xml ) i was able to run the :

$HIVE_HOME/bin/schematool -dbType derby -initSchema

script successfully

so now we are back to why i cannot log into the web page:

http://localhost:10002/

as two services are running

$HIVE_HOME/bin/hive --service metastore &
$HIVE_HOME/bin/hive --service hiveserver2 &


and no error in the log files.


thank you

format_quote

person Raymond access_time 3 years ago
Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

Hi Guy,

Most likely it is because that your Hive Metastore database schema is not correct, for example, init schema operation was not done properly.

Can you conduct the following two actions?

  1. Double check your Hive metastore configurations are correct. for example, database name, type, and username and password, etc.
  2. Check whether your metastore database is initialized successfully. For example, the following screenshot shows all the tables created by the init step in this guide. 


These tables are used by Hive to store metadata of your objects in Hive database. 

If you cannot start metastore service successfully, HiveServer2 thrift service will not be able to function properly.

G Guy access_time 3 years ago more_vert
#400 Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

hello


thank you sir for answer.

i found some stuff ... while doing the step-by-step i went alone with derby DB .. but since it didn't work. i decided to try mssql DB.. following this link:

https://kontext.tech/column/hadoop/302/configure-a-sql-server-database-as-remote-hive-metastore


found out there are steps in this page that are not in the original page.

namely this command:

cp $HIVE_HOME/conf/hive-default.xml.template $HIVE_HOME/conf/hive-site.xml

there is though a code you said to add the hive-site.xml file .. but maybe there are lines missing there ? ( this is a huge file and millions of parameters in it )


could you recheck please if there is not omitted part in the original page :

https://kontext.tech/column/hadoop/309/apache-hive-311-installation-on-windows-10-using-windows-subsystem-for-linux


thank you again for help !


Guy

format_quote

person Raymond access_time 3 years ago
Re: Apache Hive 3.1.1 Installation on Windows 10 using Windows Subsystem for Linux

Hi Guy,

Most likely it is because that your Hive Metastore database schema is not correct, for example, init schema operation was not done properly.

Can you conduct the following two actions?

  1. Double check your Hive metastore configurations are correct. for example, database name, type, and username and password, etc.
  2. Check whether your metastore database is initialized successfully. For example, the following screenshot shows all the tables created by the init step in this guide. 


These tables are used by Hive to store metadata of your objects in Hive database. 

If you cannot start metastore service successfully, HiveServer2 thrift service will not be able to function properly.

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts