linux sqoop wsl

Sqoop Installation on Windows 10 using Windows Subsystem for Linux

381 views 0 comments about 6 months ago Raymond Tang

This page summarizes the steps required to install Apache Sqoop (v1.4.7) in Windows 10 environment via Windows Subsystem for Linux (WSL).

Prerequisites

If you have already installed Hadoop 3.2.0 in WSL, ignore the following steps as you don’t need to install it again.

Follow  the following pages to install WSL in a system or non-system drive on your Windows 10 and also install Hadoop 3.2.0 in the WSL.

Now let’s start to install Apache Sqoop v1.4.7.

Download binary package

Go to Sqoop official website to find a download URL for Sqoop 1.

In the mirror site, there are usually two binaries available:

  • sqoop-[version].bin__hadoop-2.6.0.tar.gz
  • sqoop-[version].tar.gz

We are after the first one.

To me, the recommended location is http://apache.mirror.digitalpacific.com.au/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz.

Download the package using the following command:

wget http://apache.mirror.digitalpacific.com.au/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

Unzip the binary package

Unpack the binary using the following command:

tar -xvzf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C ~/hadoop

Setup environment variables

Setup SQOOP_HOME environment variables and also add the bin subfolder into PATH variable.

Run the following command to change .bashrc file:

vi ~/.bashrc

Add the following lines to the end of the file:

export SQOOP_HOME=~/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0                                                             

export PATH=$SQOOP_HOME/bin:$PATH

Source the modified file to make it effective:

source  ~/.bashrc

Test configurations

Run the following command to test the configurations:

$SQOOP_HOME/bin/configure-sqoop

Example output:

Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.

Print out version

The above warning messages are printed out because we have not configured HBase, HCAT, Accumulo or Zookeeper in the WSL distro yet.

$ sqoop version
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
/home/tangr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 2364: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: bad substitution
/home/tangr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 2459: HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS: bad substitution
2019-05-19 12:04:18,429 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017

Add comment

Comments (0)
No comments yet.