Raymond Raymond

Sqoop Installation on Windows 10 using Windows Subsystem for Linux

event 2019-05-19 visibility 3,861 comment 2 insights toc
more_vert
insights Stats

This page summarizes the steps required to install Apache Sqoop (v1.4.7) in Windows 10 environment via Windows Subsystem for Linux (WSL).

Prerequisites

If you have already installed Hadoop 3.2.0 in WSL, ignore the following steps as you don’t need to install it again.

Follow  the following pages to install WSL in a system or non-system drive on your Windows 10 and also install Hadoop 3.2.0 in the WSL.

Now let’s start to install Apache Sqoop v1.4.7.

Download binary package

Go to Sqoop official website to find a download URL for Sqoop 1.

In the mirror site, there are usually two binaries available:

  • sqoop-[version].bin__hadoop-2.6.0.tar.gz
  • sqoop-[version].tar.gz

We are after the first one.

To me, the recommended location is http://apache.mirror.digitalpacific.com.au/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz.

Download the package using the following command:

wget http://apache.mirror.digitalpacific.com.au/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

Unzip the binary package

Unpack the binary using the following command:

tar -xvzf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C ~/hadoop

Setup environment variables

Setup SQOOP_HOME environment variables and also add the bin subfolder into PATH variable.

Run the following command to change .bashrc file:

vi ~/.bashrc

Add the following lines to the end of the file:

export SQOOP_HOME=~/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0                                                             

export PATH=$SQOOP_HOME/bin:$PATH

Source the modified file to make it effective:

source  ~/.bashrc

Test configurations

Run the following command to test the configurations:

$SQOOP_HOME/bin/configure-sqoop

Example output:

Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.

Print out version

The above warning messages are printed out because we have not configured HBase, HCAT, Accumulo or Zookeeper in the WSL distro yet.

$ sqoop version
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/tangr/hadoop/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
/home/tangr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 2364: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: bad substitution
/home/tangr/hadoop/hadoop-3.2.0/libexec/hadoop-functions.sh: line 2459: HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS: bad substitution
2019-05-19 12:04:18,429 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017

More from Kontext
comment Comments
Raymond Raymond #243 access_time 5 years ago more_vert
@Sekhar You can refer to the user documentation link about sqoop import command. You need to pass in your JDBC url, user name, password, source table, etc.
format_quote

person Sekhar access_time 5 years ago

Thanks for the detailed steps for installing Hadoop, Sqoop. I was trying to sqoop data from mysql and getting different kind of errors. I followed your instructions to install on WSL. Would it be possible to provide steps/examples to sqoop data from any derby/mysql. Thank you.. 
hide_source Anonymous #115 access_time 5 years ago more_vert
Thanks for the detailed steps for installing Hadoop, Sqoop. I was trying to sqoop data from mysql and getting different kind of errors. I followed your instructions to install on WSL. Would it be possible to provide steps/examples to sqoop data from any derby/mysql. Thank you.. 

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts