This page summarizes the steps to install Spark 2.2.1 in your Windows environment.
Tools and Environment
- GIT Bash
- Command Prompt
- Windows 10
Download Binary Package
Download the latest binary from the following site:
In my case, I am saving the file to folder: F:\DataAnalytics.
UnZip binary package
Open Git Bash, and change directory (cd) to the folder where you save the binary package and then unzip:
$ cd F:\DataAnalytics
fahao@Raymond-Alienware MINGW64 /f/DataAnalytics $ tar -xvzf spark-2.2.1-bin-hadoop2.7.tgz
In my case, spark is extracted to: F:\DataAnalytics\spark-2.2.1-bin-hadoop2.7
Setup environment variables
JAVA\_HOME
Follow section ‘JAVA_HOME environment variable’ in the following page to setup JAVA_HOME
https://app.kontext.tech/docs/DataAndBusinessIntelligence/p/install-zeppelin-073-in-windows
SPARK\_HOME
Setup SPARK_HOME environment variable with value of your spark installation directory.
/project/spark/resources/00BEC7DF-53C3-5B1C-91B7-7A820F20648A.webp
PATH
Added ‘%SPARK_HOME%\bin’ to your path environment variable.
Verify the installation
Verify command
Run the following command in Command Prompt to verify the installation.
%SPARK_HOME%\bin\spark-shell
The screen should be similar to the following screenshot:
/project/spark/resources/824DC488-5525-5DDF-90F2-1072C0CE0C5E.webp
Run examples
Execute the following command in Command Prompt to run one example provided as part of Spark installation (class SparkPi with param 10).
https://spark.apache.org/docs/latest/
%SPARK_HOME%\bin\run-example.cmd SparkPi 10
The output looks like the following: /project/spark/resources/1618E2EA-FCBB-5C61-B352-A644CE5222AE.webp
Spark context UI
As printed out, Spark context Web UI available at http://172.24.144.1:4040.
The following is a screenshot of the UI:
/project/spark/resources/9186ACFA-18DB-58FC-B089-C7D695E5413F.webp
Spark developer tools
Refer to the following page if you are interested in any Spark developer tools.