Install Spark & Pyspark on Windows

Spark is a parallel data processing framework, which is used for big data analytics. It is an open-source project and is written in Scala. It supports streaming data analysis, SQL, Machine learning, and Graph processing. It has multiple APIs for R, Python, Java, and Scala.
The installation of Spark locally on windows may not be straightforward, so I decided to write a tutorial about its installation and how to call it from python. After reading this tutorial you will learn how to 1) Install Spark on windows and 2) Create a spark session using the pyspark library. The Spark version that is used for installation is Spark 3.1.2. Spark runs on Java 8/11, Scala 2.12, Python 3.6+ and R 3.5+.
Here the list of steps that we are going through together:

1. Java installation
2. Apache spark installation
3. Move the Spark files
4. Add Hadoop support
5. Set environment variables
6. Launch Spark
7. Launch a Spark session in Python

2. Apache Spark installation

The next step is to install Apache Spark, here, you can find the package of Spark prebuild with Apache Hadoop. The latest version at the writing time of this tutorial is 3.1.2, (Edit: I have seen the newer released version,3.2.0, which should not impact the steps of its installation anyway). Select the desired Spark and Hadoop version, else you can leave the default options and download the tgz file.

You may wonder why you need Hadoop. Hadoop is a distributed file system, Spark by itself doesn’t have a storage system, so if it is needed to be run in a multi-node mode, it is dependent on Hadoop or a similar package such as S3. Spark is an in-memory distributed computing engine and can run in a standalone mode without Hadoop, but this means you miss the data distribution feature. For small workloads, it is faster than Hadoop since it is using memory to process data but cannot save the data. With the help of Hadoop, it can store the data as well.

3. Move the Spark files

After downloading of tgz file, extract the file and then create a folder named Spark and move the extracted file to the folder, For example, I have created a Spark folder in my Windows directory and pasted the contents there:

4. Add Hadoop support

For running Hadoop on windows, we need an extra “.exe” file which enables Hadoop to have file access permissions, you can download this file from here, search for the Hadoop version that you have downloaded before and download relative winutils.exe, in my case I have downloaded the latest one for Hadoop 3.2.2.

After the download, you have to create a new folder, named Hadoop (or anything), inside the folder create another folder named bin, and leave the exe file there, so the structure will be "Haoop/bin/winutiles.exe". In my case, I have created the folder in the Windows directory:

5. Set environment variables

We are done with the downloads. So let’s tell the windows where to find what we have installed. Specifically, we need to set 3 environment variables for referencing Java, Spark & Hadoop.

Go to your environment variables, you can find it by clicking on Window button and typing “Environment”, your windows should automatically find it.

• Set Java Env variable
  • Click on the new variable
  • Leave the name as JAVA_HOME
  • And you need to specify where your java is located (here you can see despite that I have two versions of Java, I specified to use version 11 which is working with Spark)

• Set Spark Env variable
  • Click on the new variable
  • Leave the name as SPARK_HOME
  • Specify where your Spark folder is located

• Set Hadoop Env variable
  • Click on the new variable
  • Leave the name as HADOOP_HOME
  • Specify where your Hadoop folder is located

Then you need to add the paths of bin files in Spark, Hadoop, and Java. For this, still in the environment variables, edit your path and add the bin paths for Spark, Hadoop, and Java:

After this, save and close the window. And you are done!

6. Launch Spark

We are in the last step to check if everything is working successfully, open a Command prompt or PowerShell, and write “spark-shell”, in the case of successful installation, you should be able to see some informative outputs like this:

After seeing the above message, you can check your Spark version also by typing “spark.version”:

You can also have access to Spark Web UI by navigating to http://localhost:4040/. Here you can find useful details about spark jobs, clusters, application status, and etc.

7. Launch a Spark session in Python

Now that we have set up everything, let’s try to use Spark in Python, for this, we need to use pyspark library.

1- Install pyspark

You can download pyspark library using pip command in the notebook:

2- Create a session

To use Spark, we need to create a session. A session is going to be the entry point for Spark.

For creating a session, we can specify some configs such as the name of the app, if you want to run Spark on a cluster or local mode, etc. and then call the method for creating the session:

In the above command, the arguments are:

• appName(): name of application
• getOrCreate: if there is any Spark session, returns it else create one

If you print out the Spark variable, you can verify more details about it:

Some details about the output:

3- Check Spark version

You can check the Spark version also in the notebook, by “spark.version”, consider Spark is the name of the variable that I have created for my Spark session:

4. Other operations

For example, I want to read from a CSV file and print the scheme of the file:

Vola! hope you could install your Spark and launch it successfully in your python!

Let me know in the comments in case of any trouble or questions, happy Sparking!

References:

• https://stackoverflow.com/questions/32022334/can-apache-spark-run-without-hadoop
• https://spark.apache.org/docs/latest/
• https://www.ibm.com/cloud/blog/hadoop-vs-spark
• https://medium.com/@achilleus/spark-session-10d0d66d1d24