Advent of 2021, Day 6 – Setting up IDE
This article is originally published at https://tomaztsql.wordpress.com
Series of Apache Spark posts:
- Dec 01: What is Apache Spark
- Dec 02: Installing Apache Spark
- Dec 03: Getting around CLI and WEB UI in Apache Spark
- Dec 04: Spark Architecture – Local and cluster mode
- Dec 05: Setting up Spark Cluster
Let’s look into the IDE that can be used to run Spark.
Remember that Spark can be used with languages: Scala, Java, R, Python and each give you different IDE and different installations.
Jupyter Notebooks
Start Jupyter Notebooks and create a new notebook and you can connect to Local Spark installation.
For the testing purposes you can add code like:
spark = SparkSession.builder.set_master("spark://tomazs-MacBook-Air.local:7077")
And start working with the Spark code.
Python
In Python, you can open a PyCharm or Spyder and start working with python code:
import findspark
findspark.init("/opt/spark")
from pyspark import SparkContext
sc = SparkContext(appName="SampleLambda")
x = sc.parallelize([1, 2, 3, 4])
res = x.filter(lambda x: (x % 2 == 0))
print(res.collect())
sc.stop()
R
Open RStudio and install sparkly package, create a context and run a simple R script:
# install
devtools::install_github("rstudio/sparklyr")
spark_disconnect(sc)
# install local version
spark_install(version = "2.2.0")
# Create a local Spark master
sc <- spark_connec(master = "local")
iris_tbl <- copy_to(sc, iris)
iris_tbl
spark_disconnect(sc)
There you go. This part was fairly short but crucial for coding.
Tomorrow we will start exploring spark code. 
Compete set of code, documents, notebooks, and all of the materials will be available at the Github repository: https://github.com/tomaztk/Spark-for-data-engineers
Happy Spark Advent of 2021!
Thanks for visiting r-craft.org
This article is originally published at https://tomaztsql.wordpress.com
Please visit source website for post related comments.
