===============================
Configuring Anaconda with Spark
===============================

.. contents::
   :local:
   :depth: 2

Overview
--------

`Apache Spark <http://spark.apache.org/>`_ is an analytics engine and parallel
computation framework with Scala, Python and R interfaces. Spark can load data
directly from disk, memory and other data storage technologies such as Amazon
S3, Hadoop Distributed File System (HDFS), HBase, Cassandra, and others.

You can install Spark on a cluster using an enterprise Hadoop distribution such
as `Cloudera CDH <https://www.cloudera.com/products/apache-hadoop/key-cdh-components.html>`_
or `Hortonworks HDP <http://hortonworks.com/products/data-center/hdp/>`_.

.. _`submit-spark-job`:

Different Ways to Use Spark with Anaconda
-----------------------------------------

Spark scripts are often developed interactively and can be written as a
Python script or in a Jupyter notebook.

A PySpark script can be submitted to a Spark cluster using various methods:

* Running the script directly on the head node.
* Using the `spark-submit <https://spark.apache.org/docs/latest/submitting-applications.html>`_
  command either in Standalone mode or with the YARN resource manager.
* Interactively in an IPython shell or Jupyter Notebook on the cluster.

To run a script on the head node, simply execute ``python example.py`` on the
cluster. Alternatively, you can install Jupyter Notebook on the cluster using
Anaconda Scale. Refer to the :doc:`../install` documentation for more
information.

You can also use Anaconda Scale with enterprise Hadoop distributions such as
Cloudera CDH or Hortonworks HDP. The sections below provide details on how to
configure and use Anaconda with Spark jobs and initialize a SparkContext.

Configuring Anaconda for Spark jobs
-----------------------------------

Configuring Anaconda with the ``spark-submit`` command
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can submit Spark jobs using the ``PYSPARK_PYTHON`` environment variable that
refers to the location of the Python executable in Anaconda, for example:

.. code-block:: bash

    $ PYSPARK_PYTHON=/opt/continuum/anaconda/bin/python spark-submit pyspark_script.py

Configuring Anaconda with Jupyter Notebooks and Cloudera CDH
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can configure Jupyter Notebooks to use Anaconda Scale with Cloudera CDH
using the following Python code at the top of your notebook:

.. code-block:: python

    import os
    import sys
    os.environ["PYSPARK_PYTHON"] = "/opt/continuum/anaconda/bin/python"
    os.environ["JAVA_HOME"] = "/usr/java/jdk1.7.0_67-cloudera/jre"
    os.environ["SPARK_HOME"] = "/opt/cloudera/parcels/CDH/lib/spark"
    os.environ["PYLIB"] = os.environ["SPARK_HOME"] + "/python/lib"
    sys.path.insert(0, os.environ["PYLIB"] +"/py4j-0.9-src.zip")
    sys.path.insert(0, os.environ["PYLIB"] +"/pyspark.zip")

The above configuration was tested with Cloudera CDH 5.9 and Spark 1.6.
Depending on the version of Cloudera CDH that you have installed, you might need
to customize these paths according to the location of Java, Spark, and Anaconda
on your cluster.

Configuring Anaconda with Jupyter Notebooks and Hortonworks HDP
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can configure Jupyter Notebooks to use Anaconda Scale with Hortonworks HDP
using the following Python code at the top of your notebook:

.. code-block:: python

    import os
    import sys
    os.environ["PYSPARK_PYTHON"] = "/opt/continuum/anaconda/bin/python"
    os.environ["SPARK_HOME"] = "/usr/hdp/current/spark-client"
    os.environ["PYLIB"] = os.environ["SPARK_HOME"] + "/python/lib"
    sys.path.insert(0, os.environ["PYLIB"] +"/py4j-0.9-src.zip")
    sys.path.insert(0, os.environ["PYLIB"] +"/pyspark.zip")

The above configuration was tested with Hortonworks HDP 2.5, Apache Ambari 2.4,
and Spark 1.6. Depending on the version of Hortonworks HDP that you have
installed, you might need to customize these paths according to the location of
Spark and Anaconda on your cluster.

Creating a SparkContext
-----------------------

Once you've configured the appropriate environment variables, you can initialize
a SparkContext (in ``yarn-client`` client mode in this example) using:

.. code-block:: python

    from pyspark import SparkConf
    from pyspark import SparkContext
    conf = SparkConf()
    conf.setMaster('yarn-client')
    conf.setAppName('anaconda-pyspark')
    sc = SparkContext(conf=conf)

For more information about configuring Spark settings, refer to the
`PySpark documentation <http://spark.apache.org/docs/latest/programming-guide.html>`_.

Once you've initialized a SparkContext, you can start using Anaconda with Spark
jobs. Refer to the :doc:`../howto-overview` documentation for example Spark jobs
that use libraries from Anaconda.
