User Guide
==========

.. contents::
   :local:
   :depth: 2

Overview
--------

Using Anaconda Scale with Anaconda Adam, you can create and manage conda
packages, including Python and R, in multiple conda environments across your
nodes. You can prepend ``adam scale`` to a subset of familiar ``conda`` commands
to install and manage conda packages and environments across a cluster.

.. code-block:: none

    $ adam scale -n cluster conda --help
    Usage: adam conda [OPTIONS] COMMAND [ARGS]...

    Options:
      -h, --help          Show this message and exit.

    Commands:
      create   Create conda environment
      info     Get information about conda installation
      install  Install package(s)
      list     List conda packages
      remove   Remove package(s)
      update   Update package(s)

By default, the root conda environment is installed in
``/opt/continuum/anaconda`` on all of the cluster nodes, which can be configured
for a different location. You can manage multiple conda environments across the
same cluster (e.g., Python 2, Python 3, and R) without affecting the framework
installation of Python on the cluster nodes.

Install Conda Packages
----------------------

You can install conda packages on the cluster nodes using
``adam scale conda install``.

For example, to install ``numpy`` and ``pandas`` and their dependencies into the
root conda environment on a cluster named ``cluster``:

.. code-block:: none

  $ adam scale -n cluster conda install numpy pandas
  All nodes (x4) response:
  {
    "actions": {
      "LINK": [
        "mkl-11.3.3-0 /opt/continuum/anaconda/pkgs 1",
        "openssl-1.0.2h-1 /opt/continuum/anaconda/pkgs 1",
        "sqlite-3.13.0-0 /opt/continuum/anaconda/pkgs 1",
        "zlib-1.2.8-3 /opt/continuum/anaconda/pkgs 1",
        "python-2.7.11-5 /opt/continuum/anaconda/pkgs 1",
        "conda-env-2.5.1-py27_0 /opt/continuum/anaconda/pkgs 1",
        "numpy-1.11.0-py27_2 /opt/continuum/anaconda/pkgs 1",
        "pycosat-0.6.1-py27_1 /opt/continuum/anaconda/pkgs 1",
        "pytz-2016.4-py27_0 /opt/continuum/anaconda/pkgs 1",
        "pyyaml-3.11-py27_4 /opt/continuum/anaconda/pkgs 1",
        "requests-2.10.0-py27_0 /opt/continuum/anaconda/pkgs 1",
        "ruamel_yaml-0.11.7-py27_0 /opt/continuum/anaconda/pkgs 1",
        "setuptools-23.0.0-py27_0 /opt/continuum/anaconda/pkgs 1",
        "six-1.10.0-py27_0 /opt/continuum/anaconda/pkgs 1",
        "wheel-0.29.0-py27_0 /opt/continuum/anaconda/pkgs 1",
        "conda-4.1.4-py27_0 /opt/continuum/anaconda/pkgs 1",
        "pip-8.1.2-py27_0 /opt/continuum/anaconda/pkgs 1",
        "python-dateutil-2.5.3-py27_0 /opt/continuum/anaconda/pkgs 1",
        "pandas-0.18.1-np111py27_0 /opt/continuum/anaconda/pkgs 1"
      ],
      "PREFIX": "/opt/continuum/anaconda",
    },
    "success": true
  }

Create Conda Environments
-------------------------

You can create multiple conda environments on the cluster nodes using
``adam scale conda create``.

For example, to create a conda environment called ``py3`` with Python 3 on a
cluster named ``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda create -n py3 python=3
    All nodes (x4) response:
    {
      "actions": {
        "LINK": [
          "openssl-1.0.2h-1 1 None",
          "readline-6.2-2 1 None",
          "sqlite-3.13.0-0 1 None",
          "tk-8.5.18-0 1 None",
          "xz-5.2.2-0 1 None",
          "zlib-1.2.8-3 1 None",
          "python-3.5.1-5 1 None",
          "setuptools-23.0.0-py35_0 1 None",
          "wheel-0.29.0-py35_0 1 None",
          "pip-8.1.2-py35_0 1 None"
        ],
        "PREFIX": "/opt/continuum/anaconda/envs/py3",
        "SYMLINK_CONDA": [
          "/opt/continuum/anaconda"
        ],
      },
      "success": true
    }

Update Conda Packages
---------------------

You can update conda packages on the cluster nodes using
``adam scale conda update``.

For example, to update ``pandas`` and its dependencies in the root conda
environment on a cluster named ``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda update pandas
    All nodes (x4) response:
    {
      "actions": {
        "LINK": [
          "pandas-0.18.1-np111py27_0 1 None"
        ],
        "PREFIX": "/opt/continuum/anaconda",
        "SYMLINK_CONDA": [
          "/opt/continuum/anaconda"
        ],
        "UNLINK": [
          "pandas-0.18.0-np111py27_0"
        ],
      },
      "success": true
    }

Remove Conda Packages or Environments
-------------------------------------

You can remove conda packages on the cluster nodes using
``adam scale conda remove``.

For example, to remove ``pandas`` from the root conda environment on a cluster
named ``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda remove pandas
    All nodes (x4) response:
    {
      "actions": {
        "PREFIX": "/opt/continuum/anaconda",
        "UNLINK": [
          "pandas-0.18.1-np111py27_0"
        ],
      },
      "success": true
    }

You can also remove conda environments on the cluster nodes using
``adam scale conda remove``.

For example, to remove an environment called ``py3`` on a cluster named
``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda remove -n py3 --all
    All nodes (x4) response:
    {
      "actions": {
        "PREFIX": "/opt/continuum/anaconda/envs/py3",
        "UNLINK": [
          "openssl-1.0.2j-0",
          "pip-8.1.2-py35_0",
          "python-3.5.2-0",
          "readline-6.2-2",
          "setuptools-27.2.0-py35_0",
          "sqlite-3.13.0-0",
          "tk-8.5.18-0",
          "wheel-0.29.0-py35_0",
          "xz-5.2.2-0",
          "zlib-1.2.8-3"
        ]
      },
      "success": true
    }

List Conda Packages
-------------------

You can list conda packages on the cluster nodes using
``adam scale conda list``.

For example, to list conda packages in the root conda environment on a cluster
named ``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda list
    All nodes (x4) response:
    +-----------------+---------+--------+
    | Package         | Version | Build  |
    +-----------------+---------+--------+
    | numpy           | 1.11.0  | py27_2 |
    | conda-env       | 2.5.1   | py27_0 |
    | tk              | 8.5.18  | 0      |
    | pip             | 8.1.2   | py27_0 |
    | readline        | 6.2     | 2      |
    | setuptools      | 23.0.0  | py27_0 |
    | openssl         | 1.0.2h  | 1      |
    | pycosat         | 0.6.1   | py27_1 |
    | python-dateutil | 2.5.3   | py27_0 |
    | sqlite          | 3.13.0  | 0      |
    | pytz            | 2016.4  | py27_0 |
    | mkl             | 11.3.3  | 0      |
    | yaml            | 0.1.6   | 0      |
    | ruamel_yaml     | 0.11.7  | py27_0 |
    | zlib            | 1.2.8   | 3      |
    | pyyaml          | 3.11    | py27_4 |
    | requests        | 2.10.0  | py27_0 |
    | wheel           | 0.29.0  | py27_0 |
    | python          | 2.7.11  | 5      |
    | conda           | 4.1.4   | py27_0 |
    | pycrypto        | 2.6.1   | py27_0 |
    | six             | 1.10.0  | py27_0 |
    +-----------------+---------+--------+

Get Conda Information
---------------------

You can view information about the conda installation on the cluster nodes using
``adam scale conda info``.

For example, to view information about the root conda environment on a cluster
named ``cluster``:

.. code-block:: none

    $ adam scale -n cluster conda info
    All nodes (x4) response:
    Current conda install:

                 platform : linux-64
            conda version : 4.1.4
        conda-env version : 2.5.1
      conda-build version : not installed
           python version : 2.7.11.final.0
         requests version : 2.10.0
         root environment : /opt/continuum/anaconda  (writable)
      default environment : /opt/continuum/anaconda
         envs directories : /opt/continuum/anaconda/envs
            package cache : /opt/continuum/anaconda/pkgs
             channel URLs : https://repo.continuum.io/pkgs/free/linux-64/
                            https://repo.continuum.io/pkgs/free/noarch/
                            https://repo.continuum.io/pkgs/pro/linux-64/
                            https://repo.continuum.io/pkgs/pro/noarch/
              config file : None
             offline mode : False
        is foreign system : False

Or to view a list of conda environments on the cluster nodes:

.. code-block:: none

  $ adam scale -n cluster conda info -e
  All nodes (x4) response:
  # conda environments:
  #
  py27                     /opt/continuum/anaconda/envs/py27
  py35                     /opt/continuum/anaconda/envs/py35
  r                        /opt/continuum/anaconda/envs/r
  root                  *  /opt/continuum/anaconda


Managing Conda with Environment Definition Files
------------------------------------------------

You can also manage conda environments on the cluster nodes using
``adam scale env``:

.. code-block:: none

    $ adam scale -n cluster env
    Usage: adam scale env [OPTIONS] COMMAND [ARGS]...

    Options:
      -h, --help  Show this message and exit.

    Commands:
      create  Create conda environment
      list    List conda environments
      remove  Remove conda environment
      update  Update conda environment

This functionality uses
`conda-env <http://conda.pydata.org/docs/commands.html#conda-environment-commands>`_
and conda environment environment definition (YAML) files.

For example, the following ``environment.yml`` file describes a conda
environment named ``my-env`` with Python 2.7 and additional packages:

.. code-block:: yaml

    name: my-env
    channels:
    - defaults
    dependencies:
    - python=2.7.*
    - bokeh
    - dask
    - distributed
    - pandas

You can use ``adam scale env create`` to create a conda environment based on the
above conda environment definition file (``environment.yml``):

.. code-block:: none

    $ adam scale -n cluster env create -f environment.yml
    All nodes (x4) response:
    {
      "actions": {
        "EXTRACT": [
          "msgpack-python-0.4.8-py27_0",
          "numpy-1.11.2-py27_0",
          "psutil-5.0.0-py27_0",
          "pytz-2016.7-py27_0",
          "tblib-1.3.0-py27_0",
          "python-dateutil-2.6.0-py27_0",
          "zict-0.1.0-py27_0",
          "pandas-0.19.1-np111py27_0",
          "tornado-4.4.2-py27_0",
          "bokeh-0.12.3-py27_0",
          "dask-0.12.0-py27_0",
          "distributed-1.14.1-py27_0"
        ],
        "FETCH": [
          "msgpack-python-0.4.8-py27_0",
          "numpy-1.11.2-py27_0",
          "psutil-5.0.0-py27_0",
          "pytz-2016.7-py27_0",
          "tblib-1.3.0-py27_0",
          "python-dateutil-2.6.0-py27_0",
          "zict-0.1.0-py27_0",
          "pandas-0.19.1-np111py27_0",
          "tornado-4.4.2-py27_0",
          "bokeh-0.12.3-py27_0",
          "dask-0.12.0-py27_0",
          "distributed-1.14.1-py27_0"
        ],
        "LINK": [
          "mkl-11.3.3-0 1",
          "openssl-1.0.2j-0 1",
          "readline-6.2-2 1",
          "sqlite-3.13.0-0 1",
          "tk-8.5.18-0 1",
          "yaml-0.1.6-0 1",
          "zlib-1.2.8-3 1",
          "python-2.7.12-1 1",
          "backports-1.0-py27_0 1",
          "backports_abc-0.4-py27_0 1",
          "click-6.6-py27_0 1",
          "cloudpickle-0.2.1-py27_0 1",
          "futures-3.0.5-py27_0 1",
          "heapdict-1.0.0-py27_1 1",
          "locket-0.2.0-py27_1 1",
          "markupsafe-0.23-py27_2 1",
          "msgpack-python-0.4.8-py27_0 1",
          "numpy-1.11.2-py27_0 1",
          "psutil-5.0.0-py27_0 1",
          "pytz-2016.7-py27_0 1",
          "pyyaml-3.12-py27_0 1",
          "requests-2.11.1-py27_0 1",
          "setuptools-27.2.0-py27_0 1",
          "six-1.10.0-py27_0 1",
          "tblib-1.3.0-py27_0 1",
          "toolz-0.8.0-py27_0 1",
          "wheel-0.29.0-py27_0 1",
          "chest-0.2.3-py27_0 1",
          "jinja2-2.8-py27_1 1",
          "partd-0.3.6-py27_0 1",
          "pip-9.0.1-py27_0 1",
          "python-dateutil-2.6.0-py27_0 1",
          "singledispatch-3.4.0.3-py27_0 1",
          "ssl_match_hostname-3.4.0.2-py27_1 1",
          "zict-0.1.0-py27_0 1",
          "pandas-0.19.1-np111py27_0 1",
          "tornado-4.4.2-py27_0 1",
          "bokeh-0.12.3-py27_0 1",
          "dask-0.12.0-py27_0 1",
          "distributed-1.14.1-py27_0 1"
        ],
        "PREFIX": "/opt/continuum/anaconda/envs/my-env",
        "SYMLINK_CONDA": [
          "/opt/continuum/anaconda"
        ],
        "op_order": [
          "RM_FETCHED",
          "FETCH",
          "RM_EXTRACTED",
          "EXTRACT",
          "UNLINK",
          "LINK",
          "SYMLINK_CONDA"
        ]
      },
      "success": true
    }

Note that ``adam scale env create`` is only compatible with conda dependencies
and will ignore any ``pip`` or other non-conda dependencies that are specified
in the conda environment definition file.

You can use ``adam scale env list`` to list conda environments on the cluster:

.. code-block:: none

    $ adam scale -n cluster env list
    All nodes (x4) response:
    # conda environments:
    #
    my-env                   /opt/continuum/anaconda/envs/my-env
    root                  *  /opt/continuum/anaconda

You can use ``adam scale env update`` to update packages in an existing conda
environment based on a conda environment definition file:

.. code-block:: none

    $ adam scale -n cluster env update -f environment.yml
        All nodes (x4) response:
        {
          "message": "All requested packages already installed.",
          "success": true
        }

You can use ``adam scale env remove`` to remove a conda environment from the
cluster:

.. code-block:: none

    $ adam scale -n cluster env remove -n my-env
    All nodes (x4) response:
    {
      "actions": {
        "PREFIX": "/opt/continuum/anaconda/envs/my-env",
        "UNLINK": [
          "mkl-11.3.3-0",
          "numpy-1.11.2-py27_0",
          "openssl-1.0.2j-0",
          "pandas-0.18.0-np111py27_0",
          "pip-9.0.1-py27_0",
          "python-2.7.12-1",
          "python-dateutil-2.6.0-py27_0",
          "pytz-2016.7-py27_0",
          "readline-6.2-2",
          "setuptools-27.2.0-py27_0",
          "six-1.10.0-py27_0",
          "sqlite-3.13.0-0",
          "tk-8.5.18-0",
          "wheel-0.29.0-py27_0",
          "zlib-1.2.8-3"
        ]
      },
      "success": true
    }
