Data transformations
====================

The top bar shows a list of icons to quickly perform common data manipulations.
Mosaic transforms the data using Blaze expressions, and each of the buttons
inserts a command that can be further customized to fit your needs.

After inserting a command click *Preview* to preview the changes you are about
to make, click *Apply* to perform the transformation. A sequence of tags will
appear below the command field, these keep track of the transformations so you
can undo your previous commands by just deleting the tags.

The data in the following table was obtained performing a `by` transformation
followed by a `projection`. You can see them listed right above the table:

.. image:: /img/mosaic_intro3.png
   :align: center

It is also possible to copy the current data into a new dataset, click *Commit*
to open the *Commit Expression* box, enter a Name and a Description for the new
dataset and click *OK*.

.. image:: /img/mosaic_intro4.png
   :align: center

Now you should see the new data set listed on the left-hand column:

.. image:: /img/mosaic_intro5.png
   :align: center

The dataset on the current active tab is always denoted by `x`, but it is also
possible to use the label instead. For instance, if you open the `Iris` dataset,
whose default label on Mosaic is `iris_csv`, the following commands are
equivalent:

.. code-block:: python

   iris_csv[iris_csv.petal_length > 2]

   x[x.petal_length > 2]

Below you will find a full description of the transformations available trough
the icons on the top bar:

by
^^

.. image:: /img/mosaic_by.png
   :align: left

By is  the basic tool for Split-Apply-Combine operations. It allows to
break the table into pieces and perform reductions on each piece.

To show how `by` works the following code
will use the `species` column as categorical variable and compute the average
petal width and count how many observation there are.

Clicking the `by` button displays `by(x.name, total=x.amount.sum())` as base
command. We modify this default command to:

.. code-block:: python

  by(x.species, mean=x.petal_width.mean(), total=x.species.count())

Which produces the next table

.. image:: /img/mosaic-userguide1.png
   :align: center

projection
^^^^^^^^^^

.. image:: /img/mosaic_projection.png
   :align: left

This option allows to select a subset of the fields from the data.
To do this just use double brackets and a list of comma-separated field
names. For instance, to select out the 'species' and 'sepal_length' columns
just run

.. code-block:: python

   x[['species', 'sepal_length']]

And the corresponding output is a two-column table:

.. image:: /img/mosaic-userguide2.png
   :align: center

merge
^^^^^

.. image:: /img/mosaic_merge.png
   :align: left

Use this to merge many fields together.

.. code-block:: python

   merge(x.species, petal_area=x.petal_length * x.petal_width)

transform
^^^^^^^^^

.. image:: /img/mosaic_transform.png
   :align: left

Adds new named columns to an existing table. It takes as arguments
the table and the new columns names and data:

.. code-block:: python
   
   transform(x, petal_area=x.petal_width * x.petal_length)

The resulting table has a new column called "petal_area"

.. image:: /img/mosaic-userguide3.png
   :align: center

relabel
^^^^^^^

.. image:: /img/mosaic_relabel.png
   :align: left

Allows to change the labels for each column. The data will remain unaltered,
only the labels are updated.

.. code-block:: python

   x.relabel({'sepal_width':  'width', 'sepal_length': 'length'})

There's an alternative syntax to relabel the column, using an assignment
operator:

.. code-block:: python

   x.relabel(sepal_width = 'width', sepal_length = 'length')

In the resulting table, instead of the previous "sepal_length" and
"sepal_width" labels the new labels "length" and "width" are shown.

.. image:: /img/mosaic-userguide4.png
   :align: center
   
concat
^^^^^^

.. image:: /img/mosaic_concat.png
   :align: left

Stacks tables on common columns. It takes as arguments the collections to
concatenate. 

For example, if there are two tables with the same fields:

.. image:: /img/mosaic-userguide5.png
   :align: center

.. image:: /img/mosaic-userguide6.png
   :align: center

To combine them into a single table jut run the following command:

.. code-block:: python
   
  concat(table1, table2)

A third parameter can be passed to determine the axis to concatenate on. The
resulting table of the previous command is this:

.. image:: /img/mosaic-userguide7.png
   :align: center

join
^^^^

.. image:: /img/mosaic_join.png
   :align: left

Joins two tables on common columns. The simplest case is when both tables
have a common column with unique values. For instance, suppose you have the
following tables whose names are `table1` and `table2` respectively:

.. image:: /img/mosaic-userguide08.png
   :align: center

.. image:: /img/mosaic-userguide09.png
   :align: center

The "id" column can be used to join them by means of the next command

.. code-block:: python

   join(table1, table2, 'id')

In this case there are three arguments: the left table, the right table, and
the common column to use. It is possible to user more than one column in this
command, just pass the additional columns as extra arguments. 

In the case of the example, it creates a new table combining the previous two:

.. image:: /img/mosaic-userguide10.png
   :align: center

Additional parameters can be used to select fields from the left and right
tables, and a suffix can be set in case of duplicate columns. For more
information see the `Blaze documentation
<https://blaze.readthedocs.io/en/latest/api.html#blaze.expr.collections.Join>`_.

sort
^^^^

.. image:: /img/mosaic_sort.png
   :align: left

Allows to sort the table using one or several columns. For example, you
can use the `petal_width` column in the "Iris" table to sort the data:

.. code-block:: python

   x.sort('petal_width')

You can further sort the data using the `petal_length` and `sepal_width`
columns. In this case a list containing the column names is passed as
parameter:

.. code-block:: python

   x.sort(['petal_width', 'petal_length', 'sepal_width'])

Which produces the following table:

.. image:: /img/mosaic-userguide11.png
   :align: center

Additionally, there is an optional boolean parameter `ascending` to determine
the order of the sort.

select
^^^^^^

.. image:: /img/mosaic_select.png
   :align: left

Filter elements of expression based on predicate or condition. The syntax is
simple, just insert the condition inside brackets. For example, to filter out
those elements whose petal length is greater than 4 just run:

.. code-block:: python

   x[x.petal_length>4]

distinct
^^^^^^^^

.. image:: /img/mosaic_distinct.png
   :align: left

Scanning a particular column, it removes the duplicate elements in that column
from the table. For example, the following command removes the duplicate values
in the `petal_length` column:

.. code-block:: python

   x.distinct('petal_length')

All the entries in that column are unique in the resulting table:

.. image:: /img/mosaic-userguide12.png
   :align: center

The last two icons in the top bar are drop-down menus with extra commands to
perform row-wise and column-wise operations, respectively.

rows
^^^^

.. image:: /img/mosaic_rows.png
   :align: left

* **head** Prints the first n elements of the table. The default value for `n` is
  `10`. To see the fist 5 elements of the table the following table will do the
  trick:
  
  .. code-block:: python
  
     x.head(5)
  
  This creates the following output:
  
  .. image:: /img/mosaic-userguide13.png
     :align: center
  
* **tail** Prints the last n elements of the table. Again, the default value
  for `n` is `10`. To see the last 7 elements of the table run the command below:
  
  .. code-block:: python
  
     x.tail(7)

* **slice** Select a subset of the data providing the start and stop indexes.
  Optionally a `step` parameter can be used. The following command selects 10
  table entries, from the row 10 to the row 30, the step width is 2.
  
  .. code-block:: python 
  
     x[10: 30: 2]
  
  .. image:: /img/mosaic-userguide14.png
     :align: center
  
cols
^^^^

.. image:: /img/mosaic_cols.png
   :align: left

* **coerce** Coerce an expression to a different type. The pane at the bottom of
  the right column shows the data types of the columns in the current data set.
  In the case of the "Iris" table this is what is shows:
  
  .. image:: /img/mosaic-userguide15.png
     :align: center
  
  It is possible to coerce the numeric columns to a different type, for instance
  to switch the `sepal_length` column to `float16` just run:
  
  .. code-block:: python
  
     x.sepal_length.coerce('float16')
  
  Which generates the following output:
  
  .. image:: /img/mosaic-userguide16.png
     :align: center
  
* **isin** Checks if a set of values is in the table. The following tests if
  the values `[1.3, 1.4, 1.5]` are in the `petal_length` column and select the
  corresponding rows:
  
  .. code-block:: python
  
     x[x.petal_length.isin([1.3, 1.4, 1.5])]o
  
  .. image:: /img/mosaic-userguide17.png
     :align: center
  
* **shift** Shift a column backward or forward by N elements, negative arguments
  will switch it backward. For example, the following image shows the first 10
  entries of the `petal_length` column:
  
  .. image:: /img/mosaic-userguide18.png
     :align: center
  
  The next command switches the column 3 positions forward:
  
  .. code-block:: python
  
     x.petal_length.shift(3)
  
  This is how the new `petal_length` columns looks like:

  .. image:: /img/mosaic-userguide19.png
     :align: center
 
