[LIVY-376][DOCS] Update README with new naming and links and move Docs into separate files

[LIVY-376](https://issues.cloudera.org/browse/LIVY-376)

 * The README has been updated to match our new Apache naming and links.
 * The API Documentation has been split into separate files
 * Examples have been moved to the website https://github.com/apache/incubator-livy-website/pull/5

This PR will be followed up with a way to build out the Docs for the website

Author: Alex Bozarth <ajbozart@us.ibm.com>

Closes #21 from ajbozarth/readme.

NOTICE.txt

@@ -1,4 +1,4 @@
-Apache livy
+Apache Livy
 Copyright 2017 The Apache Software Foundation
 
 This product includes software developed at

README.md

@@ -0,0 +1,82 @@
+# Apache Livy
+
+[![Build Status](https://travis-ci.org/apache/incubator-livy.svg?branch=master)](https://travis-ci.org/apache/incubator-livy)
+
+Apache Livy is an open source REST interface for interacting with
+[Apache Spark](http://spark.apache.org) from anywhere. It supports executing snippets of code or
+programs in a Spark context that runs locally or in
+[Apache Hadoop YARN](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html).
+
+* Interactive Scala, Python and R shells
+* Batch submissions in Scala, Java, Python
+* Multiple users can share the same server (impersonation support)
+* Can be used for submitting jobs from anywhere with REST
+* Does not require any code change to your programs
+
+[Pull requests](https://github.com/apache/incubator-livy/pulls) are welcomed! But before you begin,
+please check out the [Contributing](http://livy.incubator.apache.org/community#Contributing)
+section on the [Community](http://livy.incubator.apache.org/community) page of our website.
+
+## Online Documentation
+
+Guides and documentation on getting started using Livy, example code snippets, and Livy API
+documentation can be found at [livy.incubator.apache.org](http://livy.incubator.apache.org).
+
+## Before Building Livy
+
+To build Livy, you will need:
+
+Debian/Ubuntu:
+  * mvn (from ``maven`` package or maven3 tarball)
+  * openjdk-7-jdk (or Oracle Java7 jdk)
+  * Python 2.6+
+  * R 3.x
+
+Redhat/CentOS:
+  * mvn (from ``maven`` package or maven3 tarball)
+  * java-1.7.0-openjdk (or Oracle Java7 jdk)
+  * Python 2.6+
+  * R 3.x
+
+MacOS:
+  * Xcode command line tools
+  * Oracle's JDK 1.7+
+  * Maven (Homebrew)
+  * Python 2.6+
+  * R 3.x
+
+Required python packages for building Livy:
+  * cloudpickle
+  * requests
+  * requests-kerberos
+  * flake8
+  * flaky
+  * pytest
+
+
+To run Livy, you will also need a Spark installation. You can get Spark releases at
+https://spark.apache.org/downloads.html.
+
+Livy requires at least Spark 1.6 and supports both Scala 2.10 and 2.11 builds of Spark, Livy
+will automatically pick repl dependencies through detecting the Scala version of Spark.
+
+Livy also supports Spark 2.0+ for both interactive and batch submission, you could seamlessly
+switch to different versions of Spark through ``SPARK_HOME`` configuration, without needing to
+rebuild Livy.
+
+
+## Building Livy
+
+Livy is built using [Apache Maven](http://maven.apache.org). To check out and build Livy, run:
+
+```
+git clone https://github.com/apache/incubator-livy.git
+cd livy
+mvn package
+```
+
+By default Livy is built against Apache Spark 1.6.2, but the version of Spark used when running
+Livy does not need to match the version used to build Livy. Livy internally uses reflection to
+mitigate the gaps between different Spark versions, also Livy package itself does not
+contain a Spark distribution, so it will work with any supported version of Spark (Spark 1.6+)
+without needing to rebuild against specific version of Spark.

README.rst

@@ -1,982 +0,0 @@
-Welcome to Livy
-===============
-
-.. image:: https://travis-ci.org/cloudera/livy.svg?branch=master
-    :target: https://travis-ci.org/cloudera/livy
-
-Livy is an open source REST interface for interacting with `Apache Spark`_ from anywhere.
-It supports executing snippets of code or programs in a Spark context that runs locally or in `Apache Hadoop YARN`_.
-
-* Interactive Scala, Python and R shells
-* Batch submissions in Scala, Java, Python
-* Multiple users can share the same server (impersonation support)
-* Can be used for submitting jobs from anywhere with REST
-* Does not require any code change to your programs
-
-`Pull requests`_ are welcomed! But before you begin, please check out the `Wiki`_.
-
-.. _Apache Spark: http://spark.apache.org
-.. _Apache Hadoop YARN: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
-.. _Pull requests: https://github.com/cloudera/livy/pulls
-.. _Wiki: https://github.com/cloudera/livy/wiki/Contributing-to-Livy
-
-
-Prerequisites
-=============
-
-To build Livy, you will need:
-
-Debian/Ubuntu:
-  * mvn (from ``maven`` package or maven3 tarball)
-  * openjdk-7-jdk (or Oracle Java7 jdk)
-  * Python 2.6+
-  * R 3.x
-
-Redhat/CentOS:
-  * mvn (from ``maven`` package or maven3 tarball)
-  * java-1.7.0-openjdk (or Oracle Java7 jdk)
-  * Python 2.6+
-  * R 3.x
-
-MacOS:
-  * Xcode command line tools
-  * Oracle's JDK 1.7+
-  * Maven (Homebrew)
-  * Python 2.6+
-  * R 3.x
-
-Required python packages for building Livy:
-  * cloudpickle
-  * requests
-  * requests-kerberos
-  * flake8
-  * flaky
-  * pytest
-
-
-To run Livy, you will also need a Spark installation. You can get Spark releases at
-https://spark.apache.org/downloads.html.
-
-Livy requires at least Spark 1.6 and supports both Scala 2.10 and 2.11 builds of Spark, Livy
-will automatically pick repl dependencies through detecting the Scala version of Spark.
-
-Livy also supports Spark 2.0+ for both interactive and batch submission, you could seamlessly
-switch to different versions of Spark through ``SPARK_HOME`` configuration, without needing to
-rebuild Livy.
-
-
-Building Livy
-=============
-
-Livy is built using `Apache Maven`_. To check out and build Livy, run:
-
-.. code:: shell
-
-    git clone https://github.com/cloudera/livy.git
-    cd livy
-    mvn package
-
-By default Livy is built against Apache Spark 1.6.2, but the version of Spark used when running
-Livy does not need to match the version used to build Livy. Livy internally uses reflection to
-mitigate the gaps between different Spark versions, also Livy package itself does not
-contain a Spark distribution, so it will work with any supported version of Spark (Spark 1.6+)
-without needing to rebuild against specific version of Spark.
-
-.. _Apache Maven: http://maven.apache.org
-
-
-Running Livy
-============
-
-In order to run Livy with local sessions, first export these variables:
-
-.. code:: shell
-
-   export SPARK_HOME=/usr/lib/spark
-   export HADOOP_CONF_DIR=/etc/hadoop/conf
-
-Then start the server with:
-
-.. code:: shell
-
-    ./bin/livy-server
-
-Livy uses the Spark configuration under ``SPARK_HOME`` by default. You can override the Spark configuration
-by setting the ``SPARK_CONF_DIR`` environment variable before starting Livy.
-
-It is strongly recommended to configure Spark to submit applications in YARN cluster mode.
-That makes sure that user sessions have their resources properly accounted for in the YARN cluster,
-and that the host running the Livy server doesn't become overloaded when multiple user sessions are
-running.
-
-
-Livy Configuration
-==================
-
-Livy uses a few configuration files under configuration the directory, which by default is the
-``conf`` directory under the Livy installation. An alternative configuration directory can be
-provided by setting the ``LIVY_CONF_DIR`` environment variable when starting Livy.
-
-The configuration files used by Livy are:
-
-* ``livy.conf``: contains the server configuration. The Livy distribution ships with a default
-  configuration file listing available configuration keys and their default values.
-
-* ``spark-blacklist.conf``: list Spark configuration options that users are not allowed to override.
-  These options will be restricted to either their default values, or the values set in the Spark
-  configuration used by Livy.
-
-* ``log4j.properties``: configuration for Livy logging. Defines log levels and where log messages
-  will be written to. The default configuration will print log messages to stderr.
-
-Upgrade from Livy 0.1
-=====================
-
-A few things changed between since Livy 0.1 that require manual intervention when upgrading.
-
-- Sessions that were active when the Livy 0.1 server was stopped may need to be killed
-  manually. Use the tools from your cluster manager to achieve that (for example, the
-  ``yarn`` command line tool).
-
-- The configuration file has been renamed from ``livy-defaults.conf`` to ``livy.conf``.
-
-- A few configuration values do not have any effect anymore. Notably:
-
-  * ``livy.server.session.factory``: this config option has been replaced by the Spark
-    configuration under ``SPARK_HOME``. If you wish to use a different Spark configuration
-    for Livy, you can set ``SPARK_CONF_DIR`` in Livy's environment. To define the default
-    file system root for sessions, set ``HADOOP_CONF_DIR`` to point at the Hadoop configuration
-    to use. The default Hadoop file system will be used.
-
-  * ``livy.yarn.jar``: this config has been replaced by separate configs listing specific
-    archives for different Livy features. Refer to the default ``livy.conf`` file shipped
-    with Livy for instructions.
-
-  * ``livy.server.spark-submit``: replaced by the ``SPARK_HOME`` environment variable.
-
-
-Using the Programmatic API
-==========================
-
-Livy provides a programmatic Java/Scala and Python API that allows applications to run code inside
-Spark without having to maintain a local Spark context. Here shows how to use the Java API.
-
-Add the Cloudera repository to your application's POM:
-
-.. code:: xml
-
-    <repositories>
-      <repository>
-        <id>cloudera.repo</id>
-        <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
-        <name>Cloudera Repositories</name>
-        <snapshots>
-          <enabled>false</enabled>
-        </snapshots>
-      </repository>
-    </repositories>
-
-
-And add the Livy client dependency:
-
-.. code:: xml
-
-    <dependency>
-      <groupId>com.cloudera.livy</groupId>
-      <artifactId>livy-client-http</artifactId>
-      <version>0.2.0</version>
-    </dependency>
-
-
-To be able to compile code that uses Spark APIs, also add the correspondent Spark dependencies.
-
-To run Spark jobs within your applications, extend ``org.apache.livy.Job`` and implement
-the functionality you need. Here's an example job that calculates an approximate value for Pi:
-
-.. code:: java
-
-    import java.util.*;
-
-    import org.apache.spark.api.java.*;
-    import org.apache.spark.api.java.function.*;
-
-    import com.cloudera.livy.*;
-
-    public class PiJob implements Job<Double>, Function<Integer, Integer>,
-      Function2<Integer, Integer, Integer> {
-
-      private final int samples;
-
-      public PiJob(int samples) {
-        this.samples = samples;
-      }
-
-      @Override
-      public Double call(JobContext ctx) throws Exception {
-        List<Integer> sampleList = new ArrayList<Integer>();
-        for (int i = 0; i < samples; i++) {
-          sampleList.add(i + 1);
-        }
-
-        return 4.0d * ctx.sc().parallelize(sampleList).map(this).reduce(this) / samples;
-      }
-
-      @Override
-      public Integer call(Integer v1) {
-        double x = Math.random();
-        double y = Math.random();
-        return (x*x + y*y < 1) ? 1 : 0;
-      }
-
-      @Override
-      public Integer call(Integer v1, Integer v2) {
-        return v1 + v2;
-      }
-
-    }
-
-
-To submit this code using Livy, create a LivyClient instance and upload your application code to
-the Spark context. Here's an example of code that submits the above job and prints the computed
-value:
-
-.. code:: java
-
-    LivyClient client = new LivyClientBuilder()
-      .setURI(new URI(livyUrl))
-      .build();
-
-    try {
-      System.err.printf("Uploading %s to the Spark context...\n", piJar);
-      client.uploadJar(new File(piJar)).get();
-
-      System.err.printf("Running PiJob with %d samples...\n", samples);
-      double pi = client.submit(new PiJob(samples)).get();
-
-      System.out.println("Pi is roughly: " + pi);
-    } finally {
-      client.stop(true);
-    }
-
-
-To learn about all the functionality available to applications, read the javadoc documentation for
-the classes under the ``api`` module.
-
-
-Spark Example
-=============
-
-Here's a step-by-step example of interacting with Livy in Python with the `Requests`_ library. By
-default Livy runs on port 8998 (which can be changed with the ``livy.server.port`` config option).
-We’ll start off with a Spark session that takes Scala code:
-
-.. code:: shell
-
-    sudo pip install requests
-
-.. code:: python
-
-    import json, pprint, requests, textwrap
-    host = 'http://localhost:8998'
-    data = {'kind': 'spark'}
-    headers = {'Content-Type': 'application/json'}
-    r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
-    r.json()
-
-    {u'state': u'starting', u'id': 0, u'kind': u'spark'}
-
-Once the session has completed starting up, it transitions to the idle state:
-
-.. code:: python
-
-    session_url = host + r.headers['location']
-    r = requests.get(session_url, headers=headers)
-    r.json()
-
-    {u'state': u'idle', u'id': 0, u'kind': u'spark'}
-
-Now we can execute Scala by passing in a simple JSON command:
-
-.. code:: python
-
-    statements_url = session_url + '/statements'
-    data = {'code': '1 + 1'}
-    r = requests.post(statements_url, data=json.dumps(data), headers=headers)
-    r.json()
-
-    {u'output': None, u'state': u'running', u'id': 0}
-
-If a statement takes longer than a few milliseconds to execute, Livy returns
-early and provides a statement URL that can be polled until it is complete:
-
-.. code:: python
-
-    statement_url = host + r.headers['location']
-    r = requests.get(statement_url, headers=headers)
-    pprint.pprint(r.json())
-
-    {u'id': 0,
-      u'output': {u'data': {u'text/plain': u'res0: Int = 2'},
-                  u'execution_count': 0,
-                  u'status': u'ok'},
-      u'state': u'available'}
-
-That was a pretty simple example. More interesting is using Spark to estimate
-Pi. This is from the `Spark Examples`_:
-
-.. code:: python
-
-    data = {
-      'code': textwrap.dedent("""
-        val NUM_SAMPLES = 100000;
-        val count = sc.parallelize(1 to NUM_SAMPLES).map { i =>
-          val x = Math.random();
-          val y = Math.random();
-          if (x*x + y*y < 1) 1 else 0
-        }.reduce(_ + _);
-        println(\"Pi is roughly \" + 4.0 * count / NUM_SAMPLES)
-        """)
-    }
-
-    r = requests.post(statements_url, data=json.dumps(data), headers=headers)
-    pprint.pprint(r.json())
-
-    statement_url = host + r.headers['location']
-    r = requests.get(statement_url, headers=headers)
-    pprint.pprint(r.json())
-
-    {u'id': 1,
-     u'output': {u'data': {u'text/plain': u'Pi is roughly 3.14004\nNUM_SAMPLES: Int = 100000\ncount: Int = 78501'},
-                 u'execution_count': 1,
-                 u'status': u'ok'},
-     u'state': u'available'}
-
-Finally, close the session:
-
-.. code:: python
-
-    session_url = 'http://localhost:8998/sessions/0'
-    requests.delete(session_url, headers=headers)
-
-    <Response [204]>
-
-.. _Requests: http://docs.python-requests.org/en/latest/
-.. _Spark Examples: https://spark.apache.org/examples.html
-
-
-PySpark Example
-===============
-
-PySpark has the same API, just with a different initial request:
-
-.. code:: python
-
-    data = {'kind': 'pyspark'}
-    r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
-    r.json()
-
-    {u'id': 1, u'state': u'idle'}
-
-The Pi example from before then can be run as:
-
-.. code:: python
-
-    data = {
-      'code': textwrap.dedent("""
-        import random
-        NUM_SAMPLES = 100000
-        def sample(p):
-          x, y = random.random(), random.random()
-          return 1 if x*x + y*y < 1 else 0
-
-        count = sc.parallelize(xrange(0, NUM_SAMPLES)).map(sample).reduce(lambda a, b: a + b)
-        print "Pi is roughly %f" % (4.0 * count / NUM_SAMPLES)
-        """)
-    }
-
-    r = requests.post(statements_url, data=json.dumps(data), headers=headers)
-    pprint.pprint(r.json())
-
-    {u'id': 12,
-    u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
-                u'execution_count': 12,
-                u'status': u'ok'},
-    u'state': u'running'}
-
-
-SparkR Example
-==============
-
-SparkR has the same API:
-
-.. code:: python
-
-    data = {'kind': 'sparkr'}
-    r = requests.post(host + '/sessions', data=json.dumps(data), headers=headers)
-    r.json()
-
-    {u'id': 1, u'state': u'idle'}
-
-The Pi example from before then can be run as:
-
-.. code:: python
-
-    data = {
-      'code': textwrap.dedent("""
-        n <- 100000
-        piFunc <- function(elem) {
-          rands <- runif(n = 2, min = -1, max = 1)
-          val <- ifelse((rands[1]^2 + rands[2]^2) < 1, 1.0, 0.0)
-          val
-        }
-        piFuncVec <- function(elems) {
-          message(length(elems))
-          rands1 <- runif(n = length(elems), min = -1, max = 1)
-          rands2 <- runif(n = length(elems), min = -1, max = 1)
-          val <- ifelse((rands1^2 + rands2^2) < 1, 1.0, 0.0)
-          sum(val)
-        }
-        rdd <- parallelize(sc, 1:n, slices)
-        count <- reduce(lapplyPartition(rdd, piFuncVec), sum)
-        cat("Pi is roughly", 4.0 * count / n, "\n")
-        """)
-    }
-
-    r = requests.post(statements_url, data=json.dumps(data), headers=headers)
-    pprint.pprint(r.json())
-
-    {u'id': 12,
-     u'output': {u'data': {u'text/plain': u'Pi is roughly 3.136000'},
-                 u'execution_count': 12,
-                 u'status': u'ok'},
-     u'state': u'running'}
-
-
-Community
-=========
-
- * User group: http://groups.google.com/a/cloudera.org/group/livy-user
- * Dev group: http://groups.google.com/a/cloudera.org/group/livy-dev
- * Dev slack: https://livy-dev.slack.com.
-
-   To join: http://livy-slack-invite.azurewebsites.net. Invite token: ``I'm not a bot``.
- * JIRA: https://issues.cloudera.org/browse/LIVY
- * Pull requests: https://github.com/cloudera/livy/pulls
-
-
-REST API
-========
-
-GET /sessions
--------------
-
-Returns all the active interactive sessions.
-
-Request Parameters
-^^^^^^^^^^^^^^^^^^
-
-+------+-----------------------------------+------+
-| name | description                       | type |
-+======+===================================+======+
-| from | The start index to fetch sessions | int  |
-+------+-----------------------------------+------+
-| size | Number of sessions to fetch       | int  |
-+------+-----------------------------------+------+
-
-Response Body
-^^^^^^^^^^^^^
-
-+----------+-------------------------------------+------+
-| name     | description                         | type |
-+==========+=====================================+======+
-| from     | The start index of fetched sessions | int  |
-+----------+-------------------------------------+------+
-| total    | Number of sessions fetched          | int  |
-+----------+-------------------------------------+------+
-| sessions | `Session`_ list                     | list |
-+----------+-------------------------------------+------+
-
-
-POST /sessions
---------------
-
-Creates a new interactive Scala, Python, or R shell in the cluster.
-
-Request Body
-^^^^^^^^^^^^
-
-+--------------------------+------------------------------------------------+-----------------+
-| name                     | description                                    | type            |
-+==========================+================================================+=================+
-| kind                     | The session kind (required)                    | `session kind`_ |
-+--------------------------+------------------------------------------------+-----------------+
-| proxyUser                | User to impersonate when starting the session  | string          |
-+--------------------------+------------------------------------------------+-----------------+
-| jars                     | jars to be used in this session                | List of string  |
-+--------------------------+------------------------------------------------+-----------------+
-| pyFiles                  | Python files to be used in this session        | List of string  |
-+--------------------------+------------------------------------------------+-----------------+
-| files                    | files to be used in this session               | List of string  |
-+--------------------------+------------------------------------------------+-----------------+
-| driverMemory             | Amount of memory to use for the driver process | string          |
-+--------------------------+------------------------------------------------+-----------------+
-| driverCores              | Number of cores to use for the driver process  | int             |
-+--------------------------+------------------------------------------------+-----------------+
-| executorMemory           | Amount of memory to use per executor process   | string          |
-+--------------------------+------------------------------------------------+-----------------+
-| executorCores            | Number of cores to use for each executor       | int             |
-+--------------------------+------------------------------------------------+-----------------+
-| numExecutors             | Number of executors to launch for this session | int             |
-+--------------------------+------------------------------------------------+-----------------+
-| archives                 | Archives to be used in this session            | List of string  |
-+--------------------------+------------------------------------------------+-----------------+
-| queue                    | The name of the YARN queue to which submitted  | string          |
-+--------------------------+------------------------------------------------+-----------------+
-| name                     | The name of this session                       | string          |
-+--------------------------+------------------------------------------------+-----------------+
-| conf                     | Spark configuration properties                 | Map of key=val  |
-+--------------------------+------------------------------------------------+-----------------+
-| heartbeatTimeoutInSecond | Timeout in second to which session be orphaned | int             |
-+--------------------------+------------------------------------------------+-----------------+
-
-
-Response Body
-^^^^^^^^^^^^^
-
-The created `Session`_.
-
-
-GET /sessions/{sessionId}
--------------------------
-
-Returns the session information.
-
-Response Body
-^^^^^^^^^^^^^
-
-The `Session`_.
-
-
-GET /sessions/{sessionId}/state
--------------------------------
-
-Returns the state of session
-
-Response
-^^^^^^^^
-
-+-------+-----------------------------------+--------+
-| name  | description                       | type   |
-+=======+===================================+========+
-| id    | Session id                        | int    |
-+-------+-----------------------------------+--------+
-| state | The current state of session      | string |
-+-------+-----------------------------------+--------+
-
-
-DELETE /sessions/{sessionId}
-----------------------------
-
-Kills the `Session`_ job.
-
-
-GET /sessions/{sessionId}/log
-------------------------------
-
-Gets the log lines from this session.
-
-Request Parameters
-^^^^^^^^^^^^^^^^^^
-
-+------+-----------------------------------+------+
-| name | description                       | type |
-+======+===================================+======+
-| from | Offset                            | int  |
-+------+-----------------------------------+------+
-| size | Max number of log lines to return | int  |
-+------+-----------------------------------+------+
-
-Response Body
-^^^^^^^^^^^^^
-
-+------+--------------------------+-----------------+
-| name | description              | type            |
-+======+==========================+=================+
-| id   | The session id           | int             |
-+------+--------------------------+-----------------+
-| from | Offset from start of log | int             |
-+------+--------------------------+-----------------+
-| size | Number of log lines      | int             |
-+------+--------------------------+-----------------+
-| log  | The log lines            | list of strings |
-+------+--------------------------+-----------------+
-
-
-GET /sessions/{sessionId}/statements
-------------------------------------
-
-Returns all the statements in a session.
-
-Response Body
-^^^^^^^^^^^^^
-
-+------------+-------------------+------+
-| name       | description       | type |
-+============+===================+======+
-| statements | `statement`_ list | list |
-+------------+-------------------+------+
-
-
-POST /sessions/{sessionId}/statements
--------------------------------------
-
-Runs a statement in a session.
-
-Request Body
-^^^^^^^^^^^^
-
-+------+---------------------+--------+
-| name | description         | type   |
-+======+=====================+========+
-| code | The code to execute | string |
-+------+---------------------+--------+
-
-Response Body
-^^^^^^^^^^^^^
-
-The `statement`_ object.
-
-
-GET /sessions/{sessionId}/statements/{statementId}
---------------------------------------------------
-
-Returns a specified statement in a session.
-
-Response Body
-^^^^^^^^^^^^^
-
-The `statement`_ object.
-
-
-POST /sessions/{sessionId}/statements/{statementId}/cancel
-----------------------------------------------------------
-
-Cancel the specified statement in this session.
-
-Response Body
-^^^^^^^^^^^^^
-
-+------+----------------------------+--------+
-| name | description                | type   |
-+======+============================+========+
-| msg  | is always "cancelled"      | string |
-+------+----------------------------+--------+
-
-
-GET /batches
--------------
-
-Returns all the active batch sessions.
-
-Request Parameters
-^^^^^^^^^^^^^^^^^^
-
-+------+-----------------------------------+------+
-| name | description                       | type |
-+======+===================================+======+
-| from | The start index to fetch sessions | int  |
-+------+-----------------------------------+------+
-| size | Number of sessions to fetch       | int  |
-+------+-----------------------------------+------+
-
-Response Body
-^^^^^^^^^^^^^
-
-+----------+-------------------------------------+------+
-| name     | description                         | type |
-+==========+=====================================+======+
-| from     | The start index of fetched sessions | int  |
-+----------+-------------------------------------+------+
-| total    | Number of sessions fetched          | int  |
-+----------+-------------------------------------+------+
-| sessions | `Batch`_ list                       | list |
-+----------+-------------------------------------+------+
-
-
-POST /batches
--------------
-
-Request Body
-^^^^^^^^^^^^
-
-+----------------+---------------------------------------------------+-----------------+
-| name           | description                                       | type            |
-+================+===================================================+=================+
-| file           | File containing the application to execute        | path (required) |
-+----------------+---------------------------------------------------+-----------------+
-| proxyUser      | User to impersonate when running the job          | string          |
-+----------------+---------------------------------------------------+-----------------+
-| className      | Application Java/Spark main class                 | string          |
-+----------------+---------------------------------------------------+-----------------+
-| args           | Command line arguments for the application        | list of strings |
-+----------------+---------------------------------------------------+-----------------+
-| jars           | jars to be used in this session                   | List of string  |
-+----------------+---------------------------------------------------+-----------------+
-| pyFiles        | Python files to be used in this session           | List of string  |
-+----------------+---------------------------------------------------+-----------------+
-| files          | files to be used in this session                  | List of string  |
-+----------------+---------------------------------------------------+-----------------+
-| driverMemory   | Amount of memory to use for the driver process    | string          |
-+----------------+---------------------------------------------------+-----------------+
-| driverCores    | Number of cores to use for the driver process     | int             |
-+----------------+---------------------------------------------------+-----------------+
-| executorMemory | Amount of memory to use per executor process      | string          |
-+----------------+---------------------------------------------------+-----------------+
-| executorCores  | Number of cores to use for each executor          | int             |
-+----------------+---------------------------------------------------+-----------------+
-| numExecutors   | Number of executors to launch for this session    | int             |
-+----------------+---------------------------------------------------+-----------------+
-| archives       | Archives to be used in this session               | List of string  |
-+----------------+---------------------------------------------------+-----------------+
-| queue          | The name of the YARN queue to which submitted     | string          |
-+----------------+---------------------------------------------------+-----------------+
-| name           | The name of this session                          | string          |
-+----------------+---------------------------------------------------+-----------------+
-| conf           | Spark configuration properties                    | Map of key=val  |
-+----------------+---------------------------------------------------+-----------------+
-
-Response Body
-^^^^^^^^^^^^^
-
-The created `Batch`_ object.
-
-
-GET /batches/{batchId}
-----------------------
-
-Returns the batch session information.
-
-Response Body
-^^^^^^^^^^^^^
-
-The `Batch`_.
-
-
-GET /batches/{batchId}/state
-----------------------------
-
-Returns the state of batch session
-
-Response
-^^^^^^^^
-
-+-------+-----------------------------------+--------+
-| name  | description                       | type   |
-+=======+===================================+========+
-| id    | Batch session id                  | int    |
-+-------+-----------------------------------+--------+
-| state | The current state of batch session| string |
-+-------+-----------------------------------+--------+
-
-
-DELETE /batches/{batchId}
--------------------------
-
-Kills the `Batch`_ job.
-
-
-GET /batches/{batchId}/log
---------------------------
-
-Gets the log lines from this batch.
-
-Request Parameters
-^^^^^^^^^^^^^^^^^^
-
-+------+-----------------------------------+------+
-| name | description                       | type |
-+======+===================================+======+
-| from | Offset                            | int  |
-+------+-----------------------------------+------+
-| size | Max number of log lines to return | int  |
-+------+-----------------------------------+------+
-
-Response Body
-^^^^^^^^^^^^^
-
-+------+--------------------------+-----------------+
-| name | description              | type            |
-+======+==========================+=================+
-| id   | The batch id             | int             |
-+------+--------------------------+-----------------+
-| from | Offset from start of log | int             |
-+------+--------------------------+-----------------+
-| size | Number of log lines      | int             |
-+------+--------------------------+-----------------+
-| log  | The log lines            | list of strings |
-+------+--------------------------+-----------------+
-
-
-REST Objects
-============
-
-Session
--------
-
-A session represents an interactive shell.
-
-+----------------+------------------------------------------+----------------------------+
-| name           | description                              | type                       |
-+================+==========================================+============================+
-| id             | The session id                           | int                        |
-+----------------+------------------------------------------+----------------------------+
-| appId          | The application id of this session       | String                     |
-+----------------+------------------------------------------+----------------------------+
-| owner          | Remote user who submitted this session   | String                     |
-+----------------+------------------------------------------+----------------------------+
-| proxyUser      | User to impersonate when running         | String                     |
-+----------------+------------------------------------------+----------------------------+
-| kind           | Session kind (spark, pyspark, or sparkr) | `session kind`_            |
-+----------------+------------------------------------------+----------------------------+
-| log            | The log lines                            | list of strings            |
-+----------------+------------------------------------------+----------------------------+
-| state          | The session state                        | string                     |
-+----------------+------------------------------------------+----------------------------+
-| appInfo        | The detailed application info            | Map of key=val             |
-+----------------+------------------------------------------+----------------------------+
-
-
-Session State
-^^^^^^^^^^^^^
-
-+---------------+----------------------------------+
-| value         | description                      |
-+===============+==================================+
-| not_started   | Session has not been started     |
-+---------------+----------------------------------+
-| starting      | Session is starting              |
-+---------------+----------------------------------+
-| idle          | Session is waiting for input     |
-+---------------+----------------------------------+
-| busy          | Session is executing a statement |
-+---------------+----------------------------------+
-| shutting_down | Session is shutting down         |
-+---------------+----------------------------------+
-| error         | Session errored out              |
-+---------------+----------------------------------+
-| dead          | Session has exited               |
-+---------------+----------------------------------+
-| success       | Session is successfully stopped  |
-+---------------+----------------------------------+
-
-Session Kind
-^^^^^^^^^^^^
-
-+-------------+------------------------------------+
-| value       | description                        |
-+=============+====================================+
-| spark       | Interactive Scala  Spark session   |
-+-------------+------------------------------------+
-| `pyspark`_  | Interactive Python 2 Spark session |
-+-------------+------------------------------------+
-| `pyspark3`_ | Interactive Python 3 Spark session |
-+-------------+------------------------------------+
-| sparkr      | Interactive R Spark session        |
-+-------------+------------------------------------+
-
-pyspark
-^^^^^^^
-To change the Python executable the session uses, Livy reads the path from environment variable
-``PYSPARK_PYTHON`` (Same as pyspark).
-
-Like pyspark, if Livy is running in ``local`` mode, just set the environment variable.
-If the session is running in ``yarn-cluster`` mode, please set
-``spark.yarn.appMasterEnv.PYSPARK_PYTHON`` in SparkConf so the environment variable is passed to
-the driver.
-
-pyspark3
-^^^^^^^^
-To change the Python executable the session uses, Livy reads the path from environment variable
-``PYSPARK3_PYTHON``.
-
-Like pyspark, if Livy is running in ``local`` mode, just set the environment variable.
-If the session is running in ``yarn-cluster`` mode, please set
-``spark.yarn.appMasterEnv.PYSPARK3_PYTHON`` in SparkConf so the environment variable is passed to
-the driver.
-
-Statement
----------
-
-A statement represents the result of an execution statement.
-
-+--------+----------------------+---------------------+
-| name   | description          | type                |
-+========+======================+=====================+
-| id     | The statement id     | integer             |
-+--------+----------------------+---------------------+
-| code   | The execution code   | string              |
-+--------+----------------------+---------------------+
-| state  | The execution state  | statement state     |
-+--------+----------------------+---------------------+
-| output | The execution output | statement output    |
-+--------+----------------------+---------------------+
-
-Statement State
-^^^^^^^^^^^^^^^
-
-+------------+----------------------------------------------------+
-| value      | description                                        |
-+============+====================================================+
-| waiting    | Statement is enqueued but execution hasn't started |
-+------------+----------------------------------------------------+
-| running    | Statement is currently running                     |
-+------------+----------------------------------------------------+
-| available  | Statement has a response ready                     |
-+------------+----------------------------------------------------+
-| error      | Statement failed                                   |
-+------------+----------------------------------------------------+
-| cancelling | Statement is being cancelling                      |
-+------------+----------------------------------------------------+
-| cancelled  | Statement is cancelled                             |
-+------------+----------------------------------------------------+
-
-Statement Output
-^^^^^^^^^^^^^^^^
-
-+-----------------+-------------------+----------------------------------+
-| name            | description       | type                             |
-+=================+===================+==================================+
-| status          | Execution status  | string                           |
-+-----------------+-------------------+----------------------------------+
-| execution_count | A monotonically   | integer                          |
-|                 | increasing number |                                  |
-+-----------------+-------------------+----------------------------------+
-| data            | Statement output  | An object mapping a mime type to |
-|                 |                   | the result. If the mime type is  |
-|                 |                   | ``application/json``, the value  |
-|                 |                   | is a JSON value.                 |
-+-----------------+-------------------+----------------------------------+
-
-Batch
------
-
-+----------------+-------------------------------------+-----------------+
-| name           | description                         | type            |
-+================+=====================================+=================+
-| id             | The session id                      | int             |
-+----------------+-------------------------------------+-----------------+
-| appId          | The application id of this session  | String          |
-+----------------+-------------------------------------+-----------------+
-| appInfo        | The detailed application info       | Map of key=val  |
-+----------------+-------------------------------------+-----------------+
-| log            | The log lines                       | list of strings |
-+----------------+-------------------------------------+-----------------+
-| state          | The batch state                     | string          |
-+----------------+-------------------------------------+-----------------+
-
-
-License
-=======
-
-Apache License, Version 2.0
-http://www.apache.org/licenses/LICENSE-2.0

docs/programmatic-api.md

@@ -0,0 +1,89 @@
+## Using the Programmatic API
+
+Livy provides a programmatic Java/Scala and Python API that allows applications to run code inside
+Spark without having to maintain a local Spark context. Here shows how to use the Java API.
+
+Add the Livy client dependency to your application's POM:
+
+```xml
+<dependency>
+  <groupId>org.apache.livy</groupId>
+  <artifactId>livy-client-http</artifactId>
+  <version>0.4.0-SNAPSHOT</version>
+</dependency>
+```
+
+Note: Until Livy's first Apache release you will have to
+install the livy artifacts locally using `mvn install`.
+
+To be able to compile code that uses Spark APIs, also add the correspondent Spark dependencies.
+
+To run Spark jobs within your applications, extend ``org.apache.livy.Job`` and implement
+the functionality you need. Here's an example job that calculates an approximate value for Pi:
+
+```java
+import java.util.*;
+
+import org.apache.spark.api.java.*;
+import org.apache.spark.api.java.function.*;
+
+import org.apache.livy.*;
+
+public class PiJob implements Job<Double>, Function<Integer, Integer>,
+  Function2<Integer, Integer, Integer> {
+
+  private final int samples;
+
+  public PiJob(int samples) {
+    this.samples = samples;
+  }
+
+  @Override
+  public Double call(JobContext ctx) throws Exception {
+    List<Integer> sampleList = new ArrayList<Integer>();
+    for (int i = 0; i < samples; i++) {
+      sampleList.add(i + 1);
+    }
+
+    return 4.0d * ctx.sc().parallelize(sampleList).map(this).reduce(this) / samples;
+  }
+
+  @Override
+  public Integer call(Integer v1) {
+    double x = Math.random();
+    double y = Math.random();
+    return (x*x + y*y < 1) ? 1 : 0;
+  }
+
+  @Override
+  public Integer call(Integer v1, Integer v2) {
+    return v1 + v2;
+  }
+
+}
+```
+
+To submit this code using Livy, create a LivyClient instance and upload your application code to
+the Spark context. Here's an example of code that submits the above job and prints the computed
+value:
+
+```java
+LivyClient client = new LivyClientBuilder()
+  .setURI(new URI(livyUrl))
+  .build();
+
+try {
+  System.err.printf("Uploading %s to the Spark context...\n", piJar);
+  client.uploadJar(new File(piJar)).get();
+
+  System.err.printf("Running PiJob with %d samples...\n", samples);
+  double pi = client.submit(new PiJob(samples)).get();
+
+  System.out.println("Pi is roughly: " + pi);
+} finally {
+  client.stop(true);
+}
+```
+
+To learn about all the functionality available to applications, read the javadoc documentation for
+the classes under the ``api`` module.

docs/rest-api.md

@@ -0,0 +1,726 @@
+## REST API
+
+### GET /sessions
+
+Returns all the active interactive sessions.
+
+
+#### Request Parameters
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>from</td>
+    <td>The start index to fetch sessions</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>size</td>
+    <td>Number of sessions to fetch</td>
+    <td>int</td>
+  </tr>
+</table>
+
+#### Response Body
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>from</td>
+    <td>The start index to fetch sessions</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>size</td>
+    <td>Number of sessions to fetch</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>sessions</td>
+    <td><a href="#session">Session</a> list</td>
+    <td>list</td>
+  </tr>
+</table>
+
+### POST /sessions
+
+Creates a new interactive Scala, Python, or R shell in the cluster.
+
+#### Request Body
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>kind</td>
+    <td>The session kind (required)</td>
+    <td><a href="#session-kind">session kind</a></td>
+  </tr>
+  <tr>
+    <td>proxyUser</td>
+    <td>User to impersonate when starting the session</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>jars</td>
+    <td>jars to be used in this session</td>
+    <td>List of string</td>
+  </tr>
+  <tr>
+    <td>pyFiles</td>
+    <td>Python files to be used in this session</td>
+    <td>List of string</td>
+  </tr>
+  <tr>
+    <td>files</td>
+    <td>files to be used in this session</td>
+    <td>List of string</td>
+  </tr>
+  <tr>
+    <td>driverMemory</td>
+    <td>Amount of memory to use for the driver process</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>driverCores</td>
+    <td>Number of cores to use for the driver process</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>executorMemory</td>
+    <td>Amount of memory to use per executor process</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>executorCores</td>
+    <td>Number of cores to use for each executor</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>numExecutors</td>
+    <td>Number of executors to launch for this session</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>archives</td>
+    <td>Archives to be used in this session</td>
+    <td>List of string</td>
+  </tr>
+  <tr>
+    <td>queue</td>
+    <td>The name of the YARN queue to which submitted</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>name</td>
+    <td>The name of this session</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>conf</td>
+    <td>Spark configuration properties</td>
+    <td>Map of key=val</td>
+  </tr>
+  <tr>
+    <td>heartbeatTimeoutInSecond</td>
+    <td>Timeout in second to which session be orphaned</td>
+    <td>int</td>
+  </tr>
+</table>
+
+#### Response Body
+
+The created <a href="#session">Session</a>.
+
+
+### GET /sessions/{sessionId}
+
+Returns the session information.
+
+#### Response Body
+
+The <a href="#session">Session</a>.
+
+
+### GET /sessions/{sessionId}/state
+
+Returns the state of session
+
+#### Response
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>id</td>
+    <td>Session id</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>state</td>
+    <td>The current state of session</td>
+    <td>string</td>
+  </tr>
+</table>
+
+### DELETE /sessions/{sessionId}
+
+Kills the <a href="#session">Session</a> job.
+
+
+### GET /sessions/{sessionId}/log
+
+Gets the log lines from this session.
+
+#### Request Parameters
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>from</td>
+    <td>Offset</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>size</td>
+    <td>Max number of log lines to return</td>
+    <td>int</td>
+  </tr>
+</table>
+
+#### Response Body
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>id</td>
+    <td>The session id</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>from</td>
+    <td>Offset from start of log</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>size</td>
+    <td>Max number of log lines</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>log</td>
+    <td>The log lines</td>
+    <td>list of strings</td>
+  </tr>
+</table>
+
+### GET /sessions/{sessionId}/statements
+
+Returns all the statements in a session.
+
+#### Response Body
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>statements</td>
+    <td><a href="#statement">statement</a> list</td>
+    <td>list</td>
+  </tr>
+</table>
+
+### POST /sessions/{sessionId}/statements
+
+Runs a statement in a session.
+
+#### Request Body
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>code</td>
+    <td>The code to execute</td>
+    <td>string</td>
+  </tr>
+</table>
+
+#### Response Body
+
+The <a href="#statement">statement</a> object.
+
+
+### GET /sessions/{sessionId}/statements/{statementId}
+
+Returns a specified statement in a session.
+
+#### Response Body
+
+The <a href="#statement">statement</a> object.
+
+
+### POST /sessions/{sessionId}/statements/{statementId}/cancel
+
+Cancel the specified statement in this session.
+
+#### Response Body
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>msg</td>
+    <td>is always "cancelled"</td>
+    <td>string</td>
+  </tr>
+</table>
+
+### GET /batches
+
+Returns all the active batch sessions.
+
+#### Request Parameters
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>from</td>
+    <td>The start index to fetch sessions</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>size</td>
+    <td>Number of sessions to fetch</td>
+    <td>int</td>
+  </tr>
+</table>
+
+#### Response Body
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>from</td>
+    <td>The start index of fetched sessions</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>total</td>
+    <td>Number of sessions fetched</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>sessions</td>
+    <td><a href="#batch">Batch</a> list</td>
+    <td>list</td>
+  </tr>
+</table>
+
+### POST /batches
+
+#### Request Body
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>file</td>
+    <td>File containing the application to execute</td>
+    <td>path (required)</td>
+  </tr>
+  <tr>
+    <td>proxyUser</td>
+    <td>User to impersonate when running the job</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>className</td>
+    <td>Application Java/Spark main class</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>args</td>
+    <td>Command line arguments for the application</td>
+    <td>list of strings</td>
+  </tr>
+  <tr>
+    <td>jars</td>
+    <td>jars to be used in this session</td>
+    <td>list of strings</td>
+  </tr>
+  <tr>
+    <td>pyFiles</td>
+    <td>Python files to be used in this session</td>
+    <td>list of strings</td>
+  </tr>
+  <tr>
+    <td>files</td>
+    <td>files to be used in this session</td>
+    <td>list of strings</td>
+  </tr>
+  <tr>
+    <td>driverMemory</td>
+    <td>Amount of memory to use for the driver process</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>driverCores</td>
+    <td>Number of cores to use for the driver process</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>executorMemory</td>
+    <td>Amount of memory to use per executor process</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>executorCores</td>
+    <td>Number of cores to use for each executor</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>numExecutors</td>
+    <td>Number of executors to launch for this session</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>archives</td>
+    <td>Archives to be used in this session</td>
+    <td>List of string</td>
+  </tr>
+  <tr>
+    <td>queue</td>
+    <td>The name of the YARN queue to which submitted</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>name</td>
+    <td>The name of this session</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>conf</td>
+    <td>Spark configuration properties</td>
+    <td>Map of key=val</td>
+  </tr>
+</table>
+
+#### Response Body
+
+The created <a href="#batch">Batch</a> object.
+
+
+### GET /batches/{batchId}
+
+Returns the batch session information.
+
+#### Response Body
+
+The <a href="#batch">Batch</a>.
+
+
+### GET /batches/{batchId}/state
+
+Returns the state of batch session
+
+#### Response
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>id</td>
+    <td>Batch session id</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>state</td>
+    <td>The current state of batch session</td>
+    <td>string</td>
+  </tr>
+</table>
+
+### DELETE /batches/{batchId}
+
+Kills the <a href="#batch">Batch</a> job.
+
+
+### GET /batches/{batchId}/log
+
+Gets the log lines from this batch.
+
+#### Request Parameters
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>from</td>
+    <td>Offset</td>
+    <td>Offset</td>
+  </tr>
+  <tr>
+    <td>size</td>
+    <td>Max number of log lines to return</td>
+    <td>Offset</td>
+  </tr>
+</table>
+
+#### Response Body
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>id</td>
+    <td>The batch id</td>
+    <td>id</td>
+  </tr>
+  <tr>
+    <td>from</td>
+    <td>Offset from start of log</td>
+    <td>id</td>
+  </tr>
+  <tr>
+    <td>size</td>
+    <td>Number of log lines</td>
+    <td>id</td>
+  </tr>
+  <tr>
+    <td>log</td>
+    <td>The log lines</td>
+    <td>list of strings</td>
+  </tr>
+</table>
+
+## REST Objects
+
+### Session
+
+A session represents an interactive shell.
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>id</td>
+    <td>The session id</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>appId</td>
+    <td>The application id of this session</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>owner</td>
+    <td>Remote user who submitted this session</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>proxyUser</td>
+    <td>User to impersonate when running</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>kind</td>
+    <td>Session kind (spark, pyspark, or sparkr)</td>
+    <td><a href="#session-kind">session kind</a></td>
+  </tr>
+  <tr>
+    <td>log</td>
+    <td>The log lines</td>
+    <td>list of strings</td>
+  </tr>
+  <tr>
+    <td>state</td>
+    <td>The session state</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>appInfo</td>
+    <td>The detailed application info</td>
+    <td>Map of key=val</td>
+  </tr>
+</table>
+
+
+#### Session State
+
+<table class="table">
+  <tr><th>Value</th><th>Description</th></tr>
+  <tr>
+    <td>not_started</td>
+    <td>Session has not been started</td>
+  </tr>
+  <tr>
+    <td>starting</td>
+    <td>Session is starting</td>
+  </tr>
+  <tr>
+    <td>idle</td>
+    <td>Session is waiting for input</td>
+  </tr>
+  <tr>
+    <td>busy</td>
+    <td>Session is executing a statement</td>
+  </tr>
+  <tr>
+    <td>shutting_down</td>
+    <td>Session is shutting down</td>
+  </tr>
+  <tr>
+    <td>error</td>
+    <td>Session errored out</td>
+  </tr>
+  <tr>
+    <td>dead</td>
+    <td>Session has exited</td>
+  </tr>
+  <tr>
+    <td>success</td>
+    <td>Session is successfully stopped</td>
+  </tr>
+</table>
+
+#### Session Kind
+
+<table class="table">
+  <tr><th>Value</th><th>Description</th></tr>
+  <tr>
+    <td>spark</td>
+    <td>Interactive Scala Spark session</td>
+  </tr>
+  <tr>
+    <td><a href="#pyspark">pyspark</a></td>
+    <td>Interactive Python 2 Spark session</td>
+  </tr>
+  <tr>
+    <td><a href="#pyspark3">pyspark3</a></td>
+    <td>Interactive Python 3 Spark session</td>
+  </tr>
+  <tr>
+    <td>sparkr</td>
+    <td>Interactive R Spark session</td>
+  </tr>
+</table>
+
+#### pyspark
+To change the Python executable the session uses, Livy reads the path from environment variable
+``PYSPARK_PYTHON`` (Same as pyspark).
+
+Like pyspark, if Livy is running in ``local`` mode, just set the environment variable.
+If the session is running in ``yarn-cluster`` mode, please set
+``spark.yarn.appMasterEnv.PYSPARK_PYTHON`` in SparkConf so the environment variable is passed to
+the driver.
+
+#### pyspark3
+To change the Python executable the session uses, Livy reads the path from environment variable
+``PYSPARK3_PYTHON``.
+
+Like pyspark, if Livy is running in ``local`` mode, just set the environment variable.
+If the session is running in ``yarn-cluster`` mode, please set
+``spark.yarn.appMasterEnv.PYSPARK3_PYTHON`` in SparkConf so the environment variable is passed to
+the driver.
+
+### Statement
+
+A statement represents the result of an execution statement.
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>id</td>
+    <td>The statement id</td>
+    <td>integer</td>
+  </tr>
+  <tr>
+    <td>code</td>
+    <td>The execution code</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>state</td>
+    <td>The execution state</td>
+    <td>statement state</td>
+  </tr>
+  <tr>
+    <td>output</td>
+    <td>The execution output</td>
+    <td>statement output</td>
+  </tr>
+</table>
+
+#### Statement State
+
+<table class="table">
+  <tr><th>Value</th><th>Description</th></tr>
+  <tr>
+    <td>waiting</td>
+    <td>Statement is enqueued but execution hasn't started</td>
+  </tr>
+  <tr>
+    <td>running</td>
+    <td>Statement is currently running</td>
+  </tr>
+  <tr>
+    <td>available</td>
+    <td>Statement has a response ready</td>
+  </tr>
+  <tr>
+    <td>error</td>
+    <td>Statement failed</td>
+  </tr>
+  <tr>
+    <td>cancelling</td>
+    <td>Statement is being cancelling</td>
+  </tr>
+  <tr>
+    <td>cancelled</td>
+    <td>Statement is cancelled</td>
+  </tr>
+</table>
+
+#### Statement Output
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>status</td>
+    <td>Execution status</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>execution_count</td>
+    <td>A monotonically increasing number</td>
+    <td>integer</td>
+  </tr>
+  <tr>
+    <td>data</td>
+    <td>Statement output</td>
+    <td>An object mapping a mime type to the result. If the mime type is
+    ``application/json``, the value is a JSON value.</td>
+  </tr>
+</table>
+
+### Batch
+
+<table class="table">
+  <tr><th>Name</th><th>Description</th><th>Type</th></tr>
+  <tr>
+    <td>id</td>
+    <td>The session id</td>
+    <td>int</td>
+  </tr>
+  <tr>
+    <td>appId</td>
+    <td>The application id of this session</td>
+    <td>string</td>
+  </tr>
+  <tr>
+    <td>appInfo</td>
+    <td>The detailed application info</td>
+    <td>Map of key=val</td>
+  </tr>
+  <tr>
+    <td>log</td>
+    <td>The log lines</td>
+    <td>list of strings</td>
+  </tr>
+  <tr>
+    <td>state</td>
+    <td>The batch state</td>
+    <td>string</td>
+  </tr>
+</table>