Home Explore Help
Register Sign In
liutao
/
elyra-extension
1
0
Fork 0
Files Issues 0 Pull Requests 0 Wiki
Tree: 5c3f5e3b0e
Branches Tags
elyra-extens...
/
docs
/
source
/
recipes
/
creating-a-custom-runtime-image.md

creating-a-custom-runtime-image.md 10 KB
History Raw

Creating a custom runtime container image

A runtime image provides the execution environment in which nodes are executed when a Jupyter notebook is processed as part of a pipeline. Elyra includes a number of runtime images for popular configurations, such as TensorFlow or Pytorch.

Should none of these images meet your needs, you can utilize a custom container image, as long as it meets the following pre-requisites:

  • The image is stored in a container registry in a public or private network that the container platform in which the pipeline is executed can connect to. Examples of such registries are hub.docker.com or a self-managed registry in an intranet environment.
  • The image can be pulled from the registry without the need to authenticate.
  • Python 3 is pre-installed and in the search path. Python versions that have reached their "end of life" are not supported.
  • curl is pre-installed and in the search path.

Elyra installs a set of required packages in this image prior to running Jupyter notebooks or Python scripts. If no built distribution is available for those packages, Python builds the packages from source, which might require additional software (like a compiler) to be preinstalled in the image. Building and installing those package on-the-fly might add a non-trivial overhead, and you should therefore consider pre-installing these Elyra prerequisites in the container image.

Refer to the Additional considerations section for important implementation details.

Requirements

To create a custom container image and publish it on hub.docker.com you need

Creating a basic Python runtime container image

The default Python 3 Docker image has Python and curl pre-installed and it is therefore a good starting point.

  1. Create a file named Dockerfile and add the following content.

    FROM python:3

COPY requirements.txt ./
RUN pip3 install --no-cache-dir -r requirements.txt

When you create a container image using this Dockerfile the default Python 3 Docker image is loaded and the requirements listed in requirements.txt pip-installed.

  1. in the same directory create a requirements.txt file and add the packages your notebooks depend on. For example, if your notebooks require the latest version of Pandas and Numpy, add the appropriate package names.

    pandas
numpy

Note: If your notebooks require packages that are not pre-installed on this image they need to pip-install them explicitly.

  1. Open a terminal to the location where you've created the Dockerfile and requirements.txt.

Using Docker

  1. Build the container image by running the docker build command in the terminal window, replacing my-runtime-image with the desired Docker image name.
   $ docker build -t my-runtime-image .

Verifying the basic runtime container image

The Elyra Makefile includes a task that can be used to verify that runtime images for generic components meet the basic prerequisites.

  1. Clone the Elyra source code.
   $ git clone https://github.com/elyra-ai/elyra.git
   $ cd elyra
  1. Run the validate-runtime-image task, specifying the custom container image name as parameter.
   $ make validate-runtime-image image=my-runtime-image

If no problems were detected the output should look as follows:

   ***********************************************************
   Validating container image my-runtime-image
   -----------------------------------------------------------
   => Loading container image ...
   => Checking container image my-runtime-image for curl...
   => Checking container image my-runtime-image for python3...
   => Checking notebook execution...
   ...
   => Container image my-runtime-image is a suitable Elyra runtime image

Publishing the basic runtime container image

When a notebook is processed as part of a pipeline the associated container image is downloaded from the container registry stated in the URL of the image.

For example, the following steps publish the container image you've just created on Docker Hub using docker.

  1. Log in to Docker Hub using docker login and provide your Docker id and password.
   $ docker login

  1. Run docker images and locate the image id for your Docker image. The image id uniquely identifies your Docker image.

    $ docker images

REPOSITORY         TAG      IMAGE ID            CREATED             SIZE 
my-runtime-image   latest   0d1bd98fdd84        2 hours ago         887MB
    1. Tag the container image using docker tag, replacing my-image-id, docker-id-or-org-id, and my-runtime-image as necessary. (docker-id-or-org-id is either your Docker id or, if you are a member of a team in an organization, the id of that organization.)
    $ docker tag my-image-id docker-id-or-org-id/my-runtime-image:latest

Note: For illustrative purposes this image is tagged latest, which makes it the default image. If desired, replace the tag with a specific version number or identifier, such as Vx.y.z.

  1. Publish the container image on Docker Hub by running docker push, replacing docker-id-or-org-id and my-runtime-image as necessary.

    $ docker push docker-id-or-org-id/my-runtime-image:latest

    Once the image is published on Docker Hub you can create a runtime image configuration using the Elyra UI or elyra-metadata CLI and reference the published docker-id-or-org-id/my-runtime-image:latest Docker image.

    Additional Considerations

    Prior to notebook processing Elyra modifies the associated container by changing the default execution command and installing additional packages. Please review the following section if

    Dockerfiles with CMD instructions

    If a Dockerfile includes a CMD instruction, which is used to specify defaults for an executing container, you might have to customize your notebooks. When a notebook is processed as part of a pipeline the CMD instruction is overriden, which might have side effects. The following examples illustrate two scenarios.

    Scenario 1 - override has no side-effects

    The CMD instruction launches an application that does not need to be running when the notebook is executed. For example, the official Python container images might launch the interactive Python shell by default, like so:

    ```

... CMD ["python3"]


Notebooks will work as is because Python is explicitly run during notebook processing.

#### Scenario 2 - override has side-effects

The `CMD` instruction launches an application or service that a notebook consumes. For example, a container image might by default launch an application that provides computational (or connectivity) services that notebooks rely on.

```bash
...
CMD ["python3", "/path/to/application-or-service"]

When the container started to process a notebook, the referenced application is unavailable because it wasn't automatically started. If feasible, the notebook could launch the application in the background in a code cell like so:

import os
import time
# launch application in the background
os.system("python /path/to/application-or-service &")
# wait to allow for application initialization
time.sleep(2)

Dockerfiles with ENTRYPOINT instructions

If a container is configured to run as an executable by using the ENTRYPOINT instruction in the Dockerfile, you likely have to customize your notebook.

Scenario 1 - override has side-effects

The ENTRYPOINT instruction launches an application or service that a notebook consumes.

ENTRYPOINT ["python3", "/path/to/application-or-service"]

When the container is launched to process a notebook the application or service is unavailable because it wasn't automatically started. If feasible, the notebook could launch the application or service in the background in a code cell like so:

import os
import time
# launch application in the background
os.system("python /path/to/application-or-service &")
# wait to allow for application initialization
time.sleep(2)

Conflicting package dependencies

Elyra installs additional packages in the container prior to notebook processing. If a pre-installed package is not compatible with the version requirements defined in requirements-elyra.txt, it is replaced. You should review any version discrepancies as they might lead to unexpected processing results.