aihub-dag/docs/source/recipes/running-elyra-in-air-gapped-environment.md

Running Elyra in an air gapped environment

This document outlines how to prepare an air gapped environment for Elyra.

Installation dependencies

An Elyra installation requires access to Python and Javascript packages, which must be accessible in the environment when the installation is performed.

These dependencies are in addition to any dependencies you wish to include in the installation, such as JupyterLab extensions that are not distributed with Elyra.

Runtime dependencies

When using Elyra's features to build, export, or run pipelines, additional runtime dependencies must be accessible in the environment where JupyterLab is installed and the Kubernetes cluster where the pipeline runtime environment (Kubeflow Pipelines or Apache Airflow) is installed.

In the chart above the arrows indicate whether read access, write access, or both is required.

JupyterLab environment dependencies

Elyra requires access to the following dependencies when you build, export, or submit a pipeline:

• Runtime environment: Elyra requires access to the Kubernetes cluster where Kubeflow Pipelines or Apache Airflow is running.
• GitHub repository or GitLab project: For Apache Airflow Elyra requires access to the GitHub repository or GitLab project that is configured in the runtime configuration.
• Component definitions for custom components: Elyra utilizes catalog connectors to locate and load component definitions. The connectors must be able to communicate with the configured catalog, or you will not be able to submit or export pipelines. For example, if a pipeline utilizes a component that is stored in a URL component catalog, the component's URL must be accessible via an anonymous HTTP request.
  
• S3-compatible cloud storage for generic components: During pipeline export or submission Elyra uploads pipeline artifacts to an S3 bucket. These artifacts are downloaded to the pipeline runtime environment when the pipeline is executed.

Runtime environment dependencies

During pipeline execution in the Kubeflow Pipelines or Apache Airflow environment access to the following dependencies is required:

• Container registry: All pipeline nodes are executed in containers. The runtime environment must be configured to have read access to the registries (e.g. Docker Hub) where the container images are stored that the generic and custom components are referencing.

• Elyra runtime artifacts: When processing pipeline nodes that are implemented using generic components, Elyra downloads a few dependencies to the container. By default these dependencies are located in a release-specific branch in the Elyra GitHub repository:

  https://raw.githubusercontent.com/elyra-ai/elyra/master/etc/kfp/pip.conf
  https://raw.githubusercontent.com/elyra-ai/elyra/master/elyra/kfp/bootstrapper.py
  https://raw.githubusercontent.com/elyra-ai/elyra/master/elyra/airflow/bootstrapper.py
  https://raw.githubusercontent.com/elyra-ai/elyra/master/etc/generic/requirements-elyra-py37.txt
  https://raw.githubusercontent.com/elyra-ai/elyra/master/etc/generic/requirements-elyra.txt
  

  In air-gapped environments you must store a copy of these files in a location that is accessible via an anonymous HTTP GET request and configure the following environment variables in the environment where JupyterLab is running:

  • For Kubeflow Pipelines:
    • ELYRA_PIP_CONFIG_URL (URL of .../etc/kfp/pip.conf)
    • ELYRA_BOOTSTRAP_SCRIPT_URL (URL of .../elyra/kfp/bootstrapper.py)
    • ELYRA_REQUIREMENTS_URL_PY37 (URL of .../etc/generic/requirements-elyra-py37.txt)
    • ELYRA_REQUIREMENTS_URL (URL of .../etc/generic/requirements-elyra.txt)
  • For Apache Airflow:
    • ELYRA_BOOTSTRAP_SCRIPT_URL (URL of .../elyra/airflow/bootstrapper.py)
    • ELYRA_REQUIREMENTS_URL_PY37 (URL of .../etc/generic/requirements-elyra-py37.txt)
    • ELYRA_REQUIREMENTS_URL (URL of .../etc/generic/requirements-elyra.txt)
      

• S3-compatible cloud storage for generic components: When processing pipeline nodes that are implemented using generic components, Elyra downloads the pipeline artifacts that were uploaded when the pipeline was exported or submitted.