Forem: Patrick Titzler

Introducing Elyra pipelines with custom component support

Patrick Titzler — Tue, 10 Aug 2021 13:59:36 +0000

The Elyra open source project for JupyterLab aims to simplify common data science tasks. Its most popular feature is the Visual Pipeline Editor, which is used to create pipelines without the need for coding. You can run these pipelines in JupyterLab or on Kubeflow Pipelines or Apache Airflow.

Elyra 3.0 extends the pipeline capabilities by adding experimental support for custom components. Before I dive into specifics and outline why support is still experimental in the initial releases, let's recap a few concepts.

A pipeline comprises nodes that are connected with each other to form a graph. The graph defines dependencies between the nodes, governing the order in which the nodes are executed. The example pipeline shown below executes a Python script and several Jupyter notebooks.

Nodes are implemented using components. To create the pipeline shown above, you'll need components that can execute Python scripts and Jupyter notebooks. Most components are configurable to make them re-usable. For file-based components, such a configuration might include the file name and the container image where the file is executed in.

In Elyra, the processing of Jupyter notebooks, Python scripts, and R scripts is implemented using a single component. This component is referred to as a generic component because it is supported in all runtime environments.

The pipeline editor then exposes this component under different names in the palette, which is located on the left-hand side. (You can add nodes to the pipeline by selecting a component from the palette and dropping it on the canvas.)

Pipelines that only include generic components are referred to as generic pipelines because you can run them in any runtime environment Elyra supports.

Take a look at the tutorials if you are new to Elyra and would like to learn more about how to use the Visual Pipeline Editor to create a pipeline. If you've used Elyra before, we recommend reviewing the recently published Best practices topic in the User Guide. We've only now gotten around to documenting some of the things that make your life easier!

Experimental support for custom components

Custom components are similar to generic components in that they only implement a single task, such as load data, train a model, or send an email. However, these components are only supported for Kubeflow Pipelines
and Apache Airflow and are implemented in a runtime specific form.

The screen capture below depicts the pipeline editor for Apache Airflow pipelines. The palette, shown on the left, is by default divided into two categories — one for generic components and one for custom components. Note the Airflow specific components in the second category, such as the BashOperator and the SimpleHttpOperator, which process a bash command and an HTTP request, respectively.

Pipelines that utilize custom components are called runtime specific pipelines because it is not possible to run a pipeline that was created for Kubeflow Pipelines on Apache Airflow and vice versa.

Get started with pipelines

Once you've installed Elyra, it is easy to get started with pipelines. The JupyterLab launcher now includes tiles for each pipeline type: one for generic pipelines, and one for each supported pipeline runtime platform.

Click the desired pipeline editor tile and you are ready to compose a pipeline from the components that are supported for the selected platform.

To get you going quickly, the component registry includes a few example custom components for each runtime platform. The Elyra examples GitHub repository includes information about those components and pipelines that illustrate their usage. These components are included for illustrative purposes only. Unless stated otherwise, the components were not created by the Elyra community and are therefore provided as is.

Opportunities for growth

In the initial 3.0 release, Elyra’s support for custom components is rather limited. Many features are still under development, planned for a future release, or in the backlog without a specific target release. Some of the high priority features for the next releases are:

Data exchange between custom components: Components commonly produce outputs that other components require as input. Currently, custom components are isolated from each other and cannot exchange data. (Data exchange between generic components is already supported.)
Data exchange between generic components and custom components: Same as above.
Manage component registry: Provide a UI and/or CLI that allows for the addition, editing, or deletion of components. Currently, components can only be managed manually.

For an up-to-date feature status please refer to this forum thread.

Use Watson Studio services in pipelines
Pipelines can also take advantage of external services using custom components. If you are looking for a managed solution for Watson Studio services, check out this IBM Watson Studio Pipeline article. It illustrates how to run notebooks, refine data, run AutoAI experiments, and deploy a model.

Your opportunity to help us improve Elyra

Elyra is a fairly new open source project that is currently maintained by a small community of JupyterLab enthusiasts. We welcome contributions of any kind, such as feedback, bug reports, bug fixes, features, or documentation. To learn more about how you can make a difference refer to the Getting help topic in the documentation.

On behalf of the community: Thank You!

Elyra 2.2: R support, updated CLI, and more

Patrick Titzler — Thu, 01 Apr 2021 20:43:11 +0000

It's only been a couple weeks since the Elyra open source community published version 2.1, which introduced experimental support for Apache Airflow.

Version 2.2 delivers a couple of exciting enhancements that our growing user base had on their wishlist. In this blog post I'll summarize the highlights:

New R script editor
Improved pipeline editor with support for R scripts
Deployment on Kubeflow Notebook servers
Extended command line interface with support for pipeline execution

As always, you can find the complete list of features and bug fixes that made it into the release in the changelog.

Edit and run R scripts

The Elyra Script Editor was extended to support the R language. You can therefore now create, edit, and run R scripts in JupyterLab in addition to Python scripts.

By installing the optional Language Server Protocol support for R, you can take advantage of productivity features you are likely familiar with from other IDEs, such as code linting and auto-completion.

We realize that this editor can't compete with RStudio, but its start!

If you've used the Jupyter notebooks in an enterprise deployment, you are probably familiar with the Jupyter Enterprise Gateway (JEG). In fact, even if you are not familiar with it, you might have used it. What it does - in a nutshell - is to give you the ability to run notebooks in remote kernels, allowing for better resource allocation and usage.

Watson Studio is one example of a managed enterprise service that leverages the JEG.

Because it is extending JupyterLab functionality, Elyra can take advantage of the Enterprise Gateway as well. One little known feature of Elyra though is it's ability to also allow for remote execution without the need for JEG through it's support for Kubeflow Pipelines or Apache Airflow.

If you have access to a Kubeflow Pipelines or Apache Airflow deployment, you can run R scripts (just like Python scripts and Jupyter notebooks) in those deployments directly from the editor. This is especially useful for scripts that require resources that are not available (or not sufficiently available) in your local environment.

Run R scripts in pipelines

In the Visual Pipeline Editor you can now assemble pipelines from multiple R scripts, or mix R scripts with Jupyter notebooks and Python scripts, as necessary.

You can run these pipelines locally in JupyterLab or remotely on Kubeflow Pipelines or Apache Airflow.

If you are new to Elyra pipelines, take a look at the tutorials. They guide you through the process of creating and running a pipeline in various environments.

Use Elyra to run Kubeflow-hosted notebooks

Elyra can be deployed locally or in remote environments.

A local deployment typically serves only a single user and is created by installing Elyra from PyPI, conda, source code, or pulling a ready-to-use container image.

Remote deployments, such as in a data center or the cloud, are typically used when support for many users is required.

A common approach is to deploy JupyterHub on Kubernetes and configure it for Elyra, like it is done in Open Data Hub on the Red Hat OpenShift Container platform.

If you already have Kubeflow deployed and don't want to provision a dedicated instance of JupyterHub to serve notebooks, we've got great news for you. We've recently started to publish custom Elyra container images on Docker Hub and quay.io that you can use to run JupyterLab with Elyra on Kubeflow Notebook Servers. All you need to do is specify the Elyra container image name and (version) tag when you configure a new notebook server and you are good to go!

Extended command line interface

As an extension to JupyterLab, Elyra is primarily GUI driven. However, there are certain tasks that can also be completed using the elyra-metadata command line interface:

The Elyra command line interface was extended in version 2.2 to support running of pipelines in local and remote environments. Initially this capability is only exposed through the elyra-pipeline command line interface, but work is on the way to provide a unified interface.

Run pipelines locally

Specify the run command to execute the pipeline locally, passing the pipeline file name as parameter, like so:

$ elyra-pipeline run /path/to/hello-world.pipeline

This feature is still under active development, e.g. to visualize execution progress.

Run pipelines remotely

Specify the submit command to run the pipeline on Kubeflow Pipelines or Apache Airflow, passing the pipeline file name and the runtime configuration name as parameters, like so:

$ elyra-metadata list runtimes
 Schema    Instance     Resource
 ------    --------     --------
 kfp       kfp_test_env  /.../runtimes/kfp_test_env.json 

$ elyra-pipeline submit --runtime-config kfp_test_env /path/to/hello-world.pipeline
  ...

If the pipeline was successfully submitted for execution, the command returns a GUI link that you can use to monitor the progress and a link to the cloud storage where the pipeline run artifacts are stored.

Improved usability

If you've used previous releases of Elyra, you should notice quite a few usability improvements that we've made. There's no denying that the Elyra project has matured a lot, since it was started about a year ago.

Coming up next

We've just started work for our next releases. There's plenty of stuff brewing in our lab. If you'd like to get the inside scoop, check out our discussion forum, chat with us, or join the weekly community meeting.

Automate your machine learning workflow tasks using Elyra and Apache Airflow

Patrick Titzler — Thu, 18 Mar 2021 14:19:44 +0000

In this blog post I’ll introduce a GUI-driven approach to creating Apache Airflow workflows that implement machine learning pipelines using Jupyter notebooks and Python scripts.

Apache Airflow is an open source workflow management platform that allows for programmatic creation, scheduling, and monitoring of workflows. In Apache Airflow a workflow (or pipeline) is called a Directed Acyclic Graph (DAG) that comprises of one or more related tasks. A task represents a unit of work (such as the execution of a Python script) and is implemented by an operator. Airflow comes with built-in operators, such as the PythonOperator, and can be extended using provider packages.

Let's say you want to use an Apache Airflow deployment on Kubernetes to periodically execute a set of notebooks or scripts that load and process data in preparation for machine learning model training.

Without going into details how, you could implement this workflow using the generic PythonVirtualEnvOperator to run the Python script and the special purpose PapermillOperator to run the notebook. If the latter doesn't meet your needs, you'd have to implement its functionality yourself by developing code that performs custom pre-processing, uses the papermill Python package to run the notebook, and performs custom post-processing.

An easier way to create a pipeline from scripts and notebooks is to use Elyra's Visual Pipeline Editor. This editor lets you assemble pipelines by dragging and dropping supported files onto the canvas and defining their dependencies.

Once you've assembled the pipeline and are ready to run it, the editor takes care of generating the Apache Airflow DAG code on the fly, eliminating the need for any coding.

If you are new to the Elyra open source project, take a look at the overview documentation.

The pipeline editor can also generate code that runs the pipeline in JupyterLab or on Kubeflow Pipelines for greater flexibility.

Local execution is primarily intended for use during development or if no Kubeflow Pipelines or Apache Airflow deployment is available.

This article outlines how you would go about creating a pipeline from a set of files and running it on Apache Airflow. We have published a set of tutorials that provide step-by-step instructions for each one of the supported runtime environments.

Prerequisites

JupyterLab 3.0 with the Elyra extensions v2.1 (or later) installed
Apache Airflow v1.10.8 or later but not including v2
- Must be configured to use the Kubernetes Executor with git-sync enabled
- Must be enabled for Elyra

Creating a pipeline

Pipelines are created in Elyra with the Visual Pipeline Editor by

adding Python scripts or notebooks
configuring their execution properties
connecting the files to define dependencies

Each pipeline node represents a task in the DAG and is executed in Apache Airflow with the help of a custom NotebookOp operator. The operator also performs pre- and post processing operations, that, for example, make it possible to share data between multiple tasks using shared cloud storage.

Node properties define the container image that the operator will run in, optional CPU, GPU, or RAM resource requests, file dependencies, environment variables, and output files. Output files are files that need to be preserved after the node was processed. For example, a notebook that trains a model might want to save model files for later consumption, such as deployment.

A pipeline that comprises only of file nodes (nodes that execute a Python script or Jupyter notebook) can be run as-is locally in JupyterLab, or remotely in Apache Airflow or Kubeflow Pipelines.

Future releases of Elyra might provide support for node types that are specific to a runtime platform. Pipelines that include such nodes can take advantage of platform specific features but won't be portable.

A pipeline definition does not include any target environment
information, such as the host name of the Apache Airflow webserver. This information is encapsulated in runtime [environment] configurations.

Creating a runtime configuration

In Elyra runtime configurations store metadata that describes the target environment where pipelines are executed. A runtime configuration for Apache Airflow includes

connectivity information for the Airflow web server,
details about the GitHub repository, where DAGs are stored,
connectivity information for the cloud storage service, which Elyra uses to store pipeline-run specific artifacts

Elyra supports repositories on github.com and GitHub Enterprise. Note that some of the runtime configuration information is embedded in the generated DAGs to provide the NotebookOp operator access to the configured cloud storage. Therefore you should always use a private repository to store DAGs that were produced by Elyra.

Running pipelines on Apache Airflow

Once you created a pipeline and a runtime configuration for your Apache Airflow cluster, you are ready to run the pipeline.

When you submit a pipeline for execution from the Visual Pipeline Editor, Elyra performs the following pre-processing steps:

package the input artifacts (files and dependencies) for each task in a compressed archive
upload the archives to the cloud storage bucket referenced in the runtime configuration
generate a DAG, comprising of one task for each notebook or Python script
upload DAG to the GitHub repository that Apache Airflow is monitoring

The uploaded DAG is pre-configured to run only once.

Within limits you can customize the generated DAG by exporting the pipeline instead of running it. The main difference between running and exporting a pipeline for Apache Airflow is that the latter does not upload the generated DAG file to the GitHub repository.

Monitoring a pipeline run on Apache Airflow

How soon a DAG is executed after it was uploaded to the repository depends on the git-sync refresh time interval setting and the scheduler in your Apache Airflow configuration.

Each notebook or Python script in the pipeline is executed as a task using Elyra's NotebookOp operator, in the order defined by it's dependencies.

Once a task has been processed, its outputs can be downloaded from the associated cloud storage bucket. Outputs include the completed notebooks, an HTML version of each notebook, a log file for each Python script, and files that were declared as output files.

Getting started

If you have already have access to an v1.10 Apache Airflow cluster you can start running pipelines in minutes:

Prepare your cluster for use by Elyra.
Install Elyra from PyPI or pull the ready-to-use container image.
Step through the tutorial to learn how to create a pipeline, configure a pipeline, and run/export it.

If you are interested in running pipelines on Apache Airflow on the Red Hat OpenShift Container Platform, take a look at Open Data Hub. Open Data Hub is an open source project (just like Elyra) that should include everything you need to start running machine learning workloads in a Kubernetes environment.

Using Watson Studio services in pipelines

We've introduced pipelines in Elyra to make it easy to run notebooks or scripts as batch jobs, and thus automate common repetitive tasks.

The recently launched beta of IBM Watson Studio Orchestration Flow takes this a step further. The flow orchestrator integrates with various data and AI services in Watson Studio, enabling users to ingest data, or train, test and deploy machine learning models at scale.

The orchestration flow editor, which is shown in the screen capture below, is based on the Elyra canvas open source project.

Questions and feedback

We'd love to hear from you! You can reach us in the community chatroom, the discussion forum, or by joining our weekly community call.

What's new in Elyra 2.0

Patrick Titzler — Mon, 01 Feb 2021 22:39:19 +0000

Earlier this month JupyterLab 3.0 was released, providing a slew of new features. In this blog post, I recap a few highlights and how the Elyra 2.0 extensions take advantage of them.

New to Elyra? Take a look at the overview documentation. TL;DR: Elyra provides productivity features for common data science and AI tasks.

Support for Language Servers

Many modern integrated development environments (IDEs) simplify development by providing editor features like autocomplete, "go to definition", "find references", or linting.

JupyterLab optionally supports these features by integrating with Language Servers using the jupyterlab-lsp extension. Language Servers are available for many languages, including Python and R. Note that some Language Servers might not support every convenience feature.

Elyra 2.0 includes the Python Language Server, enabling these features in the notebook editor and Python script editor, as shown below.

You can add support for other languages by installing the appropriate Language Server and its prerequisites.

Visual debugger

JupyterLab includes a visual debugger front-end that provides common debugging tasks. For example, you can set breakpoints, step through code, or inspect data. The front-end requires a kernel with support for debugging.

As of January 2021, the only Jupyter Python kernel that supports debugging is xeus-python.

Note that this kernel currently does not support all iPython features, such as cell magics.

Neither JupyterLab 3.0 nor Elyra 2.0 install xeus-python by default. You therefore have to install it separately from PyPI, conda, or build from source code.

Debugging a notebook

To debug a notebook, associate it with a kernel that supports debugging and enable the debugger. The official debugger example Python notebook provides a short introduction and is highly recommended!

Debugging a Python script

Elyra's Python editor is also enabled for debugging. To debug a Python script:

Open the script in the Python editor.
Right click on the canvas and select Create Console for Editor.
Select a kernel that supports debugging.
Enable the debugger in the console.
Set one or more breakpoints in the source code.
Run code to breakpoint and step through the code.

Elyra 2.0 also includes several enhancements that are not specific to JupyterLab 3.0.

Pipeline editor enhancements for Kubeflow Pipelines

Elyra's visual pipeline editor, which I introduced in this blog post, was extended to support custom resource allocation and the Tekton workflow engine.

Editor's note: Time flies! Elyra 2.1 (and later versions) also support running pipelines on Apache Airflow.

Customize resource allocation

When a pipeline node is executed on Kubeflow Pipelines, it is processed in a container. The amount of CPU, GPU, and memory (RAM) resources that are allocated for the container is governed by configuration settings in Kubernetes.

With the latest enhancement in Elyra, you can request additional CPU, GPU, or memory resources to speed up processing of your notebook or Python script during pipeline execution. You configure those resource requirements in the node's properties view:

Select the notebook or Python script node in the pipeline editor.
Right click, and select Properties.
Enter the desired values.

If you leave the properties empty, the Kubernetes default settings are applied.

Support for Tekton workflow engine

Kubeflow Pipelines utilizes a container-native workflow engine to execute pipelines. As of January 2021, Argo and Tekton are the only two supported engines.

When you run (or export) a pipeline in the Elyra pipeline editor, it is compiled into a workflow engine-specific format. In previous releases, Elyra could only produce Argo-compatible outputs, but this restriction is lifted in version 2.0. Since Elyra cannot automatically determine which workflow engine your Kubeflow Pipelines installation leverages, you need to specify the type when you create a runtime configuration for your Kubeflow Pipelines deployment.

If you don't specify an engine type in your configuration, Argo is assumed as default to retain backward compatibility with older Elyra releases.

If you are unsure what engine your Kubeflow Pipelines deployment utilizes, check with the administrator. If you select the wrong engine type, your pipelines will fail to execute.

Code snippets

Elyra 2.0 makes it easier to create code snippets. The code snippet editor now allows for the creation of code snippets from highlighted code.

To create a code snippet within an editor, select the desired piece of code, right click, and choose from the context menu Save as Code Snippet.

To learn more about other enhancements or bug fixes that went into Elyra 2.0 please check the changelog.

Get started

If you are new to Elyra, it's easy to get started.

Install from PyPI or conda

You can install the Elyra JupyterLab extensions from PyPI, conda, or from source code. Refer to the installation guide for details.

JupyterLab 3.0 introduces a new extension type named prebuilt extensions. These extensions don't need a JupyterLab rebuild or a Node.js installation. Unfortunately, we are unable to distribute Elyra 2.0 as prebuilt extensions, but are considering doing it for future releases.

Run JupyterLab/Elyra in a container image

The Elyra community publishes Elyra container images on DockerHub and Quay.io. Take a look at these instructions to learn more about the images and how to run them.

Get involved

During the past year we have received many suggestions and feedback from users like you. We appreciate your input and encourage you to stay involved. Open GitHub issues, reach out on gitter, post in our new forum, or join our weekly community meeting.

Use the forum to ask general questions or share things you've created, like Nick Burdakous did. He recently published a pipeline viewer for Google Chrome and Mozilla Firefox. This viewer visualizes pipeline files on GitHub, like shown in the screen shot below. The depicted pipeline was created by Nick Pentreath and presented at the Big Things conference in November 2020.

The Elyra community has lot planned for the first half of this year. Follow us here on dev.to/ibmdeveloper or on Twitter to stay in the loop.

Reusable code snippets in JupyterLab

Patrick Titzler — Fri, 13 Nov 2020 20:06:49 +0000

In my previous blog post Automate your machine learning workflow tasks using Elyra and Kubeflow Pipelines I've introduced Elyra and the Visual Pipeline Editor, which enables you to assemble machine learning pipelines from notebooks and Python scripts.

In this blog post I'll introduce the Elyra code snippet extension, which enables to to reuse arbitrary snippets of code in your notebooks, source code or markdown files in JupyterLab.

Let's say you are working on a notebook and want to include a standard header or footer (like licensing text) or some code that you've previously created. For me this typically results in a hectic search. In which notebook did I use the code? Where's the most current version of the disclaimer I need to add? While maintaining a "scratch pad" might help, native support for reusable code or documentation assets in JupyterLab holds a lot more appeal.

Installing the code snippet extension

You can install the code snippet extension as part of Elyra or individually using pip or conda. At the time of writing a rebuild of JupyterLab is required to make the extension available.

Installing the extension using pip

Install all Elyra extensions:

$ pip install elyra && jupyter lab build

Install only the code snippet extension:

$ pip install elyra-code-snippet-extension && jupyter lab build

Installing the extension using conda

Install all Elyra extensions:

$ conda install -c conda-forge elyra && jupyter lab build

Install only the code snippet extension:

$ conda install -c conda-forge elyra-code-snippet-extension && jupyter lab build

Managing and using code snippets

Code snippets are accessed by opening the code snippets tab on the JupyterLab sidebar:

The code snippets UI is divided into two parts:

a quick search panel, and
the code snippet list, providing access to code snippet specific actions, such as copy, insert, edit, and delete.

Creating a snippet

To create a snippet click + above the search bar and provide the required information. You can optionally tag snippets to make them more easily discoverable.

If your code snippet language is not listed in the drop-down list, type it in the filter box to create a new entry.

In Elyra 2.0 we've added the ability to create a code snippet from text. Select the desired text, open the context menu, and choose Save As Code Snippet.

Finding a snippet

To locate a code snippet, enter a search term and/or pick one of the listed tags.

Copying a snippet to the clipboard

You copy the snippet content to the clipboard by clicking the snippet's copy icon.

Inserting a snippet

Use drag and drop or click the snippet's insert icon to paste the snippet content in the desired destination, such as cell in a notebook or an open editor window.

Do note that the editor does not keep track of where you've inserted a snippet. Also note that snippets don't act like macros and changes to a snippet are therefore not reflected in the locations where the snippet was embedded.

Editing a snippet

You edit a snippet by clicking the snippet's pencil icon.

Deleting a code snippet

You delete a code snippet by clicking the snippet's trash icon.

Sharing code snippets

The code snippet extension currently doesn't provide any import or export capabilities. However, you can access the relevant metadata files that contain the definitions.

The definitions are stored in the $JUPYTER_DATA_DIR/metadata/code-snippets directory, where $JUPYTER_DATA_DIR refers to the Jupyter data directory, which you can identify by running the following command in a terminal window:

$ jupyter --data-dir

In my environment, the code snippets are stored in /Users/ptitzler/Library/Jupyter/metadata/code-snippets/. To back up or share my snippets, I typically create an archive from the content of this directory.

Closing thoughts

To learn more about Elyra or if you have code snippet editor feedback head on over to https://github.com/elyra-ai/elyra and get a conversation started.

Automate your machine learning workflow tasks using Elyra and Kubeflow Pipelines

Patrick Titzler — Tue, 20 Oct 2020 13:14:11 +0000

As a data scientist, Jupyter Notebooks are probably one of the most frequently used "tools" to get the job done. Whether you are loading or processing data, analyzing data, using data to train a model, or perform other tasks of the data science workflow, notebooks are usually key.

Let's say you've created a set of notebooks that load, cleanse, and analyze time-series data, which is made available periodically. Instead of running each notebook manually (or performing all tasks in a single notebook, which limits reusability of task specific code) you could create and run a re-usable machine learning pipeline:

With Elyra you can do this in JupyterLab or leverage Kubeflow Pipelines, a popular platform for building and deploying portable, scalable machine learning workflows based on Docker containers.

In Elyra 2.1 (and later releases) you can run pipelines also on Apache Airflow, as outlined in this blog post.

Before I'll outline how you can create and run pipelines, a bit of background information.

JupyterLab can be extended using extensions, making it possible for anybody to customize the user experience by providing new functionality (like a CSV file editor or a visualization), integrating services (like git for sharing and version control), or themes.

Elyra is a set of AI-centric extension for JupyterLab that aim to simplify and streamline day-to-day activities. It's main feature is the Visual Pipeline Editor, which enables you to create workflows from Python notebooks or scripts and run them locally in JupyterLab, on Kubeflow Pipelines, or Apache Airflow.

Assembling a pipeline

Pipelines are assembled in Elyra using the Visual Pipeline Editor. The pipeline assembly process generally involves

creating a new pipeline,
adding Python notebooks or Python scripts and defining their runtime properties, and
connecting the notebooks and scripts to define execution dependencies.

Creating a pipeline

Pipelines are created by opening the Elyra Pipeline Editor from the Launcher.

Adding Python notebooks and scripts to the pipeline

Python notebooks and scripts are added to the pipeline by dragging them from the JupyterLab File Browser onto the canvas.

Each notebook or file is represented by a node that includes an input port and an output port.

Runtime properties, which can be accessed via the context menu, define the execution environment (Docker image) in which the notebook or script is run during remote execution, inputs (file dependencies and environment variables), and output files.

Nodes can optionally be associated with comments to describe their purpose.

Defining dependencies between notebooks and scripts

Dependencies between notebooks or scripts are defined by connecting output ports (on the right side of a node) to input ports (on the left side of a node).

Dependencies are used to determine the order in which the nodes will be executed during a pipeline run.

The following rules are applied:

circular dependencies are not allowed
if two nodes are not connected (directly or indirectly), they can be executed in parallel
if two nodes are connected, the node producing the inputs for the other node is executed first

There are some distinct differences between how pipelines are executed in JupyterLab and on a third-party workflow orchestration framework, such as Kubeflow Pipelines.

Running pipelines in JupyterLab

Pipelines can be executed in JupyterLab as long as the environment provides access to the pipeline's prerequisites. For example, the kernels that the notebooks are associated with must be already installed, just like required packages (if they are not installed in the notebooks).

Running pipelines in the JupyterLab environment should be a viable approach if

you are assembling a new pipeline and are testing it using relatively small data volumes
pipeline tasks don't require hardware resources in excess of what's available
pipeline tasks complete in an acceptable amount of time, given existing resource constraints.

Nodes are executed as a sub-process in the JupyterLab environment and always processed sequentially.

Output files (like processed data files or training artifacts) are stored in the local file system and can be accessed using the JupyterLab File Browser.

Processed notebooks are updated in place, meaning their output cells reflect the execution results.

Python script output, such as messages sent to STDOUT or STDERR are displayed in the JupyterLab console.

Elyra currently does not provide pipeline monitoring capability in the JupyterLab UI aside from a message after processing has completed. However, the relevant information is contained in the JupyterLab console output.

To learn more about how to create a pipeline and run it in JupyterLab take a look at the Running notebook pipelines in JupyterLab tutorial.

Running pipelines on Kubeflow Pipelines

While running pipelines locally might be feasible in some scenarios, it's rather impractical if large data volumes need to be processed or if compute tasks require special purpose hardware like GPUs or TPUs to perform resource intensive calculations.

Elyra can be configured to access Kubeflow Pipeline instances (secured and unsecured) by defining a runtime configuration. Once configured the selected configuration is used to run the pipeline.

The main difference between local pipeline execution and execution on Kubeflow Pipelines is that each node is processed in an isolated Docker container, allowing for better portability, scalability, and manageability.

The following chart illustrates this for two dependent notebook nodes.

Data is exchanged between nodes using S3-compatible cloud storage. Before a notebook or Python script is executed the declared input file dependencies are automatically downloaded from cloud storage into the container. After processing has completed the declared output files are automatically uploaded from the container to cloud storage.

Since Elyra is not yet a mature project it currently relies on the Kubeflow Pipelines UI for pipeline execution monitoring.

Additional details, along with step-by-step instructions can be found in the Running notebook pipelines on Kubeflow Pipelines tutorial.

Ways to try Elyra (and pipelines)

The referenced tutorials are a great way to get started with pipelines. If you are looking for a more complex example this COVID-19 pipeline might fit the bill.

If you'd like to try out Elyra and start building your own pipelines, you have three options, outlined in the sections below:

Use a sandbox environment hosted on the cloud
Use Elyra Docker images
Install JupyterLab and Elyra on your local machine

Kubeflow Pipelines is not included in any of the Elyra installation options.

Running Elyra in a sandbox environment on the cloud

You can test drive Elyra on mybinder.org, without having to install anything. Follow this link to try out the latest stable release or the latest development version (if you feel adventurous) in a sandbox environment.

The sandbox environment contains a getting_started markdown document, which provides a short tour of the Elyra features:

A couple of things to note:

Performance can sometimes be sluggish since this is a shared environment.
The sandbox environment is not persisted and any changes you make will be lost when it is shut down.

If you have Docker installed on your machine consider using one of the Docker images instead.

Running Elyra container images

The Elyra community publishes ready-to-use container images on Docker Hub, which have JupyterLab and the Elyra extension pre-installed. Docker images are approximately one GB in size are tagged as follows:

elyra/elyra:latest is the latest stable release
elyra/elyra:x.y.z has the x.y.z release installed
elyra/elyra:dev is automatically built each time a change is committed.

Once you've decided which image to use (elyra/elyra:latest is always an excellent choice because you won't miss out on the latest features!), you can spin up a sandbox container as follows:

docker run -it -p 8888:8888\
 -v ${HOME}/jupyter-notebooks/:/home/jovyan/work\
 -w /home/jovyan/work\
 elyra/elyra:latest jupyter lab

Open your web browser to the displayed URL and you are ready to start.

To access the notebook, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/nbserver-6-open.html
    Or copy and paste one of these URLs:
        http://4d17829ecd4c:8888/?token=d690bde267ec75d6f88c64a39825f8b05b919dd084451f82
     or http://127.0.0.1:8888/?token=d690bde267ec75d6f88c64a39825f8b05b919dd084451f82

The caveat is, that in sandbox mode you cannot access existing files (such as notebooks) on your local machine and all changes you make are discarded when you shut down the container.

Therefore it's better to launch the Docker container like so, replacing ${HOME}/jupyter-notebooks/ and ${HOME}/jupyter-data-dir with the names of existing local directories:

docker run -it -p 8888:8888\
 -v ${HOME}/jupyter-notebooks/:/home/jovyan/work\
 -w /home/jovyan/work\
 -v ${HOME}/jupyter-data-dir:/home/jovyan/.local/share/jupyter\
 elyra/elyra:latest jupyter lab

This way all changes are preserved when you shut down the container and you won't have to start from scratch when you bring it up again.

Installing Elyra locally

If your local environment meets the prerequisites, you can install JupyterLab and Elyra using pip, conda, or from source code, following the instructions in the installation guide.

Closing thoughts

Elyra is a community-driven effort. We welcome contributions of any kind: bug reports, feature requests, and of course pull requests. Head on over to https://github.com/elyra-ai/elyra and let's get started.

Originally published on IBM Developer.

JupyterLab, Elyra, and quantum at JupyterCon 2020

Patrick Titzler — Fri, 02 Oct 2020 17:33:21 +0000

For this year's JupyterCon, which is held online October 5-17, we are excited to have the opportunity to share our experience with the Jupyter ecosystem and showcase some of the contributions IBM has made during the past year.

Note that the time specified in the tutorial and general session links is UTC/GMT.

Tutorial sessions (October 5-9, 2020)

Tutorial sessions, which are 90-minute, instructor-led, hands-on activities, are held October 5-9, 2020.

Developing extensions for JupyterLab
Clustering algorithms using Python and scikit-learn
Open source fundamentals

Developing extensions for JupyterLab

Introduced in JupyterLab v2 in 2019, extensions can be used to customize and extend any part of JupyterLab. For example, an extension might add a new file editor, add Git versioning support, or apply a theme. The Developing Extensions for JupyterLab tutorial on October 9 covers the basics and provides you with an opportunity to build a simple extension.

Clustering algorithms using Python and scikit-learn

If you want to learn about the theory and ideas behind unsupervised learning, attend the
Clustering algorithms using Python and scikit-learn tutorial on October 6.

Open source fundamentals

The Open source fundamentals sessions on October 9 discuss key ideas of open source development, such as licenses, governance, funding, and community.

General sessions (October 12-16, 2020)

General sessions, which are 15- to 30-minute talks, are held October 12-16, 2020.

Some of our JupyterCon sessions cover various aspects related to JupyterLab extensions, such as how to use, create, or debug them. Other sessions are dedicated to Elyra, a set of AI-centric extensions to JupyterLab, and its most prominent feature -- the Visual Pipeline Editor. Using this editor, you can assemble AI pipelines from notebooks or Python scripts and run them in JupyterLab or Kubeflows Pipelines, a popular platform for building and deploying portable, scalable machine learning workflows based on Docker containers.

So whether you are new to JupyterLab or an experienced user, there's likely something interesting for you in our sessions:

Debugging notebooks and Python scripts in JupyterLab
A generic metadata-store for JupyterLab extensions
Introduction to Elyra: an AI-centric extension for JupyterLab
What is new on Elyra: a set of AI-centric JupyterLab extensions
Explore and extend AI pipeline runtimes with Elyra and JupyterLab
Building AI pipelines with Elyra: a deep dive using COVID-19 analytics scenario
IBM Quantum Experience notebooks. Serving JupyterHub at scale for the quantum computing community

Debugging notebooks and Python scripts in JupyterLab

If you are a Jupyter Notebook user reaching out to external IDEs to debug your code and haven't had the opportunity to familiarize yourself with the JupyterLab Debugger extension, you should consider adding Karla's talk in the Jupyter Community: Practices track on October 13 to your agenda. In her session, she will discuss how to set it up and demonstrate the main debugger features.

A generic metadata-store for JupyterLab extensions

When we created Elyra, we needed a generic way to store metadata and a user-friendly approach to managing it in the JupyterLab user interface. In their talk in the Jupyter Community: Tools track on October 14, Martha explains how we implemented a generic metadata service extension front end and back end, and demonstrates how they can be utilized by other JupyterLab extensions, such as a file editor.

Introduction to Elyra: an AI-centric extension for JupyterLab

Do you have a set of notebooks (or Python scripts) that download data, process and analyze data, train a model, or perform any other machine learning workflow tasks? With the introduction of the Visual Pipeline Editor in Elyra, you can now run them as part of a pipeline to automate repetitive steps. Join Yiwen, Edward, and Saishruthi in their session on October 12 in the Jupyter Community: Tools track to learn how to use the Elyra Visual Pipeline Editor to create workflow pipelines and how to run them in JupyterLab or a Kubeflow Pipelines deployment. The demo scenario is based on a set of notebooks that process a popular time series data set from the Data Asset eXchange.

What is new on Elyra: a set of AI-centric JupyterLab extensions

Luciano and Karla provide an overview of Elyra and demonstrate the main features in their talk on October 12 in the Jupyter Community: Tools track. Covering Git integration, the code snippet editor, Python editor, Visual Pipelines Editor, and much more, this session is ideal if you are looking for ways to simplify common tasks in JupyterLab.

Explore and extend AI pipeline runtimes with Elyra and JupyterLab

Elyra's Visual Pipeline Editor makes it easy to assemble machine learning workflows from Jupyter Notebooks or Python scripts. Alan's deep dive on October 12 in the Jupyter Community: Tools track provides interesting insights into how Elyra processes the pipeline with the help of a platform like Kubeflow Pipelines.

Building AI pipelines with Elyra: a deep dive using COVID-19 analytics scenario

In their talk on October 15 in the Jupyter Community: Practices track, Luciano, Fred, and Karla demonstrate how data scientists on our team used Elyra's Visual Pipeline Editor to create and run a Jupyter Notebook-based machine learning pipeline that downloads, transforms, cleanses, and analyzes COVID-19 time series data. You can find the notebooks and pipeline referenced in this talk in the covid-notebooks GitHub repository.

IBM Quantum Experience notebooks. Serving JupyterHub at scale for the quantum computing community

Focusing on enterprise adoption of the Jupyter ecosystem, Juan covers in his session in the Enterprise Jupyter Infrastructure track on October 14 how the IBM Quantum Experience uses a JupyterHub deployment on Kubernetes to serve notebooks to the quantum computing community, which has more than 200,000 users. No quantum computing experience is required to enjoy this talk!

Can't make it to JupyterCon this year?

If you can't attend the conference this year but would like to learn more about Elyra and try it out, we've got you covered:

The overview summarizes the main features of Elyra.
The installation guide has details on how to install Elyra using pip, conda, and from source. You can also run Elyra in a Docker container, or, if you don't want to install anything at all, in the cloud at mybinder.org.
If running your notebooks or Python scripts as a pipeline in JupyterLab or on Kubeflow Pipelines is something you would like to try, take a look at the tutorials.

Would you like to get in touch with us before, during, or after the conference?

Elyra is a community-driven open source project that is maintained by a small number of developers and data scientists. If you'd like to file an enhancement request, report an issue, or simply ask a question, head over to https://github.com/elyra-ai/elyra.