Forem: //va

Developing Machine Learning IoT Apps with Node-RED and TensorFlow.js

//va — Tue, 28 Jan 2020 16:09:29 +0000

cross-post from medium.com

A low-code approach to incorporating machine learning into the Internet of Things

A comic where a guy, frowning, says to another, "My coffee machine has unfollowed me." The Internet of Things — Geek & Poke

Gartner has predicted the number of connected devices will rise to over 25 billion by 2021. And given the variety of devices out there, getting started with IoT can appear daunting. Setting up communication with or between these devices is often non-trivial. Further challenges arise if you want to integrate machine learning! Solutions need to pull together different device APIs, services, and sometimes protocols.

Node-RED with TensorFlow.js brings machine learning to IoT in an easy, low-code way. It opens new, creative approaches to enable machine learning for the Internet of Things. Image recognition, audio classification, etc. all possible on device with minimal code.

Enter Node-RED

Node-RED is a flow-based visual programming tool. With its browser-based editor you can simply wire together hardware devices, APIs and online services to create your application. You develop powerful applications by connecting nodes instead of writing code. And you can deploy them with a single click.

Node-RED Essentials playlist

Node-RED runs on local workstations, the cloud, and edge devices. It has become an ideal tool for the Raspberry Pi and other low cost hardware.

The Node-RED runtime is lightweight and built on top of Node.js. It takes full advantage of Node.js’ event-driven, non-blocking I/O model. There is also the added benefit of tapping into the most used programming language — JavaScript!

With all the Node-RED community resources and vast NPM ecosystem, you can create IoT flows in a user-friendly manner that are imaginative and help save lives.

Hello TensorFlow.js

TensorFlow.js is an open source JavaScript library. It permits you to build, train, and run machine learning models in the browser and Node.js.

TensorFlow.js intro

Often, enabling AI capabilities involves sending the data from a device to a server. The calculations happen on the server and the results returned to the device to take action. This is not ideal when data security or network reliability is a problem.

But, with TensorFlow.js there is an increase in privacy and data security. The data does not leave the device! Training and predictions can happen directly on the device collecting the data. This also makes it possible to run offline and in remote locations with no connectivity.

Node-RED meet TensorFlow.js

The combination of Node-RED and TensorFlow.js means you can build IoT apps that use machine learning simply by dragging and dropping. Drag and drop a machine learning node, wire it up, and deploy to your device.

Node-RED node with a TensorFlow.js Object Detection model

But to get to that point you need to have the TensorFlow.js nodes available. TensorFlow.js nodes are starting to show up in the Node-RED library and across GitHub repos and more are being released regularly. These nodes provide various machine learning functionality to add to your flow. But what if there isn’t a TensorFlow.js node for your machine learning task? You can create it!

The extensibility of Node-RED allows you to create custom nodes for your needs. Packing Node-RED nodes is similar to packaging Node.js modules, but with some extra information.

A Node-RED node consists of three main files:

package.json: standard file used by Node.js modules, but with an added node-red section
JavaScript file that defines the node’s behavior
HTML file that defines the node’s properties, edit dialog and help text

The JavaScript file is where you would wrap your TensorFlow.js code. It would load the TensorFlow.js model and run the prediction.

Once bundled the custom node is available to wire into a flow and deploy.

There may be challenges

As straightforward as it may appear, there can still be challenges and concerns to keep in mind.

Since, you are dealing with edge devices performance is an utmost priority. Models may be too big to load onto an edge device. Or may require specific optimizing to perform well in JavaScript.

Also, when in the node’s life-cycle should you load the model? Should you have a single node to process input/output and run prediction? Or split the work across two or three nodes?

The TensorFlow.js model you are using and the specific use case it addresses often dictate the approach and answers to a lot of these concerns.

Getting started

Combining TensorFlow.js with Node-RED lowers the barrier to entry into machine learning. From the drag-and-drop interface to the one click deploy, IoT enthusiasts and developers can incorporate machine learning in an accessible and rapid manner.

Visit the Node-RED and TensorFlow.js code pattern to check out a sample solution. Learn more by viewing the full code and deploying the tutorial.

Contribute to the Node-RED Library

Check out the Node-RED library for more TensorFlow.js nodes, flows, and collections as they become available. Anyone is welcome to contribute there so that others can learn from your work.

Veremin — A Browser-based Video Theremin

//va — Thu, 07 Feb 2019 16:16:00 +0000

cross-post from medium.com

Making music visually using TensorFlow.js, PoseNet, and the Web MIDI & Web Audio APIs

Meet Veremin!

Veremin, is a video theremin that allows anyone to make beautiful (:-?) music just by waving their arms! It makes use of TensorFlow.js, PoseNet, as well as the Web MIDI and Web Audio APIs.

Veremin is the brainchild of johncohnvt, from the MIT-IBM Watson AI Lab, who built the first rough prototype. I was then able to whip it into something that really worked!

The application attaches to the video stream from your web camera. PoseNet is used to capture the location of your hands within the video. The location then gets converted to music.

Thanks to the magic of TensorFlow.js, Veremin lives 100% in the browser and works on all modern browsers (Chrome, Safari, Firefox, IE) and platforms (OS X, iOS, Android, Windows).

And our deepest thanks to the Google Creative Lab folks who gave us a great start with their demo apps.

Play Veremin!

Just point your browser to ibm.biz/veremin on your desktop, laptop, tablet, or phone. Allow the application to use the camera when prompted and make sure the volume is up.

Stand in front of your devices camera and adjust your position so your torso fits the screen . Adjust your stance so you are centered on the vertical red line in the center of the screen and your waist is roughly even with the horizontal red line . You should see the stick version of your form in blue. Now, move both your hands above the red horizontal line. Move your right hand up and down to control the pitch and your left hand left and right to control the volume.

Now just get jiggy with it! ┌(・⌣・)┘♪

If you’re interested, you can adjust some of the parameters by clicking the settings icon in the top right of the screen. You can read more about the various control options here.

Using Veremin as a MIDI controller

If you’re feeling even more adventurous, Veremin can also be used as a MIDI controller. To do that, you must use a browser that supports MIDI output (e.g., Chrome).

Plugin in your MIDI device to your computer and launch Veremin in your browser. Then click the settings symbol in the upper right of the screen and change the Output Device to point to your MIDI output device. You should now be able to control your MIDI device which can be anything from a simple software synthesizer (e.g., SimpleSynth) to a MIDI controlled Tesla Coil (like John uses).

What’s inside Veremin?

Let’s quickly review all the technologies we use.

TensorFlow.js and PoseNet

TensorFlow.js is an open-source library for creating, training, and running machine learning models in JavaScript. It brings machine learning to the browser and is a great way to start with machine learning. Tutorials, guides, and more information for TensorFlow.js are available here.

While you can use TensorFlow.js to build and train models, the real fun comes from finding new and creative ways to interact with existing pre-trained machine learning models, like PoseNet.

The TensorFlow.js version of PoseNet allows for real-time human pose estimation in the browser. An image is passed to the model and it returns a prediction. The prediction contains a list of keypoints (i.e., right eye, left wrist, etc.) and their confidence scores. What you do with this information is left up to your imagination.

Real-time human pose estimation

Web MIDI API

The Web MIDI API allows connections to MIDI input and output devices from browsers. From the connected devices, MIDI messages can be sent or received. The MIDI message (e.g. [128, 72, 64]) is an array of three values corresponding to [command, note, velocity].

MIDI messages are received only from input devices (e.g., keyboard). And can be sent only to outputdevices (e.g., speakers). To request access to MIDI devices (and receive a list of connected inputs and outputs) a call must first be made to the requestMIDIAccess function.

Support for the Web MIDI API is unfortunately not yet wide spread. A quick getting started article for the Web MIDI API can be found here.

Web Audio API

With the Web Audio API, browsers can create sounds or work with recorded sounds. It describes a high-level API for processing and synthesizing audio in web applications.

All audio operations must occur within an AudioContext. Audio modules (i.e., AudioNodes) are created from the AudioContext and chained together to define the audio processing graph.

Working with the Web Audio API can be tricky at times. But to make it easier check out Tone.js, a Web Audio framework for creating interactive music in the browser.

Support for the Web Audio API is available across most browsers. A nice introduction to the Web Audio API can be found here.

Enjoy!

If you’re interested in the nitty gritty, head over the Veremin GitHub repository to check out the full code and learn more. The README includes instructions for deploying your own Veremin or to try it out without installing anything, visit ibm.biz/veremin.

We hope you enjoy Veremin. Please let us know what you think and share some of the beautiful music you make!

This article was written in collaboration with John Cohn (IBM Fellow at the MIT-IBM Watson AI Lab).

Bring Machine Learning to the Browser With TensorFlow.js — Part III

//va — Tue, 08 Jan 2019 16:16:00 +0000

cross-post from medium.com

Edited 2019 Mar 11 to include changes introduced in TensorFlow.js 1.0. Additional information about some of these TensorFlow.js 1.0 updates can be found here.

How to go from a web friendly format to a web application

“assorted-color textile leaves hanging decor” by DESIGNECOLOGIST on Unsplash

Welcome to part three of a series of posts where I walk you through how TensorFlow.js makes it possible to bring machine learning to the browser. First, there’s an overview of how to bring a pre-trained model into a browser application. Then you’ll find greater detail on how to convert your pre-trained model to a web friendly format. Now in this post, we step through using that web friendly model in a web application.

We continue with the Image Segmenter from the Model Asset Exchange (MAX) converted in part two of this series. The goal here is to give you a greater understanding of TensorFlow.js and how to utilize the model we made. We will create a basic web application without much style or additional libraries. To keep this article from getting too long and keep focus on TensorFlow.js, we will skip over the HTML and non-TensorFlow.js specific JavaScript code. But you can review the complete application on GitHub.

image-segmenter application output

Importing the model

The first step in importing the model to a browser readable format is to include the TensorFlow.js library in your HTML via script tag.

This will load the latest version of TensorFlow.js but you can also target a specific version or load it via NPM.

With the library loaded, a global tf variable becomes available for accessing its API. You can load the Image Segmenter model using the loadGraphModel API.

NOTE: With TensorFlow.js version before 1.0 use the loadFrozenModel for API.

Pass the URL to the dataflow graph using the appropriate API.

Using Tensorflow.js version 0.x.x:

Using TensorFlow.js version 1.x.x:

Depending on the model size, loading may take some time. Once loaded, the model is ready to accept inputs and return a prediction.

Pre-processing the input

Models will need the inputs to be of a specific type and/or size. In most cases, the input needs some sort of pre-processing before sending it to the model. For example, some models may require a one-dimensional array of a certain size while others may require more complex multi-dimensional inputs. So the input (e.g., image, sentence, etc.) would need to be pre-processed to the expected format.

For the Image Segmenter, recall, when inspecting the model graph, the input was an ImageTensor. It was of type and shape uint8[1,?,?,3].

This is a four-dimensional array of 8-bit unsigned integer values. The ? s are placeholders and can represent any length. They would correspond to the length and width of the image. The 1 corresponds to the batch size and the 3 corresponds to the length of the RGB value for a given pixel.

For 8-bit unsigned integer valid values are from 0 to 255. This corresponds with an image’s pixel RGB value which also ranges from 0 to 255. So, we should be able to take an image convert it to a multi-dimension array of RGB values and send that to the model.

To get a Tensor with the pixel values, you can use the tf.browser.fromPixels (or tf.fromPixels for TensorFlow.js versions before 1.0) function provided by TensorFlow.js. This will produce three-dimensional array with the shape [?, ?, 3] from the given HTMLImageElement. But, the Image Segmenter is expecting a four-dimensional array. To insert an extra dimension and get the shape needed, you also will need to call the expandDims function.

Using Tensorflow.js version 0.x.x:

Using Tensorflow.js version 1.x.x:

You should now have the required input data to run the model.

Running the model

Run the model by calling predict with the input data. The function takes the input Tensor(s) and some optional configuration parameters. It returns a prediction.

Computations are in batches. If needed you can run prediction on a single batch with the predictOnBatch function.

Depending on the model complexity, the prediction may take some time.

Processing the output

The type and shape of the output returned depends on the model. To do something meaningful extra processing of the prediction is most likely required.

For the Image Segmenter, the output is a segmentation map with integers between 0 and 20. The integers corresponds to one of the pre-defined labels for each pixel in the input image.

In our web application, we are going to overlay the original image with the segments found. And each segment color coded. For example, RGB (192, 0, 0) for chairs and RGB (0, 64, 0) for potted plants.

To achieve this, start with the dataSync Tensor function. The function will download the output tensor into a TypedArray. Next convert the TypedArray into a regular array with Array.from.

With a color map, go through the converted array and assign the appropriate color to each segment. Then take this data to create the desired overlay image.

You can now add the resulting image to your HTML page.

Completing the web application

To complete the application, add buttons to load the model, upload an image, and run the model. Also, add the code to overlay the input image and output prediction.

running image-segmenter application

You can find the completed project here. The repository contains the demo web application. It also includes the web friendly format from the tensorflowjs_converter. You will also find a Jupyter notebook to play with the Image Segmenter in Python.

Machine Learning in JavaScript

The ability to use machine learning technology on the web is often limited. Creating and training some models involve massive data and intense computations. The browser may not be the ideal environment. But, an exciting use case is to take models trained elsewhere then import and run them in the browser.

With TensorFlow.js you can convert some of these models into a web friendly format. Then bring them into your web application. Making machine learning in JavaScript that much easier.

To check out even more interesting applications take a look at Nick Kasten’s magicat or the TensorFlow.js version of his Magic Cropping Tool.

Bring Machine Learning to the Browser With TensorFlow.js — Part II

//va — Tue, 04 Dec 2018 16:31:01 +0000

cross-post from medium.com

Edited 2019 Mar 11 to include changes introduced in TensorFlow.js 1.0. Additional information about some of these TensorFlow.js 1.0 updates can be found here.

Converting a pre-trained model to a web friendly format

assorted-color leaves by Chris Lawton on Unsplash

If you’ve been following along, you should already have a high level understanding of how to bring a pre-trained model into a browser application. Now, let’s look at the first steps in this process in greater detail.

Before you can convert a pre-trained model to a web-friendly format and bring it to the browser, you first need a model. A great first model to start learning with is the Image Segmenter from the Model Asset Exchange (MAX). You can deploy and run the Image Segmenter model through Kubernetes or Docker Hub. To get an idea of what it does you can check out Nick Kasten’s Magic Cropping Tool.

Get the Model

You can begin by downloading and extracting the model files used in the MAX Image Segmenter. The extracted contents contain a frozen model graph. Frozen graphs encapsulates all required model data in a single file (.pb extension).

The TensorFlow.js converter supports Keras (i.e., HDF5) and TensorFlow (e.g., frozen graphs, SavedModel) models.

A few other model formats you may encounter include:

Checkpoints which contain information needed to save the current state of the model. Then resume training after loading the checkpoint. Checkpoints are not supported by the converter.
SavedModel which is the universal serialization format for TensorFlow. Unlike checkpoints, SavedModels store the model data in a language-neutral format.
HDF5 which is the format used by Keras to store model data. It is a grid format popular for storing multi-dimensional arrays of numbers.

Know the Model

It is good practice to review and understand a model before you use it. You don’t need to know every little detail about the model, but a good start is to get to know the model’s format, inputs, and outputs.

In most cases, you will have to pre-process the input(s) to the model, as well as, process the model output(s).

Learn about the model’s inputs, outputs, and operations by inspecting the model’s graph. One useful and easy-to-use visual tool for viewing machine learning models is Netron.

To inspect the Image Segmenter model, open the extracted frozen_inference_graph.pb file in Netron. You can zoom out to see the scope and size of the model. Likewise, you can zoom in to specific nodes/operations on the graph.

Without clicking on any nodes, click on the hamburger/menu icon to see the model’s properties (e.g., number of operators, input type, etc.). In addition, click on a specific node to view the properties. Alternatively, you can enter CTRL+F to open the search panel and type a specific node to jump to it.

Image Segmenter model properties

The input for the Image Segmenter is an ImageTensor of type uint8[1,?,?,3]. This is a four-dimensional array of 8-bit unsigned integer values in the shape of 1,?,?,3. The ? s are placeholders and can represent any length. They would correspond to the length and width of the image. The 1 corresponds to the batch size and the 3 corresponds to the length of the RGB value for a given pixel, which is three numbers.

Clicking on the last node (Slice), you get its name (i.e., SemanticPredictions) and attributes. The name is important to remember. You will need to provide it to the converter tool.

Image Segmenter node properties

Other available options to view information about a graph are the summarize_graph and TensorBoard tools.

Change the Model

You are now ready to run tensorflowjs_converter to get your web friendly format.

The converter is available through the command line after installing the tensorflowjs Python package

To convert the Image Segmenter specify:

SemanticPredictions for the output_node_names
tf_frozen_model for the input_format
file path to the frozen graph
directory path to store the converted model

If successful the tensorflowjs_converter outputs the dataflow graph (model.json) and shards of binary weight files. The shard files are small in size to support easier browser caching.

NOTE: If using tensorflowjs_converter version before 1.0, the output produced includes the graph (tensorflowjs_model.pb), weights manifest (weights_manifest.json), and the binary shards files.

Conversion can fail because of unsupported operations. The converted model may also be too large to be useful. In this case, there are steps you may be able to take.

To make the web friendly model smaller you can convert only part of the model graph. Any unused nodes or nodes used only during training can get stripped. It is not needed with the Image Segmenter model, but if you had to strip unused nodes, it would look something like the following:

This is also useful for failures from unsupported operations. For some unsupported operations use strip_unused to bypass the operation. You can then convert the stripped graph to get a web friendly version.

This helps get the model converted, but also adds extra work. You may need to implement the unsupported operation outside of the model. This will apply it to the input(s) to the model and/or output(s) from the model.

More options to further optimize the model are available.

More to Come…

Your pre-trained model should now be converted to the format supported by TensorFlow.js. You can load the converted format and run it in a browser environment.

Stay tuned for the follow up to this article to learn how to take the converted model and use it in a web application.

Bring Machine Learning to the Browser with TensorFlow.js — Part I

//va — Fri, 26 Oct 2018 13:31:01 +0000

cross-post from medium.com

Edited 2019 Mar 11 to include changes introduced in TensorFlow.js 1.0. Additional information about some of these TensorFlow.js 1.0 updates can be found here.

Applying a web friendly format to a pre-trained model resulting in a web application.

“assorted-color leaf hanging decor” by Chris Lawton on Unsplash

TensorFlow.js brings machine learning and its possibilities to JavaScript. It is an open source library built to create, train, and run machine learning models in the browser (and Node.js).

Training and building complex models can take a considerable amount of resources and time. Some models require massive amounts of data to provide acceptable accuracy. And, if computationally intensive, may require hours or days of training to complete. Thus, you may not find the browser to be the ideal environment for building such models.

A more appealing use case is importing and running existing models. You train or get models trained in powerful, specialized environments then you import and run the models in the browser for impressive user experiences.

Converting the model

Before you can use a pre-trained model in TensorFlow.js, the model needs to be in a web friendly format. For this, TensorFlow.js provides the tensorflowjs_converter tool. The tool converts TensorFlow and Keras models to the required web friendly format. The converter is available after you install the tensorflowjs Python package.

The tensorflowjs_converter expects the model and the output directory as inputs. You can also pass optional parameters to further customize the conversion process.

The output of tensorflowjs_converter is a set of files:

model.json — the dataflow graph
A group of binary weight files called shards. Each shard file is small in size for easier browser caching. And the number of shards depends on the initial model.

tensorflowjs_converter output files

NOTE: If using tensorflowjs_converter version before 1.0, output produced includes the graph (tensorflowjs_model.pb), weights manifest (weights_manifest.json), and the binary shards files.

Run model run

Once converted, the model is ready to load into TensorFlow.js for predictions.

Using Tensorflow.js version 0.x.x:

Using TensorFlow.js version 1.x.x:

The imported model is the same as models trained and created with TensorFlow.js.

Convert all models?

You may find it tempting to grab any and all models, convert them to the web friendly format, and run them in the browser. But this is not always possible or recommended. There are several factors for you to keep in mind.

The tensorflowjs_converter command can only convert Keras and TensorFlow models. Some supported model formats include SavedModel, Frozen Model, and HDF5.

TensorFlow.js does not support all TensorFlow operations. It currently has a limited set of supported operations. As a result, the converter will fail if the model contains operations not supported.

Thinking and treating the model as a black box is not always enough. Because you can get the model converted and produce a web friendly model does not mean all is well.

Depending on a model’s size or architecture, its performance could be less than desirable. Further optimization of the model is often required. In most cases, you will have to pre-process the input(s) to the model, as well as, process the model output(s). So, needing some understanding or inner workings of the model is almost a given.

Getting to know your model

Presumably you have a model available to you. If not, resources exist with an ever growing collection of pre-trained models. A couple of them include:

TensorFlow Models —a set of official and research models implemented in TensorFlow
Model Asset Exchange —a set of deep learning models covering different frameworks

These resources provide the model for you to download. They also can include information about the model, useful assets, and links to learn more.

You can review a model with tools such as TensorBoard. It’s graph visualization can help you better understand the model.

Another option is Netron, a visualizer for deep learning and machine learning models. It provides an overview of the graph and you can inspect the model’s operations.

visualizing a model with Netron

To be continued…

Stay tuned for the follow up to this article to learn how to pull this all together. You will step through this process in greater detail with an actual model and you will take a pre-trained model into web friendly format and end up with a web application.