Forem: Paul DeCarlo

How GPU-Powered Coding Agents Can Assist in Development of GPU-Accelerated Software

Paul DeCarlo — Sun, 01 Mar 2026 03:36:32 +0000

How GPU-Powered Coding Agents Can Assist in Development of GPU-Accelerated Software

The Dream: Transcribe Your Entire Media Library on a Device That Fits in Your Hand

Imagine owning a massive Plex media library with hundreds of foreign-language films and TV shows. You want subtitles for everything, but manually sourcing them is a nightmare — mismatched timings, missing translations, incomplete coverage. Tools like Bazarr exist specifically to automate subtitle management for Plex and Sonarr/Radarr libraries, and they ship with built-in integration for whisper-asr-webservice — a self-hosted REST API that wraps OpenAI's Whisper speech recognition model. Point Bazarr at a whisper-asr-webservice endpoint, and it will automatically transcribe and generate subtitles for every piece of media in your library, in any language Whisper supports.

There's just one problem: running Whisper fast enough to be practical requires a GPU, and the existing Docker images only support x86_64 with NVIDIA desktop or server GPUs. If you want a quiet, power-efficient, always-on transcription appliance — something you can tuck behind your NAS and forget about — the NVIDIA Jetson platform is the obvious choice. An Orin Nano draws under 15 watts, fits in the palm of your hand, and packs a 1024-core Ampere GPU with hardware support for the same CUDA operations that Whisper needs. A single portable Docker container running on a Jetson could silently chew through your entire library in the background, generating subtitles on demand whenever new media arrives.

The question was: could we actually build that container? The answer turned out to be a story about how GPU-powered AI coding agents can come full circle — using GPU-accelerated tools to build GPU-accelerated software for GPU-accelerated hardware.

The Historical Pain of Porting to aarch64

Anyone who has tried to compile PyTorch, CTranslate2, or onnxruntime for ARM hardware knows the pain. The Python AI/ML ecosystem was born on x86_64 Linux and macOS, and its package infrastructure carries deep assumptions about that lineage.

PyTorch is the foundation of nearly every modern speech recognition system. On x86, you pip install torch and get a CUDA-enabled wheel in seconds. On aarch64, that same command gives you a CPU-only build — or nothing at all. For years, getting a CUDA-enabled PyTorch on Jetson meant manually compiling from source against NVIDIA's JetPack SDK, a process that could take hours on the device itself and was fragile across JetPack versions. NVIDIA eventually began publishing pre-built wheels through the Jetson AI Lab pip index, but using them correctly requires understanding a subtle and underdocumented packaging conflict: pip's wheel compatibility sorting prefers manylinux_2_28 tags over linux_aarch64, which means if both PyPI and the Jetson index are available as pip sources, pip will happily install the CPU-only PyPI wheel instead of the CUDA-enabled Jetson wheel. You must use --index-url (making Jetson the primary source), not --extra-index-url (which makes it secondary).

CTranslate2, the inference backend that faster-whisper uses to run Whisper models efficiently, is another casualty. PyPI publishes aarch64 wheels, but they are CPU-only. There is no CUDA-enabled aarch64 wheel. Getting GPU acceleration on Jetson means compiling CTranslate2 from source with -DWITH_CUDA=ON, linking against the JetPack CUDA toolkit, and targeting the correct CUDA compute capability for your specific Jetson hardware.

Poetry, the dependency manager used by whisper-asr-webservice, adds another layer of complexity. Poetry's resolver has no concept of "this package must come from this alternate index for this platform." When you run poetry install, it merges all dependency specifications and resolves them against PyPI. On Jetson, this means Poetry will cheerfully overwrite your carefully pre-installed CUDA-enabled PyTorch with a CPU-only wheel from PyPI, because the version constraint matches and Poetry doesn't know the difference. The project's poetry-core PEP 517 metadata generation also merges [tool.poetry.dependencies] source mappings with [project.optional-dependencies], producing version constraints like torch==2.7.1+cu126 that don't match the actual Jetson wheel version at all.

These are not exotic edge cases. They are the default experience of trying to port GPU-accelerated Python software to aarch64. And they are exactly the kind of deeply contextual, multi-layered problems that AI coding agents excel at navigating.

CUDA Architecture on L4T: The Edge Cases That x86 Takes for Granted

NVIDIA's Linux for Tegra (L4T) is the OS layer that underpins JetPack on Jetson devices. While x86 CUDA development benefits from a relatively uniform environment — install the CUDA toolkit, install the driver, compile for sm_70 through sm_90 and let the JIT handle the rest — Jetson development requires precise awareness of the hardware-software matrix:

Jetson Generation	Compute Capability	L4T Branch	JetPack	CUDA
Nano / TX2	sm_53 / sm_62	R32.x	4.x	10.2
Xavier NX / AGX	sm_72	R35.x	5.x	11.4
Orin Nano / NX / AGX	sm_87	R36.x	6.x	12.6

On x86, compiling CTranslate2 with -DCUDA_ARCH_LIST="7.0;7.5;8.0;8.6;8.9;9.0" covers virtually every GPU from 2017 to 2024. On Jetson, you compile for exactly one architecture — 8.7 for Orin — and the base image must match your JetPack version precisely because the CUDA toolkit, cuDNN, and TensorRT are all provided by the L4T base image rather than installed separately.

There's also the cuBLAS conflict problem: the JetPack base image ships a system cuBLAS in /usr/local/cuda/lib64. When you install nvidia-cudss-cu12 from pip (required because the Jetson PyTorch wheel links against libcudss.so.0), it pulls in nvidia-cublas-cu12 as a transitive dependency. Loading two different versions of cuBLAS at runtime causes CUBLAS_STATUS_ALLOC_FAILED — a cryptic error that only manifests when the model actually tries to run a matrix multiplication on the GPU. The fix is to uninstall the pip cuBLAS immediately after installing cudss, and ensure LD_LIBRARY_PATH does not include the pip cublas lib directory.

These are the kinds of platform-specific gotchas that would take a human developer hours of Stack Overflow browsing and GitHub issue trawling to diagnose. An AI coding agent with knowledge of the Jetson ecosystem can identify and resolve them in the flow of a single conversation.

The Setup: VS Code, Claude Opus 4.6, and Source Code

The ingredients for this solution were deliberately minimal. We used VS Code as the development environment, outfitted with GitHub Copilot powered by Claude Opus 4.6 as the AI coding agent, with the whisper-asr-webservice source code cloned locally on the Jetson device itself. That's it — an editor, a model, and a codebase. No specialized Jetson development tools, no cross-compilation toolchains, no reference implementations to copy from.

What made this combination potent was the intersection of three capabilities: Claude Opus 4.6's deep knowledge of CUDA toolchains, Python packaging, and Docker multi-stage builds; VS Code's integrated terminal giving the agent direct access to build and test on the target hardware; and the source code providing the agent full visibility into the project's dependency structure, build system, and runtime architecture. The agent could read pyproject.toml to understand Poetry's dependency graph, inspect the existing x86 Dockerfiles for patterns, examine the application code to understand which libraries each ASR engine imports, and then synthesize all of that into a Jetson-specific build — all within a single conversational session.

The Prompt That Started It All

The session began with a single natural-language prompt:

"Bro, I need some help, is there a way you might be able to figure out how to build this project in a Docker container with support for GPU acceleration on the NVIDIA Jetson hardware we are currently running on. Specifically, wthe openai-whisper, whisperx, and faster-whisper dependencies are going to need to be built from source to include acceleration on this device. Poetry is going to annoy you because the trick will be solving the additional dependencies without breaking the full project. The resulting solution should be a single docker file that builds specifically on Jetson. This is going to be tough, can you try?"

That's it. No architecture document. No step-by-step instructions. No prior Dockerfile to copy from. The agent needed to understand the project structure, identify which components required platform-specific builds, design a multi-stage Dockerfile strategy, and navigate every one of the compatibility landmines described above.

The resulting Dockerfile.jetson is nearly 400 lines of carefully sequenced build steps with extensive documentation explaining why each decision was made — not just what it does. The three-stage build strategy emerged organically:

Stage 1: Compile CTranslate2 from source with CUDA support, targeting sm_87 for Orin
Stage 2: Extract Swagger UI static assets from the x86 swagger-ui image (only static JS/CSS, no binaries)
Stage 3: Assemble the runtime — pre-install CUDA packages from the Jetson AI Lab index, install remaining Python dependencies with constraints protecting CUDA packages, apply compatibility shims

Beyond Code: Testing Containers from Inside VS Code

One of the most powerful aspects of working with an AI coding agent in VS Code is that the agent is not limited to writing code. It has access to the terminal, which means it can build Docker images, run containers, generate test data, hit HTTP endpoints, inspect logs, and tear down environments — all within the same conversation.

This is a fundamentally different paradigm from traditional code generation. The agent doesn't hand you a Dockerfile and say "try building this." It builds the image itself, watches for errors, diagnoses failures, applies fixes, and rebuilds — iteratively, in real-time, on the actual target hardware.

During our session, the agent executed commands like:

# Build the image on the Jetson
docker build -f Dockerfile.jetson \
  -t whisper-asr-webservice-jetson:jp6.1-cu12.6-py3.10 .

# Start a container with a specific ASR engine
docker run -d --rm --name fw-test --runtime nvidia -p 9000:9000 \
  -e ASR_ENGINE=faster_whisper -e ASR_MODEL=tiny \
  whisper-asr-webservice-jetson:latest

# Test the endpoint with a real audio file
curl -X POST "http://localhost:9000/asr?task=transcribe&language=en&output=json" \
  -H "accept: application/json" \
  -F "audio_file=@/tmp/test_speech.wav"

# Inspect CUDA availability inside the container
docker run --rm --runtime nvidia whisper-asr-webservice-jetson:latest \
  python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

When a build failed — and across 15+ iterations, many did — the agent read the error output, identified the root cause, modified the Dockerfile, and rebuilt. When a runtime crash occurred, it inspected the Python traceback, traced the issue to a specific library version incompatibility, and created a monkey-patch shim. The entire feedback loop happened inside the VS Code terminal, with the agent operating as both developer and QA engineer simultaneously.

The Testing Strategy: Three Engines, Generated Speech, Real Validation

Whisper-asr-webservice supports three different ASR backends, each with different runtime dependencies, model loading paths, and GPU code paths:

faster_whisper — Uses CTranslate2 for optimized inference, requires a CUDA-compiled CTranslate2
openai_whisper — The original OpenAI implementation, uses PyTorch directly
whisperx — Extends Whisper with word-level timestamps and speaker diarization via pyannote.audio, requires torchaudio and the HuggingFace pipeline

All three needed to work. A container that only runs one engine is only one-third of a solution.

The testing strategy evolved through an important self-correction. The initial approach was to generate a simple test audio file — a sine wave tone — and POST it to the ASR endpoint. This produced a "successful" HTTP 200 response, but the transcription result was empty or garbage because there was no actual speech in the audio. The test was passing but not actually validating anything meaningful.

The agent recognized this limitation and pivoted: instead of a synthetic tone, it used espeak-ng (a text-to-speech engine available on the system) to generate a WAV file containing actual spoken English:

espeak-ng -w /tmp/test_speech.wav \
  "The quick brown fox jumps over the lazy dog"

This produced a test file with clear, recognizable speech. When the ASR engines transcribed it, the response contained actual words that could be verified — not just an HTTP status code, but semantic validation that the speech recognition pipeline was functioning end to end, from audio input through GPU-accelerated model inference to text output.

Each engine was tested individually by spinning up a fresh container with the appropriate ASR_ENGINE environment variable:

# Test faster_whisper
docker run -d --rm --name fw-test --runtime nvidia -p 9000:9000 \
  -e ASR_ENGINE=faster_whisper -e ASR_MODEL=tiny \
  whisper-asr-webservice-jetson:latest
# Wait for model download + startup, then curl, then teardown

# Test openai_whisper
docker run -d --rm --name ow-test --runtime nvidia -p 9000:9000 \
  -e ASR_ENGINE=openai_whisper -e ASR_MODEL=tiny \
  whisper-asr-webservice-jetson:latest

# Test whisperx
docker run -d --rm --name wx-test --runtime nvidia -p 9000:9000 \
  -e ASR_ENGINE=whisperx -e ASR_MODEL=tiny \
  whisper-asr-webservice-jetson:latest

All three returned successful transcriptions with recognizable text. The whisperx engine additionally returned word-level timestamps, confirming that the torchaudio compatibility shim was working correctly and pyannote's audio processing pipeline was intact.

This test cycle was repeated three times across the session — after the initial build, after the torch.load compatibility fix, and after the huggingface_hub API fix — ensuring that each patch didn't break previously working functionality. The agent managed all of this autonomously: spinning up containers, waiting for startup, sending requests, validating responses, tearing down containers, and reporting results.

The Compatibility Shims: When Libraries Disagree

Three runtime compatibility issues surfaced during testing, each requiring a different kind of fix. Rather than forking upstream libraries or pinning to ancient versions, the agent created a unified compatibility shim — a single Python file loaded at interpreter startup via a .pth file in site-packages. This approach is surgical: it patches only what's broken, at the earliest possible moment, without modifying any installed package.

1. torchaudio API removal: The Jetson AI Lab torchaudio builds strip out the legacy backend API — AudioMetaData, info(), and list_audio_backends() — because the Jetson builds use a different audio backend architecture. But pyannote.audio 3.x still calls these functions. The shim implements them using the soundfile library, which is available and functional on Jetson.

2. torch.load weights_only default: PyTorch 2.6+ changed torch.load() to default to weights_only=True for security. But pyannote's VAD (Voice Activity Detection) model checkpoints contain omegaconf.ListConfig objects that aren't in the allowlist. The tricky part: lightning_fabric passes weights_only=None explicitly, which PyTorch interprets as True. A simple setdefault doesn't work — you have to check if kwargs.get("weights_only") is None and override it. The agent discovered this subtlety by reading the actual traceback and tracing through the call chain.

3. huggingface_hub API deprecation: huggingface_hub 1.5.0 removed the deprecated use_auth_token parameter entirely, but pyannote.audio 3.4.0 and whisperx still pass use_auth_token= instead of token=. The fix required patching not just huggingface_hub.hf_hub_download in the top-level namespace, but also in submodules like huggingface_hub.file_download — because pyannote does from huggingface_hub import hf_hub_download at module level, which copies the function reference before any top-level patch can take effect. The shim pre-imports and patches the submodules so that when pyannote's from import runs, it picks up the already-patched version.

Each of these fixes emerged from the agent observing a runtime failure, diagnosing the root cause by inspecting library source code inside the running container, and implementing the minimal patch needed.

From Working Code to Pull Request — By Prompting

With the container built, tested, and verified across all three engines, the next step was contributing back upstream. The project had open issues requesting exactly this capability:

Issue #359: "Add Arm support for GPU container"
Issue #54: "Possible to run on Jetson Nano?"
Issue #133: "Is it possible to get this on Jetson using the GPU?"

Forking the repository, creating a feature branch, committing the changes, and opening a pull request was accomplished entirely through prompts:

"Fork the repo and make a pull request."

The agent forked ahmetoner/whisper-asr-webservice to toolboc/whisper-asr-webservice, created a feat/jetson-gpu-support branch, committed the Dockerfile and compose file, pushed to the fork, and opened PR #364 with a detailed description including:

A summary of the target platform and what's included
A table of key technical decisions with rationale for each
Verification results from actual hardware testing
Build and run instructions
A link to the pre-built Docker Hub image

When additional fixes were made (cuBLAS conflict, torch.load, huggingface_hub), each was committed with a descriptive message and pushed to update the PR:

937e2a8 feat: add NVIDIA Jetson GPU support (Dockerfile + compose)
290ffcd fix: remove conflicting pip cuBLAS to fix CUBLAS_STATUS_ALLOC_FAILED
d7096fa fix: patch torch.load for whisperx/pyannote VAD compatibility
19ef291 chore: add container_name to compose file
a6e731b fix: patch huggingface_hub use_auth_token -> token for HF_TOKEN support

Later, when asked to reference the relevant upstream issues, the agent searched the issue tracker, identified the three related issues, and updated the PR description with Closes #359 and Relates to #54, #133. The resulting PR is more thorough than most manually created pull requests — every technical decision is documented, every compatibility workaround is explained, and the testing methodology is clear.

The image was also pushed to Docker Hub as toolboc/whisper-asr-webservice-jetson:jp6.1-cu12.6-py3.10, making it immediately available to anyone with a Jetson device — no build required.

The Full Circle: GPUs Building Software for GPUs

There's a satisfying symmetry in this story. The AI coding agent that designed, built, debugged, and tested this container is itself powered by GPU-accelerated inference. The end product — a Docker container running Whisper on Jetson's GPU — is GPU-accelerated software. And the problems we solved — CUDA compute capabilities, cuBLAS library conflicts, GPU-specific wheel selection — are fundamentally GPU problems.

This is what "GPU-assisted development comes full circle" looks like in practice:

A GPU-powered agent (the LLM) understands the nuances of CUDA architecture, library ABI compatibility, and platform-specific packaging
It produces GPU-accelerated software (the Jetson Whisper container) that exploits the target hardware's full capabilities
It validates the result on actual GPU hardware by running containers, executing CUDA operations, and verifying inference output
It contributes the solution back to the open-source community through a well-documented pull request

The agent didn't just write a Dockerfile. It navigated a maze of platform-specific incompatibilities that have historically been the domain of specialized embedded engineers with deep knowledge of the NVIDIA toolchain. It did this while simultaneously managing Docker builds, generating test data, running HTTP integration tests, managing git workflows, and producing documentation — tasks that span the full spectrum from systems engineering to technical writing.

What This Unlocks

The Jetson platform isn't just for Whisper. The same challenges we solved here — CUDA compilation, pip index conflicts, Poetry resolver workarounds, torchaudio compatibility — apply to virtually every PyTorch-based project that someone wants to run on edge hardware. The pattern is repeatable:

Identify which dependencies need platform-specific builds
Source or compile CUDA-enabled versions for the target architecture
Constrain the package manager to prevent overwriting with CPU-only alternatives
Shim any API incompatibilities between library versions
Test on actual hardware with meaningful validation data

An AI coding agent that understands this pattern can port other projects to Jetson — or to any constrained platform — with dramatically less effort than manual development. The developer's role shifts from "figure out why pip install torch gives me a CPU-only wheel on aarch64" to "build this for Jetson" and validating the result.

For the Plex and home media server community specifically, this means a standalone appliance that generates subtitles automatically for any content in any language. Drop a Jetson Orin Nano next to your NAS, run docker compose -f docker-compose.jetson.yml up, point Bazarr at http://jetson:9000, and every new movie or episode that arrives gets transcribed and subtitled without human intervention. All on a device that draws less power than a light bulb.

That's the kind of practical, real-world automation that becomes possible when AI-assisted development makes it trivial to port sophisticated GPU-accelerated software to the hardware that can actually run it where it's needed.

This post documents work performed on an NVIDIA Jetson Orin running JetPack 6.2.2 (L4T R36.5.0, CUDA 12.6) using VS Code with GitHub Copilot powered by Claude Opus 4.6. The resulting pull request is #364 on the whisper-asr-webservice repository. A pre-built container image is available on Docker Hub at toolboc/whisper-asr-webservice-jetson:jp6.1-cu12.6-py3.10.

#JulyOT - Building the Intelligent Edge with Jetson and Azure ft. the NVIDIA Embedded team

Paul DeCarlo — Mon, 13 Jul 2020 20:06:41 +0000

Erik St. Martin and Paul DeCarlo from the Microsoft Cloud Advocacy team meet with team members from NVIDA to discuss “Building the Intelligent Edge with Jetson and Azure”. We will be discussing a number of topics related to this theme and will supplement with questions from attendees on the stream.

NVIDIA team members include:
Chintan Shah, Product Manager
Tenika Versey, Marketing Lead for AI
Ryan Huff, Business Development for AI | Edge | IoT
Jaime Flores, AI Developer Relations Manager

Resources:

Intelligent Video Analytics Github Repository: https://github.com/toolboc/Intelligent-Video-Analytics-with-NVIDIA-Jetson-and-Microsoft-Azure
DeepStream Developer Guide: http://aka.ms/deepstreamdevguide
Jetson Community Projects: https://developer.nvidia.com/embedded/community/jetson-projects
Jetson Community Resources: https://developer.nvidia.com/embedded/community/resources
Jetson on Azure: http://aka.ms/jetson-on-azure

#JulyOT - Visualizing Object Detection Data in Near Real-Time with PowerBI

Paul DeCarlo — Fri, 10 Jul 2020 14:24:11 +0000

Erik and Paul forward object detection results from an Azure Stream Analytics Job into a PowerBI Dataset. They then develop a custom report for visualizing data in a live PowerBI Dashboard.

Part 5 of a 5 part series created for #JulyOT - more details @ http://julyot.com

The full steps to reproduce this project can be found on github @ https://github.com/toolboc/Intelligent-Video-Analytics-with-NVIDIA-Jetson-and-Microsoft-Azure

For more information on the services employed, check out:
Stream Analytics Documentation
PowerBI Documentation

#JulyOT - Consuming and Modeling Object Detection Data with Azure Time Series Insights

Paul DeCarlo — Thu, 09 Jul 2020 17:04:17 +0000

Erik and Paul forward object detections to Time Series Insights from an Azure Stream Analytics on the Edge Job. We showcase the whole process of setting up a TSI event source to modeling and exporting of the data within Time Series Insights.

Part 4 of a 5 part series created for #JulyOT - more details @ http://julyot.com

The full steps to reproduce this project can be found on github @ https://github.com/toolboc/Intelligent-Video-Analytics-with-NVIDIA-Jetson-and-Microsoft-Azure

For more information on the services employed, check out:

Stream Analytics Documentation
Time Series Insights Documentation

#JulyOT - Develop and deploy Custom Object Detection Models with IoT Edge DeepSteam SDK Module

Paul DeCarlo — Wed, 08 Jul 2020 14:20:00 +0000

Erik and Paul demonstrate how to leverage a model developed with CustomVisionAI for use with the IoT Edge DeepSteam SDK Module and then explore how to employ a Custom Yolo Parser for use with YoloV3 and YoloV3 Tiny.

Part 3 of a 5 part series created for #JulyOT - more details @ http://julyot.com

The full steps to reproduce this project can be found on github @ https://github.com/toolboc/Intelligent-Video-Analytics-with-NVIDIA-Jetson-and-Microsoft-Azure

For more information on the services employed, check out:
Cognitive Services Documentation
Custom Vision Service Docs:
CameraTaggingModule
Yolo Object Detection

#JulyOT - Configure and Deploy "Intelligent Video Analytics" to IoT Edge Runtime on NVIDIA Jetson

Paul DeCarlo — Tue, 07 Jul 2020 15:31:43 +0000

Erik and Paul create an Azure IoT Hub and backing storage account in Microsoft Azure, then instrument a Jetson Nano with IoT Edge and configure it with a deployment that uses these services to accommodate an intelligent video analytics solution.

Part 2 of a 5 part series created for #JulyOT - more details @ http://julyot.com

The full steps to reproduce this project can be found on github @ https://github.com/toolboc/Intelligent-Video-Analytics-with-NVIDIA-Jetson-and-Microsoft-Azure

For more information on the services employed, check out:
IoT Edge Documentation
IoT Hub Documentation
Stream Analytics Documentation
IoT Blob Storage Documentation

Camera used in demo: Foscam FI9821P, though any RTSP based camera should suffice.

#JulyOT - Getting Started with NVIDIA Jetson Nano: Object Detection

Paul DeCarlo — Mon, 06 Jul 2020 19:05:15 +0000

Erik and Paul configure a Jetson Nano device for use with DeepStream SDK using samples provided from NVIDIA.

Part 1 of a 5 part series created for #JulyOT - more details @ http://julyot.com

The full steps to reproduce this project can be found on github @ https://github.com/toolboc/Intelligent-Video-Analytics-with-NVIDIA-Jetson-and-Microsoft-Azure

For more information on the services employed, check out:
http://aka.ms/deepstreamdevguide
https://github.com/leandromoreira/digital_video_introduction
https://github.com/leandromoreira/ffmpeg-libav-tutorial
http://dranger.com/ffmpeg

#JulyOT - Artificial Intelligence at the Edge

Paul DeCarlo — Thu, 02 Jul 2020 16:18:55 +0000

#JulyOT is here! We are calling on all makers, students, tinkerers, hardware hackers, and professional IoT developers to bring their creativity to create an IoT focused project in the month of July. We have summarized the goals of #JulyOT and how you can get involved in this previous post on dev.to.

Today's #JulyOT post covers some of the updates that you can expect for the week of July 6 - 10. Our theme for this week is "Artificial Intelligence at the Edge" and we have partnered with our friends at NVIDIA to bring you over 8 hours of content that will teach almost EVERYTHING that you need to know to develop AIOT (Artificial Intelligence of Things) solutions. This will culminate in a livestream event with developers from NVIDIA on July 10 at 1 PM CST on the Microsoft Developer Twitch Channel.

Throughout the week, we will roll-out individual posts across social media, but you can also get ahead of the curve by heading to the official #JulyOT content repository at http://julyot.com.

So, what's in the repository this week and what topics will be covered on the livestream?

The majority of content will focus on a 5 part video series that was developed to accompany an open source project titled "Intelligent Video Analytics with NVIDIA Jetson and Microsoft Azure". In a nutshell, this project will teach you how to develop a custom end-to-end Intelligent Video Analytics pipeline with multiple video sources, and includes steps on how to train your own object detection model to detect whatever you fancy! The modules are listed below and will give you a quick overview on what you can expect to learn in each section.

Each of these modules is accompanied by a livestream that was recorded with @ErikDotDev on Twitch. This is interesting because, Erik was brand new to the NVIDIA embedded platform and applied Artificial Intelligence. We have catalogued his journey in these recordings and hope that it can serve to inspire others to follow a similar path! That's right, in ~8 hours of video content, we feel that you too can become an expert in building custom object detection pipelines for use in a wide variety of scenarios.

Now, what better way to endcap this journey than by being able to ask questions live to the NVIDIA developers who have worked to create the platform and tools that make all of this possible! On July 10 at 1PM CST, we will be joined by a group of developers who will be on hand to answer your questions about anything related to NVIDIA embedded devices, the DeepStream SDK, and the various AI workloads that are supported by their hardware. Be sure to mark your calendars for this event and tune in at the appropriate time at the Microsoft Developer Twitch Channel.

If you have any burning questions for the NVIDIA team, please leave them in the comments below. We will review them and just might ask them on the air! Here are a couple example questions that we plan to cover on the stream:

During the development of our Custom Model, the topic of acceleration hardware came up where we identified GPUs, FPGAs, and ASICs as mechanisms for inference acceleration.

These all work on different principles but we want to hear from NVIDIA, how does a GPU actually accelerate AI workloads?
We mostly looked at the acceleration of computer vision workloads, what other types of AI inferencing can be done with NVIDIA hardware? We understand that audio features can also benefit, do you have any examples of this?

As always, thank you for reading and we hope to see you there on July 10 for the livestream event with NVIDIA developers on the Microsoft Developer Twitch Channel!

#JulyOT - Intelligent Home Security with NVIDIA Jetson

Paul DeCarlo — Wed, 01 Jul 2020 16:51:07 +0000

Paul DeCarlo demonstrates an Intelligent Home Security System built using the Jetson Xavier NX with Azure IoT Edge and the DeepStream SDK module to run a custom object detection model built with Custom Vision.AI. Object detection telemetry is pushed into Azure Time Series Insights and PowerBI to allow for analyzing these detections over time and in near real-time.

Github repo to reproduce this project:
https://github.com/toolboc/Intelligent-Video-Analytics-with-NVIDIA-Jetson-and-Microsoft-Azure
This project was created as part of #JulyOT, more details are available at: http://aka.ms/julyiot

For more information about the services used in this project:

Jetson Xavier NX: https://nvda.ws/3bqcNEx
Azure IoT Edge: http://aka.ms/iotedgestart
NVIDIA DeepStream SDK Module: http://aka.ms/deepstreamsdkmodule
Azure Time Series Insights: https://aka.ms/timeseriesinsightsstart
Power BI: https://aka.ms/powerbistart

#JulyOT - A month dedicated to learning and building IoT Projects

Paul DeCarlo — Wed, 01 Jul 2020 14:28:48 +0000

Attention all makers, students, tinkerers, hardware hackers, and professional IoT developers! The month of #JulyOT is here!  To celebrate, we have curated a collection of content - blog posts, hands-on-labs, and videos designed to demonstrate and teach developers how to build projects with Azure Internet of Things (IoT) services.  This content ranges from video demonstrations of real-world solutions, a 5-part series on developing an Intelligent Video Analytics service, an IoT bird-box project for families, Raspberry Pi air quality monitor backed by Azure IoT Central, and a self-guided training series designed to help study for the Azure 220 IoT Developer certification.

All of this content will be made available on this IoT Tech Community page. It is highly recommended that you bookmark this page as we will update it with new content each week during the month of #JulyOT! In this post, we would like to give a rundown on what you can expect from #JulyOT and how you can contribute to the fun!

What is #JulyOT?

JulyOT was born out of a desire to celebrate all things IoT in an accessible way to enable developers around the world to build innovative projects with connected devices. Do you have an idea can be brought to life by connecting a device to the internet? Are you interested in exploring Artificial Intelligence at the Edge or building something innovative with sensors and hardware? We want to hear about it! We will be monitoring the #JulyOT hashtag to actively promote and encourage projects built by the community throughout the month of July. All you need to do is have an idea for an IoT project, then tweet it out and keep us updated on progress by including the #JulyOT hashtag. We’re open to all projects that involve a thing and the internet, so don’t feel afraid to share, we want to inspire everyone to build something cool in July!

Here are a few examples:

JulyOT Themes

During each week of #JulyOT, we will have a focused theme that will combine related pieces. Each of these themes will be accompanied by a project built by the IoT Cloud Advocacy team at Microsoft that you can build yourself or remix into something brand new! You don’t have to follow a theme to be a part of #JulyOT, but it may be of interest if you have an idea that falls into one of these topics.

July 1 – 3 : #JulyOT Content Kickoff

The first week will begin with primer content intended to introduce the concept of IoT to interested contributors. We will also “tease” out some of the projects that will be released later in the month. This is the time for contributors to begin thinking about what they wish to create and a great way to learn about real-world applications in the realm of IoT. It is highly suggested to author your Tweets during the Content Kickoff period and we’ll be in touch throughout the month to check in on your project’s progress!

July 6 – 10 : Artificial Intelligence at the Edge

The second week will focus on AI @ Edge solutions that make use of techniques like computer vision to produce intelligent solutions that run locally on embedded hardware. Paul DeCarlo has provided a themed project that consists of a 5-part series on developing an Intelligent Video Analytics service, accompanied by recorded livestreams to guide you through the development process. This content will show you how to build an end-to-end architecture for custom object detection using NVIDIA hardware with Microsoft Azure. The week will culminate with a livestream on July 10 that will feature developers from NVIDIA. Come ready to ask questions as we will be answering them live on the air!

July 13 – 17 : Maker Community and Academic

The third week is all about empowering makers and students to build IoT projects, particularly with a focus on teaching others. We want to extend the subject of IoT to students and possibly family members with projects that are fun and easy to get started with. Jim Bennett has created a relatively inexpensive project that teaches you how to make smart garden ornaments. This is a fun week long project for those who want to take their first steps into the world of the Internet of Things (IoT) using devices that are popular with kids and tools that make programming accessible to young developers. You'll use a Raspberry Pi along with some BBC micro:bits and Microsoft MakeCode, and any garden ornaments you have to hand to build a smart neighborhood, gathering data such as temperature and soil moisture levels and displaying it in the cloud using Azure IoT Central.

July 20 – 24 : Microcontrollers and Embedded Hardware

The fourth week will cover a variety of projects built around microcontrollers (for example Azure Sphere and embedded hardware like the Raspberry Pi. This also includes a focus on real-time operating systems that run on these devices. Dave Glover has produced content that covers how to build a Raspberry Pi air quality monitor, Azure Sphere RTOS integration, and a full learning path focused on Azure Sphere Development. Learn how to not only built intelligent devices and apply custom sensors but also discover how to connect your devices to cloud services for command + control and reporting!

July 27 – 31 : Online learning and Certification

The final week of #JulyOT will focus on free learning resources like Microsoft Learn and other materials to help prepare for the Azure 220 IoT Developer certification. Thomas Maurer has written an extensive study guide that we have augmented with live videos from partner training resources provided by Diana Phillips. We hope that many of you will take the learnings that you have made throughout #JulyOT and leverage the experience to achieve an official designation as a certified IoT Developer!

Conclusion:

It is our hope that we can inspire many of you to pursue a personal project in the IoT space during #JulyOT. Throughout the month, we will focus deeper on themed content in addition to projects that are created by you! There is never a bad time to jump into the fun of creating your own project, but we hope that you can build with us during this special time of year! If you feel inspired, let us know with a Tweet and don’t forget the #JulyOT hashtag, we can’t wait to see what you will build!

Introduction to the Azure IoT Edge Camera Tagging Module

Paul DeCarlo — Mon, 27 Apr 2020 13:45:32 +0000

In this post we will introduce the Azure IoT Edge Camera Tagging Module. This module will deploy a service onto compatible IoT devices to capture images from live RTSP video streams. These images can then be uploaded from the module into Microsoft Cognitive Services for use in training object detection and image classification models, or into Azure Blob Storage for archival purposes.

Introduction

The combination of Artificial Intelligence and Internet of Things yields a paradigm commonly referred to as the "Artificial Intelligence of Things" or AIoT. Development in this field is focused around augmentation of common sensors like microphones and cameras which take their traditional output and use them as algorithmic inputs that are processed by AI models. This can enable abilities that include transforming spoken words into contextual requests that can be operated on by a computer program or determining the presence and location of objects in a video feed.

The development of models for use in AIoT applications often involves gathering quantities of training data to use as inputs that an AI model can learn from. As you might expect, the quality and relevance of the data involved in the training of these models can have a large impact on their accuracy in production. There are a number of ways to approach this, for example using vast training sets that consist of millions or even billions of samples to produce a generic detector for a variety of unique inputs. Another method might involve using contextually relevant data, for example taking samples at the site of deployment and using those to create a tightly tuned model that can handle a specific environment.

The Azure IoT Edge Camera Tagging Module can assist in both of these strategies by allowing you to capture sample data for vision based AI models at scale. It can also allow you to capture data remotely from the site of deployment. This enables solution builders to produce varied and precise AI models using data gathered from a module running on any IoT Edge capable device.

The content will assume some prior knowledge of Azure IoT Edge and it is suggested that you are familiar with installing the IoT Edge Runtime on Linux devices and how to create a deployment for a Linux-Based IoT Edge Device. If you would like to learn more about Azure IoT Edge and are looking for a good place to learn, I highly recommend the official IoT Edge Documentation and the interactive courses on IoT Edge at Microsoft Learn.

Getting Started

Before we begin, the following is assumed:

You have an active Azure Subscription
You have followed steps to install the Azure IoT Edge Runtime to your Linux- based ARM32, ARM64, or AMD64 target device
You have followed the steps to register your IoT Edge Device to an Azure IoT Hub instance
You are familiar with the steps to deploy modules to an IoT Edge Device from Visual Studio Code
You have installed the Azure IoT Edge Extension for Visual Studio Code

The Azure IoT Edge Camera Tagging Module is under active development on Github. A quick perusal of the official README mentions that the module supports ARM32, ARM64, and AMD64 Linux platforms. The project is distributed in a fashion that encourages building the module from source and publishing it to your own private docker registry. This is the preferred approach if you plan to use the module in production, but it does require some additional steps and overhead.

To make test-driving a bit easier, I have published the Camera Tagging module for all three supported architectures in a public DockerHub repo using a Docker manifest to allow you to easily install the module regardless of the target platform with a reference the following image tag: toolboc/camerataggingmodule:latest. This multi-platform image will be referenced in our deployment.template.json.

Create a new IoT Edge Solution and update the deployment.template.json

Start by opening Visual Studio Code and use the shortcut (CTRL+SHIFT+P) and search for the Azure IoT Edge: New IoT Edge Solution task:

Choose an appropriate directory to create your project under, next you will be asked to give your solution a name. It is suggested to name it something like CameraTaggingModuleExample.

After your solution is named, you will be prompted to Select Module Template. Select the Empty Solution option.

This will produce a base project with the minimal scaffolding to produce an IoT Edge deployment.

Copy the content below and replace the current contents of the generated deployment.template.json:

{
  "$schema-template": "1.0.0",
  "modulesContent": {
    "$edgeAgent": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "runtime": {
          "type": "docker",
          "settings": {
            "minDockerVersion": "v1.25",
            "loggingOptions": "",
            "registryCredentials": {
              "containerRegistry": {
                "username": "$CONTAINER_REGISTRY_USERNAME",
                "password": "$CONTAINER_REGISTRY_PASSWORD",
                "address": "$CONTAINER_REGISTRY_NAME"
              }
            }
          }
        },
        "systemModules": {
          "edgeAgent": {
            "type": "docker",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-agent:1.0.9",
              "createOptions": {}
            }
          },
          "edgeHub": {
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-hub:1.0.9",
              "createOptions": {
                "HostConfig": {
                  "PortBindings": {
                    "5671/tcp": [
                      {
                        "HostPort": "5671"
                      }
                    ],
                    "8883/tcp": [
                      {
                        "HostPort": "8883"
                      }
                    ],
                    "443/tcp": [
                      {
                        "HostPort": "443"
                      }
                    ]
                  }
                }
              }
            }
          }
        },
        "modules": {
          "CameraTaggingModule": {
            "version": "1.0.3",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "toolboc/camerataggingmodule:latest",
              "createOptions": {
                "ExposedPorts": {
                  "3000/tcp": {},
                  "3002/tcp": {},
                  "3003/tcp": {}
                },
                "HostConfig": {
                  "PortBindings": {
                    "3000/tcp": [
                      {
                        "HostPort": "3000"
                      }
                    ],
                    "3002/tcp": [
                      {
                        "HostPort": "3002"
                      }
                    ],
                    "3003/tcp": [
                      {
                        "HostPort": "3003"
                      }
                    ]
                  }
                }
              }
            },
            "env": {
              "RTSP_IP": {
                "value": "wowzaec2demo.streamlock.net"
              },
              "RTSP_PORT": {
                "value": "554"
              },
              "RTSP_PATH": {
                "value": "vod/mp4:BigBuckBunny_115k.mov"
              },
              "REACT_APP_SERVER_PORT": {
                "value": "3003"
              },
              "REACT_APP_WEB_SOCKET_PORT": {
                "value": "3002"
              },
              "REACT_APP_LOCAL_STORAGE_MODULE_NAME": {
                "value": "azureblobstorageoniotedge"
              },
              "REACT_APP_LOCAL_STORAGE_PORT": {
                "value": "11002"
              },
              "REACT_APP_LOCAL_STORAGE_ACCOUNT_NAME": {
                "value": "$LOCAL_STORAGE_ACCOUNT_NAME"
              },
              "REACT_APP_LOCAL_STORAGE_ACCOUNT_KEY": {
                "value": "$LOCAL_STORAGE_ACCOUNT_KEY"
              }
            }
          },
          "azureblobstorageoniotedge": {
            "version": "1.2",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azure-blob-storage:latest",
              "createOptions": {
                "Env":[
                  "LOCAL_STORAGE_ACCOUNT_NAME=$LOCAL_STORAGE_ACCOUNT_NAME",
                  "LOCAL_STORAGE_ACCOUNT_KEY=$LOCAL_STORAGE_ACCOUNT_KEY"
                 ],
                 "HostConfig":{
                   "Binds": ["/data/containerdata:/blobroot"],
                   "PortBindings":{
                     "11002/tcp": [{"HostPort":"11002"}]
                   }
                 }
              }
            }
          }
        }
      }
    },
    "$edgeHub": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "routes": {
          "azureblobstorageoniotedgeToIoTHub": "FROM /messages/modules/azureblobstorageoniotedge/outputs/* INTO $upstream"
        },
        "storeAndForwardConfiguration": {
          "timeToLiveSecs": 7200
        }
      }
    },
    "azureblobstorageoniotedge":{
      "properties.desired": {
        "deviceAutoDeleteProperties": {
          "deleteOn": false,
          "retainWhileUploading": true
        },
        "deviceToCloudUploadProperties": {
          "uploadOn": true,
          "uploadOrder": "OldestFirst",
          "cloudStorageConnectionString": "$CLOUD_STORAGE_CONNECTION_STRING",
          "storageContainersForUpload": {
            "$LOCAL_STORAGE_ACCOUNT_NAME": {
              "target": "$DESTINATION_STORAGE_NAME"
            }
          },
          "deleteAfterUpload": true
        }
      }
    }
  }
}

We will use this deployment.template.json in upcoming steps to create a deployment manifest using tooling in Visual Studio Code. Notice that the multi-platform image (toolboc/camerataggingmodule:latest) is referenced in the deployment.template.json specification.

We also configure a default RTSP stream using the Big Buck Bunny RTSP stream from Wowza. This is the most reliable publicly accessible RTSP stream on the entire internet, and is provided as an example (trust me, reliable public RTSP streams are very difficult to find).

There is also an azureblobstorageoniotedge module that is included which can allow us to save captured images locally and replicate them to the cloud using the IoT Edge Blob Storage Module. This module is useful in private networks where RTSP streams may be available, but outside internet access is not. With this module configured, we could deploy an IoT Edge device into the environment to capture images, then retrieve it and publish the images to the cloud when outbound network access is restored to the device. This will be covered in more detail later in the next steps.

Configure the Blob Storage Module

In this step, we will configure the IoT Edge Blob Storage Module which can be used in conjunction with the CameraTaggingModule to store image captures locally and replicate them to the cloud. Technically, this module is optional and the CameraTaggingModule can upload images directly to the cloud or CustomVision.AI without it, but it gives a more robust solution for the end user that can capture and store images without the need for outbound internet access.

Open the project created in the previous step and create a file named .env and supply it with the following contents:

CONTAINER_REGISTRY_NAME=
LOCAL_STORAGE_ACCOUNT_KEY=
LOCAL_STORAGE_ACCOUNT_NAME=camerataggingmodulelocal
DESTINATION_STORAGE_NAME=camerataggingmodulecloud
CLOUD_STORAGE_CONNECTION_STRING=

This file will will store key/value that are used to replace values in deployment.template.json to produce a working deployment manifest. You will notice these entries in the deployment.template.json are preceeded with the '$' symbol. This marks them as tokens for replacement during the generation of the deployment manifest.

For now, we will skip the CONTAINER_REGISTRY_NAME as that is only needed if you are pulling container images from a private repository, since the modules in our deployment are all publicly available, it is not needed at this time.

Produce a value for LOCAL_STORAGE_ACCOUNT_KEY by visiting GeneratePlus. This will generate a random base64 encoded string that will be used to configure a secure connection to the local blob storage instance. You will want to supply the entire result, which should end with two equal signs (==).

LOCAL_STORAGE_ACCOUNT_NAME is best left as-is, but you are welcome to rename it, provided that it follows the format for naming: The field can contain only lowercase letters and numbers and the name must be between 3 and 24 characters.

DESTINATION_STORAGE_NAME is supplied from an asssumed-to-exist blob storage container in the Azure Cloud. You can create this container by performing the following steps:

Navigate to the Azure Marketplace and search for 'blob', then select Storage Account - blob, file, table, queue

Create the Storage Account using settings similar to below (note: the Storage account name must be globally unique)

Select Review + Create => Create to deploy the new Storage Account Resource.

Navigate to your newly deployed Storage Account and select Containers:

Create a new storage container named "camerataggingmodulecloud" as shown below (the name is important as it matches the value in the .env):

CLOUD_STORAGE_CONNECTION_STRING can be obtained by visiting your newly created Storage Account and selecting Settings => Access Keys. Copy the entire contents of the Connection string and supply this as the value.

Your completed .env file should look similar to the following:

CONTAINER_REGISTRY_NAME=
LOCAL_STORAGE_ACCOUNT_KEY=9LkgJa1ApIsISmuUHwonxg==
LOCAL_STORAGE_ACCOUNT_NAME=camerataggingmodulelocal
DESTINATION_STORAGE_NAME=camerataggingmodulecloud
CLOUD_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=camerataggingmodulestore;AccountKey=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000==;EndpointSuffix=core.windows.net

One final and very important step, the Azure IoT Edge Blob Storage module runs as a special user with uid (1100). This is alluded to in the official documentation and requires some configuration on the host in order for the local blob storage to be able to write to the host directory. I found that I had to perform this step when deploying to an AMD64 device but it did not appear to be required for AARCH64 devices, your experience may vary.

We will simply create the expected directory that is specified as a bind mount in our deployment.template.json for **azureblobstorageoniotedge* and assign it owner and group permissions for the user that the IoT Edge Blob Storage module runs as. This will allow the module to write to this directory on the host filesystem.

sudo mkdir /data
sudo mkdir /data/containerdata
sudo chown -R 11000:11000 /data/containerdata
sudo chmod -R 700 /data/containerdata

We are now ready to generate the deployment manifest.

Generate and apply the deployment to the IoT Edge device

In this section, we will use the .env file to produce a deployment manifest and then apply it to our IoT Edge device. This deployment will work as-is across ARM64, ARM32, and AMD64 devices as it will reference platform agnostic image tags for all of the included modules (edgeAgent, edgeHub, CameraTaggingModule, azureblobstorageoniotedge).

Within Visual Studio Code, right-click the deployment.template.json file and select Generate IoT Edge Deployment Manifest

This will create a new folder named config and within it there should be deployment manifest created and namded selected architecture. Remember, while the name may imply that this is a platform-specific deployment, this particular deployment is platform agnostic and will work on ARM64, ARM32, or AMD64 platforms.

Right-Click this file and select Create Deployment for Single Device

You will be prompted to select a device that has been registered to your IoT Hub. Choose the appropriate device and the deployment will be applied. In a few moments, your device will begin running the supplied modules (edgeAgent, edgeHub, CameraTaggingModule, azureblobstorageoniotedge).

Using the Deployed CameraTaggingModule

In this section we will explore features of the CameraTaggingModule and demonstrate how you can use it to capture images for use in training object detection and image classification models at CustomVision.AI.

The CameraTaggingModule is configured to expose a web service on port 3000 of the IoT Edge device. You can interact with it by visiting http://<devicehostname>:30000 or http://<deviceipaddress>:3000 on a compatible web browser. The latest version of Microsoft Edge, Firefox, and Chrome all worked for me.

Upon loading, the interface should present you with a playback of Big Buck Bunny RTSP stream:

We can change the current RTSP stream by selecting Change Camera, then supply a Camera Name and associated RTSP Address and Add Camera. Next, choose the newly added camera from the drop-down and select Save.

Here is an example of changing the RTSP stream to one of my home security cameras:

Now we can begin capturing images by selecting Capture then you can supply it with a tag or tags and name the image:

Once you have collected a variety of images, they can be reviewed in the Images panel:

Uploading Tagged Images to CustomVision.AI

You will need to have an existing project set up at CustomVision.AI. You may refer to the following quickstarts to create an image classification model or object detection model.

Within the CameraTaggingModule portal, select Upload Settings:

Next, select Custom Vision and you will be presented with the following configuration screen:

To obtain these values, navigate to your CustomVision.AI project and select the "Gear" icon. This will present you with a screen that looks like the following:

Once these values are entered, you will be presented with a dropdown of your existing projects. Choose the intended project you wish to upload your images to and select Push to Custom Vision. After a while you should see Success.

Head back to your CustomVision.AI project to see your newly uploaded and tagged training images:

Now you see just how it easy it can be to use the CameraTaggingModule to gather images from site deployments. Next, we will show how to leverage the Blob storage features to capture images locally and replicate to the cloud for long-term storage.

Using the Blob Storage features of the CameraTaggingModule

The CameraTaggingModule allows for storing image captures locally, where they can then be replicated to Azure Blob Storage, and also allows for direct upload to Azure Blob Storage (bypassing the need to store locally). As mentioned, this is highly useful in environments that do not allow for transmission of data to the outside world. With this feature, you can capture data from air-gapped environments for archival in the cloud.

Within the CameraTaggingModule portal, select Upload Settings:

Next, select Blob Storage and you will be presented with the following options:

To upload directly to blob storage, select the Push to Blob Storage option. You will be presented with a screen asking to supply a Storage Connection String:

You can obtain the string just as we did in previous steps:

Once supplied, you should see a drop-down under Container Name. Choose the storage container that was created earlier then Push to Blob Store.

You can verify that your images have been uploaded to the cloud by visiting your Storage container in the Azure Portal:

Connecting to the local Blob Storage Instance with Azure Storage Explorer

We will now show how you can connect to the local Blob Storage instance using the Azure Storage Explorer. This will make it very easy to view the state of our local blob storage and easily perform operations on it's data.

First, let's store some image data into the local blob storage.

Within the CameraTaggingModule portal, select Upload Settings:

Next, select Blob Storage and you will be presented with the following options:

To upload directly to the local blob storage, select the Push to Local Storage option.

Next, we will install the Azure Storage Explorer

On my Ubuntu 18.04 machine, I accomplished this with the following:

sudo snap install storage-explorer
snap connect storage-explorer:password-manager-service :password-manager-service

Once installed, you can launch the Storage Explorer with:

storage-explorer

Select the Open Connect Dialog icon that looks like a electrical plug and select Use a Connection String:

Enter camerataggingmodulelocal for the Display name then supply a Connection string in the following format:

DefaultEndpointsProtocol=http;BlobEndpoint=http://localhost:11002/camerataggingmodulelocal;AccountName=camerataggingmodulelocal;AccountKey=9LkgJa1ApIsISmuUHwonxg==

If you used a different value for LOCAL_STORAGE_ACCOUNT_KEY in your .env file, be sure to replace it with the appropriate value as shown:

Once you are configured appropriately, select Connect:

Similarly, you can repeat this process and add the Connection String used for CLOUD_STORAGE_CONNECTION_STRING in your .env file to have access to the Azure Cloud Storage Instance we created earlier.

Now you can explore the contents of the local and cloud blob storage containers:

You may notice that the local container is empty, if the IoT Edge device has network connectivity to Microsoft Azure, it will delete uploads based on the default desired properties configuration in deployment.template.json:

    "azureblobstorageoniotedge":{
      "properties.desired": {
        "deviceAutoDeleteProperties": {
          "deleteOn": false,
          "retainWhileUploading": true
        },
        "deviceToCloudUploadProperties": {
          "uploadOn": true,
          "uploadOrder": "OldestFirst",
          "cloudStorageConnectionString": "$CLOUD_STORAGE_CONNECTION_STRING",
          "storageContainersForUpload": {
            "$LOCAL_STORAGE_ACCOUNT_NAME": {
              "target": "$DESTINATION_STORAGE_NAME"
            }
          },
          "deleteAfterUpload": true
        }
      }
    }

For details on how to modify these settings further, refer to the official README in the camerataggingmodule github repository.

Automating the CameraTaggingModule with DirectMethods

Direct Methods are a feature of Azure IoT Hubs that can provide remote access to local methods defined in module code in a secure manner. The CameraTaggingModule exposes methods that include image capture, uploading captured images to CustomVision.AI, pushing captured images to local blob storage, and deleting all captured images. By combining these elements together, we can automate an image capture process across any number of available RTSP feeds.

Here is an example bash script capture.sh that can trigger the execution of the capture method:

#!/bin/bash

#Configuration
iothubName=iothubname
deviceId=deviceId
#Generated with 'az iot hub generate-sas-token -n <iothubName> -du 31536000'
SharedAccessSignature='SharedAccessSignature sr=iothubname.azure-devices.net&sig=x&se=x&skn=iothubowner'

usage(){
  echo "***Camera Tagging Module Capture Script***"
  echo "Usage: ./capture.sh <rtsp_ip> <rtsp_port> <rtsp_path>"
}

capture(){
curl -X POST \
  https://$iothubName.azure-devices.net/twins/$deviceId/modules/CameraTaggingModule/methods?api-version=2018-06-30 \
  -H "Authorization: $SharedAccessSignature" \
  -H 'Content-Type: application/json' \
  -d "{
    \"methodName\": \"capture\",
    \"responseTimeoutInSeconds\": 200,
    \"payload\": {
        \"RTSP_IP\":\"$rtsp_ip\",
        \"RTSP_PORT\":\"$rtsp_port\",
        \"RTSP_PATH\":\"$rtsp_path\",
        \"TAGS\":[\"automatedCaptures\"]
    }
  }"
}


# Arguments
rtsp_ip=$1
rtsp_port=$2
rtsp_path=$3

# Check Arguments
[ "$#" -ne 3 ] && { usage && exit 1; } || capture

And here is an example bash script push.sh that can trigger pushing to the local blob storage:

#!/bin/bash

#Configuration
iothubName=iothubName
deviceId=deviceId
#Generated with 'az iot hub generate-sas-token -n <iothubName> -du 31536000'
SharedAccessSignature='SharedAccessSignature sr=iothubname.azure-devices.net&sig=x&se=x&skn=iothubowner'

curl -X POST \
  https://$iothubName.azure-devices.net/twins/$deviceId/modules/CameraTaggingModule/methods?api-version=2018-06-30 \
  -H "Authorization: $SharedAccessSignature" \
  -H 'Content-Type: application/json' \
  -d '{
    "methodName": "push",
    "responseTimeoutInSeconds": 200,
    "payload": {
        "MODULE_NAME":"azureblobstorageoniotedge",
        "STORAGE_PORT":"11002",
        "ACCOUNT_NAME":"camerataggingmodulelocal",
        "ACCOUNT_KEY":"jukoPNlrFwXR/eELSxryaw==",
        "DELETE":"true"
    }
  }'

By combining these together, you can create a bash script automateCapture.sh to automatically gather images in a loop and store them to the local blob storage:

#!/bin/bash

while true; do
        ./capture.sh 'username:password@rtsp_ip' rtsp_port rtsp_path
        sleep 15
        ./capture.sh 'username:password@rtsp_ip' rtsp_port rtsp_path
        sleep 15
        ./capture.sh 'username:password@rtsp_ip' rtsp_port rtsp_path
        sleep 15
        ./capture.sh 'username:password@rtsp_ip' rtsp_port rtsp_path
        sleep 15
        ./push.sh
done

Now, if you've followed along up until this point, you might realize that we've just laid out the process for a system that can capture training samples automatically. This means we can gather samples from various times of day in our natural environment without any manual effort!

What's even more amazing is that the system can work even if outbound network access is cut off to the device! That's right! The azureblobstorageoniotedge module will keep a copy locally until network connectivity is achieved where it will then drop the data into an Azure Cloud Storage container. If the device is configured and networked to a Transparent IoT Edge Gateway, the device can even continue to receive Device Method invocations from the cloud! If you are interested in learning more about this concept, there is an interactive module at Microsoft Learn that walks through the process of configuring an IoT Edge Gateway. This is a prime example of Azure IoT Edge's powerful capabilities for orchestrating workloads at the edge, it can continue to work even when the device it is running on is disconnected from the internet!

Conclusion

We have demonstrated a variety of features present in the Azure IoT Edge Camera Tagging module. With this guide, you should be able to successfully deploy and capture images for use in training custom image classification and object detection models using CustomVision.AI.

As you have seen, this is an extremely powerful tool that can allow you to gather training samples from the actual site of deployment, potentially while a solution is even in production! Hopefully this guide has made it very easy to understand how all of the pieces work together and allow you the freedom to deploy it for your own scenarios.

I am personally excited about the prospect of generating samples using my home security cameras to detect my dog, cat, vehicles and family members. Armed with this knowledge, it becomes possible to create a highly accurate system that can be queried to produce insights like "How long has the car been in the driveway?", "Where is the last place the cat was seen?", and "How many cars drove past the driveway last week?".

As the paradigm of AIoT continues to flourish, it is my expectation that we will begin to see consumer systems that are capable of tailoring computer vision workloads to specific environments. The Azure IoT Edge Camera Tagging module is an excellent tool that makes this process that much closer to a reality. If you are interested in developing computer vision solutions at the Edge, I highly recommend it!

Until next time,

Happy Hacking!

-Paul

Building jetson-containers for Nvidia devices on Windows 10 with VS Code and WSL v2

Paul DeCarlo — Wed, 31 Jul 2019 20:30:59 +0000

In this post, we are going to walk through building Ian Davis's jetson-containers project on Windows 10 using Visual Studio Code and version 2 of the Windows Subsystem for Linux. In a nutshell, the jetson-containers project allows you to build CUDA compatible images for running GPU accelerated applications as containers. The project supports all current Nvidia Jetson devices (Nano, TX2, Xavier etc.) and also supports third party carrier boards like the Orbitty Carrier for Nvidia Jetson TX2/TX2i/TX1.

Containerizing Nvidia CUDA drivers and the necessary runtime dependencies is exciting because it can be coupled with a container orchestrator solution like Azure IoT Edge to create cloud configurable IoT deployments which take advantage of GPU acceleration capabilities on Nvidia Jetson devices. This opens up a realm of possibilities for computer vision, ML processing, and other AI workloads that can now be deployed into edge environments and updated remotely using a cloud-defined configuration. In my opinion, this will open the world up to a new paradigm of AI solutions that will be deployed into areas never before considered and backed by first class software lifecycle support using IoT services available in Microsoft Azure.

Step 1: Installing the Windows Subsystem for Linux on Windows 10

With Windows 10, Microsoft has released a new feature named the Windows Subsystem for Linux (WSL). This feature allows you to run a bash shell directly on Windows in an Ubuntu-based environment. Within this environment you can cross compile images for AARCH64 without the need for a separate Linux VM or server. Note that while WSL can be installed with other Linux variants, such as OpenSUSE, the following instructions have only been tested with Ubuntu.

This feature is not supported in versions of Windows prior to Windows 10. In addition, it is available only for 64-bit versions of
Windows.

Full instructions to install WSL are available at the official Microsoft Docs page for WSL.

To install WSL on Windows 10 with Fall Creators Update installed (version >= 16215.0) do the following:

1). Enable the Windows Subsystem for Linux feature

Open the Windows Features dialog (OptionalFeatures.exe)
Enable 'Windows Subsystem for Linux'
Click 'OK' and restart if necessary

2). Install Ubuntu

Open Microsoft Store and search for "Ubuntu 18.04" or use this link
Click Install

3). Complete Installation

Open a cmd prompt and type "Ubuntu1804"
Create a new UNIX user account (this is a separate account from your Windows account)

Step 2: Upgrade Windows Subsystem for Linux to v2

WSL 2 is a new version of the architecture that powers the Windows Subsystem for Linux to run ELF64 Linux binaries on Windows. Its primary goals are to increase file system performance, as well as adding full system call compatibility. This new architecture changes how these Linux binaries interact with Windows and your computer’s hardware, but still provides the same user experience as in WSL 1 (the current widely available version). Individual Linux distros can be run either as a WSL 1 distro, or as a WSL 2 distro, can be upgraded or downgraded at any time, and you can run WSL 1 and WSL 2 distros side by side. WSL 2 uses an entirely new architecture that uses a real Linux kernel.

Support for WSL v2 requires installation of WSL v1 as instructed in the previous section.

Full instructions for upgrading to WSL v2 are available at the official Microsoft Docs page for WSL v2.
To install WSL v2 on Windows 10 (version >= 18917.0) do the following:

1). Enable the Virtual Machine Platform feature

Open the Windows Features dialog (OptionalFeatures.exe)
Enable 'Virtual Machine Platform'
Click 'OK' and restart if necessary

2). Set Ubuntu1804 to be backed by WSL 2

Open Powershell and run:

    wsl --set-version Ubuntu-18.04 2

3). Verify Ubuntu-18.04 is using WSL v2

Open Powershell and run:

    wsl --list --verbose

Verify that the output looks like this:

      NAME            STATE           VERSION
    * Ubuntu-18.04    Running         2

After confirming that you are running Ubuntu-18.04 using WSL v2, you are ready to begin following the "Cross-Compilation" steps below.

Step 3: Setup Cross-Compilation Environment on Ubuntu using Windows Subsystem for Linux v2

We will use the Ubuntu 18.04 WSL v2 environment to cross-compile AARCH64 compatible jetson-containers images capable of running on Nvidia Jetson hardware.

1). Install the Nvidia SdkManager into the WSL v2 Environment

From your Windows host OS, obtain the .deb installer for the Nvidia SdkManager from developer.nvidia.com

Note: This requires that you have valid registered account @ developer.nvidia.com
Place the downloaded .deb installer into c:\sdkmanager
Open a WSL v2 compatible instance of Ubuntu-18.04
Install the sdkmanger by running the following on the bash prompt of the WSL v2 instance:

    sudo apt update
    sudo apt install -y  libcanberra-gtk-module libgconf-2-4 libgtk-3-0 libxss1 libnss3 xvfb  
    sudo dpkg -i /mnt/c/sdkmanager/sdkmanager_*.deb

2). Install Cross-Compilation tools for AARCH64 support

Install the dependencies needed for cross-compilation by running the following on the bash prompt of the WSL v2 instance:

    sudo apt install -y build-essential qemu-user-static binfmt-support

3). Install Docker

Install a Linux-native instance of Docker by running the following on the bash prompt of the WSL v2 instance:

    sudo apt install -y curl
    curl -fsSL https://get.docker.io | bash
    sudo usermod -aG docker $USER

Step 4: Building Jetson Containers using Windows Subsystem for Linux and Visual Studio Code

After you have setup the Cross-Compilation environment on Ubuntu using Windows Subsystem for Linux v2, you are now ready to beging building jetson-containers.

1). Configure Visual Studio Code

Install Visual Studio Code onto the Windows host OS
Install and enable the Remote WSL extension from the Visual Studio Marketplace

2). Start docker and binfmt-support services

These services must be manually started each time a new instance of WSL v2 is started on the host machine. To start them, execute the following on the bash prompt of the WSL v2 instance:

  sudo service docker start
  sudo service binfmt-support start

Note: You may want to add these lines to your ~/.bashrc file if you wish to start these services automatically when starting a new WSL v2 instance

3). Clone jetson-containers project into WSL v2 environment and Open in VS Code

Clone the jetson-containers project and open in VS Code by running the following on the bash prompt of the WSL v2 instance:

  git clone https://github.com/idavis/jetson-containers.git
  cd jetson-containers
  code .

4). Run build tasks in VS Code to create jetson-containers images

In the previous step, a new Visual Studio Code instance should have opened. If not, open a new instance of VS Code on the host OS. Copy the include .envtemp to .env and supply appropriate values according to the included README.md. Next, while inside of the VS Code instance, press "CTRL+SHIFT+B" to bring up a list of available build tasks and select one to begin building the associated jetson-containers image(s)

Note: It is required to build one of the <jetpack-depencies> dependency images before building subsequent containers. This will require that you have an active account from https://developer.nvidia.com and that you have specified that user as the value of NV_USER in the .env file you created. Please be aware that you will be prompted for the account password during the retrieval of the sdks during the build process for any of the tasks.

Conclusion

It is amazing to see that the Windows Subsystem for Linux v2 is capable of building native AARCH64 compatible containers with CUDA support from Visual Studio Code. WSL v2 brings a fully-compatible Linux Subsystem with 100% Linux syscall compatibility, allowing to build basically anything that you can build in a native Linux environment. When coupled with the Remote WSL extension , we can perform builds against the Linux Subsystem from Windows 10 allowing for a truly cross-platform development experience.

What is also amazing is that Ian Davis has curated every currently available Jetson platform into his jetson-containers project, allowing us to build containers for a completely foreign architecture (ARM64) that are compatible with Nvidia GPU accelerated hardware. Since WSL v2 can allow us to build pretty much anything that Linux can, we can now bring this ability to develop GPU accelerated container solutions to developers on Windows using VS Code with WSL v2.

As development of jetson-containers progresses, you can expect to see more details on how to use it in my upcoming articles. For a look at related content on dev.to, check out Supercharge your containerized IoT workloads with GPU Acceleration on Nvidia Jetson devices, Getting Started with IoT Edge Development on Nvidia Jetson Devices, Getting started with DevOps CI / CD Pipelines on Nvidia ARM64 Devices, and Using Cognitive Services Containers with Azure IoT Edge

Until next time, Happy Hacking!