Forem: Kouji Matsui

MapLibre GL JS - Enabling large number of moveable sprites

Kouji Matsui — Wed, 03 Dec 2025 08:31:15 +0000

MapLibre's layer extension library enabling the display, movement, and modification of large numbers of dynamic sprite images.

Features:

Place, update, and remove large numbers of sprites.
Move each sprite's coordinate freely, making it easy to represent moving objects.
Specify per-sprite anchor positions for precise rendering.
Add multiple images and text to the same sprite, adjusting rotation, offset, scale, opacity, and more.
Animate sprite movement, rotation, and offsets with interpolation controls.
Control draw order via sub-layers and per-sprite ordering.
Fully imperative APIs. Updates with high-performance and extensible.
Accelerating computational processing with WASM and shaders.
MIT license.

Easy to implement video image capture with FlashCap

Kouji Matsui — Sat, 23 Apr 2022 12:19:52 +0000

Introduction

Have you ever wanted to take a video captured image and add your own custom operations to it? Have you ever looked for a library that makes it easy to implement such custom operations?

Applications that directly manipulate physical devices, such as video capture devices, create restrictions on the libraries that can be used for such operations. .NET, you need to attach native interoperability libraries. This is not a problem for hobby projects that you use only by yourself, but when you get to the stage of creating a package of applications and libraries for someone else to use, a problem emerges. For example:

Native libraries need to be compiled in C++ or similar for each of their platforms. For Windows, you may need to have separate ones for 32-bit and 64-bit, and possibly for the ARM platform in the near future. For Linux, the number of target platforms will further increase.
Packaging native libraries requires special knowledge. There may be a need to place library files in the proper locations and possibly set up custom code to load them dynamically.

In Jan 2022, my client's requirements drove the need for a camera device on Windows to take still images and perform image processing and image recognition. My initial assumption was to use a USB-connected camera (a device that supports USB Video Class) to take the still image. I wrote "still images," but in reality we also need to preview the images, so they need to be handled as video stream.

If we just want to record video, there are excellent libraries, applications, and UI controls for recording video. But at the same time, we needed such a library that would be customizable to do its own processing in the background for a single still image during the video preview, and that would have an easy and simple interface and clear packaging issues. (Sometimes referred to as "A frame grabber" as a general term.)

If we don't have it, let's make it. This is how FlashCap was born.

Note: FlashCap was born out of work, but is a fully OSS project. License follows Apache-v2.

Goals

My goal was to solve the following problems.

The interface structure should be easy enough for anyone to write code.
Extensibility to allow very low-level operations if necessary.
Support a wide range of platforms.
The higher the .NET compatibility, the better.
Not dependent on other native and/or managed libraries as much as possible.

For 3, as of the official version (1.0.0):

Windows (DirectShow API)
Windows (Video For Windows API)
Linux (V4L2 API)

are now supported. (Perhaps it can't be called multi-platform unless it supports more environments ;) For now, my personal interest is in the Android NDK.)

In addition, the following PCs have been verified:

Windows (x86, x64)
Linux (i686, x86_64, aarch64, armv7l, mipsel)

I have verified with various cameras and video capture devices. Not all combinations have been covered, but we believe this is sufficient for a first version. For a detailed list, please refer to the repository documentation.

4 also supports the current major versions of .NET (.NET 6,5, Core 3, Core 2, Framework 4,3, and Standard). If you have ever struggled with package dependencies, you will find comfort in the fact that it is so comprehensive.

Usage

So how "easy" is it to write code? First, install the FlashCap package from NuGet. An example code is shown below. It lists the video capture devices and what characteristics (resolution, FPS, format) each device has:

using FlashCap;

// Capture device enumeration:
var devices = new CaptureDevices();

foreach (var descriptor in devices.EnumerateDescriptors())
{
    // "Logicool Webcam C930e: DirectShow device, Characteristics=34"
    // "Default: VideoForWindows default, Characteristics=1"
    Console.WriteLine(descriptor);

    foreach (var characteristics in descriptor.Characteristics)
    {
        // "1920x1080 [JPEG, 30fps]"
        // "640x480 [YUYV, 60fps]"
        Console.WriteLine(characteristics);
    }
}

There is no need to determine the operating environment separately for Windows and Linux. It will automatically determine the environment internally and use the appropriate API.

Once you have narrowed down the device and its characteristics, all you have to do is open the device and start capturing:

// Open a device with a video characteristics:
var descriptor0 = devices.EnumerateDescriptors().ElementAt(0);

using var device = await descriptor0.OpenAsync(
  descriptor0.Characteristics[0],
  async bufferScope =>
  {
    // Captured into a pixel buffer from an argument.

    // Get image data (Maybe DIB/Jpeg/PNG):
    byte[] imageData = bufferScope.Buffer.ExtractImage();

    // Anything use of it...
    var ms = new MemoryStream(imageData);
    var bitmap = System.Drawing.Bitmap.FromStream(ms);

    // ...
  });

// Start processing:
device.Start();

// ...

// Stop processing:
device.Stop();

This is where the handler for the lambda expression block specified in the OpenAsync() argument is called each time a capture is made. From the bufferScope.Buffer argument (called the pixel buffer), ExtractImage() will yield a byte[] representing the image data. As in this code example, you can pass it to System.Drawing.Bitmap to generate a bitmap or output it directly to a file, and that completes the capture process.

The image data can be in the following format:

'RGB DIB': a bitmap file format such as the common foo.bmp. The format can vary from 32-bit ARGB to 8-bit palette, but current capture device specifications are almost always 24-bit or 32-bit.
'JPEG': JPEG format. Most of the time it is actually MJPEG (Motion JPEG), but since it is actually identical to JPEG, it can be treated as JPEG.

If you are familiar with the video capture situation, you may wonder what happens to the 'YUV' format.' The 'YUV' format is supported by many capture devices, but is a minor player in image data. Many bitmap decoders cannot decode the 'YUV' format. So, FlashCap will transcode 'YUV' to 'RGB DIB' by default. Of course, if you need raw image data, you can disable transcoding.

By the way, FlashCap handles video. It is not a still picture. Therefore, the above handler is called each time a frame (one still image) of a video is obtained. So, if you want a snapshot, write the code to save one still image at that timing. For example, you could always store the last image in the member field, and then save a reference to it on a click of a button UI, etc.

Topic

It should also be noted that the above handlers are called from the worker thread; WPF, Windows Forms, and other UI frameworks do not allow manipulation of the UI from the worker thread. For example, you may want to preview the resulting image data. Each of them has its own specific method. In the case of WPF, you can use a Dispatcher and write the following:

// Image previewImage;

// here executed in a worker thread
byte[] imageData = bufferScope.Buffer.ExtractImage();
var bitmap = new BitmapImage();
bitmap.BeginInit();
bitmap.CacheOption = BitmapCacheOption.OnLoad;
bitmap.StreamSource = new MemoryStream(imageData);
bitmap.EndInit();
bitmap.Freeze(); // Freeze is necessary because it was generated in a worker thread

// Execute in UI thread
previewImage.Dispatcher.BeginInvoke(
  new Action((() => previewImage.Source = bitmap)));

Overall implementation examples are available in Windows Forms, WPF and Avalonia respectively. Please refer to them.

After touching it for a while, you may be concerned about performance. Image data is very large data, and since it is video, it must be processed dozens of times per second. By default, FlashCap ignores the next image data during processing. This means that "frame dropping" occurs. To solve this, the implementation of the handler must be as fast as possible.

For example, image data can be stored once in a queue and processed in a multi-threaded manner. Such a scenario is also supported by the standard functionality of FlashCap. Add a parameter to the OpenAsync() argument:

// Open a device with a video characteristics:
var descriptor0 = devices.EnumerateDescriptors().ElementAt(0);

using var device = await descriptor0.OpenAsync(
  descriptor0.Characteristics[0],
  true,
  true,    // <-- Enable multithreading (Scattering)
  10,      // <-- Maximum queuing pixel buffers
  async bufferScope =>
  {
    // ...
  });

Specifies the maximum amount of image data queue, and whether it should be multithreaded or not. (This is called Scattering.) It is possible to remain single-threaded and increase only the maximum queue amount. Even with single threading, if the handler processing load is unstable, frame dropping may be avoided by increasing the maximum queue size. Note that in both cases, frame dropping will occur if the image data exceeds the maximum queue size.

Options are also available for those who wish to optimize processing further. If you implement your own "frame processor," you can handle almost any raw image data directly. The interface is simple but delicate and is not likely to be used in general applications. See the documentation for more details.

Conclusion

I have created a FlashCap library that completely conceals the video capture device, yet allows programmable device coverage and format selection, has a simple and easy interface, is multi-platform, and eliminates dependencies on external libraries such as native libraries. .NET runtime version requirements as broadly as possible, and also provides interface extensibility for faster capture.

You can use FlashCap to easily obtain image data with a webcam in front of you, or to implement features that are essential for sophisticated applications, such as processing the resulting image while displaying a preview.

I hope this will be helpful to everyone.

An easy way to practice version embedding on .NET

Kouji Matsui — Tue, 22 Mar 2022 05:00:05 +0000

Introduction

Have you ever wanted to visualize the version number of your application when you distribute it? There are many possible reasons:

To display the version of the application when it is launched.
To include it in diagnostic information for smooth support (and possibly output it to a log)
Other...

From my many years of experience in software support and maintenance, I have come to believe that self-diagnostic information is important when a problem occurs. One of those materials is the version number.

So, how should we manage version numbers in a .NET application? For example, consider the following console application:

// No command line arguments are given.
if (args.Length == 0)
{
  // Display help and exit
  Console.WriteLine("FooBarNet 1.0.0");  // <-- Version number
  return 0;
}

For very small code base, this method of hard-coding and embedding the version code is a good idea. However, in general, it is easy to make mistakes when thinking about releasing the application:

Forgot to update the version number.
You gave the wrong version number.
etc...

The problem does not stop with released application binaries. Another headache is the difficulty in identifying the source code version that corresponds to the binary version number.

Cooperating partners I have seen have done things like "freezing" the source code at the time of product release and compressing it, along with the compiled binaries, into a ZIP file or saving it to external storage media and labeling it.

"No, no way, in this day and age?" You might think :)

So how do we solve this problem in a "modern" way?

For source code, it is probably common practice to use a source code management system such as Git or Subversion. Some teams may store binaries at the time of release together in these repositories.

That is one way to do it, but it has the disadvantage of increasing the size of the repository. Also, there remains the problem that the first diagnostic step is not resolved when an application problem occurs.

Automatically embed version

The RelaxVersioner project described in this article is one way to solve this problem. Simply install this NuGet package in your project and automatically embed the version number in your application. No big deal, just install. It should be enough for you to try it out and see what happens.

If you want, you can also easily reference the version number as follows:

// Refer to the version number:
Console.WriteLine($"FooBarNet {ThisAssembly.AssemblyVersion}");

Looking at this code, the following questions may arise:

Doesn't the version number refer to the AssemblyVersionAttribute?
Doesn't the version number use the reflection API?
How is the version number determined in the first place?

There are several types of version numbers in .NET, and of course, RelaxVersioner applies these standard attribute classes as well. On top of that, it also defines literal strings, as shown above, for ease of use in programmatically referencing versions.

.NET standard attribute classes, so that information can be displayed even when the application binary is viewed from the Explorer:

More detailed information can be visualized by using a decompilation tool such as ILSpy:

I will then explain the core of the last question:

How is the version number determined?

If you can apply a version number simply by installing the NuGet package of RelaxVersioner, how is that version number determined?

Let us reveal the seed. It refers to a tag in the Git repository. That is, RelaxVersioner refers to the local Git repository where the project is located and uses the tag with the version number applied. If there is a tag of the form "1.2.3" or "v1.2.3" separated by dots, as shown below, the commit is recognized as that version:

Screenshots are from GitKraken.

Then, what happens in a commit without a tag is that the trailing value of the version number, separated by a dot, is automatically incremented. In the example above, a commit on a branch derived from 0.5.0 would be calculated to 0.5.1 or 0.5.3.

Let us show a slightly less simple example. Where a merge commit occurs, the version number increment is computed on the primary parent commit first, as in the following example:

Therefore, it is 0.5.1, not 0.5.6. You may think, "This would result in duplicate version numbers!", but the question is whether the application has been released or not. Once you have released your application, add a tag to it. Then the calculation will be based on the tag number.

In this case, if you want the commit in the red box to be 0.5.6, add the tag "0.5.6" to this commit. There must be a reason why you want the version number to be 0.5.6 intentionally. Perhaps you want to release it, or have released it. The term "release" here is not limited to public, but private releases as well.

This specification in RelaxVersioner is based on the idea that Git's local repository exists for the developer, not for release management. We wanted developers to be able to manipulate the commits of their own free (freedom) will, unless they decide to "release this commit. This way, you can enjoy the flexibility of Git while at the same time managing releases.

How is the version number embedded?

Some of you may have questions about how this can be accomplished simply by installing the NuGet package of RelaxVersioner. In this section, we will briefly explain how RelaxVersioner actually embeds the version number.

First, check if the directory where the .NET project exists is under a local repository in Git.
then search the Git local repository for the current checkout commit. If no commits exist at all, set the version number to 0.0.1. (either a newly created Git repository or you didn't git init it in the first place)
if the commit exists and a tag corresponding to the version number is found, use that version number. If the commit exists but does not have a tag, the tag corresponding to the version number is found by traversing the parent commits in order.
based on the found version number and the MSBuild property information group, the template source code is generated based on the rule file. The default rule file is built in, so if it does not exist, use the default. The template source code generates C#, F#, VB.NET, and C++/CLI code in a temporary directory according to the language used in the project.
ensure that the template source code is included in the MSBuild build target.

These steps are performed by MSBuild scripts and the RelaxVersioner command line tool. When the tool runs at build time, you will see a message similar to the following:

1>RelaxVersioner[2.5.5]: Generated versions code: Language=C#, Version=0.7.12
1>FlashCap -> C:\Projects\FlashCap\FlashCap\bin\Debug\net6.0\FlashCap.dll

If your project is multi-platform, you will see many of the same messages in the build log, which is not unusual, because RelaxVersioner generates template source code for each platform.

And the template source code is as follows (in C#) :

using System.Reflection;
[assembly: AssemblyVersion("1.0.21")]
[assembly: AssemblyFileVersion("2020.12.20.33529")]
[assembly: AssemblyInformationalVersion("1.0.21-561387e2f6dc90046f56ef4c3ac501aad0d5ec0a")]
[assembly: AssemblyMetadata("Date","Sun, 20 Dec 2020 09:37:39 GMT")]
[assembly: AssemblyMetadata("Branch","master")]
[assembly: AssemblyMetadata("Tags","")]
[assembly: AssemblyMetadata("Author","Kouji Matsui <k@kekyo.net>")]
[assembly: AssemblyMetadata("Committer","Kouji Matsui <k@kekyo.net>")]
[assembly: AssemblyMetadata("Message","Merge branch 'devel'")]
[assembly: AssemblyMetadata("Build","")]
[assembly: AssemblyMetadata("Generated","Sun, 20 Dec 2020 09:37:43 GMT")]
[assembly: AssemblyMetadata("Platform","AnyCPU")]
[assembly: AssemblyMetadata("BuildOn","Unix")]
[assembly: AssemblyMetadata("SdkVersion","5.0.101")]

namespace YourApp
{
  internal static class ThisAssembly
  {
    public const string AssemblyVersion = "1.0.21";
    public const string AssemblyFileVersion = "2020.12.20.33529";
    public const string AssemblyInformationalVersion = "1.0.21-561387e2f6dc90046f56ef4c3ac501aad0d5ec0a";
    public static class AssemblyMetadata
    {
      public const string Date = "Sun, 20 Dec 2020 09:37:39 GMT";
      public const string Branch = "master";
      public const string Tags = "";
      public const string Author = "Kouji Matsui <k@kekyo.net>";
      public const string Committer = "Kouji Matsui <k@kekyo.net>";
      public const string Message = "Merge branch 'devel'";
      public const string Build = "";
      public const string Generated = "Sun, 20 Dec 2020 09:37:43 GMT";
      public const string Platform = "AnyCPU";
      public const string BuildOn = "Unix";
      public const string SdkVersion = "5.0.101";
    }
  }
}

As you can see, each attribute applies and defines members that can be referenced as literal strings.

This source code is placed in a temporary directory. Generally, it is in "obj/" in the project directory. Thus, this source code will not cause unwanted contamination of the commits in the Git repository. And because the entire process is fully automated, it can also be used for continuous integration.

Packaging

Did you know that you can easily NuGet package a library project using the dotnet CLI commands? If you have the library project at hand, you can generate it with the following command:

dotnet pack -p:Configuration=Release -o artifacts FooBarComponent\FooBarComponent.csproj

This will create a package file in the artifacts directory, such as FooBarComponent.1.0.0.nupkg. Upload it to nuget.org and you are done. It's that easy.

... Easy, isn't it?

Uploading and publishing a package requires various metadata:

The version number of the package
Project site URL
Name of the authors, etc.
Source code repository URL
etc...

Among these, there are several pieces of information that should be versioned:

Package version
Version information applicable to the binaries (assembly files) contained in the package
Source code repository URL
The corresponding commit ID

The repository URL and commit ID are optional, but if this metadata can be provided, a powerful debugger feature called SourceLink becomes available.

For example, the famous Newtonsoft.Json package, which we have all used, allows you to step into methods in that package and reference implementations.

You can step into methods contained in the package and reference the implementation. IDEs such as Visual Studio can then automatically download source code from GitHub and continue debugging on the fly, as if the entire project existed locally.

To add even more value, you can use RelaxVersioner to apply versioning to NuGet packages completely automatically: install the RelaxVersioner package in your project, and run same as:

dotnet pack -p:Configuration=Release -o artifacts FooBarComponent\FooBarComponent.csproj

If you want to support SourceLink, you will need to add an additional definition to FooBarComponent.csproj, but only once. In other words, once you install RelaxVersioner, you can consistently version control everything from assembly file generation to NuGet package generation using only Git.

Conclusion

.NET applications, "RelaxVersioner" can efficiently manage version numbers using tagging in Git repositories.

This reduces the time and effort required to manage version numbers by hard-coding source code, and also allows you to manage the version of the source code itself. The applied version number can be easily referenced from the code and included in the built binary, allowing the user to reference it in the explorer.