Forem: keaukraine

Automatically picking the best ASTC textures quality

keaukraine — Fri, 29 Aug 2025 14:20:26 +0000

In this small article I’ll describe the simple script I’ve created while working on the Cartoon Lighthouse live wallpaper. You can find a live web demo of this app here and take a look at its source code here.

I’ve encountered a slight problem with encoding 31 textures into ASTC format. The thing is that usually in all our apps geometries are batched and merged to reduce the amount of draw calls. So typically we use a couple of large texture atlases for all geometries. In this scene however geometries were not batched and I decided to keep this as is since the total amount of draw calls is still rather small (around 30). So this resulted in 31 textures which were to be converted into ASTC format.

So usually I just encoded textures with some ASTC block sizes (4x4, 6x6, 8x8 and 10x10) and manually picked the one which looked good enough for my liking. While this was OK for a handful of textures it was quite a cumbersome process for 31 ones. This needed some automation, and fortunately I’ve encountered this wonderful tweet about quality improvements of Spark realtime texture encoder (make sure to check it out, the project is fascinating by itself). What was interesting in it for me is the usage of a tool called SSIMULACRA2. This open-source tool compares two images and gives it a quality score. And the algorithm is specifically tuned to look for both user-perceived and blocky compression-related artifacts in the images! That’s exactly what I needed!

Over the weekend I’ve created a bash script (bash is not my native language so I’ve used ChatGPT to help me here and there) and tested it. The algorithm of the script can be described with this pseudocode:

for each input file  
    for each ASTC block size from the lowest to the highest quality  
        encode image and save its decoded image  
        compare quality between original and decoded images  
        if quality is within threshold encode with this block size

You can optionally specify SSIMULACRA2 score but the default is 70 which can be described as “there are some compression artifacts if you look for them specifically”.

There’s additional logic for the detection of mipmaps - these must be encoded with the same compression so only the highest level is tested for quality.

There’s a limitation of the SSIMULACRA2 tool - it doesn’t validate images smaller than 8x8 pixels so these are encoded with fallback 4x4 block size.

Also I’ve played around with bash output a little so it is not so silent while working - it informs how it processes the current image, and shows scores for the processed images:

This saved not only my time - I didn’t have to check the quality of each file manually - but file sizes too! Because of my laziness I checked manually only a subset of ASTC block sizes but this script relentlessly runs through all 14 blocks supported by ASTC, choosing the best suitable one. And for some smallish textures with quite uniform images even 12x12 block size was good enough.

You can check the source code of the script here on the GitHub and adapt it for your needs. Also it has an imagemagick script to create mip maps.

How to improve MSAA performance of MTKView

keaukraine — Fri, 10 Jan 2025 18:42:26 +0000

The easiest and fastest way to use Metal in MacOS app is to use MTKView. It is a handy wrapper which initializes all low-level stuff under the hood so you can get right to the fun part — implementing actual rendering.

However, because of its simplicity it does have a couple of shortcomings and for some reasons doesn’t provide access to all internal things under its hood. One of these minor inconveniences is the way it initializes multisampled render targets.

To understand why this is important let’s explain how Metal handles MSAA. It supports multiple ways of implementing it:

You can have a multisampled render target and then resolve it to on-screen render target automatically.
You can use this multisampled render target with custom resolve shaders.
On supported hardware, you can omit mutlisampled render target and resolve it automatically directly to the final render target. This will still use multisampled render target but it will be memoryless.
The same approach but with tile shaders to resolve (apply custom tone mapping, etc).

You can find a more detailed explanation of these methods with a sample XCode project in this official Apple documentation article — https://developer.apple.com/documentation/metal/metal_sample_code_library/improving_edge-rendering_quality_with_multisample_antialiasing_msaa

What is of a particular interest for us is the memoryless multisampled render targets. They are very efficient since they are transient and reside only in (extremely fast and tiny) temporary tile memory of GPU. Because of this they don’t use main memory allocations and don’t add up to precious VRAM access bandwidth.

Here is the typical 4x MSAA rasterization process with default render pass created by MTKView:

And here is the same one but using efficient memoryless render target:

Basically, the only difference is that we substitute transient multisampled render target with the memoryless one and this results in a huge improvement in memory allocation and bandwidth. Please note that according to the Metal Feature Set tables memoryless render targets are not supported by old devices. Namely, Intel-based Macs don’t support tiled rendering and cannot use them. But if you target shiny new Apple-silicon devices then you definitely must use them because they are so extremely efficient.

The thing is that (possibly for a better support of all hardware) MTKView initializes MSAA only with classic in-memory render targets — multisampled one for the rendering and and the final one for resolving and presenting result on the screen.

In the aforementioned official Metal MSAA example you can find a proper way of initialization of memoryless MSAA resolve but it doesn’t use this handy MTKView — instead there’s quite a lot of glue code to make it work.

However I’ve found a hacky yet relatively simple and perfectly working way of initializing efficient memoryless MSAA resolve using the default MTKView wrapper view.
Let’s take a look at what configuration options MTKView does provide.
Obviously there’s a sampleCount which will initialize MSAA render targets. Also there are depthStencilPixelFormat and depthStencilStorageMode fields. And you can change depth+stencil to use memoryless storage too by setting depthStencilStorageMode=.memoryless, which also saves a lot of RAM usage and bandwidth in case you don’t need depth information of your frames.
Here’s a typical MTKView initialization code (for an Apple-silicon GPUs, which support memoryless textures):

_metalView.depthStencilPixelFormat = .depth32Float
_metalView.depthStencilStorageMode = .memoryless
_metalView.preferredFramesPerSecond = 60
_metalView.sampleCount = 4 // hard-coded 4 samples but you can query max available samples for GPU and set it accordingly

That’s cool, let’s also switch color render target to the memoryless mode too! Unfortunately for the color render target there is only colorPixelFormat available (typically set up automatically) and there is no colorStorageMode. So there’s no easy way to just set it up to use memoryless MSAA mode.

Still there’s a relatively simple way of switching it to the memoryless mode after it has been initialized!

The thing is that Metal API allows you to change the MSAA resolve texture of the current render pass. The descriptor of this render pass is provided to you by MTKView and obviously it is pre-initialized with in-memory texture.
So all you need to do is on the first frame you draw to create a memoryless render target and substitute the default resolve texture with the new one.

// New memoryless MSAA texture
var textureMsaa: MTLTexture?

.................

func yourCodeToDrawStuff() {
    // Before rendering, create and replace MSAA resolve RTT.
    do {
        let resolveTexture = view.currentRenderPassDescriptor?.colorAttachments[0].resolveTexture
        if resolveTexture != nil {
            let width = resolveTexture!.width
            let height = resolveTexture!.height
            if textureMsaa == nil || textureMsaa?.width != width || textureMsaa?.height != height {
                // Auto-purge the old unused resolve texture
                renderPassDescriptor?.colorAttachments[0].texture?.setPurgeableState(.volatile)
                textureMsaa = try create2DRenderTargetMemoryless(width: width, height: height, pixelFormat: .bgra8Unorm, metalDevice: device)
                textureMsaa?.label = "Main pass RTT"
            }
        }
    } catch {
        fatalError("Cannot create MSAA texture: \(error)")
    }
    // Use new memoryless texture
    view.currentRenderPassDescriptor?.colorAttachments[0].texture = textureMsaa

    // Do you rendering here as usual
    .....................
}

func create2DRenderTargetMemoryless(width: Int, height: Int, pixelFormat: MTLPixelFormat, metalDevice: MTLDevice) throws -> MTLTexture {
    let descriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: pixelFormat, width: width, height: height, mipmapped: false)
    descriptor.textureType = .type2DMultisample
    descriptor.sampleCount = 4 // Yes I use hard-coded 4 samples here too :)
    descriptor.usage = [.renderTarget]
    descriptor.resourceOptions = [.storageModeMemoryless]
    let result = metalDevice.makeTexture(descriptor: descriptor)
    if result != nil {
        return result!
    }
    throw RuntimeError("Cannot create texture with pixelFormat \(pixelFormat) of size \(width)x\(height)")
}

This simple trick effectively substitutes the auto-created in-memory color render target with the memoryless one.

There is one important step to do — you must set the purgeable state to volatile for the old unused render target in order for it to free memory. Otherwise even if it won’t be used it will still keep a large amount of memory allocated for it. This is an extremely powerful and easy-to-use feature of Metal API which I love — if you don’t use some resource, API can get rid of it for you automagically. You don’t have to manually delete them as in OpenGL.

Here are some final memory usage comparisons on a MacBook Air M1 with a full-screen 2560x1600 render target:

First, a default approach — MTKView with 4x MSAA in-memory resolve texture:

This multisampled texture uses 78 MB of memory which is being accessed (both write and read) on every frame!

And here is the memoryless one:

Notice the 78 MB texture is now listed in the unused resources. It actually uses 0 bytes, only listed as a “dormant” 78MB resource which could be re-allocated in case it will be reused again.
This can be confirmed in the Activity Monitor. Before optimization:

And after:

Now my app uses just under 80 MB of total RAM instead of 150! This is a good result for a full-screen 3D app — actually it about as much as just two stock MacOS calculators! (Yes you can check it yourself — Calculator uses ~40MB of RAM which seems to be a bit excessive).

Hope this little tutorial will be useful and will make your Metal app more memory and power-efficient!

Stylized Castle WebGL demo

keaukraine — Sat, 28 Oct 2023 07:07:08 +0000

This is my first attempt to create a kitbashed scene from tiling assets. I have used assets by Kenney and his Asset Forge Deluxe as an editor to create the scene. Result was exported to OBJ files and then converted into ready-to-use buffers with vertex data and indices.

Scene has a distinct stylized look with no textures used except for animated characters — knights and birds. To add depth to the scene, a simple linear vertex fog and real-time shadows are applied.

Total scene polycount with both static and dynamic objects is 95k triangles.

Static geometry

All static objects in the scene are merged into 2 large meshes to reduce the amount of draw calls. These assets don’t use textures, instead vertex colors are used. These colors are stored as indices which allows for an easier customization of color themes. Shaders accept up to 32 colors set via uniforms.

The stride for static objects vertex data is 12 bytes — 3 FP16 values are used for position, 3 normalized signed bytes for normals, and 1 unsigned byte for color. 2 unused bytes are used for 4-byte alignment of data:

I also tried using a more compact packed GL_INT_2_10_10_10_REV type for vertex positions to fit data in 8 bytes. Unfortunately, its precision was just not enough for this purpose. This data type provides precision of roughly 1 meter per 1 km. And since the scene uses quite large geometries batched into 2 meshes this precision was clearly not enough.

Shadow maps

Lighting in the scene is not baked — shadow maps are used for all objects to cast dynamic shadows. Shadow maps have no cascades since the scene is rather small. However, the detail of shadows is adjusted in a different way. The light source FOV is slightly adjusted per each camera to have more detailed shadows for close-up scenes and less detailed for overviews.

Shadow map resolution is 2048x2048 which is sufficient to create detailed enough shadows.

To smooth out hard shadow edges, the hardware bilinear filtering of shadow map is used. Please note that OpenGL ES 3.0+ / WebGL 2.0 is required for this. OpenGL ES 2.0 supports only unfiltered sampling from shadow maps which results in boolean-like comparison whether a fragment is in shadow or not. Hardware filtering is combined with 5-tap sub-texel percentage closer filtering (PCF). This results in smooth shadow edges with a relatively small amount of texture samples.

I also considered a more expensive 9-tap PCF which improved image quality in case of unfiltered shadow texture but with the filtered texture improvement over 5-tap one was negligible. So according to the golden rule of real-time graphics programming “it looks good enough”, the final filtering used in the app is 5-tap PCF with hardware filtering.

Here you can see comparison of different shadow filtering modes:

Performance

To improve performance, a couple of optimizations are used.

First is a quite typical, simple and widely used one — shadow map is updated at half framerate. This is almost unnoticeable since nothing in the scene moves too fast. For cases when the camera and light direction is about to switch to a new position it is rendered at full framerate to prevent 1-frame flickering of shadow map rendered with the old light source.

The second trick is that the PCF is not applied for distant fragments — instead a single sample of shadow texture is used. It is impossible to spot any difference in image quality in the distance because shadows are still hardware-filtered but the performance and efficiency are improved. But isn’t it considered a bad practice to use branching in the shaders? Yes and no. In general, it is not so bad on modern hardware — if used properly and not in an attempt to create some all-in-one uber-shader. Actually, it is quite often used in raymarching where it can provide a measurable performance improvement by branching out empty/occluded parts. In this particular case branching helps to save one of the most critical resources on mobile GPUs — memory bandwidth.

So how can we test if this branching actually improved things or only unnecessarily complicated shaders?

First, let’s perform a static analysis of shaders. To do this, I use the Mali Offline Compiler tool. Here are results:

Non-optimized:

Work registers: 25
Uniform registers: 16
Stack spilling: false
16-bit arithmetic: 66%
                              FMA     CVT     SFU      LS       V       T    Bound
Total instruction cycles:    0.31    0.14    0.06    0.00    1.00    1.25        T
Shortest path cycles:        0.31    0.11    0.06    0.00    1.00    1.25        T
Longest path cycles:         0.31    0.14    0.06    0.00    1.00    1.25        T

FMA = Arith FMA, CVT = Arith CVT, SFU = Arith SFU, LS = Load/Store, V = Varying, T = Texture

Conditional PCF:

Work registers: 21
Uniform registers: 16
Stack spilling: false
16-bit arithmetic: 68%
                              FMA     CVT     SFU      LS       V       T    Bound
Total instruction cycles:    0.31    0.22    0.06    0.00    1.00    1.50        T
Shortest path cycles:        0.17    0.11    0.06    0.00    1.00    0.25        V
Longest path cycles:         0.31    0.20    0.06    0.00    1.00    1.25        T

FMA = Arith FMA, CVT = Arith CVT, SFU = Arith SFU, LS = Load/Store, V = Varying, T = Texture

So according to these results, the new version is no longer texture-bound in the shortest path and still has the same cycles for the longest path. Also, the number of used registers is reduced. Looks good on paper, isn’t it?

But of course both versions of shaders perform identically in the Android app on my Pixel 7a — it always runs at stable 90 fps. So to see if GPU is less loaded, let’s run Android GPU Inspector on two versions of app and compare some metrics from profiles:

As expected, it doesn’t affect overall GPU cycles much but reduces load on texture units. As a result, the GPU now is less busy — it consumes less power and has more free resources to smoothly render home screen UI on top of live wallpaper.

Animation

All animated objects in the scene are animated procedurally. No baked skeletal or vertex animations are used. The simple shapes of these objects allow them to be animated in vertex shaders relatively easily.

I have found some inspiration for procedural animations of rats in the “King, Witch and Dragon” (https://torchinsky.me/shader-animation-unity/) and fish ABZU (https://www.youtube.com/watch?v=l9NX06mvp2E). Animations in our scene are of course simpler than the ones in these games because animated objects have stylized boxy look and therefore movements are also stylized and simplified.

Knights

Scene has 16 knights with each model made of just 48 triangles.

Animation is done in the KnightAnimatedShader.ts. Let’s take a look at how it animates the model. To do this, first it needs to detect vertices belonging to different body parts of the knight. But the vertex data doesn’t have a special “bone id” attribute for this. This cannot be done by testing vertex positions because some body parts overlap. For example, the head has 4 bottom coordinates identical to the body. Some texture coordinates also overlap so we cannot rely on them too as in the case of rats animation in the “King, Witch and Dragon” game. So I grouped the vertices for each body in the buffer, and the vertex shader determines body parts simply by comparing their gl_VertexID. Of course this is not optimal because it introduces branching but it is not overly excessive and it is done in a vertex shader for a very low-poly model. Model is grouped to have first body vertices, then head and then arms:

And here is the knight model with applied test coloring to visualize body parts:

Now that shader knows which vertex belongs to which body part it applies rotations to them provided via uniforms. Rotation pivot points are hard-coded in shader. You may notice that only the head and arms are animated. Because models don’t have separate legs they are not animated. Instead a bobbing is applied to the whole model to add a rather convincing effect of “walking”. The bobbing is simply an absolute value of the sine wave:

Birds

There are 6 eagles soaring in the sky, each model made of 70 triangles. They are flying in different circular paths.

Birds are rendered with EagleAnimatedShader.ts. Animations are done in the same way as for knights but they are simpler since only wings are animated and they are rotated synchronously. So only a single rotation timer is passed into the shader via uniform to control animation.

Flags

Scene has 3 different flags, all animated with the same FlagSmShader.ts. Technique is inspired by the wavy animation of the fish in ABZU. The simple sine wave is applied to vertices, reducing amplitude closer to the flagpole and increasing near the end of the flag. To correctly apply lighting, normals are also bent. To bend them the cosine wave of the same frequency is used since it is derivative of sine.

Wind Stripes

A small detail added to the scene as a last touch is mostly inspired by the Sea Of Thieves wind effect. In the Sea Of Thieves these stripes serve a purpose of showing wind direction to align sails so they are more straight. In our scene they are purely for the looks so they bend and twist much more.

Let’s take a look at the shader to draw them — WindShader.ts. It is even “more procedural” than the ones used for animated objects. It doesn’t use geometry buffers at all and generates triangles based on gl_VertexID. Indeed, as you can see in the source of its draw() method it doesn’t set up any buffers with vertex data. Instead it uses the hard-coded VERTICES array for two triangles declaring a single square segment. So if we need to draw 50 segments, we issue a glDrawArrays call with 50 * 3 * 2 triangle primitives. The vertex shader offsets each segment based on the gl_VertexID and tapers both ends of a final stripe using smoothstep. Then the coordinates of a resulting spline shape are shifted by an offset timer so they appear moving. Next, the shape is deformed by two sine wave noises in world-space. Color is animated to fade in and out, and offset is animated to move the stripe. All this results in a random snake-like movement of the stripes but keeps them aligned to the world-space path.

Here is a breakdown of rendering of a short 20-segments wind stripe with test coloring of segments to clearly visualize geometry:

Depth-only shaders

All objects in the scene except wind stripes cast shadows so they have to be rendered to a depth map texture using the light source camera. For this, the simplified versions of corresponding animated and static shaders are used. They perform all the same vertex transformations but don’t calculate lighting.

For example, let’s take a look at the KnightDepthShader.ts. It performs all the same vertex transformations to animate head and arms but does not calculate lighting based on normals. Even more than that, you may notice that its fragment shader is empty — it provides no color output at all. These are perfectly valid shaders in GLSL ES 3.00 since their only purpose is to write to the depth (shadow map) attachment.

Results and possible additional optimizations

Live web demo: https://keaukraine.github.io/webgl-stylized-castle/index.html

Source code: https://github.com/keaukraine/webgl-stylized-castle

Android live wallpaper app - https://play.google.com/store/apps/details?id=org.androidworks.livewallpaper.cartooncastle3d

Final web demo is ~2.2 MB which is not quite optimal because all objects are exported as 2 huge batched meshes. And there are a lot of repetitive objects like trees and cannons which are good candidates for instanced rendering.

Voxel Airplanes 3D WebGL demo

keaukraine — Mon, 19 Dec 2022 19:10:59 +0000

While working on Floating Islands live wallpaper I stumbled upon these cute voxel 3D models of airplanes by Max Parata. I already wanted to bring life to some stylized low-fi 3D art so in my mind I immediately saw how to create a stylized old-skool low-fi scene with these assets. You can watch a live WebGL demo here.
This project was quite fast to implement - the time span between the first commit with rough WIP layout and the final version is about 20 days. Yet it was quite fun to create it because during development I’ve got some fresh ideas on how to improve the scene. All additions were really minor to keep the scene as simple as possible in accordance with its art design. My brother provided valuable feedback on how to improve it and also helped with optimization of some geometries.

Scene composition

Scene is aesthetically simple so it contains just four objects: planes, ground, clouds and wind:

Next, let’s take a look at what shaders are used to render models, and how geometries of these models are optimized.

Planes

Technically planes are rendered not as voxels (each voxel cube individually) but as a ready mesh, exported from MagicaVoxel. They are not simplified using VoxCleaner to use texture atlases and reduce polycount - I decided to use them as is because it will be easier to create alternate palettes, and anyways vertex data for planes have ridiculously small memory footprint.

Each plane consists of 3 parts - plane body, glass cockpit and rotating propellers. Plane is rendered using 2 shaders - a simple directionally lit PlaneBodyLitShader.ts for body and props and its variation GlassShader.ts for glass with stylized reflections.

The specifics of plane models allow vertex data to be packed using really small data types. All plane models are small and fit in the -127…+127 bounding box. And since vertices represent voxels they are always snapped to 1x1 grid. So I chose to store vertex positions in signed bytes which have just enough precision for the job.
Also, since the palette textures used by models are of 256x1 size, the V texture coordinate is omitted and hardcoded to the 0.5 - the center of texel. Remaining U coordinate fits in one unsigned byte. One byte of padding is used to align data by 8 bytes. Here is vertex data stride for plane and prop models:

In a separate draw call, a glass with scrolling stylized fake reflection is drawn:

Glass is rendered without palette texture - its color is set via uniform. The texture passed to this shader is a mask for reflection. Its UV coordinates are calculated in the vertex shader based on model-space vertex coordinates. Of course, it is unfiltered for artistic purposes. Stride for glass models is the same but without texture coordinates:

GlassShader samples texture using textureLod with 0 mipmap level. This is done to explicitly tell the OpenGL ES driver that we access the texture without mipmaps and to reduce some overheads. You can read more about this and some other texture sampling optimizations tricks in Pete Harris blog - https://solidpixel.github.io/2022/03/27/texture_sampling_tips.html
Also glass models have small polycount so they use unsigned byte indices which also reduces memory bandwidth.

UPDATE: In the newer version of GlassShader I have found a way to optimize strides for glass models even more. Now they fit in just 4 bytes:

So how is normal stored in single byte? Since voxels can have only 6 variations of normals they can be stored as an index of an array of actual normals which are hard-coded in vertex shader. Updated shader uses this technique to reduce the amount of vertex data.

Wind stripes

For wind stripes I decided to create a shader which doesn’t perform any memory reads at all. Wind stripe has a very simple geometry - a 100x100 units quad, stretched into an appropriately thin line by model matrix. Because of its simplicity, all this geometry can be hardcoded in vertex shader code. And it doesn’t use any textures as well - fragment color is passed via uniform. You can find the implementation in WindStripeShader.ts. It uses gl_VertexID to get position for a given vertex. When this shader is used to draw a wind stripe, no buffers or textures are bound.

Technically it even can be used as a “building block” to draw more complex shapes by issuing draw calls with different rotating/scaling/shearing its base hard-coded quad geometry but this will be too inefficient.

Terrain

Terrain textures are 256x256 tiling images. They are based on aerial photos with some GIMP magic sprinkled over them - contrast adjustments and colors reduced to just 10-12. This adds a more old-skool look to them and makes each texel more pronounced.

Shader to render terrain is in DiffuseScrollingFilteredShader.ts file. Let’s take a look at it.
It is a rather simple shader which simply pans UV coordinates to create an illusion of moving ground beneath the airplane. However there is one additional thing it does, and it is texture filtering. You may be wondering what filtering is used here, ground clearly is unfiltered, it uses GL_NEAREST sampling! However, there is a custom antialiased blocky filtering used here. The thing is, that regular GL_NEAREST sampling produces a lot of aliasing on the edges of texels. This becomes especially noticeable at certain angles of the continuously rotating camera. The textureBlocky() function alleviates these aliasing artifacts while preserving that extra crispy old-skool look of unfiltered textures. Ground texture actually uses GL_LINEAR filtering and the textureBlocky() calculates sampling point to get either an interpolated filtered value at the edges or an exact unfiltered one from the center of texel for any other area.
The original author of this filtering is Permutator, and code is used under CC0 license from this shader toy - https://www.shadertoy.com/view/ltfXWS (you may find some deeper explanation of math used in this filtering technique there).
Here is a comparison (with 4x zoom) of a regular GL_NEAREST filtering vs custom blocky filtering. As you can see, both are pixelated but the latter one is not aliased.

One of the last additions to the scene is a transition between two different terrain textures. When you switch them, they don’t just toggle but instead a cute pixelated transition effect is used to smoothly switch between textures.

You can find a code for this transition in the DiffuseScrollingFilteredTransitionShader.ts file. Transition uses tiling blue noise texture for uniformly appearing square blocks on the ground. To make transition smoother, smoothstep() is used. However there is a commented out line with step() which makes transition more abrupt if you prefer it.

Clouds

Clouds don’t use this antialiased blocky filtering because they don’t rotate, are quite transparent and move relatively fast. This makes it quite hard to spot aliasing artifacts on them so they use the cheapest option available - GL_NEAREST sampling. Clouds use a custom mesh with cutouts where texture is empty. This significantly reduces overdraw compared to a regular quad mesh. Here it is visualized by disabling blending:

Result

You can see a live web demo here and if you like to have it on the home screen of your Android phone you can get a live wallpaper app on Google Play.

Source code is available on GitHub, feel free to play around with it.

As always, the web demo is heavily optimized for the fastest downloading of resources and Android app for the best efficiency and performance. The size of data transferred during initial loading of the web demo is just 189 kB, and the size of all models and textures is 1.4 MB so you can fit this data on a floppy disk.

P.S.

These WebGL demo and Android app have been made during war in Ukraine despite regular power outages caused by deliberate destruction of country’s electric infrastructure. Please support Ukraine however you can and boycott your local russian aka terrorist businesses.

Floating Islands WebGL demo 🇺🇦

keaukraine — Sat, 29 Oct 2022 15:59:37 +0000

Idea and inspiration

Idea for this 3D scene comes from the magnificent Zhangjiajie National Forest Park in China. You can clearly see where the inspiration originates from - this majestic real life location also has grassy rock pillars covered in dense clouds and when observed from above, their bottom parts are disappearing in dense fog. To improve the magical feeling to the scene, we decided to make some rocks float mid-air. This additional inspiration comes from the map Gateway to Na Pali from my favorite game Unreal. This location has floating rocks in the distant background and is placed inside a huge floating rock itself. We decided to create a scene which would have a lot of similar floating islands densely packed in one area.

AI generated concept art

We also tried to use Stable Diffusion to generate some concept art for the scene in hope AI will hallucinate some unusual points of view or might incorporate some details which we may find fitting the scene. However, all images appeared to be virtually identical. AI created a series of rather dull images - the same rocks in the same fog without any additional details. We only used a couple of them as a reference for vivid sunrise color palettes which could as well be picked from any other sources.

Scene composition

To create this scene we’ve used and reused some stylized hand-painted 3D models from packs we’ve purchased quite some ago for our previous projects. No new assets were purchased for this project. Scene uses just 3 rock models, some generic ferns and trees, and birds flying in the sky.

Render order is the following: depth pre-pass, rocks, birds, sky, soft cloud particles.

Camera path and objects placement

To make an impression of an endless random scene there were 2 options: a true random scene and a looped generated path. The first option requires object placement on the fly in front of the camera which means these positions have to be transferred to GPU dynamically. So a better option is to generate a static looped path once and draw objects along it as the camera moves.
You can find a function to generate a base spline in the positionOnSpline function in ObjectPlacement.ts file. It creates a circular looped path for the camera with oscillating radius. A couple of harmonics are applied to randomize the circle radius so it appears random but is still perfectly looped. Then, all objects are placed around this path - trees are under the camera, rocks above and to the sides.

Object positions and rotations are stored in typed Float32Array in the form of textures on GPU.
drawInstances method in Renderer.ts renders objects visible only from a certain point on the spline. Because of scene simplicity there's no need to use frustum culling - objects are drawn at a certain distance in front and back of the camera. This visibility distance is slightly larger than fog start distance so the new objects appear fully covered in fog and don't pop. Instances are ordered front-to-back so when drawn they make use of Z-buffer culling.

Only rocks and trees models are placed this way alongside the camera path. Bird flocks use hand-picked linear paths to cover the whole area of the scene with minimal paths.

Here is camera path visualized with only a subset of objects rendered in its vicinity:

Fog cubemaps

Initial implementation used fog of uniform color which looked rather bland. To add more color variation from different directions (like sun halo) we decided to use cubemaps for fog. This allows great flexibility for the artist (my brother) - he can completely change the look of the whole scene by creating a cubemap and tweaking a couple of colors in the scene preset. Cubemaps were initially created as equirectangular images since it is easy to paint them. Then we used an online tool to convert an equirectangular source image to 6 cubemap faces, and a simple ImageMagick script to fix their rotations to suit our coordinate system (Z-up).

You can find cubemap fog implementation in static constants from FogShader.ts. All fog shaders use them. Final fog coefficient used by vertex shader for color mixing also contains the height fog coefficient.

In the web demo UI you can adjust different fog parameters - start distance, transition distance, height offset and multiplier. Also, changing the scene time of day is done by using a different cubemap texture and a couple of colors for each preset.

Interestingly, after implementing this I’ve found out that fog cubemaps are widely used in Source engine, and of course this technique has also been incorporated in some indie games too.

Grass on rocks

To make rocks less dull we also apply grass texture on top of them. This technique is commonly used to simulate surfaces covered by snow or soaked with the rain. Grass texture is mixed with rock texture based on vertex normal. You can play around with the grassAmount slider in UI to see how it affects grass spread on rocks.

Source code of shader which applies grass texture on top of rocks is in FogVertexLitGrassShader.ts.

Soft clouds shader

Clouds are not instanced but are drawn one by one because transformation matrices for these objects have to be adjusted to always face the camera. There’s not that many of them so this doesn’t add too many draw calls. Actually, if GPU state is not changed (no uniforms are updated, blending mode switched, etc.) then even non-instanced rendering is quite fast on modern mobile and desktop GPUs. For test purposes we had a quick and dirty visualization of a camera spline with non-instanced rendering of 5000 small spheres and it caused no slowdowns.

There’s also one minor trick in this shader. As the camera flies through clouds, they can be abruptly culled by a near clipping plane. To prevent this, there is a simple smoothstep fading applied to fade right in front of the camera. You can find code in FogSpriteShader.

Result

You can see a live web demo here and if you like to have it on the home screen of your Android phone you can get a live wallpaper app on Google Play.

Source code is available on GitHub, feel free to play around with it.

As always, the web demo is heavily optimized for the smallest data size and Android app for the best efficiency and performance. Web version uses WebP for textures which offer better compression than PNG, better image quality than JPEG and support alpha channel even with lossy compression. Mipmaps are generated for all textures. The total gzipped size of the web demo is just 374 kB so you can copy it to a floppy disk to show to friends who have no Internet :)

Efficient WebGL vegetation rendering 🇺🇦

keaukraine — Wed, 10 Aug 2022 08:30:29 +0000

In this article I’ll explain the rendering pipeline of Spring Flowers WebGL Demo and its corresponding Android app. Also I will describe what problems we’ve encountered and what solutions we used to overcome during development and testing of the Android live wallpaper app.

You can check out the live demo page and play with various configuration options in the top right controls section.

Implementation

Scene is composed from the following main objects: sky, ground, and 3 types of grass: flowers (each containing individual instances of leaves, petals and stems), small round grass and tall animated grass. To make the scene more alive, a sphere for glare and moving ants+butterflies are also drawn.

Draw order is the following: first objects closer to the camera and larger ones to use z-buffer efficiently, then objects closer to ground, then sky and ground. Ground plane has transparent edges which blur with the background sky sphere so it is drawn the last after the sky.

For sun glare effect we draw a sphere object with a specular highlight. It is drawn last over the whole geometry without depth test. This way everything is slightly over-brightened when viewed against the sun, and glare is less prominent when the camera is not facing the sun.

Tiled culling of instances

Grass and flowers are drawn using similar shaders with a common part in them being instanced positioning. These instanced objects get their transformations from the FP32 RGB texture.

All instanced shaders use the same include COMMON_TRANSFORMS which uses 2 samples from texture to retrieve translation in XY plane, scale and rotation. Please note that rotation is stored in the form of sine and cosine of an angle to save on rotation math.

Original transformation is stored in arrays FLOWERS, GRASS1 and GRASS2 declared in GrassPositions.ts. However, these arrays have coordinates for all instances of objects, they are not split into tiles yet. For this, they are processed using sortInstancesByTiles function. It creates a new FP32 array with rearranged positions+rotations and creates an array of tiles which specify instances count and start offset in the final texture used by the shader. This ready information is stored in the TiledInstances object. This function allows to split all instances spreaded across square ground area into arbitrary N^2 tiles. In both web demo and Android app all instances are split into reasonable 4x4 tiles area with 16 tiles. Tiles have a small padding which allows them to slightly overlap. This padding has a size of grass model so the instances placed at the very edge of tile won’t disappear abruptly when tile gets culled.

To visualize how instances are split into tiles, let’s imagine a sample area with 20 randomly placed objects which we would like to cull per tile. Let’s rearrange these instances into 2x2 grid with 4 total tiles:

Here is the structure of texture containing these objects, showing tiling and data stored in each component per instance:

Here instances for tile 0 have offset=0 and count=5, for tile 1 offset=5 and count=4, and so on.

This structure allows us to draw all 20 instances in 4 draw calls and cull them in batches per tile without updating any data on the GPU.

Culling of tiles bounding boxes on CPU is also relatively cheap. It is done on each frame and you can see how many tiles and individual instances are currently rendered in “Stats” section of controls.

And reducing grass density to scale performance is also really easy with this approach because inside each tile instances are random. All we have to do is to proportionally reduce number of instances per draw call (you can use density slider in controls to test it):

There are different instanced shaders for drawing different objects. Small grass and flower petals are the simplest ones - they use a simple diffuse colored shading. Dandelion stems and leaves apply specular highlights, and the shader used to render tall grass blades also uses vertex animation for wind simulation.

Random ants

To make the ground more alive, we draw some ants on it. They are also instanced - total 68 ants are rendered in 2 draw calls.

They move in circles with a random radius and center. They are drawn in two draw calls for clockwise and counterclockwise rotation within these circular paths. You can examine the math for positioning vertices in the shader’s source code.

It will be almost impossible to notice any animation on these pretty small and very fast moving objects so we don’t animate them at all.

Butterflies

No summer can be imagined without butterflies so we added them too. They are positioned similar to ants but a sine wave is added to their height. Each instance gets its color from texture atlas with 4 different variants.

On the contrary to ants, butterflies must have animation. To animate them we don’t use any kind of baked animation. Actually, a really cheap trick is used to animate it in vertex shader. It is animated by simply moving wing tips up and down. Wing tips are determined as vertices with high absolute values of X coordinate. Of course it is not a correct circular movement of wings around the butterfly’s body - wings elongate noticeably with higher movement amplitude, but this simplifies shader math and looks convincing enough in motion, as can be seen on this image:

Android-specific optimizations

As always with our web demos, they are optimized for the smallest possible network data size and the fastest loading times. So it doesn’t use compressed or supercompressed textures. The Android app is optimized for power efficiency so it uses compressed textures (ASTC or ETC2) depending on hardware capabilities.

To further improve efficiency it uses variable rate shading (VRS) on supported hardware.

And when the app detects that the device is in energy saving mode (triggered manually or when battery is low) it will reduce FPS and will use simplified grass shader without animation to significantly reduce power draw. Additionally, in this mode the app will use more aggressive VRS.

We’ve encountered general performance issues with rendering lots of instanced geometries on low-end Android phones - the bottleneck appeared to be vertex shaders. So when the app detects it is running on a low-end device it will render grass with slightly reduced grass density and without wind animation.

Failed implementations

Before implementing this tiled rendering pipeline a couple of more naive less performant implementations have been tried and tested.

Fully randomized

The very first version of grass was with fully randomized positioning of instances. It didn’t use texture to store pre-calculated random transformation for instances but calculated them in shader instead. This introduced more complexity in vertex shaders (random and noise functions have quite some math in them). Additionally, random values were different on different GPUs which made it impossible to finely hand-pick camera paths. Take a look at this photo where we tested this version on different devices - while the code is identical, placement of instances is different:

This version had no visibility calculation or frustum culling which also affected performance.

Per-instance culling

The first naive implementation of culling has been implemented per-instance. This version already used texture to reduce vertex shader math but each instance has been tested for visibility and then texture has been updated with only visible instances.

This worked just fine on PC and on high-end Android devices but proved to be way too slow on low-end phones - CPU took about 10 ms to calculate visibility of instances. Updating the texture on the fly with glTexSubImage2D() also was unacceptably slow - it took ~20 ms. For comparison, tiled culling takes ~1 ms of CPU time on low-end devices.

Final result

Total size of the demo page is just 741 kB so you can carry it on a floppy disk.

You can play around with different parameters on the live demo page. You can alter time of day, density of grass and other settings. Double click toggles free camera mode with WASD camera movement and rotation with right mouse button down (similar to navigation in viewport in Unreal Engine).

And as usual you can get source code which is licensed under MIT license, so feel free to play around with it.

Variable Rate Shading on Adreno GPUs 🇺🇦

keaukraine — Sun, 03 Jul 2022 18:17:45 +0000

“With high screen DPI doesn’t come high GPU fillrate” — that’s the main problem of GPUs nowadays. Modern consoles struggle to sustain stable 30, let alone 60 fps on large 4k screens. The common technique to increase FPS is rendering at lower resolution with fancy upscaling techniques like DLSS and FSR. But modern VR-capable hardware has to be able to target both very high frame rates and high image quality, and upscaling does show its limitations here — depending on implementation the image will be either blurry, too sharpened or will introduce ghosting artifacts. Variable rate shading (VRS) is a temporally stable approach of improving performance with (if applied correctly) virtually unnoticeable quality reduction.

Modern mobile Adreno GPUs by Qualcomm support Variable Rate Shading, and phones with these GPUs have been available since autumn 2021. Because our live wallpapers have to be power-efficient, we have got a test device with Adreno 642L to implement this feature in our apps.

What is Variable Rate Shading

The idea behind VRS is to rasterize a single fragment and then interpolate color between adjacent pixels on screen.

A good explanation of how VRS is implemented on Adreno GPUs can be found in the official Qualcomm Developer blog here. You can understand how simple it is by looking at this image from aforementioned blog post:

VRS is better than generic downsample of the whole frame because:

It preserves geometry edges (except cases when the shape is determined by discarding fragments).
Can be adjusted per each draw call — one object can be rendered at full detail while the other one will have reduced quality.
Can be applied dynamically to keep target FPS by gradually reducing image quality.

Implementation

On Snapdragon SoCs it is implemented with QCOM_shading_rate extension. Adreno GPUs support blocks of 1x1, 1x2, 2x1, 2x2, 4x2, and 4x4 pixels. Please note that some useful dimensions like 2x4 or 4x1 are not available because they are not supported by hardware.

To apply VRS to certain objects you simply make a call to glShadingRateQCOM with desired rate before the corresponding draw calls.

To disable VRS for geometries which should preserve details and be rendered at native shading rate, simply call glShadingRateQCOM with 1x1 block size.

One of the first apps we’ve added VRS support to is Bonsai Live Wallpaper. This is a good example because it has 3 very different types of geometries ranging from perfect candidates for VRS optimizations to the very unsuitable ones.

Let’s take a look at a typical scene from the app and how different parts of image can benefit from reduced shading rate:

The best type of geometry to be optimized by VRS is the one which is blurred and has small color variation between fragments. So, for sky background we apply a quite heavy 4x2 VRS which still introduces virtually no quality degradation, especially with constantly moving cameras.

On the opposite side of the scales is leaves geometry. On the screenshot below we applied 4x4 VRS to the whole scene to showcase the issue with alpha-testing. Please note that branches, while also using the same heavy 4x4 reduction in this example, have the same smooth and anti-aliased edges, clearly showing a benefit of VRS over traditional upscaling.

Needless to say, VRS is clearly not suitable for geometries with discarded fragments.

Also because VRS is applied in screen-space, it introduces significant distortions to transparent dust particles. Their size is comparable to VRS block and they start flickering during movement. I’ve noticed a somewhat similar rendering technique used in the COD:MW game on PC when enabling half-resolution particles — sparks and other small particles flicker way too much and look very blocky.

And somewhere between these two geometries lies the ground plane. This is where we apply 2x1 rate reduction. This results in OK image quality because there’s a larger color difference between adjacent vertical pixels compared to the horizontal ones.

Where VRS definitely shines is when it is applied to geometries with very little color difference between adjacent fragments, and Bonsai wallpaper has a stylized silhouette mode where fragments use literally single color:

Here we have 3 types of shaders:

Alpha-testing for leaves. We already know that we should not apply VRS to these geometries.
Solid black silhouette and ground. The heaviest 4x4 VRS introduces literally zero quality degradation.
For the sky gradient we use 2x1 blocks. Technically it would be perfect to have a 4x1 or even 16x1 blocks because gradient changes vertically and adjacent horizontal fragments have identical color but Adreno hardware supports only 2x1 ones.

All of these applied to the scene results in identical rendering (screenshots comparison found 0 pixels difference) and 1.5x of shading speed improvement.

Dynamic quality

All our wallpapers use some ways of reducing GPU load when the battery is low. Usually this is done by limiting FPS and omitting a couple of effects.

For more efficient power usage we apply stronger VRS to certain objects in low battery mode. Tree trunks are shaded with 2x1 blocks, sky and transparent effects (light shafts and vignette) are shaded with 4x4 instead of 4x2 or 2x2 blocks. This reduction of quality is still almost unnoticeable but reduces GPU load by additional 3%.

Performance gains vs quality tradeoff

You will be hard-pressed to find any difference between original and VRS-optimized rendering — color deviation is negligible, and blocky artifacts are really hard to spot. Only ImageMagick was able to show different pixels:

Both VRS-enabled and regular rendering pipelines result in steady 120 FPS on our test device (Galaxy Samsung A52s). So we’ve run a Snapdragon Profiler to analyze performance and efficiency of the optimized build. Here are the numbers:

Bonsai 3D Live Wallpaper, regular mode:

Bonsai 3D Live Wallpaper, battery saving mode.

Bonsai 3D Live Wallpaper, silhouette mode.

In the silhouette scene we don’t use different VRS blocks for regular and power saving modes because it already uses maximum block size and still renders the image identical to non-VRS one.

Long story short, we’ve improved rendering efficiency by approximately 30% with little to (literally) none image quality reduction.

WebGL Grim Reaper demo 🇺🇦

keaukraine — Mon, 07 Mar 2022 09:18:47 +0000

A couple weeks before Halloween 2021 I browsed Sketchfab and encountered a cool 3D model of Grim Reaper by 3DRT. It has a reasonable polycount, a set of different colours and smooth animations. So the decision was made to create a Halloween-themed live wallpaper with this model. However, I was not able to finish it before Halloween because I gradually added some new effects and features which took quite some time to implement and then tweak.

You can find a live web demo here, and for persons sensitive to flickering lights a version without lightning is here. You can interact with it by clicking the mouse on screen — this will change animation. Also you can enter free-camera mode which uses WASD navigation by pressing the Enter key.

As usual, source code is available on Github.

And of course you can get an Android live wallpaper app.

Scene Composition

Scene is pretty simple so it doesn’t require any sorting of objects — carefully chosen hardcoded render order achieves minimal overdraw:

First, opaque (cloth is alpha-masked so it is also opaque) geometries are rendered. These animated objects use vertex animation with data stored in FP16 textures, so WebGL 2 is required for the demo.
After rendering opaque geometries, writing to depth is disabled with glDepthMask(false) and then transparent effects — smoke, dust and ghosts are drawn over them with blending. Sky is also drawn at this stage. Because it is the most distant object, it doesn’t have to contribute to depth — it is basically treated as a far clipping plane.

Effects

That’s where most of the time was spent — thinking of, creating, tweaking and rejecting various effects for a really simple scene with literally a single character in it.

Every time I had an idea on how to improve a look I added it to the Trello board. Then I had some time to think about it — how will it fit the scene, how to implement it, etc. So here is a breakdown of all used effects.

First, soft particles are added to the reaper. Half of them rise upwards, half of them sink down from roughly the centre of the reaper model which fluctuates a little depending on animation. Of course to get the best visual appearance soft particles are used, hence the depth pre-pass. You can read about implementation of soft particles in one of my previous articles.

Then some flickering dust is rendered. You may notice that its brightness is synchronized with lightning strikes — usually dust slowly fades in and out but at lightning strikes it is more visible.

As a final touch, a rather heavy vignette is applied. This effect blends nicely with the gloomy atmosphere, helps to draw attention to the centre of screen and to visually conceal the bland void in the corners of the screen.

There are still a couple of effect ideas noted in my Trello board but I think that adding them will only clutter the scene without adding any more noticeable eye candies.

Sky shader

Sky is used to fill in the void around the main character. To add some dynamics and movement to these empty parts of the scene it is rendered with a shader which applies simple distortion and lightning to static clouds texture.

Let’s analyse the shader code. It combines three simple effects to create a dynamic sky:

It starts with applying colour to rather bland-looking greyscale base sky texture:
Then, waves from a small distortion texture are applied (a similar but more pronounced effect can be used for water ripples). Effect is subtle but does noticeably improve overall look:
And the final touch is lightning. To recreate somewhat realistic looking lighting which cannot get through dense clouds but shines through clear areas, brightness is increased exponentially — darker parts will get very little increase in brightness while bright areas will be highlighted. Final result with all effects combined looks like this:

Timer for the lightning strikes is a periodic function of several sine waves combined, clamped to range [0…2]. I’ve used a really handy Desmos graphing calculator to visualize and tweak coefficients for this function — you can clearly see that the “spikes” of positive values create short periodic randomized bursts:

Additionally, the sky sphere slowly rotates to make the background less static.

Ghosts shader

Ghostly trails floating around the grim reaper are inspired by this Unreal Engine 4 Niagara tutorial — https://www.artstation.com/artwork/ba4mNn.

Initial idea was to use a geometry in a shape of cut-out from the cylinder side and rotate it around the centre of the reaper model. However, my brother created a shader for a more flexible approach to use a single geometry which can be rotated at arbitrary radius and stretched to arbitrary length.

To achieve this, vertex shader changes the geometry of the original mesh. It modifies X and Y coordinates of the input model, bending them around the circle of given radius. Z coordinate is not getting additional transformations. It is responsible for scaling the final effect vertically. (World space is Z-up). Shader is tailored to work with a specific model — a tessellated sheet in the XZ plane (all Y coordinates are zero):

Later, geometry was optimized to tightly fit our sprite texture in order to reduce overdraw:

Based on the math of chord length, the X and Y coordinates of bent model are:

x = R * sin(theta);
y = R * cos(theta);

where theta = rm_Vertex.x / R, and R is a bend radius. However, theta is calculated differently in the shader:

float theta = rm_Vertex.x * lengthToRadius;

lengthToRadius value is a uniform, but it is not just a reciprocal of R — we can pass values greater than 1/R to get effect length scaled (because it essentially is a pre-multiplication of rm_Vertex.x).
This minor change is done in order to eliminate redundant uniform-only math in the shader. Preliminary division of length by radius is done on the CPU and this result is passed into the shader via lengthToRadius uniform.
I’ve tried to improve this effect by applying displacement distortion in fragment shader but it appears to be virtually unnoticeable in motion. So we kept the original simpler version with static texture, which is also cheaper for the GPU.

Reduced colours filter

Not implemented in the web version, but present in Android app is a reduced colours post-processing. This gritty effect perfectly fits the overall atmosphere and adds a right mood to the scene. It is implemented not as a separate post-processing render pass but is done in the fragment shader, so rendering is still essentially single-pass.

It is based on code from Q1K3 WebGL game https://github.com/phoboslab/q1k3, and I highly recommend to read a blog post about making of seemingly impossible Q1K3 — https://phoboslab.org/log/2021/09/q1k3-making-of.

Textures compression

Android live wallpaper targets OpenGL ES 3.0+ and uses efficient ETC2 and ASTC compressed textures. However, WebGL demo is optimized only for the fastest possible loading time. I really hate when some simple WebGL demo takes forever to load its unjustifiably huge resources. Because of this, a decision not to use hardware compressed textures was made. Instead, textures are compressed as lossy WebP. Total size of all assets including HTML/CSS/JS is just 2.7 MB so it loads pretty fast.
Recently, our mountains WebGL demo has also been updated with smaller resources but it is still way larger than the Reaper one — it downloads 10.8 MB of data on initial load.

Filtering of half-float textures on different mobile GPUs

keaukraine — Sun, 01 Aug 2021 18:18:24 +0000

In this article I’ll describe what issues we’ve encountered during development of our latest Reunion live wallpaper and its WebGL demo.

All gorgeous hand-painted artwork is done by Conrad Justin, with only minor additions by me and my brother (lights for candles, dust particles, etc). You can compare our scene with an original 3D model on Sketchfab here and also make sure you take a look at the artist’s full portfolio here.

Usage of FP16 textures

In both web demo and Android app we have animated objects for characters and decorations. And for this new art style of wallpaper scene, we introduced new types of rendering techniques and custom shaders. Here we have used vertex animations which are stored in floating-point textures (so the web demo requires WebGL 2).

For our use case precision of half-float to store vertex positions is just good enough. And according to OpenGL ES 3.0 specs, FP16 textures even can be filtered, which is really handy for animations - linear interpolation between animation frames will be handled by hardware virtually for free, without calculations in shader.

Issues with textures filtering

Well, using texture filtering for interpolating animations was a good option only in theory - all you have to do is enable GL_LINEAR for texture and you’re good to go. However in practice, the arithmetic precision of filtering is not perfect and is even somewhat hardware-specific.

During development on PC everything was fine, but testing on different mobile phones with various GPUs revealed noticeable visual differences in hardware FP16 texture filtering. I suspect that the ANGLE wrapper for WebGL always uses full-precision floating point values in both shaders and textures, because we’ve already encountered precision issues in shader calculations during development of the Iceland WebGL demo - they were seen only on mobile devices with native WebGL-to-OpenGL ES translation.

Here is the reference rendering of squirrel character on PC without issues (Windows 10, Chrome or Firefox with ANGLE renderer):

And here are some results from different phones:

Galaxy A21S with Mali GPU. Vertex shivering with some gaps between triangles:

Moto E6s with PowerVR GPU. This one is closest to reference rendering on PC, but still has some gaps:

And Google Pixel 3 with Adreno 630 GPU. Shivering is also present:

Solution

The most obvious fix would be to use 32-bit GL_FLOAT textures which have better precision and should be interpolated more correctly, but unfortunately they are not filterable at all and animations look like in Quake 1 - without any interpolation:

So we decided to implement a custom linear interpolation in the vertex shader. In our implementation, textures represent animation in the direction of increasing Y coordinate, so we have to sample only two adjacent texels in this direction.

You can examine the source code of this shader here. As you can see, it uses highp precision for floats for precise and smooth interpolation of vertex positions.

First, this shader has a function getCenter() which returns a center of texel for any arbitrary coordinate. It is used to get the color of two points.

Actual filtering is done in the linearFilter() function, which samples two values half-texel higher and half-texel lower and linearly interpolates them based on how far centers of their texels are from actual sampled coordinates.

Please note that shader samples colors not exactly half-texel higher and lower but a texel height is multiplied by a value slightly smaller than 0.5 - 0.49. This is done because floating point precision is limited and sampling exactly at the edge of 2 texels might get into texels which we don’t need. This will result in a broken animation - interpolation with the previous or next frame, instead of the current one. Sampling at offsets slightly lower than half texel height eliminates this issue.

This implementation of custom texture filtering results in smooth interpolation, and it is identical on all tested platforms.

Result

You can take a look at live demo page with this custom interpolation here - https://keaukraine.github.io/webgl-reunion/

And of course full source code is available on Github - https://github.com/keaukraine/webgl-reunion

More efficient ASTC decoding

keaukraine — Sun, 04 Jul 2021 16:38:34 +0000

This will be a really short but still useful article, explaining a couple lines of code which improve performance on compatible devices.

ASTC is a very efficient texture compression format - it combines decent image quality with high compression. It helps saving a lot of memory bandwidth on modern mobile GPUs.
But can we crank it to 11 and make it run even faster? It appears we can (in certain scenarios and on supported hardware).

ARM Mali GPUs support the extension GL_EXT_texture_compression_astc_decode_mode. According to ASTC specifications, even LDR textures are decoded into 16-bit floating point values. This extension provides a possibility to switch the hardware ASTC decoder into faster mode, decoding textures into lower precision normalized 8-bit unsigned integers. This is good enough for most real-life applications, since source textures are usually 24-bit RGB or 32-bit RGBA bitmaps.
Using extension is as simple as adding 1 line of code:

GLES30.glTexParameteri(GLES30.GL_TEXTURE_2D, TEXTURE_ASTC_DECODE_PRECISION_EXT, GLES30.GL_RGBA8);

Visually there was no image quality degradation, and performance was at the same steady 60 fps. However, looking at certain metrics in the GPU profiler, we can see reduced load on the compressed texture decoder and improved texture cache access.
Here are measurements from ARM Streamline profiler for our 3D Buddha Live Wallpaper, taken on mid-range Galaxy A21s phone:

As you can see, this simple trick have noticeably improved texture lookups. As a result, it reduced memory bandwidth usage and device power consumption. So don’t be lazy and if this extension is detected, use it — it’s a minor change to code which gives virtually free performance boost on Mali GPUs.

This optimization was suggested in 3-part ARM webcast “Optimizing Android Graphics” — I highly recommend watching them.

Creating mountains landscape in OpenGL ES

keaukraine — Mon, 26 Apr 2021 19:43:27 +0000

A few days ago we released a new nature-themed app — Iceland 3D Live Wallpaper. It has an interactive WebGL demo too, which you can find here.

Terrain is based on this beautiful and detailed 3D model by Sergey Kuydin. Interestingly, this is not a real landscape of some part of Iceland. Even though it looks like a real thing, it actually is generated in World Machine. After analyzing model in Sketchfab we decided to create a live wallpaper with it, adding dynamic time of day. You should check out more of Sergey’s work, he has some high quality models and 3D scans too.

Scene Composition

Scene is created from the purchased 3D model of terrain and other assets, like textures and models for the sky hemisphere, birds and sprites. They were created and tailored to fit the scene by my brother, who also proposed some recommendations on how to optimize certain aspects of the scene, and tweaked shaders when needed. As usual, the web demo was created before the Android app because it is faster to create a web prototype than an Android app and it is way easier for me and my brother to collaborate on the web project.

To analyze scene rendering I will refer to source code. You can clone it from this repository or examine code on GitHub using links to files I will provide below.

Scene is rendered in 35 total draw calls. Order of rendering is carefully chosen to efficiently use z-buffer culling. The nearest objects are drawn first, the most distant ones last. After that we render transparent objects:

All actual draw calls are issued in drawSceneObjects() method of MountainsRenderer.ts. Let’s analyze how they are rendered.

Birds are rendered first because they can occlude both terrain and sky. They are rendered in 12 draw calls.

Next, we render terrain. Original high-poly model is simplified in Blender using the Decimate modifier to ~30k triangles which results in detailed enough geometry.

And of course to create a vast, huge mountainous landscape by reusing a single terrain model we use the same terrain skirt technique as in Dunes wallpaper (described in our previous article here, and the original implementation is in Halo Wars). The basic idea of this technique is to draw the same terrain tile mirrored at the every edge of main terrain. However, in Dunes live wallpaper there was one flaw in this. On mirrored tiles shadows from pre-rendered lightmaps were on the wrong slopes — lit by the sun. Because of the overall dunes terrain simplicity and low camera placement it was concealed and virtually unnoticeable. I must give a huge credit to u/icestep from Reddit who have found this and suggested a fix to create 4 different lightmaps for 4 possible tile orientations. But because mountains have deep, sharp shadows this cheap trick becomes clearly visible from almost any place in the scene so we had to implement a fix for this. Luckily, by clever placement of the sun (alongside one of the axes) we have to render just 2 lightmaps — for sunlight in the correct and in the flipped direction. While actual tiles are still mirrored (cameras avoid certain angles where seams are too obvious), proper lighting somewhat conceals this cheap trick with geometry from human’s eyes.

Here you can see that with correct lightmaps shadows appear on correct side of both flipped and regular tiles:

After the terrain we draw a sky hemisphere object with the basic DiffuseShader, and then draw 11 cloud sprites. Then we draw a sun sprite. These transparent objects are rendered without writing to the depth buffer. Clouds and sun have trimmed geometries for less overdraw. You can read about this optimized sprites technique here. We decided not to use soft particles for clouds because scene size allowed us to place them so that they don’t intersect with other geometries while still partially occluding some peaks. Not using soft particles is beneficial for performance because they require additional render pass to render scene depth.

Terrain shaders breakdown

The main object in the scene is obviously the terrain. And it must look good while maintaining acceptable performance. Here I’ll explain some optimizations and tricks used to achieve a balance between these two mutually exclusive goals.

Terrain shader applies the following effects to the base diffuse color:

Water reflection
Baked lightmap
Fog

This allows terrain to have crisp shadows, subtle atmospheric fog and the sun reflection in the water creeks and puddles created by thawed snow. The last one is a small detail but really improves overall scene quality when viewed against the sun:

So in addition to diffuse texture and two lightmaps (for regular and flipped tiles), this requires a separate specular channel for water. And these textures are really large ones — 4096x4096 pixels, so that’s quite a lot of data. To optimally store this information, we use only two large textures and one small auxiliary one. First texture is necessarily a diffuse map. The second one is a combined lightmap, which contains two lightmaps for regular and flipped tiles in red and green channels. Blue channel is used to store the water specular reflection map. But wait, you may say, in sunrise and sunset scenes it is clearly seen that lightmaps are colored! How RGB data can be stored in a single channel? That’s why we use that auxiliary texture. It is a small color ramp — a 256x1 gradient for coloring greyscale lightmap.

Assuming that the virtual sun is positioned alongside the same axis of the scene as the flipped skirt tiles, we can optimize this even further. This way we can actually have only two lightmaps rendered — for high and low sun position. We can treat the regular lightmap channel as a sun direction and the flipped one as a “sun is on the opposite side of sky” direction. This allows us to reuse the same “high sun” lightmap for day/night and “low sun” lightmap for sunrise/sunset, by merely swapping regular and flipped channels for different times of day.

Let’s take a look at shader source code. It is in the file TerrainWaterShader.ts. At the very end of fragment shader code you can uncomment one of 6 lines to visualize intermediate passes shown on GIF above. You may notice that shader doesn’t consume normals from any attribute and instead in calculation of specular reflection we use constant normal. This is another optimization to reduce geometry size — geometry indeed doesn’t have normals because water is placed in almost perfectly flat part of terrain and accurate vertex normal can be substituted with constant upward normal.

For skirt terrain we use a simplified version of the shader without water reflection — TerrainShader.ts.

In the MountainsRenderer in initShaders() method you may see that we create a pair of each terrain shader — with water and simplified one, both regular and flipped.

Shader precision

You may notice that the terrain fragment shader for the skirt has reduced floating point precision. For main terrain we need highp precision for correct rendering of water, and since the skirt version doesn’t have these reflections, mediump is enough.

At first this may look like a minor optimization, however it is actually a quite important one because it runs noticeably faster, meaning GPU load is reduced. Even when tested on not state-of-the-art Pixel 3 phone, both precisions result in steady 60 fps. However, reducing load on GPU means it has more free power to draw smoother UI and reduces overall power consumption which is very important for live wallpapers.

In general, highp instructions on modern mobile GPUs are twice as slow as mediump or lowp. Obviously shader has a bunch of other non-math instructions to run, so what impact does reducing precision have? While this value is different for different GPUs, we can use some tools to measure it. For example, an offline PowerVR shader compiler can be used to analyze it for this specific hardware. And targeting PowerVR Series6 GPUs we get 18 cycles for highp and 13 cycles for mediump shaders. This is 28% of performance increase for a shader which is used to draw quite a significant part of a scene’s fragments.

Targeting different OpenGL ES versions for Android

This is our first Android live wallpaper which doesn’t support OpenGL ES 2.0 at all. Only 10% of Android devices are limited to OpenGL ES 2.0 and these must be some really old, outdated devices. So we support only OpenGL ES 3.0 and up — app has two sets of resources for ES 3.0 and ES 3.2. For devices with ES 3.0 we use ETC2 textures which provide acceptable image quality at the same size as ETC1. However, compression is still not enough to keep textures small so we had to downsample them for ES 3.0. On devices with ES 3.2 we use more advanced ASTC compression for textures with better quality and better compression. This allows us to use textures of high resolution on modern devices. Here are some sample texture sizes:

Color ramp textures are uncompressed because color accuracy is critical here, but since they are really tiny they don’t use a lot of memory.

Update: In about a week after releasing app, I've compared diffuse texture compressed with ASTC 8x8 and 10x10 blocks. Higher compression definitely introduces some distortions caused by extreme compression. However, on such fuzzy images as aerial terrain imagery it is really hard to tell the difference between compression artifacts and actual random features of terrain. You can see the very similar results when compressing different images into medium-quality JPEG, which also uses fixed 8x8 pixel blocks to compress images. Images with thin sharp lines (like text and diagrams) will have notorious blocky JPEG artifacts, but you won't tell a difference between compressed and original photos of nature. So I've updated app to use even better compressed diffuse texture.

For geometry, both vertex and texture coordinates use half floats. This precision is enough for vertex coordinates, and because we use textures significantly larger than 256 we can’t use bytes for texture coordinates — 8-bit precision for 4096x4096 diffuse texture will be 16 texels.

Final result

Full source code is available on GitHub here and the live demo page is here. Click on the scene to change time of day (it may take a couple seconds to load textures), and by pressing Enter you can enter free-camera mode. Press and hold the right mouse button to look, and use WASD to move around.

And of course you can get an Android live wallpaper app from Google Play here, it is free.

Rendering dunes terrain in WebGL

keaukraine — Tue, 19 Jan 2021 09:41:23 +0000

We’ve released a new live wallpaper for Android and simultaneously published a live demo page showcasing all features of an app. You can check the webpage here.

Historically, the web demo was created first — it was used as a prototyping playground to compose a scene and to fine-tune shaders. Also, this really helps in sharing work between a team of two people without necessity to learn Android Studio for both. And when everything was polished and looked good enough, an Android app was created quite fast based on the web demo code. Porting code to Android is a quite straightforward and easy process because our WebGL framework has the same method signatures as the framework used in Android apps.

Scene composition

Scene is quite simple and contains just six objects — terrain, sky, dust particles, sun, birds, and palm trees.

To examine how objects are rendered, you can take a look at drawScene() method in DunesRenderer.ts — first we render depth map to texture (this is needed for soft particles), then render on-screen objects in front-to-back order (first closest and largest objects, then distant) to efficiently utilize z-buffer culling.
Terrain in the scene is represented as a single square tile. The base for terrain is this model purchased on CGTrader. Its polycount is reduced to 31k faces in order not to split geometry and to draw it with a single draw call. This polycount produces a reasonably good quality. However, its area is not quite large enough to create a feel of infinite sand desert — when the camera is placed slightly above terrain boundaries of square terrain its limits are clearly visible:

Apparently this reduces the range of camera movement and creates an unwanted feeling of terrain “floating” in space. To eliminate this effect and improve the immersiveness of the scene we use a technique called “terrain skirt”. We learned about it from this great GDC talk about terrain in Halo Wars. You should definitely watch the whole video as it explains a lot of other interesting and unique techniques which might come in handy. The idea behind this terrain skirt is to render the same tile at the edges of tile but mirrored away from the center of the scene. This significantly expands the area of terrain. This screenshot shows all 8 additional tiles rendered (with additional gaps to separate tiles):

You can see a mirroring of tiles at the edges where duplicate tiles connect with the main one but it is not noticeable in the final app because the camera is placed only within the main tile avoiding looking at those edges directly. We render additional tiles 1.5 times larger than original ones, effectively increasing perceived dimensions of terrain 4 times. This short clip showcases how final extended terrain looks with and without skirt:

As you can see, this simple trick creates a vast, seemingly endless terrain stretching up to horizon with very little effort and reuses existing geometries.

Dust particles

For dust effect soft particles are used. You can read more about this technique in our previous article — https://dev.to/keaukraine/implementing-soft-particles-in-webgl-and-opengl-es-3l6e.
The only object rendered to a depth texture for soft particles is the main terrain tile because that’s the only geometry particles intersect with. To make this rendering faster, the simplest fragment shader is used to render this object instead of the complex one used to render the on-screen terrain.

Dunes shader

To simulate the effect of wind creating sand waves on the dunes surface we’ve developed a quite complex shader. Let’s take a look inside of it. Please note that while we will explain GLSL code of shader, the generic techniques and approaches used in it can also be applied to recreate similar material in Unity/Unreal engines.
The code of the shader can be found in DunesShader.ts. Let’s analyze it.

Diffuse color and lightmaps

Terrain uses a quite large texture — 2048x2048 for web demo, and up to 4096x4096 in Android app. Obviously it takes quite some memory so to efficiently use it, some tricks were used. The main diffuse color for dunes is actually stored as a single-channel grayscale value in the red channel of terrain texture. Actual color of sand is specified by uColor uniform which is multiplied by grayscale diffuse value. The other 2 channels contain lightmaps for high sun (day and night) and low sun (sunrise and sunset). Since it is not possible to use uniforms for accessing texture data, two versions of shader are compiled for two lightmaps. Final diffuse color is multiplied with shadows color.

Moving sand effect

Next, let’s take a look at how the moving wind effect is created. You may notice that it is different for windward and leeward slopes of dunes. To determine which effect to apply to which slope, we calculate blending coefficients from surface normal. These coefficients are calculated per vertex and are passed into the fragment shader via vSlopeCoeff and vSlopeCoeff2 varyings. You can uncomment corresponding lines in fragment shader to visualize windward and leeward parts with different colors:

Both slopes use the same texture applied to them but windward one is more stretched. Texture coordinates for both slopes are also calculated in vertex shader to prevent dependent texture reads. Wind movement is done by adding offset to texture coordinates from uTime uniform.

Fog

The next important thing to get a realistic result is to apply atmospheric fog. For performance reasons, we use a simple linear fog which is calculated in the vertex shader. Fog range is controlled by two uniforms — fogStartDistance and fogDistance and value to be used in the fragment shader is calculated and stored in vFogAmount varying. Fragment shader applies fog color from uFogColor uniform based on the value of this varying.
Fog color is adjusted for far terrain edges to blend with sky texture. And the sky texture is also edited to have a distant haze of the same fog color in places where it should blend with the terrain.

Detail texture

Even though the overall terrain texture is quite large, it covers a large area and therefore still not detailed enough for close-ups. To make dunes less blurry and more realistic when observed from the ground we apply a detail texture to it. It is a small 256x256 texture which has 2 different sand ripples patterns in 2 channels for different slopes. Detail texture can either darken or lighten diffuse color. To achieve this, first we subtract 0.5 from the detail color so it can have negative value, and then this value is added to the final color. This way, 50% gray color in detail texture doesn’t affect diffuse color, darker values will darken it and brighter values will brighten color. Detail texture is applied the similar way as the fog — it has two uniforms to adjust cutoff distance where detail texture is not needed. You can uncomment a line in fragment shader to visualize detail texture range in red channel:

Result

You can see a live demo page here. It is interactive — you can click to change time of day. And on desktop to examine the scene from any arbitrary position you can go into free flight mode by pressing the Enter key. In this mode, to rotate the camera hold the right mouse button and to move use WASD keys, Space to go up and C to go down. Hold Shift while moving to accelerate.
Full source code is available on GitHub, if you are interested in recreating similar effects you can clone and use it for your needs — it is licensed under permissive MIT license.