Forem: ndesmic

WebGPU Engine from Scratch Part 12: Ambient Lighting

ndesmic — Mon, 22 Dec 2025 20:00:00 +0000

One persistent quality issue with the engine is that the shadows are very hard because things are either lit or not lit. In the real world there's a lot of ambient light that comes in from all angles because the rays are bouncing off the walls, clouds and other objects. We don't take this into account. Realistically, this can't be computed at all because it explodes exponentially tracing the rays as they diffuse into multiple rays. The most sophisticated renderers use things like path-tracing to aggressively estimate light. Our version will do something similar though more suitable for real-time rendering without a ray-tracing support.

The API

There's a few ways we could tackle this. We could create a new type of light "ambient" that just adds this to the scene. I don't want to go this route for a few reasons. The main one is that the shape of the API will different from other lights. In fact, I already think that maybe the different light types should be broken up into different entities rather than try to share the same interface since point lights don't need direction, directional lights to need positions and spotlights (whenever we get around to them) need position, direction and a field of view. In the case of ambient lights we either want a color (which could also encode intensity if components are > 1.0), or we want a cubemap that defines the color of the light coming in from each direction, also known as image based lighting (IBL).

It would also be scoped in a way that lights currently aren't, that is, if it's in a group, then all nested elements in that group would have it apply but not outside since we might have different ambient lights for different parts of the scene. In this case ambient light only applies to two entities, meshes and groups. It would be more annoying I think to look at the group's children to decide the ambient light rather than make it a property of the group itself. Even if we were to decide later to change that, the lookup would come from the group so the group would still have methods to get the ambient light for its children so I think that an attribute will be the best place to start.

The fun part here is dealing with the types and multi-inheritance for Transformable and the new AmbientLit class.

Adventures in jsdoc

So we're going to try to do some fancy stuff to make the types work. As usual we'll be using jsdoc to avoid any build-time or tooling (we have deno which recently re-added the bundle command so we don't actually need another tool which is nice, but maybe you don't want to use deno for web code and 0 performance overhead is still infinitely better than ts-go). It will cost us a little bit of ugly syntax though, I don't blame you if you're following along and want go full ts.

Getting interfaces to work in jsdoc is somewhat annoying. At first I thought you could just annotate the interface class with @interface but this will force you to implement the private fields as well (which doesn't actually work) so I feel that's just wrong and should be fixed. We actually have to create the interface separately which is a bit of a shame because it means keeping them up-to-date. Try as I might you can't just create the interface using jsdoc like this:

/**
 * @typedef {{ x?: number, y?: number, z?: number }} TransformValues
 *  

 * @typedef {Object} ITransformable
 * @property {Float32Array} modelMatrix
 * @property {Float32Array} worldMatrix
 * @property {(params: TransformValues) => this} translate
 * @property {(params: TransformValues) => this} scale
 * @property {(params: TransformValues) => this} rotate
 * @property {() => void} resetTransforms
 */

The reason is because we're also using ESM and jsdoc apparently was never updated to handle ESM, there's no way to export typedefs and they get block scoped in the module. So we could put it in the same file as mesh.js but that's messy and not properly segmenting them by function. But what we can do is a d.ts file to hold the types. While this is technically using typescript these d.ts files are never shipped with the code so they just exist for the editor (normally ts compiles into these file). They are easy to write because they just contain pure type typescript.

//transformable.d.ts
export type TransformValues = { x?: number, y?: number, z?: number };

export interface ITransformable {
  readonly modelMatrix: Float32Array;
  readonly worldMatrix: Float32Array;
  translate(params: TransformValues): this;
  scale(params: TransformValues): this;
  rotate(params: TransformValues): this;
  resetTransforms(): void;
}

The next problem is that we can't actually import them the normal way, again because of jsdoc ESM compatibility. So instead of:

/** @implements(import("./transformable.d.ts").ITransformable)

We need

/** @typedef {import("./transformable.d.ts").ITransformable} ITransformable
* @implements {ITransformable}
*/

You need the typedef for it to pick up. However with that it will now check assuming you add @ts-check to the top of the mesh.js file. We now have full type-checking on the interface. There are a few other type errors though. For example in bakeTransforms there's a line like this:

const transformedNormals = chunk(this.normals, this.normalSize)
    .map(values => multiplyMatrixVector(normalMatrix, values, this.normalSize)) //error on values
    .map(values => normalizeVector(values))
    .toArray();

There will be an error on value because it thinks its a any[] and not compatible with Float32Array<ArrayBuffer>. This is a nasty little hangup because while TypedArrays are compatible with Arrays in this way typescript doesn't know that. Without getting too wild we can just override the typing to work:

onst transformedNormals = chunk(this.normals, this.normalSize)
    .map(values => multiplyMatrixVector(normalMatrix, /** @type {Float32Array} */ (/** @type {unknown} */ (values)), this.normalSize))
    .map(values => normalizeVector(values))
    .toArray();

Since we can't directly cast any[] to Float32Array because they are incompatible we first cast to unknown and then to Float32Array. It's a weird typescript pattern that occasionally comes up.

Creating a multi-inheritance pattern

Now that we can type to make sure we're doing things correctly let's implement the interface. We can keep transformable.js and just create a new Transformable inside of the Mesh class (Transformable should also implement ITransformable.

/**
 * @implements {ITransformable}
 */
export class Mesh {
    //...existing code
    #transformable;

    constructor(mesh) {
        this.#transformable = new Transformable();
        //...existing code
    }

    //...existing code
    get modelMatrix(){
        return this.#transformable.modelMatrix;
    }
    get worldMatrix(){
        return this.#transformable.worldMatrix;
    }

    //Transformable
    translate(params){
        this.#transformable.translate(params);
        return this;
    }
    scale(params){
        this.#transformable.scale(params);
        return this;
    }
    rotate(params){
        this.#transformable.rotate(params);
        return this;
    }
    resetTransforms(){
        this.#transformable.resetTransforms();
    }
}

I believe this is called the "composite" pattern but it's been a while. Do the same with the AmbientLit:

//ambient-lit.d.ts
export interface IAmbientLit {
    ambientLightMap: any;
}

//amient-lit.js
//@ts-check
/** @typedef {import("./ambient-lit.d.ts").IAmbientLit} IAmbientLit */

/** @implements {IAmbientLit} */
export class AmbientLit {
    #ambientLightMap;

    constructor(options = {}){
        this.#ambientLightMap = options.ambientLightMap;
    }

    set ambientLightMap(val){
        this.#ambientLightMap = val;
    }
    get ambientLightMap(){
        return this.#ambientLightMap;
    }
}

//mesh.js
/**
 * @implements {ITransformable}
 * @implements {IAmbientLit}
 */
export class Mesh {
    #ambientLit;
    //...existing code
    constructor(){
        this.#ambientLit = new AmbientLit();
    }

    set ambientLightMap(val){
        this.#ambientLit.ambientLightMap = val;
    }
    get ambientLightMap(){
        return this.#ambientLit.ambientLightMap;
    }
}

If we need more of these patterns in the future this is how we can do them (there are other ways using more non-class objects but modern JS for some reason really wanted to push us into them so I don't want to fight it.)

A simple single color case

There are multiple ways we could do this. The simplest is to say if the ambient light is defined as a color then we add that color. The problem here is that when we expand to the IBL version we need to branch depending on the data that we have. I think the simplest way is to force everything through a unified pipeline even if simple cases are more complex. To this end, we need a way to create a cube map of a single color (in reality it will use an array of colors that are the same value). We already have createColorTexture from gpu-utils.js so let's just expand that.

/**
 * Creates a 1x1 texture of a color or a layered texture for an array of colors, colors are in float format
 * @param {GPUDevice} device
 * @param {{ label?: string, color?: [number, number, number, number], colors?: [number, number, number, number][] }}
 * @returns 
 */
export function createColorTexture(device, options = {}) {
    const colors = options.colors ?? [options.color];
    const size = {
        height: 1,
        width: 1,
        depthOrArrayLayers: colors.length
    };

    const texture = device.createTexture({
        label: options.label,
        size,
        format: 'rgba8unorm',
        usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST,
    });

    colors.forEach((color, layer) => {
        const texel = color
        ? new Uint8Array(color.map(v => v*255))
        : new Uint8Array([255, 255, 255, 255]);

        device.queue.writeTexture(
            { texture, origin: { x: 0, y: 0, z: layer } },
            texel,
            { bytesPerRow: 4, rowsPerImage: 1 },
            { width: 1, height: 1, depthOrArrayLayers: 1 }
        );
    });

    return texture;
}

There was actually an old bug here where the float values were not scaled into int values. Anyway, now if we pass it an array it will make a layered texture. But now we need to update <geo-texture> to support multiple colors. First we need to parse these attributes:

export function parseListOfFloatVector(text, length, defaultValue){
    return text?.trim()
        ? text.split(";").map(v => v.trim().split(",").map(x => parseFloat(x.trim())).slice(0, length))
        : defaultValue
}

Since , is already a separator for vector elements, we use ; as the outer list delimiter so we can define attributes with multiple colors. This is immediately useful for the debug-cubemap because we can eliminate the external images and define it inline.

async function parseTexture(textureEl){
    const name = textureEl.getAttribute("name");
    const src = textureEl.getAttribute("src");
    const srcs = parseListOrDefault(textureEl.getAttribute("srcs"));
    const color = textureEl.getAttribute("color");
    const colors = textureEl.getAttribute("colors");
    let value;
    if (src) {
        value = { entity: "texture", image: await loadImage(src), name  };
    } else if(srcs){
        value = { entity: "texture", images: await Promise.all(srcs.map(s => loadImage(s))), name } 
    } else if (color) {
        value = { entity: "texture", color: parseFloatVector(color, 4), name };
    } else if (colors){
        value = { entity: "texture", colors: parseListOfFloatVector(colors, 4), name }
    }

    return value;
}

So in this case we can use the plural colors to define a list of colors for <geo-texture>.

- <geo-texture name="debug-background" srcs="../../img/debug-cube/red-right.png, ../../img/debug-cube/red-left.png, ../../img/debug-cube/green-top.png, ../../img/debug-cube/green-bottom.png, ../../img/debug-cube/blue-back.png, ../../img/debug-cube/blue-front.png"></geo-texture>
+ <geo-texture name="debug-background" colors="1,0,0,1; 0.25,0,0,1; 0,1,0,1; 0,0.25,0,1; 0,0,1,1; 0,0,0.25,1"></geo-texture>

Nice little upgrade.

Struct packing 201

I want to pass the ambient light data to the pbr shader but the light bindgroup is already a bit packed. I see some ability to optimize the light data into a single struct so let's try that first. The first thing is to change the signature of packStruct and packArray to take options because passing empty parameters is ugly and error prone.


/**
 * @typedef {[string,GpuType | Prop[]]} Prop
 * @typedef {Prop[]} Schema
 * 
 * @param {object} data 
 * @param {Schema} schema
 * @param {{ minSize?: number, buffer?: ArrayBuffer, offset?: number }} options
 */
export function pack(data, schema, options = {}){
    const offset = options.offset ?? 0;

    if(Array.isArray(data)){
        const { totalSize: structSize } = getAlignments(getValuesFromEntriesRecursive(schema), { minSize: options.minSize });
        const outBuffer = options.buffer ?? new ArrayBuffer(structSize * data.length);

        for(let i = 0; i < data.length; i++){
            pack(data[i], schema, {
                minSize: options.minSize, 
                buffer: outBuffer, 
                offset: offset + i * structSize
            });
        }
        return outBuffer;
    } else {
        const lastSchema = schema.at(-1);
        const lastProp = data[/**@type {Prop} */(lastSchema)[0]]; 
        const count = (Array.isArray(lastProp) && Array.isArray(/** @type {Prop} */(lastSchema)[1])) ? lastProp.length : 1; //if last data and schema are arrays then it's an array
        const { offsets, totalSize } = getAlignments(getValuesFromEntriesRecursive(schema), { minSize: options.minSize, arrayCount: count });
        const outBuffer = options.buffer ?? new ArrayBuffer(totalSize);
        const dataView = new DataView(outBuffer);

        for(let i = 0; i < schema.length; i++){
            let type;
            let name;
            let value;

            if(Array.isArray(schema[i])){
                name = schema[i][0];
                type = schema[i][1];
                value = data[name];
            } else {
                type = schema[i];
                value = data;
            }

            if(value === undefined){
                throw new Error(`Value lookup for prop '${name}' failed!  Double check the prop name is correct.`);
            }
            //TODO: add other GPU Types
            const totalOffset = offset + offsets[i];
            switch(type){
                case "i32": {
                    dataView.setInt32(totalOffset, value, true);
                    break;
                }
                case "u32": {
                    dataView.setUint32(totalOffset, value, true);
                    break;
                }
                case "f32": {
                    dataView.setFloat32(totalOffset, value, true);
                    break;
                }
                case "vec2u32": {
                    dataView.setUint32(totalOffset, value[0], true);
                    dataView.setUint32(totalOffset + 4, value[1], true);
                    break;
                }
                case "vec2f32": {
                    dataView.setFloat32(totalOffset, value[0], true);
                    dataView.setFloat32(totalOffset + 4, value[1], true);
                    break;
                }
                case "vec3f32": {
                    dataView.setFloat32(totalOffset, value[0], true);
                    dataView.setFloat32(totalOffset + 4, value[1], true);
                    dataView.setFloat32(totalOffset + 8, value[2], true);
                    break;
                }
                case "vec4f32": {
                    dataView.setFloat32(totalOffset, value[0], true);
                    dataView.setFloat32(totalOffset + 4, value[1], true);
                    dataView.setFloat32(totalOffset + 8, value[2], true);
                    dataView.setFloat32(totalOffset + 12, value[3], true);
                    break;
                }
                case "mat2x2f32": {
                    dataView.setFloat32(totalOffset, value[0], true);
                    dataView.setFloat32(totalOffset + 4, value[1], true);

                    dataView.setFloat32(totalOffset + 8, value[2], true);
                    dataView.setFloat32(totalOffset + 12, value[3], true);
                    break;
                }
                case "mat3x3f32": {
                    dataView.setFloat32(totalOffset, value[0], true);
                    dataView.setFloat32(totalOffset + 4, value[1], true);
                    dataView.setFloat32(totalOffset + 8, value[2], true);

                    dataView.setFloat32(totalOffset + 16, value[3], true);
                    dataView.setFloat32(totalOffset + 20, value[4], true);
                    dataView.setFloat32(totalOffset + 24, value[5], true);

                    dataView.setFloat32(totalOffset + 32, value[6], true);
                    dataView.setFloat32(totalOffset + 36, value[7], true);
                    dataView.setFloat32(totalOffset + 40, value[8], true);
                    break;
                }
                case "mat4x4f32": {
                    dataView.setFloat32(totalOffset, value[0], true);
                    dataView.setFloat32(totalOffset + 4, value[1], true);
                    dataView.setFloat32(totalOffset + 8, value[2], true);
                    dataView.setFloat32(totalOffset + 12, value[3], true);

                    dataView.setFloat32(totalOffset + 16, value[4], true);
                    dataView.setFloat32(totalOffset + 20, value[5], true);
                    dataView.setFloat32(totalOffset + 24, value[6], true);
                    dataView.setFloat32(totalOffset + 28, value[7], true);

                    dataView.setFloat32(totalOffset + 32, value[8], true);
                    dataView.setFloat32(totalOffset + 36, value[9], true);
                    dataView.setFloat32(totalOffset + 40, value[10], true);
                    dataView.setFloat32(totalOffset + 44, value[11], true);

                    dataView.setFloat32(totalOffset + 48, value[12], true);
                    dataView.setFloat32(totalOffset + 52, value[13], true);
                    dataView.setFloat32(totalOffset + 56, value[14], true);
                    dataView.setFloat32(totalOffset + 60, value[15], true);
                    break;
                }
                default: {
                    if(Array.isArray(type)){
                        if(Array.isArray(value) && i !== (schema.length - 1)){
                            throw new Error("Array must be the last element in a struct!")
                        }
                        pack(value, /** @type {Prop[]}*/(type), { buffer: outBuffer, offset: totalOffset });
                    } else {
                        throw new Error(`Cannot pack type ${type} at prop index ${i} with value ${value}`);
                    }
                }
            }
        }
        return outBuffer;
    }
}

/**
 * @param {GpuType[]} typesToPack
 * @param {{ minSize?: number, arrayCount?: number }} options
 */
export function getAlignments(typesToPack, options = {}){
    let offset = 0;
    let maxAlign = 0;
    const offsets = new Array(typesToPack.length);

    for(let i = 0; i < typesToPack.length; i++){
        let align;
        let size;
        if(Array.isArray(typesToPack[i])){
            const alignSize = getAlignments(/** @type {GpuType[]} */(typesToPack[i]));
            align = alignSize.maxAlign;
            size = alignSize.totalSize * (options.arrayCount ?? 1);
        }else {
            const alignSize = gpuTypeAlignSize[typesToPack[i]];
            align = alignSize.align;
            size = alignSize.size;
        }   

        if(maxAlign < align){
            maxAlign = align;
        }

        offset = getPaddedSize(offset, align);
        offsets[i] = offset;
        offset += size;
    }
    return {
        offsets,
        maxAlign,
        totalSize: getPaddedSize(offset, maxAlign, options.minSize)
    };
}

I've combined all the logic into 2 functions, the main one is pack which will pack arrays or structs, it doesn't matter (the align table has also been converted to objects from tuples). I've also upgraded it to handle array-in-struct cases since it didn't before. Array alignment is based on whatever the align of the structs members themselves are. Furthermore, a struct may only have one array and it must be at the very end. This is a restriction WGSL has, and writing the updated packer I kind of see why, it's hard to know where the end of the data structure is if the arrays of arbitrary length are in the middle. With this new packer we can also do nested structs, it basically just checks if the type is an array and then if the value is a scalar it packs the nested struct, otherwise packs the nested array. The arrays themselves can be scalar value by making the type a string instead of a tuple.

I used typescript to help me update the call sites but this required adding @ts-check to gpu-engine.js and lighting the file up in red. I did find a few places where I made mistakes but there were other problems like the lack of webgpu types:

deno add @webgpu/types

//deno.json
{
  "imports": {
    "@webgpu/types": "npm:@webgpu/types@^0.1.65",
  },
  "lint": {
    "rules": {
      "exclude": [
        "no-explicit-any"
      ]
    }
  },
  "compilerOptions": {
    "noImplicitAny": false,
    "types": [
      "@webgpu/types"
    ],
    "lib": [
      "esnext",
      "dom"
    ]
  }
}

You'll see some new coercing strategies in the code as a result because I had to appease the checker. Let's update the bind group for lights:

setMainLightBindGroup(passEncoder, bindGroupLayouts, lights, shadowMaps) {
    let shadowMapIndex = 0;
    const shadowMappedLights = lights
        .entries()
        .map(([key, value]) => {
            const shadowMap = shadowMaps.get(key);
            const shadowMapAspectRatio = shadowMap.width / shadowMap.height;
            const combinedModelMatrix = multiplyMatrix(value.worldMatrix, [4,4], value.modelMatrix, [4,4]);
            return {
                typeInt: value.typeInt,
                position: multiplyMatrixVector(combinedModelMatrix, value.position, 4),
                direction: multiplyMatrixVector(combinedModelMatrix, value.direction, 4),
                color: value.color,
                shadowMap,
                projectionMatrix: shadowMap ? getLightProjectionMatrix(shadowMapAspectRatio) : getEmptyMatrix([4, 4]), //probably needs transpose
                viewMatrix: shadowMap ? getLightViewMatrix(value.direction) : getEmptyMatrix([4, 4]), //probably needs transpose
                castsShadow: value.castsShadow ? 1 : 0,
                shadowMapIndex: (value.castsShadow && shadowMap) ? shadowMapIndex++ : -1
            };
        }).toArray();
    const shadowMapsToBind = shadowMappedLights
        .filter(lightData => lightData.shadowMapIndex > -1)
        .map(lightData => lightData.shadowMap);
    const lightData = pack(
        {
            lights: shadowMappedLights,
            lightCount: shadowMappedLights.length,
        },
        [
            ["lightCount", "u32"],
            ["lights", [
                ["typeInt", "u32"],
                ["position", "vec3f32"],
                ["direction", "vec3f32"],
                ["color", "vec4f32"],
                ["projectionMatrix", "mat4x4f32"],
                ["viewMatrix", "mat4x4f32"],
                ["castsShadow", "u32"],
                ["shadowMapIndex", "i32"]
            ]]
        ],
        { minSize: 64 }
    );
    const lightBuffer = this.#device.createBuffer({
        size: lightData.byteLength,
        usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST | GPUBufferUsage.STORAGE,
        label: "main-light-buffer"
    });
    this.#device.queue.writeBuffer(lightBuffer, 0, lightData);
    const placeholderView = shadowMaps.get("placeholder").createView({ label: "placeholder-view" });
    const lightBindGroup = this.#device.createBindGroup({
        label: "main-light-bind-group",
        layout: bindGroupLayouts.get("lights"),
        entries: [
            {
                binding: 0,
                resource: {
                    buffer: lightBuffer,
                    offset: 0,
                    size: lightData.byteLength
                }
            },
            {
                binding: 1,
                resource: this.#samplers.get(DEFAULT_SHADOW_SAMPLER)
            },
            ...(getRange({ end: 3 }).map((index) => {
                const shadowMap = shadowMapsToBind[index];
                return {
                    binding: index + 2, //manually offset bind index
                    resource: shadowMap ? shadowMap.createView({ label: `shadow-view-${index}` }) : placeholderView
                };
            })),
        ]
    });
    passEncoder.setBindGroup(2, lightBindGroup);
}

We can now combine two of the bindings into one which is nice. You'll also need to update the pipeline too (not shown) as well as the shader (also not shown) but these should be fairly obvious.

Passing the ambient light map

First we need to parse it off the element.

//geo-markup-parser.js
function updateMeshAttributes(meshEl, mesh) {
    //..

    const ambientLightMap = meshEl.getAttribute("ambient-light-map");
    const ambientLightMap = meshEl.getAttribute
    //...
}

Simple. We'll also need to setup a placeholder in-case we don't define it.

//gpu-engine.js
async initializeScene(scene) {
    //...
    this.#textures.set(PLACEHOLDER_CUBEMAP, createColorTexture(this.#device, { label: "placeholder-cubemap", colors: [
        [0,0,0,1],[0,0,0,1],[0,0,0,1],[0,0,0,1],[0,0,0,1],[0,0,0,1]
    ] }));
    //..
}

It's important that it's black because we'll just use it as normal, it just won't contribute anything.

setMainLightBindGroup(passEncoder, bindGroupLayouts, lights, shadowMaps, ambientLightMap) {
    //...
    const lightBindGroup = this.#device.createBindGroup({
        label: "main-light-bind-group",
        layout: bindGroupLayouts.get("lights"),
        entries: [
            //...
            {
                binding: 6,
                resource: this.#textures.get(ambientLightMap ?? PLACEHOLDER_CUBEMAP).createView({ dimension: "cube", label: "ambient-light-cube-view"}) 
            }
        ]
    });
    //...
}

As the final binding we'll pass in the cubemap defined as the ambient light map or the placeholder if none.

Lastly let's update the shader:

//pbr.wgsl
@group(2) @binding(0) var<storage, read> light_data: LightData;
@group(2) @binding(1) var shadow_sampler: sampler_comparison;
@group(2) @binding(2) var shadow_map_0: texture_depth_2d;
@group(2) @binding(3) var shadow_map_1: texture_depth_2d;
@group(2) @binding(4) var shadow_map_2: texture_depth_2d;
@group(2) @binding(5) var shadow_map_3: texture_depth_2d;
+@group(2) @binding(6) var ambient_light_map: texture_cube<f32>;

@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32>
{   
    var surface_albedo = textureSample(albedo_map, albedo_sampler, frag_data.uv).rgb;
    var roughness_from_map = textureSample(roughness_map, roughness_sampler, frag_data.uv).x;
    var roughness = max(mix(material.roughness, roughness_from_map, f32(material.use_specular_map)), 0.0001);
    var f0 = mix(vec3(0.04, 0.04, 0.04), material.base_reflectance, material.metalness);
    var total_color = vec3(0.0);
    var normal = round_small_mag_3(normalize(frag_data.normal));

    let i = 0u;
    for(var i: u32 = 0; i < light_data.light_count; i++){
        let light = light_data.lights[i];
        let light_distance = length(light.position - frag_data.world_position.xyz);
        var to_light = vec3(0.0);

        switch light.light_type {
            case 0: { //point
                to_light = normalize(light.position - frag_data.world_position.xyz);
            }
            case 1: { //directional
                to_light = normalize(-light.direction);
            }
            default: {}
        }

        let attenuation = 1.0 / pow(light_distance, 2.0);
        let radiance = light.color.rgb * attenuation;
        let lit_color = get_bdrf(
            surface_albedo, 
            f0, 
            roughness, 
            material.metalness,
            normal, 
            radiance, 
            to_light,
            scene.camera_position, 
            frag_data.world_position.xyz
        );
        let diffuse_factor = max(dot(normal, to_light), 0.0);
        let shadow = get_shadow(frag_data.world_position, diffuse_factor);
        let shadowed_color = lit_color * shadow;

+       let ambient_light = textureSample(ambient_light_map, albedo_sampler, normal);

        total_color += shadowed_color;
+       total_color += ambient_light.rgb;
    }

    let tone_mapped_color = total_color / (total_color + vec3(1.0));
    return vec4(pow(total_color, vec3(1.0/2.2)), 1.0);
}

I'm recycling the albedo sampler since it a simple bilinear filter but maybe we could pass one in too, I just don't see a point.

The effect is interesting on a white teapot:

This is not yet correct but you can kinda see what it's going for using the very simple debug environment. It looks like we're getting indirect lighting from the walls. However there are two big issues, one is that we're not representing the other object, the rug and getting leakage from the floor. The other is that we're only measuring light from a single point along the normal, realistically we need all of the light from the hemisphere around each point. The nice part is that we can precompute this with a slightly different sort of cubemap built specifically for this purpose called an irradiance map.

Light probes

To build this map we'll need to have a concept of a light probe which is kinda like a virtual light measuring camera that sits in the environment. For the sake of authoring it will be part of the markup although I'm not excited about that because it is a specific concept tied to this implementation, I just don't have a better way to model it yet.

The basic idea will look like this:

<geo-probe type="irradiance" position="0, 0, 0" output-name="debug-irradiance-map" samples="10"></geo-probe>

We have an element called <geo-probe> that has a type (maybe we'll make other types for things like specular light), a position in the environment, a texture name to output to and the number of samples which will be how many samples we take per texel (this number will impact quality but we can't take every angle in the hemisphere because that would require infinite compute so we take a finite number of samples and average them).

//geo-markup-parser.js
function parseProbe(probeEl){
    const name = probeEl.getAttribute("name");
    const type = probeEl.getAttribute("type");
    const outputName = probeEl.getAttribute("outputName");
    const samples = parseInt(probeEl.getAttribute("samples"), 10);
    const position = parseFloatVector(probeEl.getAttribute("position"), 3);
    const resolution = parseInt(probeEl.getAttribute("resolution"));

    const probe = new Probe({
        name,
        type,
        outputName,
        position,
        samples,
        resolution
    });

    return probe;
}

Hopefully the plumbing is pretty natural now so I'm going to skip over that but all we need to do for now is to add it to the probe set. We also need the texture to draw to.

//gpu-engine.js
initializeProbe(probe, defaultName){
    const key = probe.name ?? defaultName;
    this.#probes.set(key, probe);
    this.#textures.set(`${key}-cubemap`, this.#device.createTexture({
        size: [probe.resolution, probe.resolution, 6],
        format: "rgba32float",
        usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
    }));
}

Refactoring the pipeline

I'm not going to go too deep into the changes because they are large but the basic code has not changed. I was finding it hard to work the existing pipelines because they had a lot of code cluttering gpu-engine.js and having to add a 4th was just too much. So instead that code was moved into classes under the pipelines folder. Each class implements and interface with createPipeline which sets up the pipeline descriptor, and render which does the rendering for that pipeline since these are always paired. This also means that the bind group code moves in for each pipeline class. There's a bunch of changes for types and to fit the shared interface, using types really helps to make sure everything is still working.

Since they are all bundled up this makes it much easier to add new pipelines and declutters a lot. It also means things like tracking the bindgroups doesn't require extra data structures. Worth it but it was a lot of work (maybe LLMs could have done okay at this task?), the number of file changes in the PR will increase a lot because of this, but it's mostly code moving, not changing, however one change was to stop using multiple command encoders (this was a mistake I didn't catch early on) and put all passes into a single one which I'm sure has big performance improvements.

Rendering the scene as a cubemap

For the next part I'm going to add a new phase to the engine. Before we start rendering we'll have a phase where the preprocessing stuff happens. This way we can separate out those things and maybe show a loading indicator while we wait for them to process.

In this preprocess step we can setup another pipeline to create the irradiance map. The first part of this step will be very similar to the main pipeline in terms of passes and bindings but it will render a cubemap. This is sort of a caching step as the next step will be to collect light sample data for each texel of the irradiance map using that cubemap rendering.

One of the biggest differences for rendering is we'll need to get the cameras for each side of the cube map. This is done by taking the probes and making a view matrix in each of the 6 directions. The projection matrix is always a 1:1 ratio so we just need one resolution measurement and it always encompasses 90 degrees. The near and far plane can be adjusted as necessary, perhaps parameterized (I didn't go that far).

//probe.js
import { getProjectionMatrix } from "..//utilities/vector.js";

export class Probe {
    static FIELD_OF_VIEW = 90;
    static NEAR = 0.01;
    static FAR = 5;

    #name;
    #type;
    #position;
    #outputName;
    #samples;
    #resolution;

    constructor(probe){
        this.name = probe.name;
        this.type = probe.type;
        this.position = probe.position;
        this.outputName = probe.outputName;
        this.samples = probe.samples;
        this.resolution = probe.resolution ?? 32;
    }

    static getProjectionMatrix(resolution) {
        return getProjectionMatrix(
            resolution,
            resolution,
            Probe.FIELD_OF_VIEW,
            Probe.NEAR,
            Probe.FAR
        );
    }

    set name(val){
        this.#name = val;
    }
    get name(){
        return this.#name;
    }

    set type(val){
        this.#type = val;
    }
    get type(){
        return this.#type;
    }

    set position(val){
        if(val.length === 3){
            this.#position = new Float32Array([...val, 1]);
        } else {
            this.#position = new Float32Array(val);
        }
    }
    get position(){
        return this.#position;
    }

    set outputName(val){
        this.#outputName = val;
    }
    get outputName(){
        return this.#outputName;
    }

    set samples(val){
        this.#samples = val;
    }
    get samples(){
        return this.#samples;
    }

    set resolution(val){
        this.#resolution = val;
    }
    get resolution(){
        return this.#resolution;
    }
}

    /**
     * @param {GPUDevice} device
     * @param {Mesh | Group} root
     * @param {AttachmentViews} attachmentViews }
     * @param {{
     *  meshContainers: Map<Mesh, MeshContainer>,
     *  lights: Map<string | symbol, Light>,
     *  shadowMaps: Map<string | symbol,  GPUTexture>,
     *  textures: Map<string | symbol, GPUTexture>,
     *  samplers: Map<string | symbol, GPUSampler>,
     *  materials: Map<string | symbol, Material>,
     *  probes: Map<string | symbol, Probe>,
     *  cameras: Map<string | symbol, Camera>,
     *  primaryCamera: Camera,
     *  background: IBackground,
     *  commandEncoder: GPUCommandEncoder
     * }} info
     */
    render(device, root, attachmentViews, info) {
        const commandEncoder = info.commandEncoder ?? device.createCommandEncoder({
            label: "cubemap-render-command-encoder",
        });

        const width = 720;
        const height = 720;
        const sceneCubeMap = device.createTexture({
            size: [height, width, 6],
            format: "rgba8unorm",
            usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING,
            dimension: "2d"
        });
        const sceneDepthMap = device.createTexture({
            label: "depth-texture",
            size: {
                width,
                height,
                depthOrArrayLayers: 1
            },
            format: "depth32float",
            usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
        });

        info.textures.set("ir-test", sceneCubeMap);

        for (const probe of info.probes.values()) {
            for (let i = 0; i < 6; i++) {
                const viewMatrix = getWorldToCameraMatrixFromDirection(probe.position, CUBEMAP_DIRECTIONS[i]);
                const colorView = sceneCubeMap.createView({ dimension: "2d", baseArrayLayer:  i, arrayLayerCount: 1, label: `cubemap-render-scene-cube-map-${i}` });
                const depthView = sceneDepthMap.createView({ label: "cubemap-render-scene-depth-map" });
                const attachmentViews = { colorView, depthView };

                const innerInfo = { ...info, commandEncoder, 
                    primaryCamera: new Camera({ 
                        name: `cubemap-render-map-direction-${i}`,
                        position: probe.position,
                        direction: CUBEMAP_DIRECTIONS[i],
                        fieldOfView: Probe.FIELD_OF_VIEW, 
                        near: Probe.NEAR, 
                        far: Probe.FAR, 
                        screenWidth: probe.resolution, 
                        screenHeight: probe.resolution,
                        isPerspective: true
                    })
                };

                this.#shadowPipeline?.render(device, root, attachmentViews, innerInfo);
                this.renderScene(device, commandEncoder, root, attachmentViews, innerInfo, viewMatrix, probe, i);
                this.#backgroundPipeline?.render(device, info.background.mesh, attachmentViews, innerInfo);
            }
        }
        if(!info.commandEncoder){
            device.queue.submit([commandEncoder.finish()]);
        }
    }

For each probe, we'll iterate over all 6 sides and render and image and finally arrange all 6 of those images into the cubemap. The first thing was just to copy the pbr.wgsl because most of it is still relevant. I removed the ambient light map (since it doesn't exist in this phase) and using the six different view matrices as the camera rendered to the six layers of the cubemap texture. The nice part about abstracting the pipelines is we can now nest them inside one-another so we can just reuse the shadow and background pipeline as-is. To test, I moved the probe to 0,0,-2 and showed the +Z side showing the teapot to test out that the shader is working. The output looks like this:

The whole cubemap looks like this:

Once we can render all 6 sides into a cubemap we then need to use that cube map and then do the work to sample the light coming into our point.

Creating the irradiance map

Finally we can start sampling. We take some parameter that defines the sample count and carve it up across the two dimensions we sample over, in this case we're using spherical coordinates and so it's theta and phi. To get the number of steps per dimension we take the square root of the total samples.

Each sample needs to be scaled by cos(phi). This is basically dot(N, to_light) to get the weighted amount of light given the normal. This could theoretically could be done during rendering like we do with diffuse shading but it's easier and slightly cheaper to precompute this and bake it into the irradiance map. We also need to scale by sin(phi). This term comes from the fact that we are integrating over a spherical surface grid, while d_phi is constant d_theta isn't because as phi gets us closer to the poles the size of the arcs traced is smaller.

Finally we weight each sample by d_phi * d_theta the area of the little patch it represents (these would add up to the total hemisphere area).

const PI = 3.14159265359;
const TWO_PI = 2 * PI;
const HALF_PI = PI / 2;

struct VertexOut {
    @builtin(position) frag_position : vec4<f32>,
    @location(0) clip_position: vec4<f32>,
    @location(1) uv: vec2<f32>
};
struct Int {
    value: u32
};

@group(0) @binding(0) var environment_sampler: sampler;
@group(0) @binding(1) var environment_map: texture_cube<f32>;
@group(0) @binding(2) var<uniform> face: Int;

@vertex
fn vertex_main(@location(0) position: vec2<f32>) -> VertexOut
{
    var output : VertexOut;
    output.frag_position = vec4(position, 0.0, 1.0);
    output.clip_position = vec4(position, 0.0, 1.0);
    output.uv = vec2(position.x * 0.5 + 0.5, 1.0 - (position.y * 0.5 + 0.5));
    return output;
}

fn get_direction_from_uv_and_index(i: u32, uv: vec2<f32>) -> vec3<f32>
{
    let xy = uv * 2.0 - 1.0;

    switch i {
        case 0: {
            return vec3(1.0, -xy.y, -xy.x);
        }
        case 1: {
            return vec3(-1.0, -xy.y, xy.x);
        }
        case 2: {
            return vec3(xy.x, 1.0, xy.y);
        }
        case 3: {
            return vec3(xy.x, -1.0, -xy.y);
        }
        case 4: {
            return vec3(xy.x, -xy.y, 1.0);
        }
        case 5: {
            return vec3(-xy.x, -xy.y, -1.0);
        }
        default {
            //shouldn't pass this
            return vec3(0.0, 0.0, 0.0);
        }
    }
}

@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32> {
    let normal = normalize(get_direction_from_uv_and_index(face.value, frag_data.uv));
    var irradiance = vec3(0.0);

    const sample_count = 128u;

    let samples_per_dimension = u32(ceil(sqrt(f32(sample_count))));
    let d_theta = TWO_PI / f32(samples_per_dimension);
    let d_phi = HALF_PI / f32(samples_per_dimension);

    for(var i: u32 = 0; i < samples_per_dimension; i = i + 1u){
        var theta = (f32(i) + 0.5) * d_theta;
        for(var j: u32 = 0; j < samples_per_dimension; j = j + 1u){
            var phi = (f32(j) + 0.5) * d_phi;
            let sin_phi = sin(phi);
            let cos_phi = cos(phi);
            let x = cos_phi * cos(theta);
            let y = sin_phi;
            let z = cos_phi * sin(theta);
            let local_dir = vec3(x,y,z);

            let up = normal;
            let tangent = select(vec3<f32>(0.0, 1.0, 0.0), vec3<f32>(1.0, 0.0, 0.0), abs(normal.y) > 0.99);
            let right = normalize(cross(tangent, up));
            let forward = cross(up, right);
            let direction = normalize(local_dir.x * right + local_dir.y * normal + local_dir.z * forward);
            let weight = cos_phi * sin_phi;
            irradiance += textureSample(environment_map, environment_sampler, direction).rgb * weight;
        }
    }

    irradiance *= d_phi * d_theta;
    return vec4(irradiance, 1.0);
}

The normals are perturbed by the local direction vector and we need to convert them back into world space. There's a line in there (the select) which accounts for times when the normal is too close to the up direction to form a valid cross product (we had this problem earlier when doing space transforms, this time it's in the shader itself). The resulting irradiance map looks like this:

Looking at this we can see a few things going on (this is also way higher resolution than we actually need...). What we are getting seems reasonable. If we look at the mid points they are sampling from the areas we expect (we see green at the top of each vertical plane for example). However, the sampling is pretty bad. We can see clear starburst patterns on the top and bottom where the very consistent spacing that leaves gaps. We also see rings in the center were the sample density is much greater. This is useable but maybe we can do better.

Cosine Weighted Distribution

So one way we can fix the issue is by using cosine weighted distribution. The idea here is that we should actually sample more the closer we are to the normal. The reason is because light that is closer to the normal direction contributes much more than light on the edges (dot product). This means that we'll get closer to the true color if we focus our sampling there. The other part is that we can be a little more random. The start bursts are likely because our sampling is too ridged and periodic, if it was a little more chaotic then we'll likely get fewer gaps. This leads us to a new problem, WGSL doesn't give us anything to produce random numbers, so how can we deal with that? Luckily there's a paper that addresses just this topic: https://indico.cern.ch/event/93877/contributions/2118070/attachments/1104200/1575343/acat3_revised_final.pdf

Using some fancy bitmath we can get pseudo-random numbers that are good enough for the job entirely on the GPU itself. I'm not going to go into it but just translate this to something we can use. Here's a shader to generate "random" noise:

const U32_MAX: u32 = 4294967295u;
const SCREEN_WIDTH = 320;

fn seed_per_thread(id: u32) -> u32 {
    return id * 1099087573u;
}
fn taus_step(z: u32, s1: u32, s2: u32, s3: u32, m: u32) -> u32 {
    let b = ((z << s1) ^ z) >> s2;
    return ((z & m) << s3) ^ b;
}
fn lcg_step(z: u32, a: u32, c: u32) -> u32 {
    return a * z + c;
}
fn hybrid_taus(z1: u32, z2: u32, z3: u32, z4: u32) -> f32 {
    let r = taus_step(z1, 13u, 19u, 12u, 4294967294u) ^
            taus_step(z2, 2u, 25u, 4u, 4294967288u) ^
            taus_step(z3, 3u, 11u, 17u, 4294967280u) ^
            lcg_step(z4, 1664525u, 1013904223u);
    return f32(r) * 2.3283064365387e-10;
}

fn rand_uint(last_r: u32) -> u32 {
    let z1 = taus_step(last_r, 13u, 19u, 12u, 429496729u);
    let z2 = taus_step(last_r, 2u, 25u, 4u, 4294967288u);
    let z3 = taus_step(last_r, 3u, 11u, 17u, 429496280u);
    let z4 = lcg_step(1664525, last_r, 1013904223u);
    return (z1 ^ z2 ^ z3 ^ z4);
}

fn uint_to_normalized_float(value: u32) -> f32 {
    return f32(value) / (f32(U32_MAX) + 1.0); //+1 to account for 0
}


@group(0) @binding(0) var my_sampler: sampler;
@group(0) @binding(1) var my_texture: texture_2d<f32>;

struct VertexOut {
    @builtin(position) position : vec4<f32>,
    @location(0) uv : vec2<f32>
};

@vertex
fn vertex_main(@location(0) position: vec2<f32>, @location(1) uv: vec2<f32>) -> VertexOut
{
    var output : VertexOut;
    output.position = vec4<f32>(position, 0.0, 1.0);
    output.uv = uv;
    return output;
}

@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32>
{
    let id = u32(frag_data.position.x) + (u32(frag_data.position.y) * SCREEN_WIDTH);
    let seed = seed_per_thread(id);

    let int_r = rand_uint(seed);
    let int_g = rand_uint(int_r);
    let int_b = rand_uint(int_g);

    let r = uint_to_normalized_float(int_r);
    let b = uint_to_normalized_float(int_g);
    let g = uint_to_normalized_float(int_b);

    return vec4(r, g, b, 1.0);
}

This follows the the paper using shifts, ors and magic primes. Since we don't have a thread id the seed per thread is generated by taking the pixel position and flattening it into a single index int. To get each successive number we need to pass in the last value we got. I looked into stateful ways to do this but they seemed more trouble than they were worth. The downside is you have to keep track and limit uses of functional style programming so you can retain the intermediate uint. The noise we get looks pretty decent to me.

We can then take this random number generation and apply it to our sampling routine.

const PI = 3.14159265359;
const TWO_PI = 2 * PI;
const HALF_PI = PI / 2;
const U32_MAX: u32 = 4294967295u;

struct VertexOut {
    @builtin(position) frag_position : vec4<f32>,
    @location(0) clip_position: vec4<f32>,
    @location(1) uv: vec2<f32>
};
struct IrradianceParams {
    face_index: u32,
    screen_width: u32,
    sample_count: u32
};

fn seed_per_thread(id: u32) -> u32 {
    return id * 1099087573u;
}
fn taus_step(z: u32, s1: u32, s2: u32, s3: u32, m: u32) -> u32 {
    let b = ((z << s1) ^ z) >> s2;
    return ((z & m) << s3) ^ b;
}
fn lcg_step(z: u32, a: u32, c: u32) -> u32 {
    return a * z + c;
}
fn hybrid_taus(z1: u32, z2: u32, z3: u32, z4: u32) -> f32 {
    let r = taus_step(z1, 13u, 19u, 12u, 4294967294u) ^
            taus_step(z2, 2u, 25u, 4u, 4294967288u) ^
            taus_step(z3, 3u, 11u, 17u, 4294967280u) ^
            lcg_step(z4, 1664525u, 1013904223u);
    return f32(r) * 2.3283064365387e-10;
}

fn rand_uint(last_r: u32) -> u32 {
    let z1 = taus_step(last_r, 13u, 19u, 12u, 429496729u);
    let z2 = taus_step(last_r, 2u, 25u, 4u, 4294967288u);
    let z3 = taus_step(last_r, 3u, 11u, 17u, 429496280u);
    let z4 = lcg_step(1664525u, last_r, 1013904223u);
    return (z1 ^ z2 ^ z3 ^ z4);
}

fn uint_to_normalized_float(value: u32) -> f32 {
    return f32(value) / (f32(U32_MAX) + 1.0);
}

@group(0) @binding(0) var environment_sampler: sampler;
@group(0) @binding(1) var environment_map: texture_cube<f32>;
@group(0) @binding(2) var<uniform> params: IrradianceParams;

@vertex
fn vertex_main(@location(0) position: vec2<f32>) -> VertexOut
{
    var output : VertexOut;
    output.frag_position = vec4(position, 0.0, 1.0);
    output.clip_position = vec4(position, 0.0, 1.0);
    output.uv = vec2(position.x * 0.5 + 0.5, 1.0 - (position.y * 0.5 + 0.5));
    return output;
}

fn get_direction_from_uv_and_index(i: u32, uv: vec2<f32>) -> vec3<f32>
{
    let xy = uv * 2.0 - 1.0;

    switch i {
        case 0: {
            return vec3(1.0, -xy.y, -xy.x);
        }
        case 1: {
            return vec3(-1.0, -xy.y, xy.x);
        }
        case 2: {
            return vec3(xy.x, 1.0, xy.y);
        }
        case 3: {
            return vec3(xy.x, -1.0, -xy.y);
        }
        case 4: {
            return vec3(xy.x, -xy.y, 1.0);
        }
        case 5: {
            return vec3(-xy.x, -xy.y, -1.0);
        }
        default {
            //shouldn't pass this
            return vec3(0.0, 0.0, 0.0);
        }
    }
}

@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32> {
    let id = u32(frag_data.frag_position.x) + (u32(frag_data.frag_position.y) * params.screen_width);
    let seed = seed_per_thread(id);
    let normal = normalize(get_direction_from_uv_and_index(params.face_index, frag_data.uv));

    var irradiance = vec3(0.0);
    var rand_int_a = rand_uint(seed); //seed value
    var rand_int_b = rand_uint(rand_int_a); //seed value

    for(var i: u32; i < params.sample_count; i = i + 1u){
        rand_int_a = rand_uint(rand_int_b);
        rand_int_b = rand_uint(rand_int_a);
        let theta = uint_to_normalized_float(rand_int_a) * TWO_PI;
        let r_square = uint_to_normalized_float(rand_int_b);
        let radius = sqrt(r_square);
        let local_tangent = radius * cos(theta);
        let local_bitangent = radius * sin(theta);
        let local_normal = sqrt(1 - r_square);
        let local_dir = vec3(local_tangent, local_bitangent, local_normal);
        let up = normal;
        let tangent = select(vec3<f32>(0.0, 1.0, 0.0), vec3<f32>(1.0, 0.0, 0.0), abs(normal.y) > 0.99);
        let right = normalize(cross(tangent, up));
        let forward = cross(up, right);
        let world_direction = normalize(local_dir.x * right + local_dir.y * forward + local_dir.z * up);
        irradiance += textureSample(environment_map, environment_sampler, world_direction).rgb;
    }

    irradiance /= f32(params.sample_count);
    irradiance *= PI;
    return vec4(irradiance, 1.0);
}

The first thing is we are entirely in spherical coordinates which makes things much easier. We get a normalized random float for theta and the radius term. The tangent and bitangent of the sample are just the spherical coordinates converted. The sqrt(1-r^2) term is more interesting. The reason is because to model the hemisphere we can think of it as sampling from a circle of area 1 at the base x^2 + Y^2 = 1. It's a cross section of the hemisphere in the tangent/bitangent plane. However as we get higher in elevation (the normal direction) the circle shrinks at a rate of sqrt(1-z^2) so at hemisphere radius 1, the circle has radius 0, and at hemisphere radius 0 the circle has radius 1. This is what essentially biases the samples toward the normal direction

The resulting map is a lot better:

Note to get the same brightness we need to multiply by pi which comes from the volume under the hemisphere and we divide to normalize it back to 1 for the probability density function. If we don't you'll get a much darker map. But with the same amount of samples, this is much, much better.

But this cubemap is fairly expensive to render and it would be nice if we could lower the resolution or sample count. Lowering the resolution looks bad though (64x64 with 128 samples):

It feels like we could could run a blur filter over it to make it look better (I won't be detailing this part because I already have another post on convolutions but take a look at the convolution-pipeline.js in the code if you want to see it). It's also not perfect as you can see the visible seams where the edge behavior is incorrect. Realistically this doesn't improve much and will need more sophisticated methods to get looking better.

I also parameterized the pipelines better so things like resolution of the cubemaps weren't hardcoded. If you see differences in the final code that's why.

The final output is a little pixelated but adjustable with parameters.

Outputting a cubemap

Something helpful is a the ability to output the cubemap or a texture. I made a whole pipeline to do this:

//@ts-check

/**
 * @typedef {import("../../../entities/camera.js").Camera} Camera
 * @typedef {import("../../../entities/material.js").Material} Material
 * @typedef {import("../../../entities/probe.js").Probe} Probe
 * @typedef {import("../../../entities/mesh.js").Mesh} Mesh
 * @typedef {import("../../../entities/light.js").Light} Light
 * @typedef {import("../../../entities/group.js").Group} Group
 * @typedef {import("../../../entities/background.d.ts").IBackground} IBackground
 * @typedef {import("../../../entities/pipeline.d.ts").IPipeline} IPipeline
 * @typedef {import("../../../entities/pipeline.d.ts").MeshContainer} MeshContainer
 * @typedef {import("../../../entities/pipeline.d.ts").AttachmentViews} AttachmentViews
 */

import { uploadShader } from "../../../utilities/wgpu-utils.js";

/**
 * @implements {IPipeline}
 */
export class DisplayTexturePipeline {
    #bindGroupLayouts = new Map();
    #pipeline;
    #sampler;
    #vertexBuffer;

    /**
     * 
     * @param {GPUDevice} device 
     * @param {{ textureType?: "texture" | "depthmap" | "cubemap"}} options 
     */
    async createPipeline(device, options = {}) {
        const textureType = options.textureType ?? "texture";
        const vertexBufferDescriptor = [{
            attributes: [
                {
                    shaderLocation: 0,
                    offset: 0,
                    format: "float32x2"
                }
            ],
            arrayStride: 8,
            stepMode: "vertex"
        }];

        let samplerType;
        let sampleType;
        let viewDimension;
        let url;

        switch(textureType){
            case "texture": {
                samplerType = "filtering";
                sampleType = "float";
                viewDimension = "2d";
                url = "../shaders/display-texture.wgsl";
                break;
            }
            case "cubemap": {
                samplerType = "filtering";
                sampleType = "float";
                viewDimension = "2d-array";
                url = "../shaders/display-cubemap.wgsl";
                break;
            }
            case "depthmap": {
                samplerType = "non-filtering";
                sampleType = "depth";
                viewDimension = "2d";
                url = "../shaders/display-texture.wgsl";
                break;
            }
        }

        const relativeShaderUrl = import.meta.resolve(url);
        const shaderModule = await uploadShader(device, relativeShaderUrl);

        this.#bindGroupLayouts.set("texture", device.createBindGroupLayout({
            entries: [
                {
                    binding: 0,
                    visibility: GPUShaderStage.FRAGMENT,
                    sampler: {
                        type: samplerType
                    }
                },
                {
                    binding: 1,
                    visibility: GPUShaderStage.FRAGMENT,
                    texture: {
                        sampleType,
                        viewDimension 
                    }
                }
            ]
        }));

        const pipelineLayout = device.createPipelineLayout({
            label: "display-texture-pipeline-layout",
            bindGroupLayouts: this.#bindGroupLayouts.values().toArray()
        });

        const pipelineDescriptor = {
            label: "display-texture-pipeline",
            layout: pipelineLayout,
            vertex: {
                module: shaderModule,
                entryPoint: "vertex_main",
                buffers: vertexBufferDescriptor
            },
            fragment: {
                module: shaderModule,
                entryPoint: "fragment_main",
                targets: [
                    { format: "rgba8unorm" }
                ]
            },
            primitive: {
                topology: "triangle-list"
            }
        };

        this.#pipeline = device.createRenderPipeline(pipelineDescriptor);


        this.#sampler = options.sampler;
        const vertices = new Float32Array([
            -1.0, -1.0,
            3.0, -1.0,
            -1.0, 3.0
        ]);

        this.#vertexBuffer = device.createBuffer({
            label: "display-texture-export-tri",
            size: vertices.byteLength,
            usage: GPUBufferUsage.VERTEX | GPUBufferUsage.COPY_DST,
        });
        device.queue.writeBuffer(this.#vertexBuffer, 0, vertices);
    }

    /**
     * @param {GPUDevice} device
     * @param {Mesh} root
     * @param {AttachmentViews} attachmentViews }
     * @param {{
     *  meshContainers: Map<Mesh, MeshContainer>,
     *  lights: Map<string | symbol, Light>,
     *  shadowMaps: Map<string | symbol, GPUTexture>,
     *  textures: Map<string | symbol, GPUTexture>,
     *  samplers: Map<string | symbol, GPUSampler>,
     *  materials: Map<string | symbol, Material>,
     *  cameras: Map<string | symbol, Camera>,
     *  primaryCamera: Camera;
     *  probes: Map<string | symbol, Probe>,
     *  background: IBackground
     *  commandEncoder: GPUCommandEncoder
     * }} info
     * @param {GPUTextureView?} args
     */
    render(device, root, attachmentViews, info, args) {
        const commandEncoder = info.commandEncoder ?? device.createCommandEncoder({
            label: "display-texture-command-encoder"
        });

        const passEncoder = commandEncoder.beginRenderPass({
            label: "display-texture-render-pass",
            //@ts-expect-error WGPU types are not good
            colorAttachments: [
                {
                    storeOp: "store",
                    loadOp: "clear",
                    clearValue: { r: 0, g: 0, b: 0, a: 1 },
                    view: attachmentViews.colorView
                }
            ]
        });

        const textureBindGroup = device.createBindGroup({
            label: "display-texture-bind-group",
            layout: this.#bindGroupLayouts.get("texture"),
            entries: [
                { binding: 0, resource: this.#sampler },
                { binding: 1, resource: args},
            ]
        });

        passEncoder.setPipeline(this.#pipeline);
        passEncoder.setBindGroup(0, textureBindGroup);
        passEncoder.setVertexBuffer(0, this.#vertexBuffer);
        passEncoder.draw(3);
        passEncoder.end();

        if(!info.commandEncoder){
            device.queue.submit([commandEncoder.finish()]);
        }
    }
}

It works just like the old depth map debugger but expanded using our new pipeline abstraction. To refresh we simply draw a triangle large enough to fill the whole screen and then for each fragment we sample the texel.

struct VertexOut {
    @builtin(position) frag_position : vec4<f32>,
    @location(0) clip_position: vec4<f32>,
    @location(2) uv: vec2<f32>
};
@group(0) @binding(0) var my_sampler: sampler;
@group(0) @binding(1) var my_texture: texture_2d<f32>;

@vertex
fn vertex_main(@location(0) position: vec2<f32>) -> VertexOut
{
    var output : VertexOut;
    output.frag_position =  vec4(position, 1.0, 1.0);
    output.clip_position = vec4(position, 1.0, 1.0);
    output.uv = vec2(position.x * 0.5 + 0.5, 1.0 - (position.y * 0.5 + 0.5));
    return output;
}

@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32> {
    let value = textureSample(my_texture, my_sampler, frag_data.uv);
    return value;
}

For the cube map we create a view as a texture-2d-array (because cubemaps are sample by direction vectors). Then for each patch of pixels we sample a layer of the texture:

struct VertexOut {
    @builtin(position) frag_position : vec4<f32>,
    @location(0) clip_position: vec4<f32>,
    @location(2) uv: vec2<f32>
};
@group(0) @binding(0) var my_sampler: sampler;
@group(0) @binding(1) var my_texture: texture_2d_array<f32>;

@vertex
fn vertex_main(@location(0) position: vec2<f32>) -> VertexOut
{
    var output : VertexOut;
    output.frag_position =  vec4(position, 1.0, 1.0);
    output.clip_position = vec4(position, 1.0, 1.0);
    output.uv = vec2(position.x * 0.5 + 0.5, 1.0 - (position.y * 0.5 + 0.5));
    return output;
}

@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32> {
    let tile_width = 0.25;
    let tile_height = 0.33;
    var layer: i32 = -1;
    var origin: vec2<f32> = vec2(0.0, 0.0);

    //top
    if(frag_data.clip_position.x > 0.0 && frag_data.clip_position.x < 0.5 && frag_data.clip_position.y <= 1.0 && frag_data.clip_position.y > 0.33) {
        layer = 2;
        origin = vec2(tile_width * 2, 0.0);
    //back
    } else if(frag_data.clip_position.x >= -1.0 && frag_data.clip_position.x < -0.5 && frag_data.clip_position.y <= 0.33 && frag_data.clip_position.y > -0.33) {
        layer = 5;
        origin = vec2(0.0, tile_height);
    //left
    } else if(frag_data.clip_position.x >= -0.5 && frag_data.clip_position.x < 0.0 && frag_data.clip_position.y <= 0.33 && frag_data.clip_position.y > -0.33) {
        layer = 1;
        origin = vec2(tile_width, tile_height);
    //front
    } else if(frag_data.clip_position.x >= 0.0 && frag_data.clip_position.x < 0.5 && frag_data.clip_position.y <= 0.33 && frag_data.clip_position.y > -0.33) {
        layer = 4;
        origin = vec2(tile_width * 2, tile_height);
    //right
    } else if(frag_data.clip_position.x >= 0.5 && frag_data.clip_position.y <= 0.33 && frag_data.clip_position.y > -0.33) {
        layer = 0;
        origin = vec2(tile_width * 3, tile_height);
    //bottom
    } else if(frag_data.clip_position.x > 0.0 && frag_data.clip_position.x < 0.5 && frag_data.clip_position.y <= -0.33) {
        layer = 3;
        origin = vec2(tile_width * 2, tile_height * 2);
    }

    let dims = textureDimensions(my_texture, 0);
    let cubemap_px = vec2<f32>(dims.xy);
    let tile_px = vec2<f32>(tile_width * cubemap_px.x, tile_height * cubemap_px.y); //get tile in terms of texture pixels
    let half_texel = vec2<f32>(0.5) / tile_px;       

    let local_uv = clamp((frag_data.uv - origin) / vec2(tile_width, tile_height), vec2(half_texel), vec2(1.0 - half_texel));

    let sample = textureSample(my_texture, my_sampler, local_uv, layer);

    if(layer == -1){
        return vec4(0.0, 0.0, 0.0, 1.0);
    }
    return sample;
}

There are 2 rows and 4 columns of titles. For each tile, we take the position inside the tile and scale it to get the local UV to sample. If we're not in a valid layer then we just return black. The order here is important. If you try to sample the texture in an if block you'll get errors because you can't sample textures in a way in which all threads would not sample the texture at the same time. I guess that makes sense thinking about how GPUs work but it's a tad unexpected. You also might wonder about the half texel business. We need to calculate what half a texel is in the space of the cubemap layer and then add or subtract from the end point so that we do not sample at the very end which can produce ugly seams. I think in this case it could be because of the repeat on the sampler instead of mirror, but in any case it's nice to not be dependent on the sampler since it was parameterized (mostly for nearest neighbor scenarios).

Further Exploration

There were a lot of things I didn't get to because this chapter was already too long. Apparently spherical harmonics can be used to compress the data we have. I don't know anything about that but it seems interesting. In a real scene objects move around so being able to generate lighting for these objects in real time would be an interesting upgrade. Apparently the most common way is to generate a lattice of probes and interpolate between them. It would also be nice to cache the preprocessing stuff so it doesn't have to compute that on every load.

There's also some cleanup to do. It would be nice if the main pipeline was combined with the background pipeline. It would also be nice if the cubemap render was able to just call the main pipeline in general. Maybe at that point a probe just becomes a new type of camera. They could be used for other things like point lights or for reflections. Ideas for another day.

Conclusion

This one took a while to learn about. I really wasn't expecting to jump headfirst into global illumination but I just took my original idea for adding ambient light and then researched what people actually tend to do. It's a lot to take in though but it's pretty cool that this works and it way higher fidelity than I originally intended. I might have to take a break after this though as I kinda burned out at the finish line and there's a ton of refactorings nagging me that I can hopefully knock by themselves out and stay mostly on topic for once.

The code

https://github.com/ndesmic/geo/compare/v0.9...v0.10

Links

https://learnopengl.com/PBR/IBL/Diffuse-irradiance
https://indico.cern.ch/event/93877/contributions/2118070/attachments/1104200/1575343/acat3_revised_final.pdf

WebGPU Engine from Scratch Part 11 - 2: True Scene Graph

ndesmic — Tue, 16 Dec 2025 04:52:26 +0000

Improvements to the markup

Resolution

At this point we can play around a little more without having to worry about breaking things because we can write visual tests for them. But one of the first features to come to mind is making the rendering smaller because the tests don't need large screenshots taking up so much space. The wc-geo component had a height and width attribute but we never actually wired those up. Let's fix that.

//wc-geo.js
set width(value) {
    this.#width = parseInt(value, 10);
    if (this.dom?.canvas) {
        this.dom.canvas.width = this.#width;
    }
}
set height(value) {
    this.#height = parseInt(value, 10);
    if (this.dom?.canvas) {
        this.dom.canvas.height = this.#height;
    }
}

This will change the props and the canvas when the height or width changes. This will also cause some internal problems in the engine (which will render a black rectangle) because the resources will now have to resize so let's fix those too.

Really the only thing that needs to change at this point is the depth texture resolution. We'll split it off from the initializeTextures method.

//gpu-engine.js
async initializeTextures(textures) {
    for (const [key, texture] of Object.entries(textures)) {
        if (texture.image ?? texture.images) {
            this.#textures.set(key, await uploadTexture(this.#device, texture.image ?? texture.images, { label: `${key}-texture` }));
        } else if (texture.color) {
            this.#textures.set(key, createColorTexture(this.#device, { color: texture.color, label: `${key}-texture` }));
        }
    }
    //default textures
    this.initDepthTexture();
    this.#textures.set(PLACEHOLDER_TEXTURE, createColorTexture(this.#device, { label: "placeholder-texture" }));
}
initDepthTexture() {
    this.#textures.set(DEPTH_TEXTURE, this.#device.createTexture({
        label: "depth-texture",
        size: {
            width: this.#canvas.width,
            height: this.#canvas.height,
            depthOrArrayLayers: 1
        },
        format: "depth32float",
        usage: GPUTextureUsage.RENDER_ATTACHMENT
    }));
}

Then we update the depth texture with a method if things change.

updateCanvasSize(){
    this.initDepthTexture();
}

Since it's tied to the canvas itself we don't need to pass in the height and width as those are already updated but there's an argument from an API standpoint we should manage resolution apart from the canvas. I'm not going to bother with that though.

We'll also set some state info to know if the engine was initialized. This way we know the different between setting the first time and if it's actually changing the engine in progress.

//gpu-engine.js
#isIntialized = false

async initializeScene(scene) {
    this.initializeCameras(scene.cameras);
    await this.initializeTextures(scene.textures);
    this.initializeMaterials(scene.materials);
    this.initializeSamplers();
    await this.initializeMeshes(scene.meshes);
    this.initializeGroups(scene.groups);
    this.initializeLights(scene.lights);
    await this.initializePipelines();
    this.initializePipelineMeshes(scene.pipelineMeshes);
    this.#isInitialized = true;
}

get isInitialized(){
    return this.#isInitialized;
}

Could be combined into a state machine with isRunning since they aren't entirely orthogonal. Also not going to bother with that.

Finally we can finish our setters.

//wc-geo.js
set width(value) {
    this.#width = parseInt(value, 10);
    if (this.dom?.canvas) {
        this.dom.canvas.width = this.#width;
    }
    if(this.engine?.isInitialized){
        this.engine.updateCanvasSize();
    }
}
set height(value) {
    this.#height = parseInt(value, 10);
    if (this.dom?.canvas) {
        this.dom.canvas.height = this.#height;
    }
    if(this.engine?.isInitialized){
        this.engine.updateCanvasSize();
    }
}

We do need to be defensive with the sets because the attribute will change before connectedCallback so those object won't even exist the first time around.

A true scene graph

The previous version was hacked together using keys as a crutch to pick-out nested meshes and groups. What we really want is a full tree from the top-down. It became too hard to explain step-by-step so we'll do a big migration and explain what happened:

//geo-markup-parser.js
import { Camera } from "../entities/camera.js";
import { Material } from "../entities/material.js";
import { Mesh } from "../entities/mesh.js";
import { Group } from "../entities/group.js";
import { Light } from "../entities/light.js";
import { loadImage } from "../utilities/image-utils.js";
import { fetchObjMesh } from "./data-utils.js";
import { surfaceGrid, quad, cube } from "./mesh-generator.js";

function parseVector(text, length = 4, defaultValue = null) {
    return text
        ? text.split(",").map(x => parseFloat(x.trim())).slice(0, length)
        : defaultValue
}

function parseIntOrDefault(text, defaultValue = null) {
    return text
        ? parseInt(text, 10)
        : defaultValue;
}

function parseFloatOrDefault(text, defaultValue = null) {
    return text
        ? parseFloat(text)
        : defaultValue;
}

function parseListOrDefault(text, defaultValue = null) {
    return text
        ? text.split(",").map(x => x.trim())
        : defaultValue
}

function updateMeshAttributes(meshEl, mesh) {
    const normalize = meshEl.hasAttribute("normalize");
    if (normalize) {
        mesh.normalizePositions();
    }

    const resizeUvs = parseIntOrDefault(meshEl.getAttribute("resize-uvs"));
    if (resizeUvs) {
        mesh.resizeUvs(resizeUvs)
    }

    const material = meshEl.getAttribute("material");
    mesh.setMaterial(material);

    const attributes = parseListOrDefault(meshEl.getAttribute("attributes"));
    if (attributes) {
        mesh.useAttributes(attributes);
    }

    const translate = parseVector(meshEl.getAttribute("translate"), 3);
    if (translate) {
        mesh.translate({ x: translate[0], y: translate[1], z: translate[2] });
    }

    const rotate = parseVector(meshEl.getAttribute("rotate"), 3);
    if (rotate) {
        mesh.rotate({ x: rotate[0], y: rotate[1], z: rotate[2] });
    }

    const scale = parseVector(meshEl.getAttribute("scale"), 3);
    if (scale) {
        mesh.scale({ x: scale[0], y: scale[1], z: scale[2] });
    }

    //must come last because it updates mesh
    const bakeTransforms = meshEl.hasAttribute("bake-transforms");
    if (bakeTransforms) {
        mesh.bakeTransforms();
    }
}

function parseCamera(cameraEl, options = {}) {
    return new Camera({
        name: cameraEl.getAttribute("name"),
        position: parseVector(cameraEl.getAttribute("position"), 3),
        screenHeight: cameraEl.getAttribute("height") ?? options.defaultHeight,
        screenWidth: cameraEl.getAttribute("width") ?? options.defaultWidth,
        fieldOfView: cameraEl.getAttribute("fov") ?? 90,
        near: cameraEl.getAttribute("near") ?? 0.01,
        far: cameraEl.getAttribute("far") ?? 5,
        isPerspective: !cameraEl.hasAttribute("is-orthographic")
    });
}

async function parseTexture(textureEl){
    const name = textureEl.getAttribute("name");
    const src = textureEl.getAttribute("src");
    const srcs = parseListOrDefault(textureEl.getAttribute("srcs"));
    const color = textureEl.getAttribute("color");
    let value;
    if (src) {
        value = { entity: "texture", image: await loadImage(src), name  };
    } else if(srcs){
        value = { entity: "texture", images: await Promise.all(srcs.map(s => loadImage(s))), name } 
    } else if (color) {
        value = { entity: "texture", color: parseVector(color, 4), name };
    }

    return value;
}

function parseMaterial(materialEl) {
    const roughnessMap = materialEl.getAttribute("roughness-map");
    const albedoMap = materialEl.getAttribute("albedo-map");

    return new Material({
        name: materialEl.getAttribute("name"),
        albedoMap: albedoMap,
        useRoughnessMap: !!roughnessMap,
        roughness: parseFloatOrDefault(materialEl.getAttribute("roughness")),
        metalness: parseFloatOrDefault(materialEl.getAttribute("metalness")),
        baseReflectance: parseVector(materialEl.getAttribute("base-reflectance"), 3)
    });
}

async function parseMesh(meshEl) {
    const reverseWinding = meshEl.hasAttribute("reverse-winding");
    const src = meshEl.getAttribute("src");
    const mesh = await fetchObjMesh(src, { reverseWinding });

    updateMeshAttributes(meshEl, mesh);

    return mesh;
}

function parseSurfaceGrid(meshEl) {
    const rowCount = parseInt(meshEl.getAttribute("row-count"), 10);
    const colCount = parseInt(meshEl.getAttribute("col-count"), 10);
    const mesh = new Mesh(surfaceGrid(rowCount, colCount));

    updateMeshAttributes(meshEl, mesh);

    return mesh;
}

function parseQuad(meshEl){
    const mesh = new Mesh(quad());
    updateMeshAttributes(meshEl, mesh);
    return mesh;
}

function parseCube(meshEl){
    const mesh = new Mesh(cube());
    updateMeshAttributes(meshEl, mesh);
    return mesh;
}

/**
 * 
 * @param {HTMLElement} groupEl 
 */
async function parseGroup(groupEl, options){
    const children = await Promise.all(Array.from(groupEl.children).map(async c => {
        switch(c.tagName){
            case "GEO-BACKGROUND": {
                return parseBackground(c);
            }
            case "GEO-MESH": {
                return (await parseMesh(c));
            }
            case "GEO-SURFACE-GRID": {
                return parseSurfaceGrid(c);
            }
            case "GEO-QUAD": {
                return parseQuad(c);
            }
            case "GEO-CAMERA": {
                return parseCamera(c, options);
            }
            case "GEO-CUBE": {
                return parseCube(c)
            }
            case "GEO-GROUP": {
                return (await parseGroup(c, options))
            }
            case "GEO-LIGHT": {
                return parseLight(c)
            }
            case "GEO-TEXTURE": {
                return parseTexture(c);
            }
            case "GEO-MATERIAL": {
                return parseMaterial(c);
            }
            default: {
                throw new Error(`Group doesn't support ${c.tagName} children`)
            }
        }
    }));

    const group = new Group({
        children
    });

    const translate = parseVector(groupEl.getAttribute("translate"), 3);
    if (translate) {
        group.translate({ x: translate[0], y: translate[1], z: translate[2] });
    }

    const rotate = parseVector(groupEl.getAttribute("rotate"), 3);
    if (rotate) {
        group.rotate({ x: rotate[0], y: rotate[1], z: rotate[2] });
    }

    const scale = parseVector(groupEl.getAttribute("scale"), 3);
    if (scale) {
        group.scale({ x: scale[0], y: scale[1], z: scale[2] });
    }

    return group;
}

function parseLight(lightEl) {
    const light = new Light({
        type: lightEl.getAttribute("type") ?? "point",
        color: parseVector(lightEl.getAttribute("color"), 4, [1, 1, 1, 1]),
        direction: parseVector(lightEl.getAttribute("direction"), 3, [0, 0, 0]),
        castsShadow: lightEl.hasAttribute("casts-shadow")
    });

    const translate = parseVector(lightEl.getAttribute("translate"), 3);
    if (translate) {
        light.translate({ x: translate[0], y: translate[1], z: translate[2] });
    }

    const rotate = parseVector(lightEl.getAttribute("rotate"), 3);
    if (rotate) {
        light.rotate({ x: rotate[0], y: rotate[1], z: rotate[2] });
    }

    return light;
}

function parseBackground(backgroundEl){
    if(!backgroundEl) return null;
    return {
        entity: "background",
        environmentMap: backgroundEl.getAttribute("environment-map"),
        sampler: backgroundEl.getAttribute("sampler")
    };
}

export async function parseScene(element) {
    const sceneRoot = await parseGroup(element, {
        defaultWidth: element.width,
        defaultHeight: element.height
    });

    return {
        sceneRoot,
    };
}

Here we've eliminated everything from the parseScene output except the top most group to make it fully recursive. This means we can just store each element as a child of a group which is much simpler. We do this by checking each nested element and seeing if it's something we can parse and then parse it. I also changed key to name for most elements as it overall made more sense to do so. The other benefit of the tree structure is that things like lights no longer need keys as we can generate those automatically.

//gpu-engine.js
async initializeScene(scene) {
        this.initializeSamplers();

        this.initializeGroup(scene.sceneRoot, 0);
        //default textures
        this.initDepthTexture();
        this.#textures.set(PLACEHOLDER_TEXTURE, createColorTexture(this.#device, { label: "placeholder-texture" }));

        this.#shadowMaps.set("placeholder", this.#device.createTexture({
            label: "placeholder-depth-texture",
            size: { width: 1, height: 1, depthOrArrayLayers: 1 },
            format: "depth32float",
            usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
        }));

        await this.initializePipelines();
        this.#isInitialized = true;
    }
    initializeCameras(camera) {
        this.#cameras.set(camera.name, camera);
    }
    initializeTexture(texture) {
        if (texture.image ?? texture.images) {
            this.#textures.set(texture.name, uploadTexture(this.#device, texture.image ?? texture.images, { label: `${texture.name}-texture` }));
        } else if (texture.color) {
            this.#textures.set(texture.name, createColorTexture(this.#device, { color: texture.color, label: `${texture.name}-texture` }));
        }
    }
    initDepthTexture() {
        this.#textures.set(DEPTH_TEXTURE, this.#device.createTexture({
            label: "depth-texture",
            size: {
                width: this.#canvas.width,
                height: this.#canvas.height,
                depthOrArrayLayers: 1
            },
            format: "depth32float",
            usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
        }));
    }
    initializeMaterial(material) {
        this.#materials.set(material.name, material);
    }
    initializeSamplers() {
        this.#samplers.set(DEFAULT_SAMPLER, this.#device.createSampler({
            addressModeU: "repeat",
            addressModeV: "repeat",
            magFilter: "linear",
            minFilter: "linear"
        }));
        this.#samplers.set(DEFAULT_NEAREST_SAMPLER, this.#device.createSampler({
            addressModeU: "repeat",
            addressModeV: "repeat",
            magFilter: "nearest",
            minFilter: "nearest"
        }));
        this.#samplers.set(DEFAULT_SHADOW_SAMPLER, this.#device.createSampler({
            label: "shadow-map-default-sampler",
            compare: "less",
            magFilter: "linear",
            minFilter: "linear"
        }));
        this.#samplers.set("shadow-map-debug", this.#device.createSampler({
            label: "shadow-map-debug-sampler",
            compare: undefined,
        }));
    }
    initializeMesh(mesh, key) {
        const { vertexBuffer, indexBuffer } = uploadMesh(this.#device, mesh, { label: `${key}-mesh` });
        this.#meshContainers.set(mesh, { mesh, vertexBuffer, indexBuffer });
    }
    initializeGroup(group, key) {
        for (let i = 0; i < group.children.length; i++) {
            const child = group.children[i];
            if (child instanceof Camera){
                this.initializeCameras(child);
            } else if(child instanceof Mesh) {
                this.initializeMesh(child);
            } else if(child instanceof Light){
                this.initializeLight(child, `${key}-${i}`);
            } else if (child instanceof Group) {
                this.initializeGroup(child, `${key}-${i}`);
            } else if (child instanceof Material){
                this.initializeMaterial(child);
            } else if (child.entity === "texture"){
                this.initializeTexture(child);
            } else if (child.entity === "background"){
                this.initializeBackground(child);
            } else {
                throw new Error(`Don't know what this entity is ${JSON.stringify(child)}`)
            }
        }
        this.#sceneRoot = group;
    }
    initializeLight(light, key) {
        this.#lights.set(key, light)

        if(light.castsShadow){
            this.#shadowMaps.set(key, this.#device.createTexture({
                label: `shadow-map-${key}`,
                size: {
                    width: 2048,
                    height: 2048,
                    depthOrArrayLayers: 1
                },
                format: "depth32float",
                usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
            }));
        }
    }
    async initializePipelines() {
        {
            const pipeline = await getMainPipeline(this.#device);

            this.#pipelines.set("main", {
                pipeline,
                bindGroupLayouts: new Map([
                    ["scene", pipeline.getBindGroupLayout(0)],
                    ["materials", pipeline.getBindGroupLayout(1)],
                    ["lights", pipeline.getBindGroupLayout(2)]
                ]),
                bindMethod: this.setMainBindGroups.bind(this)
            });
        }
        {
            const pipeline = await getShadowMapPipeline(this.#device);

            this.#pipelines.set("shadow-map", {
                pipeline,
                bindGroupLayouts: new Map([
                    ["scene", pipeline.getBindGroupLayout(0)],
                ]),
                bindMethod: this.setShadowMapBindGroups.bind(this)
            });
        }
        {
            const pipeline = await getBackgroundPipeline(this.#device);

            this.#pipelines.set("background", {
                pipeline: pipeline,
                bindGroupLayouts: new Map([
                    ["scene", pipeline.getBindGroupLayout(0)],
                    ["materials", pipeline.getBindGroupLayout(1)]
                ]),
                bindMethod: this.setBackgroundBindGroups.bind(this)
            });
        }
    }
    initializeBackground(background) {
        if (background) {
            const mesh = new Mesh(screenTri()).useAttributes(["positions"]);
            const { vertexBuffer, indexBuffer } = uploadMesh(this.#device, mesh, { label: `background-tri-mesh` });
            this.#meshContainers.set(mesh, { mesh, vertexBuffer, indexBuffer });
            this.#background = {
                mesh,
                environmentMap: background.environmentMap,
                sampler: background.sampler === "nearest" ? DEFAULT_NEAREST_SAMPLER : DEFAULT_SAMPLER
            };
        }
    }
    //...etc

In gpu-engine.js we traverse through that tree and start adding the entities into their maps as we see them. For things like background (a new entity, see below) we'll just override so only the last one is used. There could be more validation here. Also with more complex parsing we could restrict certain entities to the top-level since it doesn't actually make sense to nest backgrounds and textures but we allow it anyway.

We can rip out the concept of pipelineMesh too. Snything in the scene root will go through the main pipeline. We'll hardcode the rest, at present there are only 3, main, shadow-map, and background and I don't see that changing much, at least in a way that can be configured dynamically.

Adding a background

Speaking of backgrounds, I want the ability to set the background. This means the cubemap that the scene sits inside. We already have the background pipeline so we should be able to leverage that. As you can see above we parse it into the scene object and set it in the engine. There was a modification to the texture parsing to handle multi-layer textures using the srcs attribute instead of src.

Looking at initializeBackground, we need the fullscreen tri for the background to sit on. Keep in mind we only need the positions, and don't use the other attributes. It'll be a little weird to hold the mesh reference to use as a key but that's how we'll deal with it. We could do something more complex with samplers (like take the actual min/mag filters) but I don't see the need right now. To render it, renderScene has it's pipeline loop unrolled to do the two phases for main geometry and background.

//gpu-engine.js
renderScene() {
    const commandEncoder = this.#device.createCommandEncoder({
        label: "main-command-encoder"
    });
    const camera = this.#cameras.get("main");
    const depthView = this.#textures.get(DEPTH_TEXTURE).createView({ label: "depth-texture-view"});
    const colorView = this.#context.getCurrentTexture().createView({ label: "color-texture-view"});
    {
        const passEncoder = commandEncoder.beginRenderPass({
            label: `main-render-pass`,
            colorAttachments: [
                {
                    storeOp: "store",
                    loadOp: "clear",
                    clearValue: { r: 0.1, g: 0.3, b: 0.8, a: 1.0 },
                    view: colorView
                }
            ],
            depthStencilAttachment: {
                view: depthView,
                depthClearValue: 1.0,
                depthStoreOp: "store",
                depthLoadOp: "clear"
            }
        });
        const pipelineContainer = this.#pipelines.get("main");
        passEncoder.setPipeline(pipelineContainer.pipeline);
        const renderRecursive = (meshOrGroup) => {
            if (meshOrGroup instanceof Group) {
                for (const child of meshOrGroup.children) {
                    renderRecursive(child)
                }
            } else if(meshOrGroup instanceof Mesh) {
                const meshContainer = this.#meshContainers.get(meshOrGroup);
                pipelineContainer.bindMethod(passEncoder, pipelineContainer.bindGroupLayouts, camera, meshContainer.mesh, this.#lights, thi
                passEncoder.setVertexBuffer(0, meshContainer.vertexBuffer);
                passEncoder.setIndexBuffer(meshContainer.indexBuffer, "uint16");
                passEncoder.drawIndexed(meshContainer.mesh.indices.length);
            }
        }
        renderRecursive(this.#sceneRoot);
        passEncoder.end();
    }
    if(this.#background){
        const passEncoder = commandEncoder.beginRenderPass({
            label: `background-render-pass`,
            colorAttachments: [
                {
                    storeOp: "store",
                    loadOp: "load",
                    view: colorView
                }
            ],
            depthStencilAttachment: {
                view: depthView,
                depthStoreOp: "store",
                depthLoadOp: "load"
            }
        });
        const pipelineContainer = this.#pipelines.get("background");
        passEncoder.setPipeline(pipelineContainer.pipeline);

        const meshContainer = this.#meshContainers.get(this.#background.mesh);
        pipelineContainer.bindMethod(passEncoder, pipelineContainer.bindGroupLayouts, camera, meshContainer.mesh, this.#lights, this.#shado
        passEncoder.setVertexBuffer(0, meshContainer.vertexBuffer);
        passEncoder.setIndexBuffer(meshContainer.indexBuffer, "uint16");
        passEncoder.drawIndexed(meshContainer.mesh.indices.length);
        passEncoder.end();
    }
    this.#device.queue.submit([commandEncoder.finish()]);
}

For completeness here's the change to the backgroundTextureBindGroup

setBackgroundTextureBindGroup(passEncoder, bindGroupLayouts) {
    const textureBindGroup = this.#device.createBindGroup({
        layout: bindGroupLayouts.get("materials"),
        entries: [
            { binding: 0, resource: this.#samplers.get(this.#background.sampler) },
            { binding: 1, resource: this.#textures.get(this.#background.environmentMap).createView({ dimension: "cube" }) },
        ]
    });
    passEncoder.setBindGroup(1, textureBindGroup);
}

For visual tests I added a new cube map with primary colors on each side so it's immediately apparent which way the camera is facing.

Note: If you caught it in the video above, there's a bug here that cost me like an hour of debugging later on.

Adding lights to groups

To start with this it would be nice to manipulate the cameras and lights the same way as the meshes. To do so lets create a base class Transformable that lets us perform common transforms and then the light and camera can inherit those abilities. These are just cut/pasted from mesh so you can update that class to inherit accordingly:

//transformable.js
import { getTranslationMatrix, getRotationXMatrix, getRotationYMatrix, getRotationZMatrix, getScaleMatrix, multiplyMatrix, getIdentityMatrix } from "../utilities/vector.js";

export class Transformable {
    #transforms = [];
    #worldMatrix = getIdentityMatrix();

    get modelMatrix() {
        return this.#transforms.reduce((mm, tm) => multiplyMatrix(tm, [4,4], mm, [4,4]), getIdentityMatrix());
    }
    get worldMatrix() {
        return this.#worldMatrix;
    }
    /**
     * @param {Float32Array} value 
     */
    set worldMatrix(value){
        this.#worldMatrix = value;
    }

    translate({ x = 0, y = 0, z = 0 }) {
        this.#transforms.push(getTranslationMatrix(x, y, z));
        return this;
    }
    scale({ x = 1, y = 1, z = 1 }) {
        this.#transforms.push(getScaleMatrix(x, y, z));
        return this;
    }
    rotate({ x, y, z }) {
        //there's an order dependency here... something something quaternions...
        if (x) {
            this.#transforms.push(getRotationXMatrix(x));
        }
        if (y) {
            this.#transforms.push(getRotationYMatrix(y));
        }
        if (z) {
            this.#transforms.push(getRotationZMatrix(z));
        }
        return this;
    }
    resetTransforms() {
        this.#transforms = [];
    }
}

Be aware that I also changed the getX methods into actual getters because it's a bit cleaner and I don't expect them to need parameters.

For the light we extend it:

//light.js
import { Transformable } from "./transformable.js";

export class Light extends Transformable { ... }

I noticed a bug where we were directly setting the private fields in the constructor of the Light, those should have actually gone through the setters (eg this.position = options.position this.#position = options.position). We'll also need to fix up the direction property:

set direction(val) {
    if(val.length === 3){
        this.#direction = new Float32Array([...val, 0]);
    } else {
        this.#direction = new Float32Array(val);
    }
}

We're using homogeneous coordinates (4-value) and this is necessary for the matrix multiplies to work. If we only pass in 3 values assume the last one is a 0 (remember that because it's a direction, not a position we give it a 0 instead of a 1). I'm going to try to be more consistent about using these coordinates in the engine to hopefully make things more consistent and explicit.

In the parsing we've already taken care of the transforms.

We will use the recursive group traversal to add them to a set. Unlike geometry, the semantic tree doesn't matter as much for lights, even if they are in different subtrees they can still impact the scene so we still iterate through them in a list (potentially we could be smarter in the future using complex heuristics). We need to make sure since they are still present along with some other entities in the scene graph we don't try to render them.

//gpu-engine.js - renderShadowMaps and renderScene
const renderRecursive = (meshOrGroup) => {
    if (meshOrGroup instanceof Group) {
        for (const child of meshOrGroup.children) {
            renderRecursive(child)
        }
    } else if(meshOrGroup instanceof Mesh) {
        const meshContainer = this.#meshContainers.get(meshOrGroup);
        pipelineContainer.bindMethod(passEncoder, pipelineContainer.bindGroupLayouts, camera, meshContainer.mesh, this.#lights, this.#shadowMaps)
        passEncoder.setVertexBuffer(0, meshContainer.vertexBuffer);
        passEncoder.setIndexBuffer(meshContainer.indexBuffer, "uint16");
        passEncoder.drawIndexed(meshContainer.mesh.indices.length);
    }
}

When we render a scene we apply the transform to the light. Unlike the mesh it's easier to apply this on the CPU since there's only one vector to be transformed we gain nothing from doing it on the GPU.

//gpu-engine.js - setMainLightBindGroup
//...stuff
const shadowMappedLights = lights
            .entries()
            .map(([key, value]) => {
                const shadowMap = shadowMaps.get(key);
                const shadowMapAspectRatio = shadowMap.width / shadowMap.height;

                return {
                    typeInt: value.typeInt,
                    position: multiplyMatrixVector(value.position, multiplyMatrix(getTranspose(value.worldMatrix, [4,4]), [4,4], value.modelMatrix, [4,4]), 4),
                    direction: multiplyMatrixVector(value.direction, multiplyMatrix(getTranspose(value.worldMatrix, [4,4]), [4,4], value.modelMatrix, [4,4]), 4),
                    color: value.color,
                    shadowMap,
                    projectionMatrix: shadowMap ? getLightProjectionMatrix(shadowMapAspectRatio) : getEmptyMatrix([4, 4]),
                    viewMatrix: shadowMap ? getLightViewMatrix(value.direction) : getEmptyMatrix([4, 4]),
                    castsShadow: value.castsShadow ? 1 : 0,
                    shadowMapIndex: (value.castsShadow && shadowMap) ? shadowMapIndex++ : -1
                };
            }).toArray();
//stuff...

The transform needs to be transposed. This is because the convention in WGSL is col-major and our matrices are row-major (this is mostly apparent because one gets the correct answer and the other rotates the opposite direction so if you see that this might be the answer). This will get a correct result.

Adding cameras to groups

Cameras can also be added to groups to be transformed with them but it will require a big refactor to the current camera system.

//camera.js
import { getOrthoMatrix, getProjectionMatrix, getCameraToWorldMatrixFromDirection, UP, subtractVector, normalizeVector, getWorldToCameraMatrixFromDirection, multiplyMatrixVector, addVector } from "../utilities/vector.js";
import { sphericalToCartesian, cartesianToSpherical } from "../utilities/math-helpers.js";
import { Transformable } from "./transformable.js";

export class Camera extends Transformable {
    #name;
    #position = new Float32Array([0,0,-1,1]);
    #direction;
    #screenWidth;
    #screenHeight;
    #near;
    #far;
    #left;
    #right;
    #top;
    #bottom;
    #fieldOfView;
    #isPerspective;

    /**
     * 
     * @param {{
     *   position: ArrayLike,
     *   screenWidth: number,
     *   screenHeight: number,
     *   left: number,
     *   right: number,
     *   top: number,
     *   bottom: number,
     *   near: number,
     *   far: number,
     *   fieldOfView: number,
     *   isPerspective: boolean
     * }} camera 
     */
    constructor(camera){
        super();
        if(!camera.name){
            throw new Error("Camera must have a name");
        }
        this.name = camera.name;
        this.position = camera.position;

        if(camera.direction){
            this.direction = normalizeVector(camera.direction);
        } else {
            this.lookAt(camera.target ?? new Float32Array([0,0,0,1]));
        }

        this.#screenWidth = camera.screenWidth;
        this.#screenHeight = camera.screenHeight;
        this.#left = camera.left;
        this.#right = camera.right;
        this.#top = camera.top;
        this.#bottom = camera.bottom;
        this.#near = camera.near;
        this.#far = camera.far;
        this.#fieldOfView = camera.fieldOfView;
        this.#isPerspective = camera.isPerspective;

        if (this.#isPerspective && (this.#screenWidth === undefined || this.#screenHeight === undefined || this.#near === undefined || this.#far === undefined || this.#fieldOfView === undefined)){
            throw new Error(`Missing required value for perspective projection`);
        }
        if (!this.#isPerspective && (this.#left === undefined || this.#right === undefined || this.#near === undefined || this.#far === undefined || this.#top === undefined || this.#bottom === undefined)) {
            throw new Error(`Missing required value for ortho projection`);
        }
    }
    lookAt(target){
        const normalizedTarget = normalizeVector(target.length === 3
            ? new Float32Array([...target, 1])
            : new Float32Array(target));
        this.direction = subtractVector(normalizedTarget, this.position); 
    }

    moveTo(x, y, z){
        this.position = [x,y,z,1];
    }

    moveBy({ x = 0, y = 0, z = 0 }){
        this.position = addVector(this.position, new Float32Array([x,y,z,1]));
    }

    panBy({ right = 0, up = 0, forward = 0 }){
        const cameraToWorld = getCameraToWorldMatrixFromDirection(this.position, this.direction);

        const delta = multiplyMatrixVector(new Float32Array([right, up, forward, 0]), cameraToWorld, 4);
        this.position = addVector(this.position, delta); 
    }

    orbitBy({ radius = 0, lat = 0, long = 0 }, target){
        const [currentLat, currentLng, r] = this.getOrbit(target); 
        const newLat = currentLat + lat;
        const newLong = currentLng - long;
        const newRadius = Math.max(0.1, r + radius);
        this.position = sphericalToCartesian([newLat, newLong, newRadius]);
    }

    getOrbit(target){
        const homgeneousTarget = target.length === 3 ? new Float32Array([...target, 1]) : new Float32Array(target);
        const targetDelta = subtractVector(this.position, homgeneousTarget);
        return cartesianToSpherical(targetDelta);
    }

    get viewMatrix(){
        return getWorldToCameraMatrixFromDirection(this.#position, this.#direction, UP);
    }

    get projectionMatrix(){
        return this.#isPerspective 
            ? getProjectionMatrix(this.#screenHeight, this.#screenWidth, this.#fieldOfView, this.#near, this.#far)
            : getOrthoMatrix(this.#left, this.#right, this.#bottom, this.#top, this.#near, this.#far);
    }

    get fieldOfView() {
        return this.#fieldOfView;
    }

    /**
     * 
     * @param {ArrayLike<number>} position 
     */
    set position(val){
        if(val.length === 3){
            this.#position = new Float32Array([...val, 1]);
        } else {
            this.#position = new Float32Array(val);
        }
    }
    get position(){
        return this.#position;
    }

    set direction(val){
        if(val.length === 3){
            this.#direction = new Float32Array([...val, 0]);
        } else {
            this.#direction = new Float32Array(val);
        }
    }
    get direction(){
        return this.#direction;
    }

    set name(val){
        this.#name = val;
    }
    get name(){
        return this.#name;
    }
}

We'll need to add the name to the Camera class so that we can keep track of it. Unlike Light which could have also used an explicit key, Cameras are more important to identify and carrying the name around as floating data would just get messier. The big thing here is that the Camera no longer has a target it's based entirely on direction with new methods to lookAt a certain position. This is because a direction is more intuitive to transform.

I've changed the methods that generate view matrices to be more explicitly named and to take homogeneous coordinates.

//vector.js

export function crossVector(a, b, isHomogeneous = false) {
    if(isHomogeneous){
        return new Float32Array([
            a[1] * b[2] - a[2] * b[1],
            a[2] * b[0] - a[0] * b[2],
            a[0] * b[1] - a[1] * b[0],
            0
        ]);
    }
    return new Float32Array([
        a[1] * b[2] - a[2] * b[1],
        a[2] * b[0] - a[0] * b[2],
        a[0] * b[1] - a[1] * b[0],
    ]);
}

/**
 * Creates the matrix to convert world space coordinates into camera space looking a particular direction
 * @param {Float32Array} position position of camera, homogeneous (4-value)
 * @param {Float32Array} direction direction of camera, homogeneous (4-value)
 * @param {Float32Array?} up direction of up, homogeneous (4-value)
 * @returns 
 */
export function getWorldToCameraMatrixFromDirection(position, direction, up = UP) {
    const forward = normalizeVector(direction);

    if(Math.abs(dotVector(forward, up)) > 0.999){
        up = Math.abs(forward[1]) < 0.999 ? UP : FORWARD;
    }

    const right = normalizeVector(crossVector(up, forward, true));
    const newUp = crossVector(forward, right, true);


    return new Float32Array([
        right[0], newUp[0], forward[0], 0,
        right[1], newUp[1], forward[1], 0,
        right[2], newUp[2], forward[2], 0,
        -dotVector(position, right), -dotVector(position, newUp), -dotVector(position, forward), 1
    ]);
}

/**
 * Creates the matrix to convert camera space coordinates into world space looking a particular direction
 * @param {Float32Array} position position of camera, homogeneous (4-value)
 * @param {Float32Array} direction direction of camera, homogeneous (4-value)
 * @param {Float32Array?} up direction of up, homogeneous (4-value)
 */
export function getCameraToWorldMatrixFromDirection(position, direction, up = UP) {
    const forward = normalizeVector(direction);

    // choose a stable up if forward is (nearly) parallel to provided up
    if (Math.abs(dotVector(forward, up)) > 0.999) {
        up = Math.abs(forward[1]) < 0.999 ? UP : FORWARD;
    }

    const right = normalizeVector(crossVector(up, forward, true));
    const newUp = crossVector(forward, right, true);

    return new Float32Array([
        right[0], newUp[0], forward[0], position[0],
        right[1], newUp[1], forward[1], position[1],
        right[2], newUp[2], forward[2], position[2],
        0,        0,        0,         1
    ]);
}

/**
 * Creates the matrix to convert world space coordinates into camera space looking at a particular target
 * @param {Float32Array} position position of camera, homogeneous (4-value)
 * @param {Float32Array} direction direction of camera, homogeneous (4-value)
 * @param {Float32Array?} up direction of up, homogeneous (4-value)
 * @returns 
 */
export function getWorldToCameraMatrixFromTarget(position, target, up = UP){
    return getWorldToCameraMatrixFromDirection(position, subtractVector(target, position), up);
}

There were a number of bugs I noticed moving to a different environment map. The biggest is that when creating the inverse of the view matrix I needed to remove the translation bits otherwise the movement of the cubemap gets wonky. This sorta makes sense because the cubemap is infinite distance so moving shouldn't impact it, only rotations.

setBackgroundSceneBindGroup(passEncoder, bindGroupLayouts, camera) {
    const viewRotationOnly = camera.viewMatrix.slice();
    viewRotationOnly[12] = 0;
    viewRotationOnly[13] = 0;
    viewRotationOnly[14] = 0;
    const inverseViewMatrix = getInverse(viewRotationOnly, [4, 4])
    const sceneBuffer = this.#device.createBuffer({
        size: inverseViewMatrix.byteLength,
        usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
        label: "background-scene-buffer"
    });
    this.#device.queue.writeBuffer(sceneBuffer, 0, inverseViewMatrix);
    const sceneBindGroup = this.#device.createBindGroup({
        label: "background-scene-bind-group",
        layout: bindGroupLayouts.get("scene"),
        entries: [
            {
                binding: 0,
                resource: {
                    buffer: sceneBuffer,
                    offset: 0,
                    size: inverseViewMatrix.byteLength
                }
            }
        ]
    });
    passEncoder.setBindGroup(0, sceneBindGroup);
}

Hopefully now everything comes together and still works as expected:

There are probably some spots I missed in my constant refactoring to call out. Please see the code changes for full details.

Convention Bugs

One issue I had was that with some transforms the light/normals/camera would rotate the opposite direction. There were several areas where I had to clean up some convention because I wasn't careful, and wasn't paying too much attention to which direction the transforms happened. Most of these stemmed the difference between WGSL's matrix ordering. In WGSL matrices are column major but in javascript they are row major (because that's how we define them). This means they need to transposed when passed to the GPU to be in the right order. I wasn't doing this in all cases which causes reversed rotations. I'm still not with the view and projection matrices but that's mostly because I didn't do a full investigation of what's happening there but I suspect it's wrong. Perhaps in the future this needs to be solved the the struct packer. To be clear, the convention is right-handed rotations. This means if you take your right hand and point your thumb along the rotation axis pointing in the position direction your fingers will curl the direction of the rotation. If you see weird rotation bugs double check these conventions.

Conclusion

We cleaned up the code and make things just a little bit more consistent which should make adding future features easier. With the scene graph we could do interesting things like scoping too. These refactors would be a lot harder to dig into without tests so I'm glad we have them even if they are a bit messy.

Code

https://github.com/ndesmic/geo/pull/5

WebGPU Engine from Scratch Part 11 - 1: Visual Testing

ndesmic — Mon, 13 Oct 2025 07:00:00 +0000

Note: This chapter was split into 2 parts because the second half deals with a different topic I thought would be easy be really wasn't.

Now that things are starting to get a bit complex, manual testing isn't really an option, it's simply too easy to break things so I'd like a way to confirm past features still work after refactors and such. Unfortunately, while I've billed this series as having no dependencies this is one where it really makes no sense to write from scratch (that could be it's own series though). So we'll use playwright for this task as I've found it to work well in the past and it's conforms pretty closely to puppeteer which I've also used a bit.

One small issue is that I'm not using node. Deno has support for most things in node but this one is a little tricky because it does some weird things. If you are using node you can skip a lot of the next section which helps deal with playwright's idiosyncrasies.

Install Playwright with Deno

Firstly we start with:

deno add npm:playwright

deno add npm:@playwright/test

This will install playwright and the associated test framework. playwright has other setup stuff it must do like download browsers and such so we also need to run

deno run playwright install

You might want to slap a -A on that because it's going to do a lot of scary file work.

We'll setup out new test in a folder called "visual-tests":

//basic.spec.js
import { test, expect } from "@playwright/test";

test("demo loads", async ({ page }) => {
    await page.goto("localhost:4507");
    await expect(page).toHaveTitle("Geo");
});

If we try to run it, it won't work for a few reasons. One we need to give it permissions because that's how Deno works. Sadly, because it's accessing lots of stuff outside of the project folder you can't really scope the permissions down, so I gave up and gave it all read permissions and write permissions. This is bad, but it's a node package and they do that. Here's the deno.json:

{
  "permissions": {
    "playwright": {
      "env": true,
      "sys": ["osRelease", "cpus"],
      "read": true,
      "write": true,
      "run": true
    }
  },
  "nodeModulesDir": "auto",
  "tasks": {
    "start": "deno run -A https://deno.land/std/http/file_server.ts",
    "test:one": "deno test ./js/utilities/buffer-utils.test.js",
    "test:visual": "deno run -P=playwright npm:playwright test visual-tests/"
  },
  "imports": {
    "@playwright/test": "npm:@playwright/test@^1.55.0",
    "@std/expect": "jsr:@std/expect@^1.0.16",
    "@std/testing": "jsr:@std/testing@^1.0.14",
    "playwright": "npm:playwright@^1.55.0"
  }
}

You will need nodeModulesDir: "auto" because playwright wants to use the node_modules directory to do stuff. Now as long as the tests are scoped to the visual-tests directory so they don't pick up the unit-tests it should run with:

deno task test:visual

As long as the server is running this should work.

Use a playwright config

Let's use a playwright.config.js instead as this will give us access to more features.

//deno.json
{
  "permissions": {
    "playwright": {
      "env": true,
      "sys": ["osRelease", "cpus"],
      "read": true,
      "write": true,
      "run": true,
      "net": true
    }
  },
  "nodeModulesDir": "auto",
  "tasks": {
    "start": "deno run -A https://deno.land/std/http/file_server.ts",
    "test:one": "deno test ./js/utilities/buffer-utils.test.js",
    "test:visual": "deno run -P=playwright npm:playwright test --config ./visual-tests/playwright.config.js"
  },
  "imports": {
    "@playwright/test": "npm:@playwright/test@^1.55.0",
    "@std/expect": "jsr:@std/expect@^1.0.16",
    "@std/testing": "jsr:@std/testing@^1.0.14",
    "playwright": "npm:playwright@^1.55.0"
  }
}

We need some more permissions for net and then we will point it at the playwright config in the visual-test folder. We no longer need the path as the config will contain that.

//./visual-tests/playwright.config.js
import { defineConfig } from '@playwright/test';

export default defineConfig({
    testMatch: './visual-tests/**/*.spec.js',
    // Run your local dev server before starting the tests
    webServer: {
        command: "deno task start",
        url: 'http://localhost:4507',
        reuseExistingServer: "http://localhost:4507",
        stdout: 'ignore',
        stderr: 'pipe',
    },
});

This will filter to just the .spec.js tests in the visual-tests folder. Keep in mind since the config is in the visual-tests folder, everything is relative to that. The web server command will run if none is found on localhost:4507 which is where the deno file server I've setup for dev and test is.

This will not actually work, at least on the current Deno on windows. There seems to be some problem with the way the stio streams get picked up by the runner. We need to remove the stdout and stderr settings in the config. Then we need to wrap the deno commands up:

#./visual-tests/start.ps1
#wrapper for deno webserver on windows because playwright doesn't like to run it directly

deno run -A https://deno.land/std/http/file_server.ts

You could also use cmd if powershell isn't your speed.

#!/usr/bin/env sh
#./visual-tests/start.sh
#wrapper for deno webserver on mac/linux because playwright doesn't like to run it directly
deno run -A https://deno.land/std/http/file_server.ts

//./visual-tests/playwright.config.js
import { defineConfig } from '@playwright/test';

export default defineConfig({
    testMatch: '**/*.spec.js',
    // Run your local dev server before starting the tests
    webServer: {
        command: process.platform === 'win32'
            ? `powershell -ExecutionPolicy ByPass -File "start.ps1"`
            : `sh -c "./start.sh"`,
        url: 'http://localhost:4507',
        reuseExistingServer: !process.env.CI,
    },
});

This will wrap the process and lets it work and work on multiple platforms.

Writing a test

I'm going to group each test with an html page which will render the test. So each folder under visual-tests will be one test with a .spec.js file and a .html file.

<!-- ./visual-tesats/should-render-teapot/should-render-teapot.html -->
<!doctype html>
<html lang="en">
    <head>
        <title>Geo - Should render teapot</title>
    </head>
    <body>
        <h1>Geo - Should render teapot</h1>
        <wc-geo>
            <geo-camera key="main" position="0, 1, 2"></geo-camera>

            <geo-texture key="red-fabric" src="../../img/red-fabric/red-fabric-base.jpg"></geo-texture>
            <geo-texture key="red-fabric-roughness" src="../../img/red-fabric/red-fabric-roughness.jpg"></geo-texture>
            <geo-texture key="gold" color="0, 0, 0, 1"></geo-texture>

            <geo-material key="red-fabric" roughness-map="red-fabric-roughness" albedo-map="red-fabric"></geo-material>
            <geo-material key="gold" roughness="0.2" metalness="1" base-reflectance="1.059, 0.773, 0.307" albedo-map="gold"></geo-material>

            <geo-group key="teapot-rug-1" pipeline="main">
                <geo-mesh 
                    key="teapot" 
                    normalize
                    bake-transforms 
                    reverse-winding 
                    src="../../objs/teapot.obj" 
                    resize-uvs="2" 
                    material="gold" 
                    attributes="positions, normals, uvs">
                </geo-mesh>
                <geo-surface-grid 
                    key="rug" 
                    row-count="2" 
                    col-count="2" 
                    translate="0, -0.25, 0" 
                    bake-transforms 
                    material="red-fabric" 
                    attributes="positions, normals, uvs">
                </geo-surface-grid>
            </geo-group>

            <geo-light key="light1" type="directional" color="1, 1, 1, 1" direction="0, -1, 1" casts-shadow></geo-light>
        </wc-geo>
        <script src="../../js/components/wc-geo.js" type="module"></script>
    </body>
</html>

//./visual-tesats/should-render-teapot/should-render-teapot.spec.js
import { test, expect } from "@playwright/test";

test("Should render teapot", async ({ page }) => {
    await page.goto("http://localhost:4507/visual-tests/should-render-teapot/should-render-teapot.html");
    await expect(page).toHaveTitle("Geo - Should render teapot");
    await expect(page).toHaveScreenshot();
});

Now if we run deno task test:visual it should start the webserver if not already running and run the tests. At this point we have two problems. One is that there is no screenshot, but that will be easily solved when one is created the first time. The bigger issue is the page doesn't render correctly because the path to the pbt.wgsl was hardcoded such that it can't be accessed from the new folder.

The path we are looking for is in js/engines/gup-engine/pipelines.js. What we should probably do is fix the path lookup so it is relative to the module, rather than relative to the page (this is closer to how ESM behaves and means we can still hardcode it as long as the structure doesn't change).

//pipelines.js
- const shaderModule = await uploadShader(device, import.meta.resolve("./shaders/pbr.wgsl"));
+ const shaderModule = await uploadShader(device, import.meta.resolve("../../../shaders/pbr.wgsl"));

- const shaderModule = await uploadShader(device, "./shaders/shadow-map.wgsl");
+ const shaderModule = await uploadShader(device, import.meta.resolve("../../../shaders/shadow-map.wgsl"));

We also need to move the background pipeline in here since we never got around to that.

//pipelines.js
export async function getBackgroundPipeline(device) {
    const vertexBufferDescriptor = [{
        attributes: [
            {
                shaderLocation: 0,
                offset: 0,
                format: "float32x2"
            }
        ],
        arrayStride: 8,
        stepMode: "vertex"
    }];

    const shaderModule = await uploadShader(device, import.meta.resolve("../../../shaders/space-background.wgsl"));

    const pipelineDescriptor = {
        label: "background-pipeline",
        vertex: {
            module: shaderModule,
            entryPoint: "vertex_main",
            buffers: vertexBufferDescriptor
        },
        fragment: {
            module: shaderModule,
            entryPoint: "fragment_main",
            targets: [
                { format: "rgba8unorm" }
            ]
        },
        primitive: {
            topology: "triangle-list"
        },
        depthStencil: {
            depthWriteEnabled: true,
            depthCompare: "less-equal",
            format: "depth32float"
        },
        layout: "auto"
    };

    return device.createRenderPipeline(pipelineDescriptor);
}

I'm not going to explicitly write out how to update gpu-engine.js for this.

Anyway once the code is fixed we can run the test. It will create a base image from which to compare future runs. Unfortunately, we still have more problems. If you actually look at the image you'll see the screen is white, but if you actually open it in Chrome it renders normally. This is because playwright doesn't enable the GPU so it's not actually rendering anything!

Debugging

One thing I found helpful is to have a baseline test that makes sure the environment itself is setup correctly.

//basic.spec.js
import { test, expect } from "@playwright/test";

test("Environment should be setup correctly", async ({ page }) => {
    page.goto("chrome://gpu");
    await expect(page).toHaveTitle("GPU Internals");
    await page.evaluate(async () => {
        document.querySelector("info-view").shadowRoot.querySelector(".info-table:first-child tr:first-of-type").style.display = "none";
    });
    await expect(page.getByText('WebGPU: Hardware accelerated')).toBeVisible();
    await expect(page).toHaveScreenshot({ maxDiffPixelRatio: 0.05 });
});

This basically ensures we can hit a page in the browser like our previous samples, but it also takes a screen shot of chrome://gpu which you can inspect to see what GPU settings are actually enabled and if they aren't displaying correctly. We look for "WebGPU: Hardware accelerated" so we know that it's turned on. The evaluate script hides a row that shows the current date which will make the test always fail but we also increase the maxDiffPixelRatio because version changes can make small text changes too. You can play around with the tolerances or hide other elements.

To enable the GPU we need to set a flag.

//./visual-tests/playwright.config.js
import { defineConfig } from '@playwright/test';

export default defineConfig({
    use: {
        channel: "chrome",
        launchOptions: {
            // force GPU hardware acceleration
            // (even in headless mode)
            args: [
                '--enable-gpu'
            ]
        }
    },
    testMatch: '**/*.spec.js',
    // Run your local dev server before starting the tests
    webServer: {
        command: process.platform === 'win32'
            ? `powershell -ExecutionPolicy ByPass -File "start.ps1"`
            : `sh -c "./start.sh"`,
        url: 'http://localhost:4507',
        reuseExistingServer: !process.env.CI,
    },
});

Now running the tests should produce a screenshot of the teapot that we can compare future versions against. If you need to regenerate the screenshot run deno task test:visual --update-snapshots. Note that even after the fixes sometimes I found that playwright just kinda dies on waiting for a pipe or unexpected closure. I don't think much can be done about that on our end.

Clipping just the element

The screen shots are of the page but the rest of the page really doesn't matter. We can alter the test slightly to just get the element.

//./visual-tesats/should-render-teapot/should-render-teapot.spec.js
import { test, expect } from "@playwright/test";

test("Should render teapot", async ({ page }) => {
    await page.goto("http://localhost:4507/visual-tests/should-render-teapot/should-render-teapot.html");
    await expect(page).toHaveTitle("Geo - Should render teapot");
    const canvas = await page.locator("canvas");
    await expect(canvas).toHaveScreenshot();
});

Note that I found that playwright took screenshots that were 1px taller than the actual element. Clipping based on the bounding box didn't help so I'm not really sure what to do about that, just some bug in the underlying libraries I guess.

Waiting for the element

Sometime you might need to wait for the first render because the screenshot will happen too fast. One way is to add an element to the page once it's finished rendering.

import { test, expect } from "@playwright/test";

test("Should transform group mesh", async ({ page }) => {
    await page.goto("http://localhost:4507/visual-tests/should-transform-group-mesh/should-transform-group-mesh.html");
    await expect(page).toHaveTitle("Geo - Should transform group mesh");
    const canvas = await page.locator("canvas");
    const rendered = await page.locator(".rendered"); //llok for this
    expect(rendered).toBeVisible();
    await expect(canvas).toHaveScreenshot();
});

We can add this from the element:

<script>
    const geo = document.querySelector("wc-geo");
    geo.onRender = () => {
        if (!document.querySelector(".rendered")) {
            const div = document.createElement("div");
            div.className = "rendered";
            div.textContent = "rendered";
            document.body.append(div);
        }
    }
</script>

We'll also need to add the onRender callback to wc-geo:

//wc-geo.js
async connectedCallback() {
    //...
    this.engine = new Engine({
        canvas: this.dom.canvas,
        onRender: this.onRender
    });
    //...
}

onRender() {} //stub in case it's not used

And finally to the engine itself:

//gpu-engine.js
#onRender;

constructor(options) {
    //...
    this.#context = options.canvas.getContext("webgpu");
    //...
}

render() {
    this.renderShadowMaps();
    //this.renderDepthBuffer(this.#shadowMaps.get("light1").createView());
    this.renderScene();
    this.#onRender?.();
}

Conclusion

This setup kinda works but isn't great. I still get lots of handle invalid errors when running tests though. Might have better luck with playwright if we switch back to node. One of my bigger concerns is the amount of code and performance of tests. We'll probably want a lot and I don't know how well this will scale. An idea might be to use deno's WebGPU capability itself to render the scenes for the tests which would remove the entire browser and ultimately much more flaky setup of attaching to the browser debug socket which is how Playwright -> Puppeteer work under the hood. This however would require that we rebuild the screenshot testing. Perhaps a future chapter could look at that.

In the next half, we'll finish the scene graph implementation so everything works more consistently and the code is a bit cleaner.

WebGPU Engine from Scratch Part 10: Markup Language and Scene Graph

ndesmic — Mon, 06 Oct 2025 19:00:00 +0000

One thing that is becoming really annoying while doing manual tests is that I have to setup scenes. These are done by adding objects to the various dictionaries in the gpu-engine.js file which involves lots of scrolling and you can't see the scene description as a whole. This time I want to create an SVG-like scene descriptions for this purpose.

The scene as an object

The engine shouldn't deal with anything like this, it should just take raw scene data. It does but it's all over the place so let's combine it all into one object that we can pass in. For each initialize method we pass in the object. At this point I'm not sure if they should be classes themselves or just the object passed to the class constructor. For now they are classes since I don't see an issue with having those external.

Cameras

initializeCameras(cameras){
    for(const [key, camera] of Object.entries(cameras)){
        this.#cameras.set(key, camera);
    }
}

Nothing too interesting here, just iterate over the object to turn it into the map.

Textures

export const DEPTH_TEXTURE = Symbol("depth-texture");
export const PLACEHOLDER_TEXTURE = Symbol("placeholder-texture");

async initializeTextures(textures) {
    for(const [key, texture] of Object.entries(textures)){
        if(texture.image ?? texture.images){
            this.#textures.set(key, await uploadTexture(this.#device, texture.image ?? texture.images, { label: `${key}-texture` }));
        } else if(texture.color){
            this.#textures.set(key, createColorTexture(this.#device, { color: texture.color, label: `${key}-texture` }));
        }
    }
    //default textures
    this.#textures.set(DEPTH_TEXTURE, this.#device.createTexture({
        label: "depth-texture",
        size: {
            width: this.#canvas.width,
            height: this.#canvas.height,
            depthOrArrayLayers: 1
        },
        format: "depth32float",
        usage: GPUTextureUsage.RENDER_ATTACHMENT
    }));
    this.#textures.set(PLACEHOLDER_TEXTURE, createColorTexture(this.#device, { label: "placeholder-texture" }));
}

There are two types of textures. Those that come from images and those that are colors so we handle those cases based on which key is passed in. For some semantic ergonomics we support image or images but they are normalized in either case. We also have our 2 textures that are necessary for the pipeline to function. I've changed those to use symbol keys so they don't get overridden from the outside unintentionally. These need to be applied to the Material class which references them as well. I've pushed the async fetching outside of the engine because that's really not it's concern. With that I've updated uploadTexture to not do the fetching:

//wgpu-utils.js
/**
 * Loads an image url, uploads to GPU and returns texture ref.
 * Cubemaps defined like [+X, -X, +Y, -Y, +Z, -Z]
 * @param {GPUDevice} device 
 * @param {HTMLImageElement | HTMLImageElement[]} imageOrImages
 * @param {{ label?: string }} options 
 */
export function uploadTexture(device, imageOrImages, options = {}) {
    const images = [].concat(imageOrImages);
    const size = {
        width: images[0].width,
        height: images[0].height,
        depthOrArrayLayers: images.length
    };

    const texture = device.createTexture({
        label: options.label,
        size,
        dimension: "2d",
        format: `rgba8unorm`,
        usage: GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
    });

    images.forEach((img, layer) => {
        device.queue.copyExternalImageToTexture(
            {
                source: img,
                flipY: true
            },
            {
                texture,
                origin: [0, 0, layer]
            },
            {
                width: img.width,
                height: img.height,
                depthOrArrayLayers: 1
            }
        );
    });

    return texture;
}

Materials

initializeMaterials(materials) {
    for (const [key, material] of Object.entries(materials)) {
        this.#materials.set(key, material);
    }
}

Materials are simple.

Samplers

I don't see a good reason right now to let these be set from the outside so they are left alone. But if we do pass them in we need to build a wrapper object for them and iterate over them like usual. I did add symbols for the default and shadow default sampler and those references need to be updated.

Meshes

initializeMeshes(meshes){
    for(const [key, mesh] of Object.entries(meshes)){
        const { vertexBuffer, indexBuffer } = uploadMesh(this.#device, mesh, { label: `${key}-mesh` });
        this.#meshContainers.set(key, { mesh, vertexBuffer, indexBuffer });
    }
}

These are pretty simple too but we need to make sure to keep GPU related activities like uploadMesh in the engine itself. We'd also like to push async network stuff outside of the engine so fetchObj is in the component.

Lights

initializeLights(lights){
    for (const [key, light] of Object.entries(lights)) {
        this.#lights.set(key, light)
    }
    for(const key of this.#lights.keys()){
        this.#shadowMaps.set(key, this.#device.createTexture({
            label: `shadow-map-${key}`,
            size: {
                width: 2048,
                height: 2048,
                depthOrArrayLayers: 1
            },
            format: "depth32float",
            usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
        }));
    }
    this.#shadowMaps.set("placeholder", this.#device.createTexture({
        label: "placeholder-depth-texture",
        size: { width: 1, height: 1, depthOrArrayLayers: 1 },
        format: "depth32float",
        usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
    }));
}

Nothing fancy here either except we need to setup the shadow maps. Shadow maps are an internal only thing so I'm not assigning symbols to anything. Since the ShadowMappedLight class is passed in this might indicate some internal leakage...

Pipelines

These shouldn't be exposed.

Pipeline - Mesh associations

initializePipelineMeshes(pipelineMeshes){
    for(const [key, pipelineMesh] of Object.entries(pipelineMeshes)){
        this.#pipelineMesh.set(key, pipelineMesh);
    }
}

Nothing to it. Although this means that the author needs to know about our pipelines and what they are named. Should pipelines be exposed or maybe we need some other abstraction so we can hide the pipeline stuff?

Here's how the new initialization works:

await this.engine.initialize({
    scene: {
        cameras: {
            "main": new Camera({
                position: [0.5, 0.2, -0.5],
                screenHeight: this.dom.canvas.height,
                screenWidth: this.dom.canvas.width,
                fieldOfView: 90,
                near: 0.01,
                far: 5,
                isPerspective: true
            })
        },
        textures: {
            "marble": { image: await loadImage("./img/marble-white/marble-white-base.jpg") },
            "marble-roughness": { image: await loadImage("./img/marble-white/marble-white-roughness.jpg") },
            "red-fabric": { image: await loadImage("./img/red-fabric/red-fabric-base.jpg") },
            "red-fabric-roughness": { image: await loadImage("./img/red-fabric/red-fabric-roughness.jpg") },
            "gold": { color: [0, 0, 0, 1] },
        },
        materials : {
            "marble": new Material({
                texture: "marble",
                useSpecularMap: true,
                specularMap: "marble-roughness"
            }),
            "red-fabric": new Material({
                texture: "red-fabric",
                useSpecularMap: true,
                specularMap: "red-fabric-roughness"
            }),
            "gold": new Material({
                texture: "gold",
                useSpecularMap: false,
                roughness: 0.2,
                metalness: 1,
                baseReflectance: [1.059, 0.773, 0.307]
            })
        },
        meshes: {
            "teapot": (await fetchObjMesh("./objs/teapot.obj", { reverseWinding: true }))
                .useAttributes(["positions", "uvs", "normals"])
                .normalizePositions()
                .resizeUvs(2)
                .setMaterial("gold"),
            "rug": new Mesh(surfaceGrid(2, 2))
                .useAttributes(["positions", "uvs", "normals"])
                .translate({ y: -0.25 })
                .bakeTransforms()
                .setMaterial("red-fabric")
        },
        lights: {
            "light": new ShadowMappedLight({
                type: "directional",
                color: [1.0, 1.0, 1.0, 1],
                direction: [0, -1, 1],
                hasShadow: true,
            })
        },
        pipelineMeshes: {
            "main": ["teapot", "rug"]
        }
    }
});

It's all nicely packed into one structure. It's not perfect. ShadowMappedLight probably shouldn't be part of this, and we know about the pipelines which if this was a different engine these implementations might not be a thing. So there will likely be more work to do here in the future to not leak that into the scene representation.

Building the markup

Might as well use HTML because it's already parsed as DOM. We don't really need custom elements or anything, just the ability to read it. In order to get the key for each entity it will be required that they all have an attribute key. I had originally went with id before deciding that was a bad idea because it conflates DOM ids with our entities key naming.

We do this with a new utility function parseScene which takes a wc-geo element as an parameter.

export async function parseScene(element) { //more below

Cameras

//geo-markup-parser.js
function parseCamera(cameraEl, options) {
    const key = getKey(cameraEl, "camera");

    return [
        key,
        new Camera({
            position: parseVector(cameraEl.getAttribute("position"), 3),
            screenHeight: cameraEl.getAttribute("height") ?? options.defaultHeight,
            screenWidth: cameraEl.getAttribute("width") ?? options.defaultHWidth,
            fieldOfView: cameraEl.getAttribute("fov") ?? 90,
            near: cameraEl.getAttribute("near") ?? 0.01,
            far: cameraEl.getAttribute("far") ?? 5,
            isPerspective: !cameraEl.hasAttribute("is-orthographic")
        })
    ]
}

//geo-markup-parser.js - parseScene
const cameras = Object.fromEntries(Array.from(element.querySelectorAll("geo-camera"))
    .map(c => parseCamera(c, { defaultHeight: element.dom.canvas.height, defaultHWidth: element.dom.canvas.width })));

The only interesting thing here is that we need the default screenHeigth and screenWidth which come from the canvas. It's a bit of hack to just read it out.

Textures

// geo-markup-parser.js
async function parseTexture(textureEl){
    const key = getKey(textureEl, "texture")
    const src = textureEl.getAttribute("src");
    const color = textureEl.getAttribute("color");
    let value;
    if (src) {
        value = { image: await loadImage(src) };
    } else if (color) {
        value = { color: parseVector(color, 4) };
    }

    return [key, value];
}

// geo-markup-parser.js - parseScene
const textures = Object.fromEntries(await Promise.all(Array.from(element.querySelectorAll("geo-texture"))
        .map(parseTexture)));

Textures have two types, color and image url. Unfortunately they are async which complicates things (ideally async things should not happen in the parser but whatever).

Materials

For the materials I decided we'll stick with the PBR paradigm going forward and so instances of string "texture" are renamed "albedoMap" and "specularMap" to "roughnessMap".

// geo-markup-parser.js
function parseMaterial(materialEl) {
    const key = getKey(materialEl, "material");
    const roughnessMap = materialEl.getAttribute("roughness-map");
    const albedoMap = materialEl.getAttribute("albedo-map");

    return [
        key,
        new Material({
            name: key,
            albedoMap: albedoMap,
            useRoughnessMap: !!roughnessMap,
            roughness: parseFloatOrDefault(materialEl.getAttribute("roughness")),
            metalness: parseFloatOrDefault(materialEl.getAttribute("metalness")),
            baseReflectance: parseVector(materialEl.getAttribute("base-reflectance"), 3)
        })
    ]
}

// geo-markup-parser.js - parseScene
const materials = Object.fromEntries(Array.from(element.querySelectorAll("geo-material"))
    .map(parseMaterial));

Nothing interesting otherwise.

Meshes

The surface grid can be it's own element since it's a generative primitive mesh. I've renamed the height and width parameters to rowCount and colCount respectively because they align better with what's happening and don't imply orientation.

// geo-markup-parser.js
async function parseMesh(meshEl) {
    const key = getKey(meshEl, "mesh");
    const reverseWinding = meshEl.hasAttribute("reverse-winding");
    const src = meshEl.getAttribute("src");
    const mesh = await fetchObjMesh(src, { reverseWinding });

    updateMeshAttributes(meshEl, mesh);

    return [key, mesh];
}

// geo-markup-parser.js
function parseSurfaceGrid(meshEl) {
    const key = getKey(meshEl, "surface-grid");
    const rowCount = parseInt(meshEl.getAttribute("row-count"), 10);
    const colCount = parseInt(meshEl.getAttribute("col-count"), 10);
    const mesh = new Mesh(surfaceGrid(rowCount, colCount));

    updateMeshAttributes(meshEl, mesh);

    return [key, mesh];
}

// geo-markup-parser.js - sceneParser
const meshes = Object.fromEntries(await Promise.all(Array.from(element.querySelectorAll("geo-mesh"))
    .map(parseMesh)));
const surfaceGrids = Object.fromEntries(Array.from(element.querySelectorAll("geo-surface-grid"))
    .map(parseSurfaceGrid));

Lights

After thinking about it harder, the ShadowMappedLight doesn't need to exist after all. Since we can calculate the view matrix and projection matrix with a few constants and non-light values we can make these into helper functions. The only exception is hasShadow which can be moved into the Light itself because it does make sense for it to be used in other engine contexts. I did that and renamed it to castsShadow because it makes more grammatical sense

//light-utils.js
const distance = 0.75;
const center = [0, 0, 0];
const frustumScale = 2;

export function getLightViewMatrix(direction) {
    const lightPosition = scaleVector(subtractVector(center, direction), distance);
    return getLookAtMatrix(lightPosition, center);
}

export function getLightProjectionMatrix(aspectRatio) {
    const right = aspectRatio * frustumScale;
    return getOrthoMatrix(-right, right, -frustumScale, frustumScale, 0.1, Math.min(distance * 2, 2.0));
}

Even this is dubious because it's just some transforms that are near-generic. Could also be removed but I want some level of abstraction because I feel in the future this might handle more than it does like point and spotlights.

// geo-markup-parser.js
function parseLights(lightEl) {
    const key = getKey(lightEl, "light");
    const light = new Light({
        type: lightEl.getAttribute("type") ?? "point",
        color: parseVector(lightEl.getAttribute("color"), 4, [1, 1, 1, 1]),
        direction: parseVector(lightEl.getAttribute("direction"), 3, [0, 0, 0]),
        castsShadow: lightEl.hasAttribute("casts-shadow")
    });

    return [key, light];
}

// geo-markup-parser.js - parseScene
const lights = Object.fromEntries(Array.from(element.querySelectorAll("geo-light"))
        .map(parseLights));

Pipeline mesh

I still don't like it but we'll take it from the meshes themselves.

// geo-markup-parser.js
function getPipelineMesh(meshEl) {
    const pipeline = meshEl.getAttribute("pipeline");
    const meshKey = meshEl.getAttribute("key");

    return {
        pipeline,
        meshKey
    };
}

// geo-markup-parser.js - parseScene
const pipelineMeshes = Array.from(element.querySelectorAll("geo-mesh, geo-surface-grid"))
        .map(getPipelineMesh);

Finally we return the rest:

// geo-markup-parser.js - parseScene
if (cameras.length === 0) {
    throw new Error("Need a 'main' camera defined");
}
return {
    cameras,
    textures,
    materials,
    meshes: { ...meshes, ...surfaceGrids },
    lights,
    pipelineMeshes
};

This code is not fantastic. We don't have a lot of validation or error messages that would be nice. This won't work on the server without a DOM polyfill either. It might be nice if some resources could be anonymous and just nested, like textures under materials for example. But it gets the job done. If there's more need maybe we can do a bit more.

Grouping and scene graph

One thing that would be nice is some grouping though. The idea here is that we group related parts of a scene so we can do things like apply transforms to all of them. We can start with a group class.

//group.js
export class Group {
    #children = [];
    #transforms = [];

    constructor(options) {
        this.#children = options.children;
    }

    get children() {
        return this.#children;
    }

    getModelMatrix() {
        return this.#transforms.reduce((mm, tm) => multiplyMatrix(tm, [4, 4], mm, [4, 4]), getIdentityMatrix());
    }

    translate({ x = 0, y = 0, z = 0 }) {
        this.#transforms.push(getTranslationMatrix(x, y, z));
        return this;
    }
    scale({ x = 1, y = 1, z = 1 }) {
        this.#transforms.push(getScaleMatrix(x, y, z));
        return this;
    }
    rotate({ x, y, z }) {
        //there's an order dependency here... something something quaternions...
        if (x) {
            this.#transforms.push(getRotationXMatrix(x));
        }
        if (y) {
            this.#transforms.push(getRotationYMatrix(y));
        }
        if (z) {
            this.#transforms.push(getRotationZMatrix(z));
        }
        return this;
    }
}

This just collects meshes and has a transform array. Technically we could add more types of things to the group like lights and cameras but for simplicity today we'll just consider meshes. I'm also not going to immediately deal with subclassing yet but it makes sense that "transformable" things share one.

To parse a group

//geo-markup-parser.js
/**
 * 
 * @param {HTMLElement} groupEl 
 */
async function parseGroup(groupEl){
    const key = getKey(groupEl, "group");
    const children = await Promise.all(Array.from(groupEl.children).map(async c => {
        switch(c.tagName){
            case "GEO-MESH": {
                return (await parseMesh(c))[1];
            }
            case "GEO-SURFACE-GRID": {
                return parseSurfaceGrid(c)[1];
            }
            default: {
                throw new Error(`Group doesn't support ${c.tagName} children`)
            }
        }
    }))

    const group = new Group({
        children
    });

    return [key, group];
}

//geo-markup-parser.js - parseScene
const groups = Object.fromEntries(await Promise.all(Array.from(element.children).filter(c => c.tagName === "GEO-GROUP")
    .map(parseGroup)));

Parsing is simple but I've modified it so that it only checks one level deep. This will be a limitation for now but it sets up recursion. Note that the children we only need the value, not the key. Once recursion is setup the keys will be less important. Likewise, meshes will not check the whole tree, just the top level.

//geo-markup-parser.js - parseScene
const meshes = Object.fromEntries(await Promise.all(Array.from(element.children).filter(c => c.tagName === "GEO-MESH")
    .map(parseMesh)));

And hook it up in the engine:

//gpu-engine.js - intialize
this.initializeGroups(scene.groups);

//gpu-engine.js
initializeGroups(groups){
    for(const [key, group] of Object.entries(groups)){
        this.initializeMeshes(group.children);
        this.#groups.set(key, group);
    }
}

At this point nothing should change.

To render the groups we need to make the rendering recursive.

//gpu-engine.js - renderShadowMaps
renderShadowMaps(){
    const commandEncoder = this.#device.createCommandEncoder({
        label: "shadow-map-command-encoder"
    });
    const shadowMapPipelineContainer = this.#pipelines.get("shadow-map");
    for(const [key, light] of this.#lights){
        let isFirstPass = true;
        const passEncoder = commandEncoder.beginRenderPass({
            label: `shadow-map-render-pass`,
            colorAttachments: [],
            depthStencilAttachment: {
                view: this.#shadowMaps.get(key).createView(),
                depthClearValue: 1.0,
                depthStoreOp: "store",
                depthLoadOp: isFirstPass ? "clear" : "load",
            }
        });
        passEncoder.setPipeline(shadowMapPipelineContainer.pipeline);
        const renderRecursive = (meshOrGroup) => {
            if(meshOrGroup instanceof Group){
                for(const child of meshOrGroup.children){
                    renderRecursive(child)
                }
            } else {
                const shadowMap = this.#shadowMaps.get(key);
                const meshContainer = this.#meshContainers.get(meshOrGroup);
                shadowMapPipelineContainer.bindMethod(passEncoder, shadowMapPipelineContainer.bindGroupLayouts, light, shadowMap, meshOrGroup);
                passEncoder.setVertexBuffer(0, meshContainer.vertexBuffer);
                passEncoder.setIndexBuffer(meshContainer.indexBuffer, "uint16");
                passEncoder.drawIndexed(meshContainer.mesh.indices.length);
            }
        }
        for (const meshName of this.#pipelineMesh.get("main")) {
            const group = this.#groups.get(meshName);
            renderRecursive(group);
        }
        passEncoder.end();
        isFirstPass = false;
    }
    this.#device.queue.submit([commandEncoder.finish()]);
}

//gpu-engine.js - renderScene
renderScene(){
    const commandEncoder = this.#device.createCommandEncoder({
        label: "main-command-encoder"
    });
    const camera = this.#cameras.get("main");
    let isFirstPass = true;
    const depthView = this.#textures.get(DEPTH_TEXTURE).createView();
    for (const [pipelineName, meshNames] of this.#pipelineMesh.entries()) {
        const passEncoder = commandEncoder.beginRenderPass({
            label: `${pipelineName}-render-pass`,
            colorAttachments: [
                {
                    storeOp: "store",
                    loadOp: isFirstPass ? "clear" : "load",
                    clearValue: { r: 0.1, g: 0.3, b: 0.8, a: 1.0 },
                    view: this.#context.getCurrentTexture().createView()
                }
            ],
            depthStencilAttachment: {
                view: depthView,
                depthClearValue: 1.0,
                depthStoreOp: "store",
                depthLoadOp: isFirstPass ? "clear" : "load"
            }
        });
        const pipelineContainer = this.#pipelines.get(pipelineName);
        passEncoder.setPipeline(pipelineContainer.pipeline);
        const renderRecursive = (meshOrGroup) => {
            if(meshOrGroup instanceof Group){
                for(const child of meshOrGroup.children){
                    renderRecursive(child)
                }
            } else {
                const meshContainer = this.#meshContainers.get(meshOrGroup);
                pipelineContainer.bindMethod(passEncoder, pipelineContainer.bindGroupLayouts, camera, meshContainer.mesh, this.#lights, this.#shadowMaps);
                passEncoder.setVertexBuffer(0, meshContainer.vertexBuffer);
                passEncoder.setIndexBuffer(meshContainer.indexBuffer, "uint16");
                passEncoder.drawIndexed(meshContainer.mesh.indices.length);
            }
        }
        for (const meshName of meshNames) {
            const group = this.#groups.get(meshName);
            renderRecursive(group);
        }
        passEncoder.end();
        isFirstPass = false;
    }
    this.#device.queue.submit([commandEncoder.finish()]);
}

Note that since meshes no longer have specific keys, I changed the lookup to work on the mesh object itself.

//gpu-engine.js - initializeMeshes
initializeMeshes(meshes){
    for(const [key, mesh] of Object.entries(meshes)){
        const { vertexBuffer, indexBuffer } = uploadMesh(this.#device, mesh, { label: `${key}-mesh` });
        this.#meshContainers.set(mesh, { mesh, vertexBuffer, indexBuffer });
    }
}

This again should render the same scene provided we update the markup.

<wc-geo>
    <geo-camera key="main" position="0.5, 0.2, -0.5"></geo-camera>
    <geo-texture key="marble" src="./img/marble-white/marble-white-base.jpg"></geo-texture>
    <geo-texture key="marble-roughness" src="./img/marble-white/marble-white-roughness.jpg"></geo-texture>
    <geo-texture key="red-fabric" src="./img/red-fabric/red-fabric-base.jpg"></geo-texture>
    <geo-texture key="red-fabric-roughness" src="./img/red-fabric/red-fabric-roughness.jpg"></geo-texture>
    <geo-texture key="gold" color="0, 0, 0, 1"></geo-texture>
    <geo-material key="marble" roughness-map="marble-roughness" albedo-map="marble"></geo-material>
    <geo-material key="red-fabric" roughness-map="red-fabric-roughness" albedo-map="red-fabric"></geo-material>
    <geo-material key="gold" roughness="0.2" metalness="1" base-reflectance="1.059, 0.773, 0.307" albedo-map="gold"></geo-material>
    <geo-group key="teapot-rug" pipeline="main" rotate="1.5707963267948966, 0, 0">
        <geo-mesh 
            key="teapot" 
            normalize
            bake-transforms 
            reverse-winding 
            src="./objs/teapot.obj" 
            resize-uvs="2" 
            material="gold" 
            attributes="positions, normals, uvs">
        </geo-mesh>
        <geo-surface-grid 
            key="rug" 
            row-count="2" 
            col-count="2" 
            translate="0, -0.25, 0" 
            bake-transforms 
            material="red-fabric" 
            attributes="positions, normals, uvs">
        </geo-surface-grid>
    </geo-group>

    <geo-light key="light1" type="directional" color="1, 1, 1, 1" direction="0, -1, 1" casts-shadow></geo-light>

I've added the pipeline to the group instead. The one thing that doesn't work is the rotation because we need to recursively setup the transform.

Transforms on groups

Groups should be recursive in order to make full use of them. In order to do that we should start parsing them recursively:

// geo-markup-parser.js
/**
 * 
 * @param {HTMLElement} groupEl 
 */
async function parseGroup(groupEl){
    const key = getKey(groupEl, "group");
    const children = await Promise.all(Array.from(groupEl.children).map(async c => {
        switch(c.tagName){
            case "GEO-MESH": {
                return (await parseMesh(c))[1];
            }
            case "GEO-SURFACE-GRID": {
                return parseSurfaceGrid(c)[1];
            }
            case "GEO-QUAD": {
                return parseQuad(c);
            }
            case "GEO-CUBE": {
                return parseCube(c)
            }
            case "GEO-GROUP": {
                return (await parseGroup(c))[1]
            }
            default: {
                throw new Error(`Group doesn't support ${c.tagName} children`)
            }
        }
    }));

    const group = new Group({
        children
    });

    const translate = parseVector(groupEl.getAttribute("translate"), 3);
    if (translate) {
        group.translate({ x: translate[0], y: translate[1], z: translate[2] });
    }

    const rotate = parseVector(groupEl.getAttribute("rotate"), 3);
    if (rotate) {
        group.rotate({ x: rotate[0], y: rotate[1], z: rotate[2] });
    }

    const scale = parseVector(groupEl.getAttribute("scale"), 3);
    if (scale) {
        group.scale({ x: scale[0], y: scale[1], z: scale[2] });
    }

    return [key, group];
}

It's not very clean because we have keys we aren't using but it'll do for now. In the engine we should initialize things recursively.

//gpu-engine.js
initializeMesh(mesh, key) {
    const { vertexBuffer, indexBuffer } = uploadMesh(this.#device, mesh, { label: `${key}-mesh` });
    this.#meshContainers.set(mesh, { mesh, vertexBuffer, indexBuffer });
}
initializeMeshes(meshes) {
    for (const [key, mesh] of Object.entries(meshes)) {
        this.initializeMesh(mesh, key);
    }
}
initializeGroups(groups) {
    for (const [key, group] of Object.entries(groups)) {
        this.initializeGroup(group, key);
    }
}
initializeGroup(group, key) {
    for (const child of group.children) {
        if (child instanceof Mesh) {
            this.initializeMesh(child);
        } else if (child instanceof Group) {
            this.initializeGroup(child);
        }
    }
    this.#groups.set(key, group);
}

This still allows things to work the other way for now but if we restricted everything to a group this would become cleaner.

In order to actually apply the transforms we'll create a new concept of a world matrix per entity. (I also changed the getModelMatrix method to just be a getter since I doubt we'll need parameters for those)

//mesh.ts
#worldMatrix = getIdentityMatrix();
//mesh.js
get worldMatrix() {
    return this.#worldMatrix;
}
/**
 * @param {Float32Array} value 
 */
set worldMatrix(value){
    this.#worldMatrix = value;
}

This is world matrix tells the object its parent transforms. The idea here is that the world matrix is set recursively. At each group we multiple the world matrix by the model matrix of the group, this becomes the world matrix for each of that group's child entities. By recursively multiplying them they will get us coordinates in the top-most space, the true global world space.

//group.ts
import { getIdentityMatrix, getRotationXMatrix, getRotationYMatrix, getRotationZMatrix, getScaleMatrix, getTranslationMatrix, multiplyMatrix } from "../utilities/vector.js";

export class Group {
    #children = [];
    #transforms = [];
    #worldMatrix = getIdentityMatrix();

    constructor(options) {
        this.#children = options.children;
    }

    get children() {
        return this.#children;
    }

    get modelMatrix() {
        return this.#transforms.reduce((mm, tm) => multiplyMatrix(tm, [4, 4], mm, [4, 4]), getIdentityMatrix());
    }

    get worldMatrix(){
        return this.#worldMatrix;
    }

    set worldMatrix(value){
        this.#worldMatrix = value;
        this.updateWorldMatrix();
    }

    updateWorldMatrix(){
        const worldMatrix = multiplyMatrix(this.modelMatrix, [4,4], this.#worldMatrix, [4,4]);
        for(const child of this.#children){
            child.worldMatrix = worldMatrix;
        }
    }

    translate({ x = 0, y = 0, z = 0 }) {
        this.#transforms.push(getTranslationMatrix(x, y, z));
        this.updateWorldMatrix();
        return this;
    }
    scale({ x = 1, y = 1, z = 1 }) {
        this.#transforms.push(getScaleMatrix(x, y, z));
        this.updateWorldMatrix();
        return this;
    }
    rotate({ x, y, z }) {
        //there's an order dependency here... something something quaternions...
        if (x) {
            this.#transforms.push(getRotationXMatrix(x));
        }
        if (y) {
            this.#transforms.push(getRotationYMatrix(y));
        }
        if (z) {
            this.#transforms.push(getRotationZMatrix(z));
        }
        this.updateWorldMatrix();
        return this;
    }
}

Here, whenever a group is updated with a transform it will recursively update its children, and if they are groups, their children. This is the magic that makes nesting work.

The mesh-level multiplication will happen in the shader, so we'll pass this matrix in as well.

//gpu-engine.js
setMainSceneBindGroup(passEncoder, bindGroupLayouts, camera, mesh) {
    const scene = {
        viewMatrix: camera.getViewMatrix(),
        projectionMatrix: camera.getProjectionMatrix(),
        modelMatrix: getTranspose(mesh.modelMatrix, [4, 4]), //change to col major
        worldMatrix: mesh.worldMatrix,
        normalMatrix: getTranspose(
            getInverse(
                trimMatrix(
                    multiplyMatrix(mesh.worldMatrix, [4, 4], mesh.modelMatrix, [4, 4]),
                    [4, 4],
                    [3, 3]
                ),
                [3, 3]
            ),
            [3, 3]),
        cameraPosition: camera.getPosition()
    };
    const sceneData = packStruct(scene, [
        ["viewMatrix", "mat4x4f32"],
        ["projectionMatrix", "mat4x4f32"],
        ["modelMatrix", "mat4x4f32"],
        ["worldMatrix", "mat4x4f32"],
        ["normalMatrix", "mat3x3f32"],
        ["cameraPosition", "vec3f32"]
    ]);
    const sceneBuffer = this.#device.createBuffer({
        size: sceneData.byteLength,
        usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
        label: "main-scene-buffer"
    });
    this.#device.queue.writeBuffer(sceneBuffer, 0, sceneData);
    const sceneBindGroup = this.#device.createBindGroup({
        label: "main-scene-bind-group",
        layout: bindGroupLayouts.get("scene"),
        entries: [
            {
                binding: 0,
                resource: {
                    buffer: sceneBuffer,
                    offset: 0,
                    size: sceneData.byteLength
                }
            }
        ]
    });
    passEncoder.setBindGroup(0, sceneBindGroup);
}

The main change here is adding the world matrix to the struct we pass in and multiplying the world matrix with the model matrix before doing all the transforms to get the correct normal matrix.

On the shader side it's just one more multiplication:

@vertex
fn vertex_main(@location(0) position: vec3<f32>, @location(1) uv: vec2<f32>, @location(2) normal: vec3<f32>) -> VertexOut
{
    var output : VertexOut;
    output.frag_position =  scene.projection_matrix * scene.view_matrix * scene.world_matrix * scene.model_matrix * vec4<f32>(position, 1.0);
    output.world_position = scene.world_matrix * scene.model_matrix * vec4<f32>(position, 1.0);
    output.uv = uv;
    output.normal = scene.normal_matrix * normal;

    return output;
}

Now, we could just bake this transform data right into the model matrix itself before passing it in. I don't have a great answer for why I'm not doing that other than to spell it out more but I expect at some point this will return to a single matrix at least as far as the shader is concerned.

As a test I nested two groups doing different rotations:

<wc-geo>
    <geo-camera key="main" position="0, 0, -2"></geo-camera>
    <geo-texture key="marble" src="./img/marble-white/marble-white-base.jpg"></geo-texture>
    <geo-texture key="marble-roughness" src="./img/marble-white/marble-white-roughness.jpg"></geo-texture>
    <geo-texture key="red-fabric" src="./img/red-fabric/red-fabric-base.jpg"></geo-texture>
    <geo-texture key="red-fabric-roughness" src="./img/red-fabric/red-fabric-roughness.jpg"></geo-texture>
    <geo-texture key="gold" color="0, 0, 0, 1"></geo-texture>
    <geo-material key="marble" roughness-map="marble-roughness" albedo-map="marble"></geo-material>
    <geo-material key="red-fabric" roughness-map="red-fabric-roughness" albedo-map="red-fabric"></geo-material>
    <geo-material key="gold" roughness="0.2" metalness="1" base-reflectance="1.059, 0.773, 0.307" albedo-map="gold"></geo-material>
    <geo-group key="teapot-rug-0" pipeline="main" rotate="0, 1.5707963267948966, 0">
    <geo-group key="teapot-rug-1" rotate="1.5707963267948966, 0, 0">
        <geo-mesh 
            key="teapot" 
            normalize
            bake-transforms 
            reverse-winding 
            src="./objs/teapot.obj" 
            resize-uvs="2" 
            material="gold" 
            attributes="positions, normals, uvs">
        </geo-mesh>
        <geo-surface-grid 
            key="rug" 
            row-count="2" 
            col-count="2" 
            translate="0, -0.25, 0" 
            bake-transforms 
            material="red-fabric" 
            attributes="positions, normals, uvs">
        </geo-surface-grid>
    </geo-group>
    </geo-group>

    <geo-light key="light1" type="directional" color="1, 1, 1, 1" direction="-1, 0, 0" casts-shadow></geo-light>
</wc-geo>

Which yields this image:

Debugging

One way I tried to the test the result was to make a cube shape and then shine light on it from one side. This would test that the normals are correct after rotation. But I had a problem. When shining light from one side I would get an image like this:

With 2 sides lit. This was puzzling and took a while to figure out since if I just drew the colors of the normal vectors it looked correct. The problem here is that there is precision errors. Even very slight near-zero agitation of the normals to point toward the light will cause the material to look very lit. To fix this issue I had to make a function to round low values toward 0.

fn round_small_mag_3(v: vec3<f32>) -> vec3<f32> {
  return select(v, vec3<f32>(0.0), abs(v) < vec3<f32>(1e-6));
}

This might not be necessary for "real" scenes since it's unlikely the planes are exactly orthogonal to light in that way and that the viewer would notice in a complex scene. However for something like a technical rendering like this it definitely looks off. So we can just apply this function to the normal value to clean up those almost-zero values. vec3<f32>(1e-6) is the threshold per element, you can play around with the value to get it to work depending on how bad the precision issue is.

Conclusion

It's nice working on a project nobody else uses because you can always go in an fix all your stupid naming conventions and cleanup bad abstractions. While functionally this didn't add very much nor was super complicated it really helps to have a clean codebase.

Of course the code here is a little iffy because it was added a bit ad-hoc to see what conventions stick and will need some cleanup.

Code

https://github.com/ndesmic/geo/pull/4/files

WebGPU Engine from Scratch Part 9: Shadow Maps

ndesmic — Mon, 15 Sep 2025 19:00:00 +0000

This was a topic I wanted to get to for a while. There's 2 common ways (but other less common ways) to do shadows for 3D graphics. Raytracing is the most accurate and simple but very expensive. When we can't afford that, the stock standard approach is shadow mapping because it's inexpensive and works well enough. This is what most games you've seen use and you can usually tell by the characteristic artifacts where shadows look "pixelated" and that's what I want to implement. There are tons of tweaks to shadow maps to improve results but for now I'm just looking at the simpler version.

How it works

The intuition here is that we treat each light like a camera and then we take a picture of the scene with it. This picture is just a depth map. Then when we render the scene for each point we compare to see if the light camera saw it, that is, the world space position of the fragment as calculated from the depth map is the same distance from the light as the world space position as calculated by the actual camera. If it is then the fragment is not in shadow. If the distance in the depth map distance is shorter then that means it was occluded by something else between the position and the camera.

Directional lighting

We have two light types, directional and point. Directional is the easier of the two to deal with so we'll start with that. Directional lighting when modeled as a camera is doesn't have any perspective because it's infinitely wide (we do have to decide a finite cut-off for the depth map dimensions though). So it's really just an orthographic camera pointed the direction of the light. So let's do that.

Housekeeping

First some housekeeping. We'll need to build a new pipeline but because these take up a lot of space I want to start moving them out of the main engine class. Now each engine will have it's own folder so that it can have sub-resources that are specific to it. Small path changes.

Building the shadow map

We need to build the new pipeline that creates the shadow map next. This is basically just rendering the geometry with no shader. Since we turned on depth mapping we'll automatically get that output even though we didn't rendering anything in particular. We'll also need the appropriate bindings for the vertex transforms. Most of this can be borrowed from the camera code but we do need to decide how to set the bounds for the projection matrix.

For the direction light in order to build the view matrix we need to pick a position. Since it's orthographic it shouldn't really matter, just as long as it's far enough out to encompass the scene. I've kept it slightly generic using constants even though we can simplify if we are always looking at the origin. 0.5 is a good enough a distance to show the teapot but it needs to be manually calibrated per scene for now. Therefore we also need to pick the bounds of the view frustum. I decided to go with -2 to 2 with a depth of 1 (this will be updated later because it turns out it's not a great choice). I created a new entity called ShadowMappedLight with this logic. The reason is because it makes sense to group this with the light but it's implementation specific, for example ray-tracing does not need these concepts so I subclassed it.

//shadow-mapped-light.js
import { Light } from "./light.js";
import { getLookAtMatrix, getOrthoMatrix, scaleVector, subtractVector } from "../utilities/vector.js";

const distance = 0.5;
const center = [0, 0, 0];
const frustumScale = 2;

export class ShadowMappedLight extends Light {
    getViewMatrix(light) {
        const lightPosition = scaleVector(subtractVector(center, this.direction), distance);
        return getLookAtMatrix(lightPosition, center);
    }

    getProjectionMatrix(aspectRatio) {
        const right = aspectRatio * frustumScale;
        return getOrthoMatrix(-right, right, -frustumScale, frustumScale, 0.1, distance * 2);
    }
}

One thing we need to be careful about that I didn't notice before is the when the camera, or in this case, the light, is looking down the Y axis this will cause the view matrix to get filled with NaNs because they will cross multiply a zero vector. In this case we need to choose a different up. I've modified the getLookAtMatrix to handle this case:

//vector.js
export function getLookAtMatrix(position, target, up = UP) {
    const forward = normalizeVector(subtractVector(target, position));

    if(Math.abs(dotVector(forward, up)) > 0.999){
        up = Math.abs(forward[1]) < 0.999 ? UP : FORWARD;
    }

    const right = normalizeVector(crossVector(up, forward));
    const newUp = crossVector(forward, right);


    return new Float32Array([
        right[0], newUp[0], forward[0], 0,
        right[1], newUp[1], forward[1], 0,
        right[2], newUp[2], forward[2], 0,
        -dotVector(position, right), -dotVector(position, newUp), -dotVector(position, forward), 1
    ]);
}

If the vector is too similar to UP then we choose FORWARD instead. Note that if the direction simply contained a Y component but not a full Y component like [0,-1,1] we should still choose UP even though the dot product is 1/-1. This should make things a bit more robust.

We also need to get the aspect ratio of the shadow map to generate the view frustum bounds so that we're not distorting the image. I'm not sure if it's strictly necessary but it puts me at ease.

The rest is near identical to the main scene bindgroup while removing stuff that we don't need.

//gpu-engine.js
//create the matrices from above...
const scene = {
    viewMatrix,
    projectionMatrix,
    modelMatrix: getTranspose(mesh.getModelMatrix(), [4, 4]), //change to col major?
    normalMatrix: getTranspose(
        getInverse(
            trimMatrix(
                mesh.getModelMatrix(),
                new Float32Array([1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]),
                [4, 4],
                [3, 3]
            ),
            [3, 3]
        ),
        [3, 3])
};
const sceneData = packStruct(scene, [
    ["viewMatrix", "mat4x4f32"],
    ["projectionMatrix", "mat4x4f32"],
    ["modelMatrix", "mat4x4f32"],
    ["normalMatrix", "mat3x3f32"]
]);
const sceneBuffer = this.#device.createBuffer({
    size: sceneData.byteLength,
    usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
    label: "shadow-map-scene-buffer"
});
this.#device.queue.writeBuffer(sceneBuffer, 0, sceneData);
const sceneBindGroup = this.#device.createBindGroup({
    label: "shadow-map-scene-bind-group",
    layout: bindGroupLayouts.get("scene"),
    entries: [
        {
            binding: 0,
            resource: {
                buffer: sceneBuffer,
                offset: 0,
                size: sceneData.byteLength
            }
        }
    ]
});
passEncoder.setBindGroup(0, sceneBindGroup);

When debugging this I highly recommend rendering an actual scene because the depth buffer is a little more finicky to look at even with our debug tool. Speaking of which, I added an option shouldScaleGamma

//debug-utils.js
/**
 * 
 * @param {GPUDevice} device 
 * @param {*} context 
 * @param {{ shouldGammaScale?: boolean }} options 
 * @returns 
 */
export function setupExtractDepthBuffer(device, context, options = {}) {

//debug-utils.js
const shaderModule = device.createShaderModule({
    label: "depth-buffer-export-shader",
    code: `
        struct VertexOut {
            @builtin(position) frag_position : vec4<f32>,
            @location(0) clip_position: vec4<f32>,
            @location(2) uv: vec2<f32>
        };
        @group(0) @binding(0) var depthSampler: sampler;
        @group(0) @binding(1) var depthTex: texture_depth_2d;
        @vertex
        fn vertex_main(@location(0) position: vec2<f32>) -> VertexOut
        {
            var output : VertexOut;
            output.frag_position =  vec4(position, 1.0, 1.0);
            output.clip_position = vec4(position, 1.0, 1.0);
            output.uv = position.xy * 0.5 + vec2<f32>(0.5, 0.5);
            return output;
        }
        @fragment
        fn fragment_main(fragData: VertexOut) -> @location(0) vec4<f32> {
            let depth = textureSample(depthTex, depthSampler, fragData.uv);

            ${options.shouldGammaScale
                ? `
                let gamma_depth = pow(depth, 10.0);
                return vec4<f32>(gamma_depth, gamma_depth, gamma_depth, 1.0);
                `
                : `
                return vec4<f32>(depth, depth, depth, 1.0);
                `
            }
        }
    `
});

This is because this depth buffer doesn't really need it because we're more appropriately using the depth range (I could probably have made this a parameter at the render call instead of setup but then I'd have to add bindings and I was lazy). Any way here's the depth buffer.

Hmmmm...it is correct for being taken on the side ([0,0,-1]) but it's upside down (note if you want to get in close shorten the distance constant). This is because the depth texture has Y = 0 at the top and moves down, we didn't notice this last time because it was symmetric. It requires a small fix to the shader to flip the Y-axis:

output.uv = vec2(position.x * 0.5 + 0.5, 1.0 - (position.y * 0.5 + 0.5));

At first I thought this was just a visualization bug but, nope, even when we using the shadow map we need to flip the Y (V value). I spent a lot of time debugging that one.

Binding the shadow maps

Now we have the map and we need to use it. One annoying limitation is that depth textures cannot have layers as this would probably be the most intuitive way to have multiple lights casting shadows. Instead we have to create a finite number of bindings. If we have too many then it won't run, if we have too few then we need to fill them with dummy depth textures.

//gpu-engine.js - initializeLights
this.#shadowMaps.set("dummy", this.#device.createTexture({
    label: "dummy-depth-texture",
    size: { width: 1, height: 1, depthOrArrayLayers: 1 },
    format: "depth32float",
    usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
}));

We also need a sampler. This one is for depth so it needs to do comparisons:

//gpu-engine.js - initializeSampler
this.#samplers.set("shadow-map-default", this.#device.createSampler({
    label: "shadow-map-default-sampler",
    compare: "less",
    magFilter: "nearest",
    minFilter: "nearest"
}));

Then we can setup the bindgroup. We need access to the same transforms we used to make the shadow map so that we can take the world space position and recover the coordinates in light view-projection space, which can then be used to get the UV space in the texture. You could also combine the transform matrices so you don't have to pass both because we don't use them individually, but for now I split it out to be explicit.

//gpu-engine.js
setMainShadowBindGroup(passEncoder, bindGroupLayouts, lights, shadowMaps){
    const lightsArray = lights.entries().toArray();
    if(lightsArray.length > 4){
        throw new Exception("This engine can only handle shadows from 4 lights max");
    }
    const lightTransforms = [];
    const shadowMapsToBind = new Array(4).fill(null);
    let i = 0;
    for(const [key, light] of lightsArray){
        const shadowMap = shadowMaps.get(key);
        shadowMapsToBind[i++] = shadowMap;
        if(shadowMap){
            const shadowMapAspectRatio = shadowMap.width / shadowMap.height;
            lightTransforms.push({
                projectionMatrix: light.getProjectionMatrix(shadowMapAspectRatio),
                viewMatrix: light.getViewMatrix()
            })
        } else {
            lightTransforms.push({
                projectionMatrix: getEmptyMatrix([4, 4]),
                viewMatrix: getEmptyMatrix([4, 4])
            });
        }
    }
    const lightTransformData = packArray(lightTransforms, [
        ["projectionMatrix", "mat4x4f32"],
        ["viewMatrix", "mat4x4f32"]
    ]);
    const lightTransformBuffer = this.#device.createBuffer({
        size: lightTransformData.byteLength,
        usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST | GPUBufferUsage.STORAGE,
        label: "main-shadow-light-transform-buffer"
    });
    this.#device.queue.writeBuffer(lightTransformBuffer, 0, lightTransformData);
    const dummyView = shadowMaps.get("dummy").createView();
    const shadowBindGroup = this.#device.createBindGroup({
        label: "main-shadow-bind-group",
        layout: bindGroupLayouts.get("shadows"),
        entries: [
            {
                binding: 0,
                resource: this.#samplers.get("shadow-map-default")
            },
            {
                binding: 1,
                resource: {
                    buffer: lightTransformBuffer,
                    offset: 0,
                    size: lightTransformData.byteLength
                }
            },
            ...(shadowMapsToBind.map((value, i) => {
                return {
                    binding: i + 2,
                    resource: value ? value.createView() : dummyView
                };
            }))
        ]
    });
    passEncoder.setBindGroup(3, shadowBindGroup);
}

There's a lot of boilerplate for filling dummy values if the number of lights or shadow maps isn't aligned. The key thing is we need to create the transforms and pack them, then attach the shadow maps or dummy views until we reach the max (4). This is hard-coded manual since there's no way to bind array light structures for depth maps. Make sure to call this from setMainBindGroups with the right parameters.

//gpu-engine.js - setMainBindGroups
this.setMainShadowBindGroup(passEncoder, bindGroupLayouts, lights, shadowMaps)

The shader

We'll start with 4 possible lights as our scenes shouldn't get too complicated yet. In the shader we'll setup this bindgroup.

const shadow_max = 4;
struct LightTransform {
    projection_matrix: mat4x4<f32>,
    view_matrix: mat4x4<f32>,
}

@group(3) @binding(0) var shadow_sampler: sampler_comparison;
@group(3) @binding(1) var<storage, read> light_transforms: array<LightTransform>;
@group(3) @binding(1) var shadow_map_0: texture_depth_2d;
@group(3) @binding(2) var shadow_map_1: texture_depth_2d;
@group(3) @binding(3) var shadow_map_2: texture_depth_2d;
@group(3) @binding(4) var shadow_map_3: texture_depth_2d;

Notice the sample_comparison. This lets it know it's a comparison type and this is also necessary for "auto" bind group layouts to work correctly (before in debug utils we made manual bind group layouts). Reminder if you are debugging to check to make sure you actually use them or the bindings get erased!

Shadow comparison

We need to have access to the light-space transform to convert a world space coordinate into a light space one (and then turn that into a shadow map UV). Once we have that, in the fragment shader we can work backward to get the the UV to sample.

fn get_shadow(world_position: vec4<f32>) -> f32 {
    var shadow = 1.0;
    for(var i = 0; i < shadow_max; i++){
        let shadow_space = light_transforms[i].projection_matrix * light_transforms[i].view_matrix * world_position;
        let shadow_projection = shadow_space.xyz / shadow_space.w;
        let shadow_uv = shadow_projection.xy * 0.5 + vec2<f32>(0.5);
        let shadow_depth = shadow_projection.z;
        switch(i){
            case 0: { shadow *= textureSampleCompare(shadow_map_1, shadow_sampler, shadow_uv, shadow_depth - 0.05); }
            case 1: { shadow *= textureSampleCompare(shadow_map_1, shadow_sampler, shadow_uv, shadow_depth - 0.05); }
            case 2: { shadow *= textureSampleCompare(shadow_map_2, shadow_sampler, shadow_uv, shadow_depth - 0.05); }
            case 3: { shadow *= textureSampleCompare(shadow_map_3, shadow_sampler, shadow_uv, shadow_depth - 0.05); }
            default: { 
            } //shouldn't happen...
        }
    }
    return shadow;
}

//pbr.wgsl
@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32> {
    //...other stuff
    var shadow = get_shadow(frag_data.world_position);
    total_color *= shadow; //add this line

    var tone_mapped_color = total_color / (total_color + vec3(1.0));
    return vec4(pow(total_color, vec3(1.0/2.2)), 1.0);
}

We first multiple by the light projection-view matrix which gives us the light space coordinates and then divide by w to get the projection to put it into the equivalent of "screen" space for the light (-1 to 1). We can then transform that into UVs because we know that the edges will lineup with the shadow map if we convert coordinates from (-1, 1) to (0, 1). The z value is the distance in world space from the light to the fragment.

If we know how far away the light is supposed to be, we can compare that with the value in the shadow map. If the shadow map shows less then it was occluded and it means that there is a shadow on that fragment. The value we get from textureSampleCompare isn't boolean though, it's 0 or 1, where 1 is not occluded and 0 is occluded (remember the depth compare in the sampler is less, so if the value in the provided is less than the provided value it returns 1.0). Note that it's strictly less, because equal means that it's in shadow (and minus precision issues this is what we expect, the shadow map should be equal or less, in practicality though greater is possible and that's not in shadow). Although, if we changed the shadow map sampler to have linear scaling the value can actually vary between 0 and 1 depending on how many samples passed, this will give us softer shadows. The output of this is simply multiplied by light to get the final result (this is actually not correct, but it's a starting point, it actually needs to be per light).

If everything was correct (always double check your bindings are actually uploading the resources you think they are!) we'll get something like this (light direction is [0,-1,0]):

This is because the places the shadow map samples are slightly different than the ones in the scene so on steep faces when might get parts that are higher or lower that correspond to the same shadow map text. We can add a bias value to play around with which basically adds a tolerance so that if the distance is a little longer to still count it as shadow:

shadow *= textureSampleCompare(shadow_map_1, shadow_sampler, shadow_uv, shadow_depth - 0.05);

Cranking it to 0.05 (which is really high) reduces but doesn't fully alleviate the artifacts on the most oblique of angles.

If we use linear filtering it somewhat improves but is still a bit weird.

We can also up the resolution from 1280x720 to 4096x4096:

This make the shadow higher resolution and reduces aliasing. As for the holes, that's because the there are holes in the model itself. Playing around with it more (bias values, light position, shadow map frustum etc) not everything looks quite right. I get a sharp cutoff at a certain point.

The shadow artifacts come from the bias but the direction light (-1,-1,0) show be shadowing the back part of the rug. As explained in the section about debug visualizations I forgot to flip the Y coordinate on the shadow map. This took a while to figure out. I actually had to project the depth texture onto the scene in order to find this. This isn't easy because depth textures don't like to be sampled normally unless you do manual bind layouts. In the end I was forced to switch from autobinding because there didn't seem to be a way to indicate a sampler is non-filtering. I guess it gives a little more control in the end but it's a lot of boilerplate to carry around. Here's the fixed version:

-let shadow_uv = shadow_projection.xy * 0.5 + vec2<f32>(0.5);
+let shadow_uv = vec2(shadow_projection.x * 0.5 + 0.5, 1.0 - (shadow_projection.y * 0.5 + 0.5));

Multiple lights

So the code above has a lot of boilerplate to accommodate multiple lights but there's only one light. In fact, the code itself is wrong because we need an association of the light to the shadow map because they should only occlude themselves, not all lights, so we can't just multiply the final output. This created some more refactoring and code shuffling. The lights should actually carry their transforms so it's easier to iterate over them and the shadow maps are really tied to the light itself. They should also have a property that indicates whether or not they cast shadows at all since in a complex scene we don't want to pay for too many shadow casts so we can just ignore non-important lights.

//shadow-mapped-light.js
import { Light } from "./light.js";
import { getLookAtMatrix, getOrthoMatrix, scaleVector, subtractVector } from "../utilities/vector.js";

const distance = 0.75;
const center = [0, 0, 0];
const frustumScale = 2;

export class ShadowMappedLight extends Light {
    #hasShadow = false;

    constructor(options){
        super(options);
        this.#hasShadow = options.hasShadow;
    }
    getViewMatrix() {
        const lightPosition = scaleVector(subtractVector(center, this.direction), distance);
        return getLookAtMatrix(lightPosition, center);
    }

    getProjectionMatrix(aspectRatio) {
        const right = aspectRatio * frustumScale;
        return getOrthoMatrix(-right, right, -frustumScale, frustumScale, 0.1, Math.min(distance * 2, 2.0));
    }
    set hasShadow(value){
        this.#hasShadow = value;
    }
    get hasShadow(){
        return this.#hasShadow;
    }
}

Once we have this, we can dump the the whole shadow bind group entirely. This is actually better since I found that the max bindgroups on my computer is only 4 so it's better to not use more than we need. Everything gets put in the the light bind group instead.

setMainLightBindGroup(passEncoder, bindGroupLayouts, lights, shadowMaps){
    let shadowMapIndex = 0;
    const shadowMappedLights = lights
        .entries()
        .map(([key, value]) => {
            const shadowMap = shadowMaps.get(key);
            const shadowMapAspectRatio = shadowMap.width / shadowMap.height;
            return {
                typeInt: value.typeInt,
                position: value.position,
                direction: value.direction,
                color: value.color,
                shadowMap,
                projectionMatrix: shadowMap ? value.getProjectionMatrix(shadowMapAspectRatio) : getEmptyMatrix([4,4]),
                viewMatrix: shadowMap ? value.getViewMatrix() : getEmptyMatrix([4,4]),
                hasShadow: value.hasShadow ? 1 : 0,
                shadowMapIndex: (value.hasShadow && shadowMap) ? shadowMapIndex++ : -1
            };
        }).toArray();
    const shadowMapsToBind = shadowMappedLights
        .filter(lightData => lightData.shadowMapIndex > -1)
        .map(lightData => lightData.shadowMap);
    const lightData = packArray(shadowMappedLights,
    [ 
        ["typeInt", "u32"],
        ["position", "vec3f32"],
        ["direction", "vec3f32"],
        ["color", "vec4f32"],
        ["projectionMatrix", "mat4x4f32"],
        ["viewMatrix", "mat4x4f32"],
        ["hasShadow", "u32"],
        ["shadowMapIndex", "i32"]
    ]
    , 64);
    const lightBuffer = this.#device.createBuffer({
        size: lightData.byteLength,
        usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST | GPUBufferUsage.STORAGE,
        label: "main-light-buffer"
    });
    this.#device.queue.writeBuffer(lightBuffer, 0, lightData);
    const lightCountBuffer = this.#device.createBuffer({
        size: 4,
        usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
        label: "main-light-count-buffer"
    });
    this.#device.queue.writeBuffer(lightCountBuffer, 0, new Int32Array([lights.size]));
    const placeholderView = shadowMaps.get("placeholder").createView({ label: "placeholder-view" });
    const lightBindGroup = this.#device.createBindGroup({
        label: "main-light-bind-group",
        layout: bindGroupLayouts.get("lights"),
        entries: [
            {
                binding: 0,
                resource: {
                    buffer: lightBuffer,
                    offset: 0,
                    size: lightData.byteLength
                }
            },
            {
                binding: 1,
                resource: {
                    buffer: lightCountBuffer,
                    offset: 0,
                    size: 4
                }
            },
            {
                binding: 2,
                resource: this.#samplers.get("shadow-map-default")
            },
            ...(getRange({ end: 3 }).map((index) => {
                const shadowMap = shadowMapsToBind[index];
                return {
                    binding: index + 3,
                    resource: shadowMap ? shadowMap.createView({ label: `shadow-view-${index}`}) : placeholderView
                };
            })),
            {
                binding: 7,
                resource: this.#samplers.get("shadow-map-debug")
            }
        ]
    });
    passEncoder.setBindGroup(2, lightBindGroup);
}

We can do this with just two lists. First we iterate over the lights and pack their new properties in. Remember that bools need to be converted to ints. We also need to keep track of which shadow map slot it being used since some of the lights might not cast shadows. The second list is used to figure out which slots are actually filled and by which shadow map. Like last time, we bind 4 slots and any that don't have shadow maps will get a dummy shadow map (I actually changed these variable names and keys to "placeholder" because it was a better word to describe them).

Now on to the shader.

@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32>
{   
    var surface_albedo = textureSample(albedo_map, albedo_sampler, frag_data.uv).rgb;
    var roughness_from_map = textureSample(roughness_map, roughness_sampler, frag_data.uv).x;
    var roughness = max(mix(material.roughness, roughness_from_map, f32(material.use_specular_map)), 0.0001);
    var f0 = mix(vec3(0.04, 0.04, 0.04), material.base_reflectance, material.metalness);
    var total_color = vec3(0.0);
    var normal = normalize(frag_data.normal);
    var shadow = get_shadow(frag_data.world_position);

    var i = 0;
    for(var i: u32 = 0; i < light_count.count; i++){
        var light = lights[i];
        var light_distance = length(light.position - frag_data.world_position.xyz);
        var to_light = vec3(0.0);

        switch light.light_type {
            case 0: {
                to_light = normalize(light.position - frag_data.world_position.xyz);
            }
            case 1: {
                to_light = normalize(-light.direction);
            }
            default: {}
        }

        var attenuation = 1.0 / pow(light_distance, 2.0);
        var radiance = light.color.rgb * attenuation;
        var lit_color = get_bdrf(
            surface_albedo, 
            f0, 
            roughness, 
            material.metalness,
            normal, 
            radiance, 
            to_light,
            scene.camera_position, 
            frag_data.world_position.xyz
        );

        let shadowed_color = lit_color * shadow;
        total_color += shadowed_color;
    }

    var tone_mapped_color = total_color / (total_color + vec3(1.0));
    return vec4(pow(total_color, vec3(1.0/2.2)), 1.0);
}

The main changes here is that the shadow multiplication is moved inside the look to apply the each light individually and then that result is added to the light. get_shadow has minor changes to get the projections from the light object and to use the has_shadow property to skip calculating the shadow if it's not needed for that light.

fn get_shadow(world_position: vec4<f32>) -> f32 {
    var shadow = 1.0;
    var i = 0;
    for(var i: u32 = 0; i < light_count.count; i++){
        if lights[i].has_shadow == 0 {
            continue;
        }
        let shadow_space = lights[i].projection_matrix * lights[i].view_matrix * world_position;
        let shadow_projection = shadow_space.xyz / shadow_space.w;
        let shadow_uv = vec2(shadow_projection.x * 0.5 + 0.5, 1.0 - (shadow_projection.y * 0.5 + 0.5));
        let shadow_depth = shadow_projection.z;
        switch(lights[i].shadow_map_index){
            case 0: { shadow *= textureSampleCompare(shadow_map_0, shadow_sampler, shadow_uv, shadow_depth - 0.01); }
            case 1: { shadow *= textureSampleCompare(shadow_map_1, shadow_sampler, shadow_uv, shadow_depth - 0.01); }
            case 2: { shadow *= textureSampleCompare(shadow_map_2, shadow_sampler, shadow_uv, shadow_depth - 0.01); }
            case 3: { shadow *= textureSampleCompare(shadow_map_3, shadow_sampler, shadow_uv, shadow_depth - 0.01); }
            default: { } //shouldn't happen...
        }
    }
    return shadow;
}

This took a little debugging to make sure all the values were actually being bound and packed and that the shadow maps were rendering correctly before being bound. We can test that in fact multiple directional light cast shadows:

We can play around with turning off shadows, moving the lights and changing colors to convince ourselves it works.

Cleaning up the shadows a bit

I'm still not too happy with the look of the shadows and the banding (also called shadow "acne") so I wanted to try something to help reduce that at oblique angles. I tried scaling the bais factor based on the dot product of the light direction and the normal (eg more perpendicular would get more bias).

let diffuse_factor = max(dot(normal, to_light), 0.0);
let bias = mix(shadow_bias, 0.0, diffuse_factor);

At first I got this wrong because I mixed in the wrong order (similar lines tend toward 1.0 so that should tend to 0.0). The result now looks better, but I also found that just setting the bias to constant 0.05 with the changed shadow code just works now. Whatever.

We can actually optimize a little by letting the GPU handle this. We can do this with by setting some values on the depth stencil render attachment when making the shadow map. To add constant bias, we can set depthBias. This is similar to subtracting bias in the main pass but instead we add bias to the depth map when it gets rendered. To do that bias where we consider the slop of the fragment we use depthBiasSlopScale instead. These are where the depth buffer is defined in the pipeline definition (not the render pass).

//pipelines.js - getShadowMapPipeline
depthStencil: {
    depthWriteEnabled: true,
    depthCompare: "less",
    depthBias: 1000000,
    depthBiasSlopeScale: 2.0,
    format: "depth32float"
}

I removed the bias term and used the above instead and it got the same result. Tuning the depthBias value is kinda stupid because the buffer is non-linear so best I could tell you guess and check so I don't know how sustainable this is or how to calculate it correctly. But it works for now.

Where to go from here

At this point we have working shadows for directional lights which are the easiest but also the ones that are most useful (eg sunlight). We don't have spotlights at this point so we can skip that for now (it's the same but with perspective transform). We do have point lights which are more complicated because we basically have to render 6 views for a cubemap but after that it's the same. Curious what limits that would butt up against. There's also lots of techniques to optimize the size to visual quality (cascade shadow maps, perspective shadow maps etc).

Code

https://github.com/ndesmic/geo/compare/v0.6...v0.7

References

https://www.youtube.com/watch?v=Jhopq2lkzMQ Interactive Graphics 16 - Shadow Mapping
https://www.youtube.com/watch?v=x2FHHU50ktQ Shadow Mapping Unleashed! // OpenGL Tutorial #38
https://learn.microsoft.com/en-us/windows/win32/dxtecharts/common-techniques-to-improve-shadow-depth-maps Common Techniques to Improve Shadow Depth Maps

WebGPU Engine from Scratch Part 8: Physically Based Lighting (PBR)

ndesmic — Mon, 08 Sep 2025 19:00:00 +0000

I wasn't expecting to do this so soon but the whole roughness texture stuff really got to me so I decided to see what it would take to improve the lighting to PBR (Physically based rendering). Luckily, not very hard. While we do need to look at equations it's mostly a matter of plugging them in, no aligned buffer packing or anything crazy.

Most of the materials I looked through started with the rendering equation. To be honest, I don't really care for the rendering equation because it makes everything look way more complicated than it needs to be. I understand it's fundamental value though but I'm not going to bother printing it because we aren't going to look at. Suffice to say, all that matters is what it means: the color of a pixel is the result of all incoming light and the amount of light reflected back to the viewer which involves the dot product of the light direction and the normal.

This is mostly what we've been doing so far, but PBR is a bit more strict in that it respects energy conservation. That is, all light has to be accounted for either absorbed, reflected or transmitted. It is impossible for there to be more light given off than there was coming in. For the classic Blinn-Phong this isn't true because we had factors we could change for the specular, tuning them artistically. PBR doesn't have that, it gives a set of physical properties (roughness and reflectivity) we use calculate both the diffuse and specular light.

The other part is that it's based on micro-facets. This means that instead of a flat plane the surface is modeled as having little bumps that reflect light in different directions. What's nice is that this is not really different than calculating the light reflected by a flat plane but since each microfacet is smaller than a pixel we just use statistical scaling. If 30% of the micro facets of a pixel face the viewer then you essentially multiply the value by 0.3.

BDRF

The BDRF is the Bidirectional Radiance Function and it describes how light gets reflected off the surface. There are multiple BDRFs of which Blinn-Phong is one. In fact, in practice it's split up the same way into diffuse and specular parts.

Here s and d are scaling values and R_s and R_d are functions that describe the diffuse and specular parts respectively. The key to energy conservation here is the constraint that s + d = 1. They cannot add up to more that the total light. Note: the Cook-Torrence model also contains a Fresnel term. You only need to apply Fresnel once, so the s term will be removed in this case.

Diffuse

Normal Lambertian diffuse seems to be the most common and it's exactly what we've been doing.

Albedo is the surface color
L is the vector towards the light
N is the surface normal
LightColor is the color of the light

That is the diffuse light is just the incoming light times the dot product of the surface normal and the vector pointing toward the light. (note that the dot product will get factored out later).

Specular

This is where it gets a little more interesting. We'll be using the Cook-Torrence model because that seems to be one of the most common but there are others.

V is the direction toward the viewer
N is the normal of the surface
L is the direction to the light
D, G and F are functions

D (the normal distribution function)

This function describes how the normals of microfacets are oriented on the surface. What we want to know is what ratio of them point toward the viewer. For this there are several but a common one I saw was GGX (Trowbridge-Reitz).

alpha is roughness (yay! we found it)
H is the half vector (same as Blinn Phong)
N is still the surface normal

I honestly have no idea how this one works under the hood but as long as it works I'm not going to question it.

G (the geometry function)

This function describes how the light bounces between the surface bump, specifically to create cases where there is self-shadowing. So if a ray comes in and bounces between two peaks it might not reflect back the viewer as might be expected if we didn't consider the bouncing cases. The most common one here seemed to be Shlick-GGX.

k is alpha/2 in some renderers, where alpha is roughness. Might also see k = (roughness + 1)^2 / 8 .
N is the surface normal
X is either the vector to the light or to the viewer.

We actually have to apply this twice once where X = toLight and the other where X = toCamera.

L is the vector toward the light
V is the vector toward the viewer

F (Fresnel)

This is the amount of light that is reflected versus refracted. We use the Schlick approximation for it.

F_0 is the base reflectance (ie how reflective it is when looking straight down the normal vector). It is an RGB vector.
V is the vector toward the viewer
H is the half vector between the view direction and the light direction

The F_0 value is something you can look up for common materials. You can also find the Index of Refraction (IOR) for the material and use the equation:

r is the index of refraction

I also saw that it's common to just use a hardcoded value of 0.04 for all dielectrics (non-metals). My guess is because the range is so small and 2 orders of magnitude less than those of metals.

The last part of the BDRF is figuring out s and d from the original BDRF equation. By definition, d = 1 - s. And s will be the Fresnel value shown above. With that, we have all equations and all variables filled so we can implement it in the shader.

Implementation

struct VertexOut {
    @builtin(position) frag_position : vec4<f32>,
    @location(0) world_position: vec4<f32>,
    @location(1) uv : vec2<f32>,
    @location(2) normal : vec3<f32>
};
struct Scene {
    view_matrix: mat4x4<f32>,
    projection_matrix: mat4x4<f32>,
    model_matrix: mat4x4<f32>,
    normal_matrix: mat3x3<f32>,
    camera_position: vec3<f32>
}
struct Light {
    light_type: u32,
    position: vec3<f32>,
    direction: vec3<f32>,
    color: vec4<f32>
}
struct LightCount {
    count: u32
}
struct SpecularMaterial {
    use_specular_map: u32,
    roughness: f32
}

@group(0) @binding(0) var<uniform> scene : Scene;

@group(1) @binding(0) var albedo_sampler: sampler;
@group(1) @binding(1) var albedo_map: texture_2d<f32>;
@group(1) @binding(2) var<uniform> specular: SpecularMaterial;
@group(1) @binding(3) var roughness_sampler: sampler;
@group(1) @binding(4) var roughness_map: texture_2d<f32>;

@group(2) @binding(0) var<storage, read> lights: array<Light>;
@group(2) @binding(1) var<uniform> light_count: LightCount;

//schlick
fn get_fresnel(f0: vec3<f32>, to_view: vec3<f32>, half_vector: vec3<f32>) -> vec3<f32> {
   return f0 + (vec3(1.0) - f0) * pow(1 - max(dot(to_view, half_vector), 0), 5.0);
}

//lambert
fn get_diffuse(surface_albedo: vec3<f32>, light_color: vec3<f32>, normal: vec3<f32>, to_light: vec3<f32>) -> vec3<f32> {
    return surface_albedo * max(dot(normal, to_light), 0.0) * light_color;
}

//GGX
fn get_normal_distribution(roughness: f32, normal: vec3<f32>, half_vector: vec3<f32>) -> f32 {
    var roughness_squared = pow(roughness, 2.0);
    var n_dot_h = max(dot(normal, half_vector), 0.0);
    var denominator = pow(n_dot_h * n_dot_h *  (roughness_squared - 1.0) + 1.0, 2.0);
    return roughness_squared / ((3.1415 * denominator) + 0.00001);
}

//GGX-schlick
fn get_geometry(to_view: vec3<f32>, to_light: vec3<f32>, normal: vec3<f32>, roughness: f32) -> f32 {
    var k = roughness / 2; //might be different in other renderers
    //var k = pow(roughness + 1, 2.0) / 8; 
    var n_dot_l = max(dot(normal, to_light), 0.0);
    var geometry_light = n_dot_l / ((n_dot_l * (1.0 - k) + k) + 0.00001);
    var n_dot_v = max(dot(normal, to_view), 0.0);
    var geometry_view = n_dot_v / ((n_dot_v * (1.0 - k) + k) + 0.00001);

    return geometry_light * geometry_view;
}

//cook-torrence
fn get_specular(fresnel: vec3<f32>, to_view: vec3<f32>, to_light: vec3<f32>, normal: vec3<f32>, half_vector: vec3<f32>, roughness: f32) -> vec3<f32> {
    var d = get_normal_distribution(roughness, normal, half_vector);
    var g = get_geometry(to_view, to_light, normal, roughness);
    var f = fresnel;
    var v_dot_n = max(dot(to_view, normal), 0.0);
    var l_dot_n = max(dot(to_light, normal), 0.0);
    return (d * g * f) / (4 * v_dot_n * l_dot_n);
}

fn get_bdrf(surface_albedo: vec3<f32>, f0: vec3<f32>, roughness: f32, normal: vec3<f32>, light_color: vec3<f32>, light_pos: vec3<f32>, camera_pos: vec3<f32>, frag_pos: vec3<f32>) -> vec3<f32>{
    var to_view = normalize(camera_pos - frag_pos);
    var to_light = normalize(light_pos - frag_pos);
    var half_vector = normalize(to_view + to_light);
    var fresnel = get_fresnel(f0, to_view, half_vector);
    var ks = fresnel;
    var kd = vec3(1.0) - ks;

    var diffuse = get_diffuse(surface_albedo, light_color, normal, to_light);
    var specular = get_specular(fresnel, to_view, to_light, normal, half_vector, roughness);

    return kd * diffuse + specular;
}

@vertex
fn vertex_main(@location(0) position: vec3<f32>, @location(1) uv: vec2<f32>, @location(2) normal: vec3<f32>) -> VertexOut
{
    var output : VertexOut;
    output.frag_position =  scene.projection_matrix * scene.view_matrix * scene.model_matrix * vec4<f32>(position, 1.0);
    output.world_position = scene.model_matrix * vec4<f32>(position, 1.0);
    output.uv = uv;
    output.normal = scene.normal_matrix * normal;
    return output;
}
@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32>
{   
    var surface_albedo = textureSample(albedo_map, albedo_sampler, frag_data.uv).rgb;
    var roughness_from_map = textureSample(roughness_map, roughness_sampler, frag_data.uv).x;
    var roughness = mix(specular.roughness, roughness_from_map, f32(specular.use_specular_map));
    var f0 = vec3(0.04, 0.04, 0.04);
    var total_color = vec3(0.0);
    var normal = normalize(frag_data.normal);

    for(var i: u32 = 0; i < light_count.count; i++){
        var light = lights[i];
        total_color += get_bdrf(surface_albedo, f0, roughness, normal, light.color.rgb, light.position, scene.camera_position, frag_data.world_position.xyz);
    }

    return vec4(total_color, 1.0);
}

A few things here. I changed the names of some things, "specular" is now "roughness" and the base texture is now call "albedo" which is the more common name to use. Anywhere you saw a dot product, it's max(dot(X, Y), 0.0). This is prevent negative dot products which might cause weird bugs. I also add 0.00001 to each denominator to prevent accidental divide by zero.

As for the overall result. It seems decent to me.

Light attenuation

One thing we didn't really apply is light attenuation. That is, the light get less bright the further away you are. We can add that term.

@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32>
{   
    var surface_albedo = textureSample(albedo_map, albedo_sampler, frag_data.uv).rgb;
    var roughness_from_map = textureSample(roughness_map, roughness_sampler, frag_data.uv).x;
    var roughness = mix(specular.roughness, roughness_from_map, f32(specular.use_specular_map));
    var f0 = vec3(0.04, 0.04, 0.04);
    var total_color = vec3(0.0);
    var normal = normalize(frag_data.normal);

    for(var i: u32 = 0; i < light_count.count; i++){
        var light = lights[i];
+       var light_distance = length(light.position - frag_data.world_position.xyz);
+       var attenuation = 1.0 / pow(light_distance, 2.0);
+       var radiance = light.color.rgb * attenuation;
+       total_color += get_bdrf(surface_albedo, f0, roughness, normal, radiance, light.position, scene.camera_position, frag_data.world_position.xyz);
-total_color += get_bdrf(surface_albedo, f0, roughness, normal, light.color, light.position, scene.camera_position, frag_data.world_position.xyz);
    }

    return vec4(total_color, 1.0);
}

Light falls off over distance squared so we just multiply by 1 over that. This is the inverse-power law.

The scene gets a lot darker.

Gamma correction

Some sources I looked at performed gamma correct, which is a process of stretching the rgb space so there is more room for dark colors and better maps the range to human vision. A simple approximation is to raise the color to the power of 1/2.2 .

After some reading it seems we might need to apply this manually. We are drawing to a rgba8unorm texture canvas which is in linear rgb space so there should not be any auto conversion to srgb (gamma corrected rgb) if I understand correctly. In fact, it seems you can't configure a canvas context with rgba8unorm-srgb format anyway, at least in Chrome. We'll try it.

In addition we should tone map as well. This just takes the unbounded color which could have components of greater than 1 which we cannot display and reduces them down into the range of 0 to 1. We do this with Reinhard's operator which is var tone_mapped_color = color / (color + vec3(1.0) This should be done before gamma correction.

@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32>
{   
    var surface_albedo = textureSample(albedo_map, albedo_sampler, frag_data.uv).rgb;
    var roughness_from_map = textureSample(roughness_map, roughness_sampler, frag_data.uv).x;
    var roughness = mix(specular.roughness, roughness_from_map, f32(specular.use_specular_map));
    var f0 = vec3(0.04, 0.04, 0.04);
    var total_color = vec3(0.0);
    var normal = normalize(frag_data.normal);

    for(var i: u32 = 0; i < light_count.count; i++){
        var light = lights[i];
        var light_distance = length(light.position - frag_data.world_position.xyz);
        var attenuation = 1.0 / pow(light_distance, 2.0);
        var radiance = light.color.rgb * attenuation;
        total_color += get_bdrf(surface_albedo, f0, roughness, normal, radiance, light.position, scene.camera_position, frag_data.world_position.xyz);
    }

-   return vec4(total_color, 1.0);
+   var tone_mapped_color = total_color / (total_color + vec3(1.0));
+   return vec4(pow(total_color, vec3(1.0/2.2)), 1.0);
}

We can see more of the dark details although artistically it kinda sucks.

We can play around with the light's brightness by using a color vector with value that are greater than 1.

Turned up the brightness with color [2.5, 2.5, 2.5, 1]. Not too bad.

Metals

The last things we want to consider are metals. These behave slightly differently in the PBR model because metals have much higher F0 values and they vary by color channel. Let's try making a gold teapot to test it out. We'll need to add 2 new properties to the material:

//material.js
export class Material {
  //...stuff
  constructor(options){
    //...stuff
    this.#metalness = options.metalness ?? 0.0;
    this.#baseReflectance = options.baseReflectance ?? [0.04, 0.04, 0.04];
  }
  #metalness;
  #baseReflectance;
  set metalness(val){
    this.#metalness = val;
  }
  get metalness(){
    return this.#metalness;
  }
  set baseReflectance(val){
    this.#baseReflectance = new Float32Array(val);
  }
}

metalness is how much the material is metal (this will most likely be 0 or 1 but we'll provide a range because it's easier). baseReflectance is the F0 value in RGB.

physicallybased.info says a good F0 for gold is: 1.059,0.773,0.307.

In the shader we'll add those props to the Material struct (formerly SpecularMaterial, "specular" was redundant, instances called "specular" were renamed to "material").

struct Material {
    use_specular_map: u32,
    roughness: f32,
+   metalness: f32,
+   base_reflectance: vec3<f32>
}

We use the metalness value in the BDRF:

-fn get_bdrf(surface_albedo: vec3<f32>, f0: vec3<f32>, roughness: f32, normal: vec3<f32>, light_color: vec3<f32>, light_pos: vec3<f32>, camera_pos: vec3<f32>, frag_pos: vec3<f32>) -> vec3<f32>{
+fn get_bdrf(surface_albedo: vec3<f32>, f0: vec3<f32>, roughness: f32, metalness: f32, normal: vec3<f32>, light_color: vec3<f32>, light_pos: vec3<f32>, camera_pos: vec3<f32>, frag_pos: vec3<f32>) -> vec3<f32>{
    var to_view = normalize(camera_pos - frag_pos);
    var to_light = normalize(light_pos - frag_pos);
    var half_vector = normalize(to_view + to_light);
    var fresnel = get_fresnel(f0, to_view, half_vector);
    var ks = fresnel;
-   var kd = (vec3(1.0) - ks);
+   var kd = (vec3(1.0) - ks) * (1.0 - metalness);

    var diffuse = get_diffuse(surface_albedo, light_color, normal, to_light);
    var specular = get_specular(fresnel, to_view, to_light, normal, half_vector, roughness);
    return kd * diffuse + specular;
}

Things that are metal don't diffuse so we essentially turn it off if the metalness is 1. We also vary the f0 value by interpolating between the dielectric value and the passed in value:

@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32>
{   
    var surface_albedo = textureSample(albedo_map, albedo_sampler, frag_data.uv).rgb;
    var roughness_from_map = textureSample(roughness_map, roughness_sampler, frag_data.uv).x;
-   var roughness = max(mix(material.roughness, roughness_from_map, f32(material.use_specular_map)), 0.0);
+   var roughness = max(mix(material.roughness, roughness_from_map, f32(material.use_specular_map)), 0.0);
-   var f0 = vec3(0.04, 0.04, 0.04);
+   var f0 = mix(vec3(0.04, 0.04, 0.04), material.base_reflectance, material.metalness);
    var total_color = vec3(0.0);
    var normal = normalize(frag_data.normal);

    for(var i: u32 = 0; i < light_count.count; i++){
        var light = lights[i];
        var light_distance = length(light.position - frag_data.world_position.xyz);
        var attenuation = 1.0 / pow(light_distance, 2.0);
        var radiance = light.color.rgb * attenuation;
        total_color += get_bdrf(surface_albedo, f0, roughness, normal, radiance, light.position, scene.camera_position, frag_data.world_position.xyz);
    }

    //return vec4(total_color, 1.0);
    var tone_mapped_color = total_color / (total_color + vec3(1.0));
    return vec4(pow(total_color, vec3(1.0/2.2)), 1.0);
}

I also had to clamp the roughness because at 0 the normal distribution function goes to 0 meaning no specular at all. Perfect mirrors don't behave well. Speaking of not behaving, this is the result:

The gold actually looks pretty good! But the shadowing is very harsh. This is because the metal only has specular so where the light doesn't hit will get shadowed to the extreme because there is no color transition as we'd get with diffuse lighting.

Note on Ambient Lighting

In reality the scene will have light from more directions the subtle bouncing of light off surfaces like the rug. This can be modeled using expensive and sophisticated ways like path-tracing. I think I'll leave this topic alone though since it seems to be a bit large.

Directional Lights

Along with point lights we'd like to have directional lights. These are lights that just case one direction but have infinite size. Like the sun from far away. Might as well add them now while we're in here. We already have the enum for it defined on light we just need a few tweaks.


-fn get_bdrf(surface_albedo: vec3<f32>, f0: vec3<f32>, roughness: f32, metalness: f32, normal: vec3<f32>, light_color: vec3<f32>, light_pos: vec3<f32>, camera_pos: vec3<f32>, frag_pos: vec3<f32>) -> vec3<f32>{
+fn get_bdrf(surface_albedo: vec3<f32>, f0: vec3<f32>, roughness: f32, metalness: f32, normal: vec3<f32>, light_color: vec3<f32>, to_light: vec3<f32>, camera_pos: vec3<f32>, frag_pos: vec3<f32>) -> vec3<f32>{
    var to_view = normalize(camera_pos - frag_pos);
-   var to_light = normalize(light_pos - frag_pos);
    var half_vector = normalize(to_view + to_light);
    var fresnel = get_fresnel(f0, to_view, half_vector);
    var ks = fresnel;
    var kd = (vec3(1.0) - ks) * (1.0 - metalness);

    var diffuse = get_diffuse(surface_albedo, light_color, normal, to_light);
    var specular = get_specular(fresnel, to_view, to_light, normal, half_vector, roughness);

    return kd * diffuse + specular;
}

We don't need light_pos except to calculate the vector to_light. So we'll calculate that in the main loop.

@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32>
{   
    var surface_albedo = textureSample(albedo_map, albedo_sampler, frag_data.uv).rgb;
    var roughness_from_map = textureSample(roughness_map, roughness_sampler, frag_data.uv).x;
    var roughness = max(mix(material.roughness, roughness_from_map, f32(material.use_specular_map)), 0.0001);
    var f0 = mix(vec3(0.04, 0.04, 0.04), material.base_reflectance, material.metalness);
    var total_color = vec3(0.0);
    var normal = normalize(frag_data.normal);

    for(var i: u32 = 0; i < light_count.count; i++){
        var light = lights[i];
        var light_distance = length(light.position - frag_data.world_position.xyz);
+       var to_light = vec3(0.0);

+       switch light.light_type {
+           case 0: {
+               to_light = normalize(light.position - frag_data.world_position.xyz);
+           }
+           case 1: {
+               to_light = normalize(-light.direction);
+           }
+           default: {}
+       }

        var attenuation = 1.0 / pow(light_distance, 2.0);
        var radiance = light.color.rgb * attenuation;


        total_color += get_bdrf(
            surface_albedo, 
            f0, 
            roughness, 
            material.metalness,
            normal, 
            radiance, 
+           to_light,
            scene.camera_position, 
            frag_data.world_position.xyz
        );
    }

    var tone_mapped_color = total_color / (total_color + vec3(1.0));
    return vec4(pow(total_color, vec3(1.0/2.2)), 1.0);
}

This, as far as I can tell, requires branching mainly because there will be 3 states (I'm just not dealing with spotlights yet). So we switch on the light type. If it's type 0 (point) we use the previous method of getting the vector from the surface to the point. In the case of a directional light we just need the direction, but to_light is from the surface to the light so it's actually just the inverse of the light direction. Then we plug in as normal.

Here is the teapot scene with an overhead directional light ([0, -1, 0]). It's very bright because it extends in all directions.

Code

https://github.com/ndesmic/geo/compare/v0.5...v0.6

References

WebGPU Engine from Scratch Part 7: Specular Lighting

ndesmic — Mon, 01 Sep 2025 19:00:00 +0000

Since we added diffuse lighting we might as well add specular too. The teapot is a marble texture that should be shiny. Let's start with a fixed value and build up to specular maps which were included with the texture I downloaded.

First we'll start with an entity. We'll add a few properties to Material:

useSpecularMap - a boolean for whether or not we're using a specular map
specularMap - a specular map (we're not saying which kind but for now it'll be hardcoded as a roughness map since that's what my textures came with. eg. gloss = 1 - value). This could be expanded in the future. They are also full color so a sampled value is the same as a glossColor.
specularSampler - the sampler for the specular map
glossColor - the constant gloss color if not using a specular map. This encodes gloss value per color channel, not the color of the highlight directly.

//material.js
export class Material {
    #useSpecularMap;
    #specularMap;
    #specularSampler;
    #glossColor;

    #texture;
    #textureSampler;
    #name;

    constructor(options){
        this.name = options.name;
        this.useSpecularMap = options.useSpecularMap ?? false;
        this.specularMap = options.specularMap ?? "dummy";
        this.specularSampler = options.specularSampler ?? "default";
        this.glossColor = options.glossColor ?? new Float32Array(1,1,1,1);
        this.texture = options.texture ?? "dummy";
        this.textureSampler = options.textureSampler ?? "default";
    }

    set name(val){
        this.#name = val;
    }
    get name(){
        return this.#name;
    }
    set useSpecularMap(val){
        this.#useSpecularMap = val;
    }
    get useSpecularMap(){
        return this.#useSpecularMap;
    }
    set specularMap(val){
        this.#specularMap = val;
    }
    get specularMap(){
        return this.#specularMap;
    }
    set specularSampler(val){
        this.#specularSampler = val;
    }
    get specularSampler(){
        return this.#specularSampler;
    }
    set glossColor(val){
        this.#glossColor = new Float32Array(val);
    }
    get glossColor(){
        return this.#glossColor;
    }
    set texture(val){
        this.#texture = val;
    }
    get texture() {
        return this.#texture;
    }
    set textureSampler(val){
        this.#textureSampler = val;
    }
    get textureSampler(){
        return this.#textureSampler;
    }
}

The way this will work is that useSpecularMap will be set depending on if we are using a constant or a specular map. specularMap is the map if we're using it and specularSampler samples from it. If we aren't using the map then we use glossColor. Note that like texture the specular map is a string reference. I also added textureSampler for completeness (also a string).

Also note the the default values. Specular map "dummy" will be a new texture that functions as a placeholder when we're not using one so the binding still works. Sampler "default" is what we'll rename sampler "main" to because it's a good default for most things (bilinear UVs).

We can initialize a Material like this:

//gpu-engine.js
initializeMaterials(){
    this.#materials.set("marble", new Material({
        texture: "marble",
        useSpecularMap: false,
        glossColor: new Float32Array([4.0,4.0,4.0,1])
    }));
    this.#materials.set("red-fabric", new Material({
        texture: "red-fabric"
    }));
}

(May there's an argument for auto setting useSpecularMap if using a value, but I like it explicit right now). For the dummy texture we'll use this:

//wgpu-utils.js
/**
 * Creates a 1x1 texture of a color
 * @param {GPUDevice} device 
 * @returns 
 */
export function createColorTexture(device, options = {}) {
    const texture = device.createTexture({
        label: options.label,
        size: [1, 1],
        format: 'rgba8unorm',
        usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST,
    });

    const texel = options.color ?? new Uint8Array([255, 255, 255, 255]);

    device.queue.writeTexture(
        { texture: texture },
        texel,
        { bytesPerRow: 4 },
        { width: 1, height: 1, depthOrArrayLayers: 1 }
    );

    return texture;
}

And set it:

//gpu-engine.js - initializeTextures
this.#textures.set("dummy", createColorTexture(this.#device, { label: "dummy-texture" }));

This will be called after initializeTextures. Next let's create the bindings. Since things are changing I renamed setMainTextureBindGroup to setMainMaterialBindGroup.

setMainMaterialBindGroup(passEncoder, bindGroupLayouts, mesh){
    const material = this.#materials.get(mesh.material);
    const specular = {
        useSpecularMap: material.useSpecularMap ? 1 : 0, //0 => constant, 1 => map
        specularValue: material.specularValue,
        specularColor: material.specularColor
    }
    const specularData = packStruct(specular, [
        ["useSpecularMap", "u32"],
        ["glossColor", "vec4f32"]
    ]);
    const specularBuffer = this.#device.createBuffer({
        size: specularData.byteLength,
        usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
        label: "main-specular-buffer"
    });
    this.#device.queue.writeBuffer(specularBuffer, 0, specularData);
    const materialBindGroup = this.#device.createBindGroup({
        layout: bindGroupLayouts.get("materials"),
        entries: [
            { binding: 0, resource: this.#samplers.get(material.textureSampler) },
            { binding: 1, resource: this.#textures.get(material.texture).createView() },
            { 
                binding: 2, 
                resource: {
                    buffer: specularBuffer,
                    offset: 0,
                    size: specularData.byteLength
                }
            },
            { binding: 3, resource: this.#samplers.get(material.specularSampler) },
            { binding: 4, resource: this.#textures.get(material.specularMap).createView()}
        ]
    });
    passEncoder.setBindGroup(1, materialBindGroup);
}

We've upgraded it to take in the extra parameters from the mesh's material. As an aside I got caught up for a really long time based on a terrible error message. If you forget .createView() on the texture you will get an error about buffer not existing on the resource. I kept thinking it was binding 2 (it doesn't tell you which) but I guess it happens for texture bindings under the covers. Really confusing. https://issues.chromium.org/issues/40287743. You also, as always, need to use the bindings otherwise they get erased by the compiler.

Specular Maps

When we use specular maps there are now some conventions we need to consider. The first is that the maps we have are "roughness" maps and these are the inverse of a gloss map, 0 is very shiny, 1 is fully diffuse. The other is that the images themselves are black and white. This actually works out since our pipeline does specular per color channel but it's possible a specular map could only have a single value and be embedded in one channel like red. We'll cross these bridges if we get to them.

To use specular maps we need to provide them to shader:

initializeMaterials(){
    this.#materials.set("marble", new Material({
        texture: "marble",
        useSpecularMap: true,
        specularMap: "marble-roughness"
    }));
    this.#materials.set("red-fabric", new Material({
        texture: "red-fabric",
        useSpecularMap: true,
        specularMap: "red-fabric-roughness"
    }));
}

And finally the shader:


struct VertexOut {
    @builtin(position) frag_position : vec4<f32>,
    @location(0) world_position: vec4<f32>,
    @location(1) uv : vec2<f32>,
    @location(2) normal : vec3<f32>
};
struct Scene {
    view_matrix: mat4x4<f32>,
    projection_matrix: mat4x4<f32>,
    model_matrix: mat4x4<f32>,
    normal_matrix: mat3x3<f32>,
    camera_position: vec3<f32>
}
struct Light {
    light_type: u32,
    position: vec3<f32>,
    direction: vec3<f32>,
    color: vec4<f32>
}
struct LightCount {
    count: u32
}
struct SpecularMaterial {
    use_specular_map: u32,
    gloss_color: vec4<f32>
}

@group(0) @binding(0) var<uniform> scene : Scene;

@group(1) @binding(0) var main_sampler: sampler;
@group(1) @binding(1) var texture: texture_2d<f32>;
@group(1) @binding(2) var<uniform> specular: SpecularMaterial;
@group(1) @binding(3) var specular_sampler: sampler;
@group(1) @binding(4) var specular_map: texture_2d<f32>;

@group(2) @binding(0) var<storage, read> lights: array<Light>;
@group(2) @binding(1) var<uniform> light_count: LightCount;

@vertex
fn vertex_main(@location(0) position: vec3<f32>, @location(1) uv: vec2<f32>, @location(2) normal: vec3<f32>) -> VertexOut
{
    var output : VertexOut;
    output.frag_position =  scene.projection_matrix * scene.view_matrix * scene.model_matrix * vec4<f32>(position, 1.0);
    output.world_position = scene.model_matrix * vec4<f32>(position, 1.0);
    output.uv = uv;
    output.normal = scene.normal_matrix * normal;
    return output;
}
@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32>
{   
    //assume it's a roughness map
    var gloss_from_map = vec4(1.0) - textureSample(specular_map, specular_sampler, frag_data.uv);
    var gloss = mix(specular.gloss_color.rgb, gloss_from_map.rgb, f32(specular.use_specular_map));

    var surface_color = textureSample(texture, main_sampler, frag_data.uv);
    var total_diffuse = vec4(0.0);
    var total_specular = vec4(0.0);

    var i = 0;
    for(var i: u32 = 0; i < light_count.count; i++){
        //diffuse
        var light = lights[i];
        var to_light = normalize(light.position - frag_data.world_position.xyz);
        var diffuse_intensity = max(dot(normalize(frag_data.normal), to_light), 0.0);
        total_diffuse += light.color * vec4(diffuse_intensity, diffuse_intensity, diffuse_intensity, 1);

        //specular
        var to_camera = normalize(scene.camera_position - frag_data.world_position.xyz);
        var half_vector = normalize(to_light + to_camera);
        var base_specular = vec3(clamp(dot(half_vector, frag_data.normal), 0.0, 1.0));
        var specular_intensity = pow(base_specular, gloss.rgb);
        specular_intensity *= vec3(f32(gloss.r > 0.0), f32(gloss.g > 0.0), f32(gloss.b > 0.0)); //disable if specular_value is 0
        total_specular += light.color * vec4(specular_intensity.r, specular_intensity.g, specular_intensity.b, 1); 
    }

    //return gloss_from_map;
    return surface_color * (total_diffuse + total_specular);
}

First we convert roughness to gloss. Then we use mix to interpolate between the constant value and the map value (this saves us from having to use an if which causes branching in the shader which is bad). This works because it can only ever be 0 or 1, fully one or the other. The rest is pretty much the same as the WebGL version. There is a term specular_intensity *= vec3(f32(gloss.r > 0.0), f32(gloss.g > 0.0), f32(gloss.b > 0.0)); which fully turns off specular if the value is 0. Then we add it all up.

The result works but is a little underwhelming because the the scale is all wrong. Turns out roughness maps don't exactly translate to our gloss values (no surprise). We could add scaling factors to fudge it, but perhaps we'll revisit it with a real PBR (Physically Based Rendering) implementation later. For now I'm okay with passing in constants.

Since it's easy to flip back and forth here are my values:

initializeMaterials(){
    this.#materials.set("marble", new Material({
        texture: "marble",
        useSpecularMap: false,
        glossColor: [4,4,4,1],
        specularMap: "marble-roughness"
    }));
    this.#materials.set("red-fabric", new Material({
        texture: "red-fabric",
        useSpecularMap: false,
        glossColor: [0,0,0,1],
        specularMap: "red-fabric-roughness"
    }));
}

Good enough for now.

Code

https://github.com/ndesmic/geo/releases/tag/v0.5

WebGPU Engine from Scratch Part 6: Diffuse Lighting

ndesmic — Mon, 25 Aug 2025 19:00:00 +0000

The scene is bland and could use some lighting so let's implement that. First we start with the Light entity:

//light.js
/**
 * @typedef {"point" | "directional" | "spot" } LightType
 */

export class Light {
    /**@type {LightType} */
    #type;
    #position;
    #direction;
    #color;

    constructor(light){
        this.#type = light.type ?? "point";
        this.#position = light.position ?? [0, 0, 0, 0];
        this.#direction = light.direction ?? [0, 0, 0, 0];
        this.#color = light.color ?? [1,1,1,1];
    }

    set type(val){
        this.#type = new Float32Array(val);
    }
    get type(){
        return this.#type;
    }

    set position(val){
        this.#position = new Float32Array(val);
    }
    get position(){
        return this.#position;
    }

    set direction(val){
        this.#direction = new Float32Array(val);
    }
    get direction(){
        return this.#direction;
    }

    set color(val){
        this.#color = new Float32Array(val);
    }
    get color(){
        return this.#color;
    }
}

Similar to the WebGL version. Then let's create a light

initializeLights(){
    this.#lights.set("light", new Light({
        type: "point",
        position: sphericalToCartesian([ Math.PI / 4, Math.PI / 4, 2]),
    }));
}

This is the same as the other initialization method and need this.#lights set as a Map. Then we need to pass the light to the shader in a bindgroup.

I struggled for a while to get the correct size for the buffer and the correct offsets. Lots of little mistakes. I finally came up with a genericized way to pack structs:

//buffer-utils.js
/**
 * @typedef {[string,GpuType]} Prop
 * @typedef {Prop[]} Schema
 * @param {object} data 
 * @param {Schema} schema 
 */
export function packStruct(data, schema, minSize){
    const { offsets, totalSize } = getAlignments(schema.map(s => s[1]), minSize);
    const buffer = new ArrayBuffer(totalSize);
    const dataView = new DataView(buffer);

    for(let i = 0; i < schema.length; i++){
        const [name, type] = schema[i];
        const value = data[name];
        //TODO: add other GPU Types
        switch(type){
            case "u32": {
                dataView.setUint32(offsets[i], value, true);
                break;
            }
            case "vec3f32": {
                dataView.setFloat32(offsets[i], value[0], true);
                dataView.setFloat32(offsets[i] + 4, value[1], true);
                dataView.setFloat32(offsets[i] + 8, value[2], true);
                break;
            }
            case "vec4f32": {
                dataView.setFloat32(offsets[i], value[0], true);
                dataView.setFloat32(offsets[i] + 4, value[1], true);
                dataView.setFloat32(offsets[i] + 8, value[2], true);
                dataView.setFloat32(offsets[i] + 12, value[3], true);
                break;
            }
            default: {
                throw new Error(`Cannot pack type ${type} at prop index ${i} with value ${value}`);
            }
        }
    }

    return buffer;
}

This is only partially complete as it only covers the types that I need for the light right now but filling the rest in should be easy and an exercise to the reader (and my future self). With that we can pack a light into a buffer.

//buffer-utils.js
/**
 * 
 * @param {Light} light 
 */
export function packLight(light){
    const schema = [
        ["typeInt", "u32"],
        ["position", "vec3f32"],
        ["direction", "vec3f32"],
        ["color", "vec4f32"]
    ];
    return packStruct(light, schema, 64);
}

This illustrates how to use packStruct. It only works for shallow structs but you give tuples of name and type. The name looks up the property in the object. It's an array because I want to maintain strict ordering so there are no surprises.

We can then set the bind group. I've made it a 3rd group and added it to setMainBindGroups.

setMainLightBindGroup(passEncoder, bindGroupLayouts, lights){
    const lightData = packLight(lights.get("light"));
    const bufferSize = lightData.byteLength;
    const lightBuffer = this.#device.createBuffer({
        size: bufferSize,
        usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST | GPUBufferUsage.STORAGE,
        label: "main-light-buffer"
    });
    this.#device.queue.writeBuffer(lightBuffer, 0, lightData);
    const lightCountBuffer = this.#device.createBuffer({
        size: 4,
        usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
        label: "main-light-count-buffer"
    });
    this.#device.queue.writeBuffer(lightCountBuffer, 0, new Int32Array([lights.size]));
    const lightBindGroup = this.#device.createBindGroup({
        label: "main-light-bind-group",
        layout: bindGroupLayouts.get("lights"),
        entries: [
            {
                binding: 0,
                resource: {
                    buffer: lightBuffer,
                    offset: 0,
                    size: bufferSize
                }
            },
            {
                binding: 1,
                resource: {
                    buffer: lightCountBuffer,
                    offset: 0,
                    size: 4
                }
            }
        ]
    });
    passEncoder.setBindGroup(2, lightBindGroup);
}

This process should be well understood by now. We are passing a light (we'll see soon that this can generalize to many lights) and the count of lights (the reason why explained below). Also keep in mind that lightBuffer has usage GPUBufferUsage.STORAGE. We'll test things out in a shader based on our previous one.

//textured-lit.wgsl
struct VertexOut {
    @builtin(position) position : vec4<f32>,
    @location(0) uv : vec2<f32>
};
struct Uniforms {
    view_matrix: mat4x4<f32>,
    projection_matrix: mat4x4<f32>,
    model_matrix: mat4x4<f32>,
    normal_matrix: mat3x3<f32>,
    camera_position: vec3<f32>
}
struct Light {
    light_type: u32,
    position: vec3<f32>,
    direction: vec3<f32>,
    color: vec4<f32>
}
struct LightCount {
    count: u32
}

@group(0) @binding(0) var<uniform> uniforms : Uniforms;
@group(1) @binding(0) var main_sampler: sampler;
@group(1) @binding(1) var texture: texture_2d<f32>;
@group(2) @binding(0) var<storage, read> lights: array<Light>;
@group(2) @binding(1) var<uniform> light_count: LightCount;

@vertex
fn vertex_main(@location(0) position: vec3<f32>, @location(1) uv: vec2<f32>) -> VertexOut
{
    var output : VertexOut;
    output.position =  uniforms.projection_matrix * uniforms.view_matrix * uniforms.model_matrix * vec4<f32>(position, 1.0);
    output.uv = uv;
    return output;
}
@fragment
fn fragment_main(fragData: VertexOut) -> @location(0) vec4<f32>
{
    var light = lights[0];
    var lc = light_count.count;

    var tex = textureSample(texture, main_sampler, fragData.uv);
    return light.color;
}

We need to define the struct for the Light which is pretty straight-forward. What's a little more odd is LightCount which holds a single u32. LightCount must be a struct. We cannot bind directly to scalar values for whatever reason so they always need to be boxed in a struct. The other interesting part here is how we define an array of lights with var<storage, read> lights: array<Light>; This is cool because unlike WebGL we can actually pass arrays of dynamic length, however it is limited in that the length is set by the binding and we do not get a length in the shader value which is why we need to pass the count manually. This array construct is also why the buffer usage was tagged with GPUBufferUsage.STORAGE because it's treated differently by the GPU.

In the fragment shader we're just doing a test to see if we can output the light color (white). This will help ensure that we got all the data packing correct. We need some dummy variables for light count and the texture otherwise they get erased for no-usage on compile and we'll get errors (I hate this about WebGPU).

If everything went well and we pray to the alignment gods we should see something like this.

Cool.

With that hooked up correctly now we can actually get to making our shader use normal diffuse lighting.

struct VertexOut {
    @builtin(position) frag_position : vec4<f32>,
    @location(0) world_position: vec4<f32>,
    @location(1) uv : vec2<f32>,
    @location(2) normal : vec3<f32>
};
struct Uniforms {
    view_matrix: mat4x4<f32>,
    projection_matrix: mat4x4<f32>,
    model_matrix: mat4x4<f32>,
    normal_matrix: mat3x3<f32>,
    camera_position: vec3<f32>
}
struct Light {
    light_type: u32,
    position: vec3<f32>,
    direction: vec3<f32>,
    color: vec4<f32>
}
struct LightCount {
    count: u32
}

@group(0) @binding(0) var<uniform> uniforms : Uniforms;
@group(1) @binding(0) var main_sampler: sampler;
@group(1) @binding(1) var texture: texture_2d<f32>;
@group(2) @binding(0) var<storage, read> lights: array<Light>;
@group(2) @binding(1) var<uniform> light_count: LightCount;

@vertex
fn vertex_main(@location(0) position: vec3<f32>, @location(1) uv: vec2<f32>, @location(2) normal: vec3<f32>) -> VertexOut
{
    var output : VertexOut;
    output.frag_position =  uniforms.projection_matrix * uniforms.view_matrix * uniforms.model_matrix * vec4<f32>(position, 1.0);
    output.world_position = uniforms.model_matrix * vec4<f32>(position, 1.0);
    output.uv = uv;
    output.normal = normal;
    return output;
}
@fragment
fn fragment_main(frag_data: VertexOut) -> @location(0) vec4<f32>
{   
    var surface_color = textureSample(texture, main_sampler, frag_data.uv);
    var diffuse = vec4(0.0);

    for(var i: u32 = 0; i < light_count.count; i++){
        var light = lights[i];
        var light_position = vec4(light.position.xyz, 1.0);
        var to_light = normalize(light_position - frag_data.world_position);
        var light_intensity = max(dot(normalize(frag_data.normal), to_light.xyz), 0.0);
        diffuse += light.color * vec4(light_intensity, light_intensity, light_intensity, 1);
    }

    return surface_color * diffuse;
}

We need to modify the pipeline to pass in the normals and pass that through the vertex shader along with the world position of the fragment. In the fragment shader we get the base color and apply each light in sequence. We get the direction of the light and then dot that with the normal to figure out how much gets reflected (if the light is the same direction as the normal it all does). Then we take the light color, surface color and intensity and multiply it together for the final color. Repeat for each light.

This works with one light but what an actual array of lights? We need a small modification.

//buffer-utils.js
export function packArray(data, schema, minSize){
    const { totalSize: structSize } = getAlignments(schema.map(s => s[1]), minSize);
    const totalSize = structSize * data.length;
    const buffer = new ArrayBuffer(totalSize);
    for(let i = 0; i < data.length; i++){
        packStruct(data[i], schema, minSize, buffer, i * structSize);
    }
    return buffer;
}

To pack an array we need to take the max align of the struct and make sure each element aligns on that boundary. Luckily the padding from packing a struct is the same thing so we just need to put it into a single buffer. To do this, I modified packStruct to optionally take a buffer and offset for where to place the packed bytes. Not worth showing here but you can look at the final code. Then we just modify packLight into packLights and use packArray instead.

When we apply it something weird happens.

Here I have a red light at 0, 0, -1. This should be slightly in front of the camera and on the surface mesh as seen from above (toward -Y) we can see that it is.

But the bottom of the teapot is light up, not the side. The -X and +X work just fine though, but it appears as though Y and Z are flipped! It took a while to figure out but it's because when we bake the transforms we didn't do the normals! Recall that to properly transform normals we need to get the top left 3x3 of the model matrix and get the transpose of the inverse (in this case we do not need to multiply by the view matrix since it's in model space). This also requires use to alter the matrix-vector multiplication because we had assumed it was always going to be a 4x4 times a 4, but now we need variable length.

//vector.js
/**
 * 
 * @param {Float32Array} vector 
 * @param {Float32Array} matrix
 * @param {number} size
 * @returns 
 */
export function multiplyMatrixVector(vector, matrix, size) {
    const newVector = new Float32Array(size);
    for(let i = 0; i < size; i++){
        let sum = 0;
        for(let j = 0; j < size; j++){
            sum += vector[j] * matrix[i * size + j] 
        }
        newVector.set([sum], i);
    }

    return newVector;
}

The whole bakeTransforms

//mesh.js
bakeTransforms(){
    //positions
    const modelMatrix = this.getModelMatrix();
    const transformedPositions = chunk(this.positions, this.positionSize)
        .map(values => {
            const lengthToPad = 4 - values.length;
            switch(lengthToPad){
                case 1:{
                    return [...values, 1.0]
                }
                case 2:{
                    return [...values, 0.0, 1.0];
                }
                case 3: {
                    return [...values, 0.0, 0.0, 1.0];
                }
                default: {
                    return [0.0, 0.0, 0.0, 1.0];
                }
            }
        })
        .map(values => multiplyMatrixVector(values, modelMatrix, this.positionSize + 1)) //need homogenous coordinates for positions
        .toArray();
    const normalMatrix = getTranspose(
        getInverse(
            trimMatrix(modelMatrix, [4,4], [this.normalSize,this.normalSize]), 
        [this.normalSize,this.normalSize]), 
    [this.normalSize,this.normalSize]);
    const transformedNormals = chunk(this.normals, this.normalSize)
        .map(values => multiplyMatrixVector(values, normalMatrix, this.normalSize))
        .map(values => normalizeVector(values))
        .toArray();
    //collect
    const newPositionsBuffer = new Float32Array(this.vertexLength * this.positionSize);
    for(let i = 0; i < transformedPositions.length; i++){
        newPositionsBuffer.set(transformedPositions[i].slice(0, this.positionSize), i * this.positionSize)
    }
    const newNormalsBuffer = new Float32Array(this.vertexLength * this.normalSize);
    for (let i = 0; i < transformedNormals.length; i++) {
        newNormalsBuffer.set(transformedNormals[i].slice(0, this.normalSize), i * this.normalSize)
    }
    this.positions = newPositionsBuffer;
    this.normals = newNormalsBuffer;
    this.resetTransforms();
    return this;
}

Before we are done we need to double check a few things that tripped me up:

The normalMatrix should not be multiplied by the view matrix, it should be in world space (if everything was view space it would work but it's more work).
Packing a mat3x3 is unintuitively the same as packing a mat4x4 with zeros at the end of each column. Each column needs padding. This took ages to debug because it's yet another alignment rule.
Matrices are column major meaning they index like M[col][row]. This is flipped 90 from how we do it in js so you need to double check it's what your thing.

I also when ahead and started using pack struct to pack the matrices and renamed "uniforms" to "scene" because it sounded better. After sorting all the alignment bugs we can finally have 2 lights that properly work.

Code

https://github.com/ndesmic/geo/releases/tag/v0.5

WebGPU Engine from Scratch Part 5: More Pipeline Improvements

ndesmic — Mon, 18 Aug 2025 19:00:00 +0000

I was hoping to get to new features but it seems like there was just too much I wanted to fix up in the pipeline because it was getting really taxing to try different things. So it seems this will be another chapter of more scattershot improvements but admittedly they are more boring in nature. There's enough that I didn't want to bog down a feature post with too much enhancement diversion.

Surface Grid Mesh

I want to re-add the formerly named "terrain mesh" but I'm going to rename it "surface grid" because it's a little more general (I also fixed a bug in the uvSphere where the index buffer was too big).

/**
 * Generates a flat surface made up of multiple quads, faces +Y, each quad is 1x1
 * @param {number} height 
 * @param {number} width 
 */
export function surfaceGrid(height, width){
    const vertexLength = (height + 1) * (width + 1);
    const positions = new Float32Array(vertexLength * 3);
    const uvs = new Float32Array(vertexLength * 2);
    const normals = new Float32Array(vertexLength * 3);
    const tangents = new Float32Array(vertexLength * 3);
    const indices = new Int16Array(height * width * 6);

    let z = -(height / 2);

    for (let row = 0; row < height + 1; row++) {
        let x = -(width / 2);
        for (let col = 0; col < width + 1; col++) {
            positions.set([
                x, 0, z 
            ], (row * (width + 1) + col) * 3);
            uvs.set([
                col / width, row / height
            ], (row * (width + 1) + col) * 2);
            normals.set([
                0, 1, 0
            ], (row * (width + 1) + col) * 3);
            tangents.set([
                1, 0, 0
            ], (row * (width + 1) + col) * 3)
            x++;
        }
        z++;
    }

    for(let row = 0; row < height; row++){
        for(let col = 0; col < width; col++){
            const index = row * (width + 1) + col;
            indices.set([
                index, index + 1, index + width + 2, //take into account the extra vert at end of row
                index, index + width + 2, index + width + 1
            ], (row * width + col) * 6);
        }
    }

    return {
        positions,
        uvs,
        normals,
        indices,
        tangents,
        vertexLength
    };
}

This is much cleaned up. It doesn't create extra overlapping vertices anymore and the code is more straight-forward. This will let us generate some floor so we can see the shadows.

Mesh enhancements

Another thing to do is added some nice-to-haves to the Mesh class as it's currently frustrating flipping back and forth between test models because they are inconsistent or we might use different sorts of attributes.

First I renamed length to vertexLength for clarity. Then I normalized all the properties to always have a default array so they can't be undefined and that the size attributes will return 0 if there is no corresponding data. Then I added a method useAttributes which lets you prune attribute data you aren't using.

//mesh.js
useAttributes(attrNames){
    for(const [attrName, _attrSizeName] of Mesh.attributeOrdering){
        if(!attrNames.includes(attrName)){
            this[attrName] = null;
        }
    }
    return this;
}

This is useful because we can save some space but also it means the mesh can handle the attribute sizes and thus we don't need to pass that in to packMesh.

//buffer-utils.js
/**
 * 
 * @param {{ positions: Float32Array, colors?: Float32Array, uvs?: Float32Array, normals?: Float32Array, attributeLength: number, positionSize?: number, colorSize?: number, uvSize?: number, normalSize?: number, tangentSize?: number }} mesh 
 */
export function packMesh(mesh){
    const stride = (mesh.positionSize ?? 0) + (mesh.colorSize ?? 0) + (mesh.uvSize ?? 0) + (mesh.normalSize ?? 0) + (mesh.tangentSize ?? 0); //stride in terms of indices (not bytes, assume F32s)
    const buffer = new Float32Array(stride * mesh.attributeLength);

    const positionOffset = 0;
    const colorOffset = mesh.positionSize ?? 0;
    const uvOffset = colorOffset + (mesh.colorSize ?? 0);
    const normalOffset = uvOffset + (mesh.uvSize ?? 0);
    const tangentOffset = uvOffset + (mesh.uvSize ?? 0);

    for(let i = 0; i < mesh.attributeLength; i++){
        packAttribute(buffer, mesh.positions, i, positionOffset, mesh.positionSize, stride);
        packAttribute(buffer, mesh.colors, i, colorOffset, mesh.colorSize, stride);
        packAttribute(buffer, mesh.uvs, i, uvOffset, mesh.uvSize, stride);
        packAttribute(buffer, mesh.normals, i, normalOffset, mesh.normalSize, stride);
        packAttribute(buffer, mesh.tangents, i, tangentOffset, mesh.tangentSize, stride);
    }

    return buffer;
}

So now we can use our surface mesh like this:

{
    const mesh = new Mesh(surfaceGrid(1, 1))
        .useAttributes(["positions", "uvs"]);
    const { vertexBuffer, indexBuffer } = uploadMesh(this.#device, mesh, {
        label: "floor-mesh"
    });
    this.#meshInfos.set("floor", { vertexBuffer, indexBuffer, mesh });
}

And get the vertexBufferLayout like this (#meshes was renamed to #meshContainers because I kept thinking it contained Mesh objects):

const vertexBufferLayout = getVertexBufferLayout(this.#meshContainers.get("floor").mesh);

Baking transforms

The next thing I want to do is allow us to "bake" transforms. That is, directly modify the mesh position data. This can happen during a loading phase of the pipeline if necessary but it allows us to normalize the meshes better. We already do some normalization for scale to shrink everything down to a unit volume. Baking does a similar thing. Since the models to start at 0 they tend to occupy the upper right distant octant (3d quadrant). We'd rather have them always start centered so what we can do is apply that transform and then bake it.

//mesh.js
bakeTransforms(){
    const modelMatrix = this.getModelMatrix();
    const transformedPositions = chunk(this.positions, this.positionSize)
        .map(values => {
            const lengthToPad = 4 - values.length;
            switch(lengthToPad){
                case 1:{
                    return [...values, 1.0]
                }
                case 2:{
                    return [...values, 0.0, 1.0];
                }
                case 3: {
                    return [...values, 0.0, 0.0, 1.0];
                }
                default: {
                    return [0.0, 0.0, 0.0, 1.0];
                }
            }
        })
        .map(values => multiplyMatrixVector(values, modelMatrix))
        .toArray();
    //collect
    const newPositionsBuffer = new Float32Array(this.vertexLength * this.positionSize);
    for(let i = 0; i < transformedPositions.length; i++){
        newPositionsBuffer.set(transformedPositions[i].slice(0, this.positionSize), i * this.positionSize)
    }
    this.positions = newPositionsBuffer;
    this.resetTransforms();
    return this;
}

We have to do a little bit of transformation to deal with different position sizes and we need to apply the matrix point-by-point which means we have to re-collect the points into vectors from the linear positions property which is kinda annoying. Then we create a new position buffer and assign it. We should also remove the transforms since they are now applied to the data itself, it wouldn't make sense to keep them around.

The iterator helper to chunk:

//iterator-utils.js
export function* chunk(iterator, size) {
    let chunk = new Array(size);
    let i = 0;
    for (const element of iterator) {
        chunk[i] = element;
        i++;
        if (i === size) {
            yield chunk;
            chunk = new Array(size);
            i = 0;
        }
    }
    if (i > 0) {
        yield chunk;
    }
}

The intermediate Float32Arrays are annoying to deal with but it's fine, I don't think it's a performance problem just a verbose code one.

Now that we can bake transforms, when we normalize positions we should also center the model.

//mesh.js
/**
 * Normalizes positions to be unit volume and centers
 * @param {{ scale?: boolean, center?: boolean }} options
 * @returns 
 */
normalizePositions(options = {}){
    const shouldCenter = options.center ?? true;
    const shouldScale = options.scale ?? true;
    const max = new Array(this.positionSize).fill(-Infinity);
    const min = new Array(this.positionSize).fill(Infinity);
    for(let i = 0; i < this.vertexLength; i++){
        for(let j = 0; j < this.positionSize; j++){
            const coord = this.#positions[i * this.positionSize + j];
            if(coord > max[j]){
                max[j] = coord
            }
            if(coord < min[j]){
                min[j] = coord;
            }
        }
    }

    const length = subtractVector(max, min);
    const maxLength = Math.max(...length);

    let currentCenter;
    if(shouldScale){
        for(let i = 0; i < this.positions.length; i++){
            this.#positions[i] /= maxLength;
        }
        currentCenter = addVector(divideVector(min, maxLength), divideVector(divideVector(length, maxLength), 2));
    } else {
        currentCenter = addVector(min, divideVector(length, 2));
    }
    if(shouldCenter){
        for (let i = 0; i < this.positions.length; i++) {
            const dimension = i % this.positionSize;
            this.#positions[i] -= currentCenter[dimension];
        }
    }
    return this;
}

To reiterate, this takes the min and max along each direction (previously there was a bug here where I always started at 0), finds the distance between the min and max to get the max length per dimension. To normalize we divide everything by the maxLength. To center we need to get the center point which is the min + length / 2 per dimension. If we scaled then we need to divide by max length to get the updated min and length. Once we have the center we subtract it from each point. This will allow us to easies center non-normalized models. I removed the parameter to set the scale value because we can now use the bakeTransforms to do that.

We can take extra stuff out of uploadObj, rename and move it to data-utils.js. We'll use the mesh directly to add transforms and normalization.

//data-utils.js
/**
 * Loads an .obj file
 * @param {GPUDevice} device 
 * @param {string} url 
 * @param {{ color?: [number, number, number, number], reverseWinding?: boolean, label?: string }} options
 * @returns 
 */
export async function fetchObjMesh(url, options = {}) {
    const response = await fetch(url);
    if (!response.ok) throw new Error(`Could not fetch obj content from ${url}`);
    const objText = await response.text();
    const objContent = loadObj(objText, { color: options.color, reverseWinding: options.reverseWinding });
    const mesh = new Mesh(objContent);
    return mesh;
}

Finally we can get the teapot, scaled, centered and rotated when loaded

//gpu-engine.js
{
    const mesh = await fetchObjMesh("./objs/teapot-low.obj", { reverseWinding: true });
    mesh.useAttributes(["positions", "uvs"])
        .normalizePositions()
        .rotate({ x: -Math.PI / 2 })
        .bakeTransforms();
    const { vertexBuffer, indexBuffer } = await uploadMesh(this.#device, mesh, { label: "teapot" });
    this.#meshContainers.set("teapot", { vertexBuffer, indexBuffer, mesh });
}

Materials

So we can render a textured teapot but what about something else in the same scene? In this case we'll need to make sure that the textures get parameterized into bind groups. We'll map the meshes with a property called "material". For now this will just be the name of the texture.

I found some free textures and wired them up.

//gpu-engine.js
async initializeTextures(){
    this.#textures.set("marble", await uploadTexture(this.#device, "./img/marble-white/marble-white-base.jpg"));
    this.#textures.set("red-fabric", await uploadTexture(this.#device, "./img/red-fabric/red-fabric-base.jpg"));
    //...other stuff here
}

And associate them to meshes

async initializeMeshes(){
    {
        const mesh = await fetchObjMesh("./objs/teapot-low.obj", { reverseWinding: true });
        mesh.useAttributes(["positions", "uvs"])
            .normalizePositions()
            .rotate({ x: -Math.PI / 2 })
            .bakeTransforms()
+           .setMaterial("marble");
        const { vertexBuffer, indexBuffer } = await uploadMesh(this.#device, mesh, { label: "teapot" });
        this.#meshContainers.set("teapot", { vertexBuffer, indexBuffer, mesh });
    }
    {
        const mesh = new Mesh(surfaceGrid(2, 2))
            .useAttributes(["positions", "uvs"])
            .translate({ y: -0.24 })
+           .setMaterial("red-fabric");
        const { vertexBuffer, indexBuffer } = uploadMesh(this.#device, mesh, {
            label: "floor-mesh"
        });
        this.#meshContainers.set("floor", { vertexBuffer, indexBuffer, mesh });
    }
    // {

setMaterial is identical to set material but chainable. Lastly, when setting the texture bind group we pass in the mesh (which setMainBindgroups already has) so we can access the material

setMainTextureBindGroup(passEncoder, bindGroupLayouts, mesh){
    const textureBindGroup = this.#device.createBindGroup({
        layout: bindGroupLayouts.get("textures"),
        entries: [
            { binding: 0, resource: this.#samplers.get("main") },
-           { binding: 1, resource: this.#textures.get("earth").createView() }
+           { binding: 1, resource: this.#textures.get(mesh.material).createView() },
        ]
    });
    passEncoder.setBindGroup(1, textureBindGroup);
}

This is enough to render two things.

Of course without lighting this looks pretty bad. Next time we'll clean it up.

Code

https://github.com/ndesmic/geo/releases/tag/v0.5

WebGPU Engine from Scratch Part 4: Updating the Pipeline

ndesmic — Mon, 11 Aug 2025 19:00:00 +0000

For this chapter I wanted to take a pause. The code before was spelled out but could be a little more reusable. We'll clean it up now and add a few of the extra features which should get us to near parity with the old WebGL renderer. It'll be a hodge-podge of things.

Creating a texture upload helper

Creating textures whether they are 2d or cubemaps is almost identical so we can create a helper to do this.

//wgpu-utils.js
import { loadImage } from "./image-utils.js";

/**
 * Loads an image url, uploads to GPU and returns texture ref.
 * Cubemaps defined like [+X, -X, +Y, -Y, +Z, -Z]
 * @param {GPUDevicee} device 
 * @param {string | string[]} urlOrUrls 
 * @param {*} options 
 */
export async function uploadTexture(device, urlOrUrls, options) {
    const urls = [].concat(urlOrUrls);
    const images = await Promise.all(urls.map(url => loadImage(url)));

    const size = {
        width: images[0].width,
        height: images[0].height,
        depthOrArrayLayers: images.length
    };

    const texture = device.createTexture({
        size,
        dimension: "2d",
        format: `rgba8unorm`,
        usage: GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
    });

    images.forEach((img, layer) => {
        device.queue.copyExternalImageToTexture(
            {
                source: img,
                flipY: true
            },
            {
                texture,
                origin: [0, 0, layer]
            },
            {
                width: img.width,
                height: img.height,
                depthOrArrayLayers: 1
            }
        );
    });

    return texture;
}

Then everything gets a lot simpler

async initializeTextures(){
    this.#textures.set("earth", await uploadTexture(this.#device, "./img/earth.png"));
    this.#textures.set("space", await uploadTexture(this.#device, [
        "./img/space_right.png",
        "./img/space_left.png",
        "./img/space_top.png",
        "./img/space_bottom.png",
        "./img/space_front.png",
        "./img/space_back.png"
    ]));
}

options is kept incase we need them later.

Creating a mesh upload helper

We should do the same for meshes.

//wgpu-utils.js
/**
 * This version is incomplete, see version in example below!
 * @param {GPUDevice} device 
 * @param {Mesh} mesh 
 * @param {{ positionLength: number, uvLength?: number, label?: string }} options 
 */
export function uploadMesh(device, mesh, options){
    const vertices = packMesh(mesh, { positionLength: options.positionLength, uvLength: options.uvLength });

    const vertexBuffer = device.createBuffer({
        label: options.label,
        size: vertices.byteLength,
        usage: GPUBufferUsage.VERTEX | GPUBufferUsage.COPY_DST,
    });
    device.queue.writeBuffer(vertexBuffer, 0, vertices);

    const indexBuffer = device.createBuffer({
        label: `${options.label}-indices`,
        size: mesh.indices.byteLength,
        usage: GPUBufferUsage.INDEX | GPUBufferUsage.COPY_DST,
    });
    device.queue.writeBuffer(indexBuffer, 0, mesh.indices);

    return {
        vertexBuffer,
        indexBuffer
    };
}

Then meshes get simpler too

initializeMeshes(){
    {
        const mesh = new Mesh(uvSphere(8))
        const { vertexBuffer, indexBuffer } = uploadMesh(this.#device, mesh, { positionLength: 3, uvLength: 2, label: "earth-mesh" });
        this.#meshes.set("earth", { vertexBuffer, indexBuffer, mesh });
    }
    {
        const mesh = new Mesh(quad());
        const { vertexBuffer, indexBuffer } = uploadMesh(this.#device, mesh, { positionLength: 3, label: "background-mesh" });
        this.#meshes.set("background", { vertexBuffer, indexBuffer, mesh });
    }
}

However we should also apply the optimization from the cube map chapter to reduce the screen quad to a single triangle:

//mesh-generator.js
export function screenTri(){
    return {
        positions: new Float32Array([
            -1.0, -1.0,
            3.0, -1.0,
            -1.0, 3.0
        ]),
        uvs: new Float32Array([
            0.0, -1.0,
            3.0, 1.0,
            0.0, 3.0,
        ]),
        indices: [0,1,2],
        length: 3
    }
}

We could even embed this in the shader itself (something WebGL couldn't do!) but that gets inconsistent with the mesh pipeline (we'd need to invent a virtual mesh or something) so we'll leave it external for now. But this has a problem because the index buffer needs to be a multiple of 4 (3xint16 = 6 bytes).

export function uploadMesh(device, mesh, options){
    const vertices = packMesh(mesh, { positionLength: options.positionLength, uvLength: options.uvLength });

    const vertexBuffer = device.createBuffer({
        label: options.label,
        size: vertices.byteLength,
        usage: GPUBufferUsage.VERTEX | GPUBufferUsage.COPY_DST,
    });
    device.queue.writeBuffer(vertexBuffer, 0, vertices);

    const paddedIndexSize = getPaddedSize(mesh.indices.byteLength, 4);
    const indexBuffer = device.createBuffer({
        label: `${options.label}-indices`,
        size: paddedIndexSize,
        usage: GPUBufferUsage.INDEX | GPUBufferUsage.COPY_DST,
    });
    let indexData = mesh.indices;
    if(paddedIndexSize !== mesh.indices.byteLength){
        indexData = new ArrayBuffer(paddedIndexSize);
        const indicesAsUint = new Uint16Array(indexData);
        indicesAsUint.set(mesh.indices, 0);
    }

    device.queue.writeBuffer(indexBuffer, 0, indexData);

    return {
        vertexBuffer,
        indexBuffer
    };
}

This will adhere to the padding for this case.

Simplify Pipelines

Pipelines are still pretty manual at this point so we can't do a lot to simplify them because the shader code and vertex descriptors are most of the actual code. But we can move to autobinding which should at least clean up the code a bit. We're really not doing anything fancy enough to warrant manual binding.

initializePipelines(){
    {
        const vertexBufferDescriptor = [{
            attributes: [
                {
                    shaderLocation: 0,
                    offset: 0,
                    format: "float32x3"
                },
                {
                    shaderLocation: 1,
                    offset: 12,
                    format: "float32x2"
                }
            ],
            arrayStride: 20,
            stepMode: "vertex"
        }];
        const shaderModule = this.#device.createShaderModule({
            code: `
                struct VertexOut {
                    @builtin(position) position : vec4<f32>,
                    @location(0) uv : vec2<f32>
                };
                struct Uniforms {
                    view_matrix: mat4x4<f32>,
                    projection_matrix: mat4x4<f32>,
                    model_matrix: mat4x4<f32>,
                    normal_matrix: mat3x3<f32>,
                    camera_position: vec3<f32>
                }

                @group(0) @binding(0) var<uniform> uniforms : Uniforms;
                @group(1) @binding(0) var main_sampler: sampler;
                @group(1) @binding(1) var earth_texture: texture_2d<f32>;
                @vertex
                fn vertex_main(@location(0) position: vec3<f32>, @location(1) uv: vec2<f32>) -> VertexOut
                {
                    var output : VertexOut;
                    output.position =  uniforms.projection_matrix * uniforms.view_matrix * uniforms.model_matrix * vec4<f32>(position, 1.0);
                    output.uv = uv;
                    return output;
                }
                @fragment
                fn fragment_main(fragData: VertexOut) -> @location(0) vec4<f32>
                {
                    return textureSample(earth_texture, main_sampler, fragData.uv);
                }
            `
        });
        const pipelineDescriptor = {
            label: "main-pipeline",
            vertex: {
                module: shaderModule,
                entryPoint: "vertex_main",
                buffers: vertexBufferDescriptor
            },
            fragment: {
                module: shaderModule,
                entryPoint: "fragment_main",
                targets: [
                    { format: "rgba8unorm" }
                ]
            },
            primitive: {
                topology: "triangle-list",
                frontFace: "ccw",
                cullMode: "back"
            },
            layout: "auto"
        };
        const pipeline = this.#device.createRenderPipeline(pipelineDescriptor);
        this.#pipelines.set("main", {
            pipeline,
            bindGroupLayouts: new Map([
                ["uniforms", pipeline.getBindGroupLayout(0)],
                ["textures", pipeline.getBindGroupLayout(1)]
            ]),
            bindMethod: this.setMainBindGroups.bind(this)
        });
    }
    {
        const vertexBufferDescriptor = [{
            attributes: [
                {
                    shaderLocation: 0,
                    offset: 0,
                    format: "float32x3"
                }
            ],
            arrayStride: 12,
            stepMode: "vertex"
        }];
        const shaderModule = this.#device.createShaderModule({
            code: `
                struct VertexOut {
                    @builtin(position) frag_position : vec4<f32>,
                    @location(0) clip_position: vec4<f32>
                };
                @group(0) @binding(0) var<uniform> inverse_view_matrix: mat4x4<f32>;
                @group(1) @binding(0) var main_sampler: sampler;
                @group(1) @binding(1) var space_texture: texture_cube<f32>;
                @vertex
                fn vertex_main(@location(0) position: vec3<f32>) -> VertexOut
                {
                    var output : VertexOut;
                    output.frag_position =  vec4(position, 1.0);
                    output.clip_position = vec4(position, 1.0);
                    return output;
                }
                @fragment
                fn fragment_main(fragData: VertexOut) -> @location(0) vec4<f32>
                {
                    var pos = inverse_view_matrix * fragData.clip_position;
                    return textureSample(space_texture, main_sampler, pos.xyz);
                }
            `
        });
        const pipelineDescriptor = {
            label: "background-pipeline",
            vertex: {
                module: shaderModule,
                entryPoint: "vertex_main",
                buffers: vertexBufferDescriptor
            },
            fragment: {
                module: shaderModule,
                entryPoint: "fragment_main",
                targets: [
                    { format: "rgba8unorm" }
                ]
            },
            primitive: {
                topology: "triangle-list",
                frontFace: "ccw",
                cullMode: "back"
            },
            layout: "auto"
        };
        const pipeline = this.#device.createRenderPipeline(pipelineDescriptor);
        this.#pipelines.set("background", {
            pipeline: pipeline,
            bindGroupLayouts: new Map([
                ["uniforms", pipeline.getBindGroupLayout(0)],
                ["textures", pipeline.getBindGroupLayout(1)]
            ]),
            bindMethod: this.setBackgroundBindGroups.bind(this)
        });
    }
}

This reduces the code by a fair bit but we lose a little control. Unclear if we'll need to add it back. The other thing I want to do here is allow the shaders to be external because I don't like dealing with inline strings.

//wgpu-utils.js
/**
 * 
 * @param {GPUDevice} device 
 * @param {string} url 
 * @param {*} options 
 * @returns 
 */
export async function uploadShader(device, url, options = {}) {
    const response = await fetch(url);
    if (!response.ok) throw new Error(`Could not fetch text content from ${url}`);
    const code = await response.text();

    const shaderModule = device.createShaderModule({
        label: options.label ?? url,
        code
    });

    const compilationInfo = await shaderModule.getCompilationInfo();
    if(compilationInfo.messages.length > 0){
        throw new Error(`Failed to compile shader ${url}.`);
    }

    return shaderModule;
}

I'm not really handling the error messages because those are already sent to the console in Chrome, but we could do some error formatting if we really wanted. Then we can clean up a little more:

-const shaderModule = this.#device.createShaderModule({
-   code: `
-       ...
-   `
-})

+const shaderModule = await uploadShader(this.#device, "./shaders/shader.wgsl");

Optimizing drawing

As in the cubemap chapter we want to optimize the drawing so that we aren't drawing over the background but only filling the pixel once. For that we need depth buffer.

//gpu-engine.js - initializeTextures
async initializeTextures(){
    //...
    this.#textures.set("depth", this.#device.createTexture({
        size: {
            width: this.#canvas.width,
            height: this.#canvas.height,
            depthOrArrayLayers: 1
        },
        format: "depth32float",
        usage: GPUTextureUsage.RENDER_ATTACHMENT
    }));
}

Here we're creating a new texture that's the same size as the canvas and giving it a 32-bit depth value (essentially a black/white image). This will be the depth buffer. I chose this over depth24plus because apparently it's more compatible and I was having issues debugging. We need to set it so that if we try to draw something behind what's already on screen we cancel that pixel (you want less-equal rather than less because the background is always drawn at max value 1.0 which is also the clear value and we want that to stick).

//gpu-engine.js initializePipelines
const pipelineDescriptor = {
    label: "main-pipeline",
    vertex: {
        module: shaderModule,
        entryPoint: "vertex_main",
        buffers: vertexBufferDescriptor
    },
    fragment: {
        module: shaderModule,
        entryPoint: "fragment_main",
        targets: [
            { format: "rgba8unorm" }
        ]
    },
    primitive: {
        topology: "triangle-list",
        frontFace: "ccw",
        cullMode: "back"
    },
+   depthStencil: {
+       depthWriteEnabled: true,
+       depthCompare: "less-equal",
+       format: "depth32float"
+   },
    layout: "auto"
};

We add the depth stencil to the pipeline (both pipelines! only one shown!). Lastly we actually need to use it in the pipeline. This poses a bit of a problem because we want to clear on the first pass and load on the subsequent passes which means that render passes must now be 1-to-1 with pipelines so we need to shuffle the code around a bit.

render() {
    const commandEncoder = this.#device.createCommandEncoder({
        label: "main-command-encoder"
    });
    const camera = this.#cameras.get("main");
    let isFirstPass = true;
    const depthView = this.#textures.get("depth").createView();
    for(const [pipelineName, meshNames] of this.#pipelineMesh.entries()){
        const passEncoder = commandEncoder.beginRenderPass({
            label: `${pipelineName}-render-pass`,
            colorAttachments: [
                {
                    storeOp: "store",
                    loadOp: "load",
                    view: this.#context.getCurrentTexture().createView()
                }
            ],
            depthStencilAttachment: {
                view: depthView,
                depthClearValue: 1.0,
                depthStoreOp: "store",
                depthLoadOp: isFirstPass ? "clear" : "load"
            }
        });
        const pipelineContainer = this.#pipelines.get(pipelineName);
        passEncoder.setPipeline(pipelineContainer.pipeline);
        for(const meshName of meshNames){
            const meshContainer = this.#meshes.get(meshName);
            pipelineContainer.bindMethod(passEncoder, pipelineContainer.bindGroupLayouts, camera, meshContainer.mesh);
            passEncoder.setVertexBuffer(0, meshContainer.vertexBuffer);
            passEncoder.setIndexBuffer(meshContainer.indexBuffer, "uint16");
            passEncoder.drawIndexed(meshContainer.mesh.indices.length);
        }
        passEncoder.end();
        isFirstPass = false;
    }
    this.#device.queue.submit([commandEncoder.finish()]);
}

The first time through we need to clear the depth buffer (1.0 means the deepest values) and then on subsequent passes we will use the existing values. Since it compares less-equal when it writes the pixel, if the new value is less or equal, it will overwrite and if not it will discard. Be aware that you should only call createView() on the depth texture once. If you do it per pass it seems to reset the values and that mistake took a while to debug. One last things is the flip the order so the Earth is drawn first.

initializePipelineMesh(){
    this.#pipelineMesh.set("main", ["earth"]);
    this.#pipelineMesh.set("background", ["background"]);
}

Now there is no overdraw. The Earth is drawn and the background is drawn in all the space around that has the default 1.0 (the depth is given by the builtin(position)'s z value).

Debugging the depth test

While implementing this I kept getting the wrong result and it's actually tricky to debug. What we want is to be able to get an black/white image that represents the depth texture so we can see if it actually wrote correctly. The problem is that the depth test texture isn't directly accessible.

//debug utils
export function setupExtractDepthBuffer(device, context) {
    const vertices = new Float32Array([
        -1.0, -1.0,
        3.0, -1.0,
        -1.0, 3.0
    ]);

    const vertexBuffer = device.createBuffer({
        label: "depth-buffer-export-tri",
        size: vertices.byteLength,
        usage: GPUBufferUsage.VERTEX | GPUBufferUsage.COPY_DST,
    });
    device.queue.writeBuffer(vertexBuffer, 0, vertices);

    const vertexBufferDescriptor = [{
        attributes: [
            {
                shaderLocation: 0,
                offset: 0,
                format: "float32x2"
            }
        ],
        arrayStride: 8,
        stepMode: "vertex"
    }];

    const shaderModule = device.createShaderModule({
        label: "depth-buffer-export-shader",
        code: `
            struct VertexOut {
                @builtin(position) frag_position : vec4<f32>,
                @location(0) clip_position: vec4<f32>,
                @location(2) uv: vec2<f32>
            };

            @group(0) @binding(0) var depthSampler: sampler;
            @group(0) @binding(1) var depthTex: texture_depth_2d;

            @vertex
            fn vertex_main(@location(0) position: vec2<f32>) -> VertexOut
            {
                var output : VertexOut;
                output.frag_position =  vec4(position, 1.0, 1.0);
                output.clip_position = vec4(position, 1.0, 1.0);
                output.uv = position.xy * 0.5 + vec2<f32>(0.5, 0.5);
                return output;
            }

            @fragment
            fn fragment_main(fragData: VertexOut) -> @location(0) vec4<f32> {
                let depth = textureSample(depthTex, depthSampler, fragData.uv);
                let gamma_depth = pow(depth, 10.0);
                return vec4<f32>(gamma_depth, gamma_depth, gamma_depth, 1.0); // grayscale output
            }
        `
    });

    const sampler = device.createSampler({
        compare: undefined
    });

    const bindGroupLayout = device.createBindGroupLayout({
        entries: [
            {
                binding: 0,
                visibility: GPUShaderStage.FRAGMENT,
                sampler: {
                    type: "non-filtering"
                }
            },
            {
                binding: 1,
                visibility: GPUShaderStage.FRAGMENT,
                texture: {
                    sampleType: "depth",
                    viewDimension: "2d"
                }
            }
        ]
    });

    const pipelineLayout = device.createPipelineLayout({
        label: "depth-buffer-export-pipeline-layout",
        bindGroupLayouts: [
            bindGroupLayout
        ]
    });

    const pipelineDescriptor = {
        label: "depth-buffer-export-pipeline",
        vertex: {
            module: shaderModule,
            entryPoint: "vertex_main",
            buffers: vertexBufferDescriptor
        },
        fragment: {
            module: shaderModule,
            entryPoint: "fragment_main",
            targets: [
                { format: "rgba8unorm" }
            ]
        },
        primitive: {
            topology: "triangle-list"
        },
        layout: pipelineLayout
    };

    const pipeline = device.createRenderPipeline(pipelineDescriptor);

    return (depthBufferView) => {
        const commandEncoder = device.createCommandEncoder({
            label: "depth-buffer-export-command-encoder"
        });

        const passEncoder = commandEncoder.beginRenderPass({
            label: `depth-buffer-export-render-pass`,
            clearValue: { r: 0, g: 0, b: 0, a: 1 },
            colorAttachments: [
                {
                    storeOp: "store",
                    loadOp: "clear",
                    view: context.getCurrentTexture().createView()
                }
            ]
        });

        const textureBindGroup = device.createBindGroup({
            label: "depth-buffer-export-bind-group",
            layout: bindGroupLayout,
            entries: [
                { binding: 0, resource: sampler },
                { binding: 1, resource: depthBufferView },
            ]
        });

        passEncoder.setPipeline(pipeline);
        passEncoder.setBindGroup(0, textureBindGroup);
        passEncoder.setVertexBuffer(0, vertexBuffer);
        passEncoder.draw(3);
        passEncoder.end();

        device.queue.submit([commandEncoder.finish()]);
    }
}

I won't explain this too much since we've already gone over similar pieces. The function returns a function. This is so we can setup the buffers in a setup command and not recreate them on each draw call. We call the function and then call the return value when we want to draw the depth buffer to the screen. We pass in the depth texture and sample from it. In order to do this we need to add a new usage flag for TEXTURE_BINDING to the depth texture:

this.#textures.set("depth", this.#device.createTexture({
    label: "depth-texture",
    size: {
        width: this.#canvas.width,
        height: this.#canvas.height,
        depthOrArrayLayers: 1
    },
    format: "depth32float",
    usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING //for debug
    }));

The key here is that we're sampling a texture_depth_2d which gives us a single float. Getting the various formats to agree was hard (and why I went with depth32float). I also scale the values let gamma_depth = pow(depth, 10.0); because otherwise they would be too small to see, you'd just get a white screen.

We set it up in intialize with

this.#extractDepthBuffer = setupExtractDepthBuffer(this.#device, this.#context);

We can call this at the bottom of the render method with (this won't be in the final code since it was for debugging)

this.#extractDepthBuffer(depthView);`

This will produce an image like the following

Where the darker spots are closer to the viewer and pure white is the furthest value we can see.

Setting up the screen recorder

Also for the sake of producing captures I wanted to re-add the screen capture functionality. This time I've set it to use Shift+R to make it easier to toggle.

//wc-geo.js

onRKeyPressed(e) {
    if (!e.shiftKey) return; //record function is shift+R
    this.#isRecording = !this.#isRecording;
    if (this.#isRecording) {
        this.dom.message.textContent = "Recording video";
        const stream = this.dom.canvas.captureStream(25);
        this.#mediaRecorder = new MediaRecorder(stream, {
            mimeType: 'video/webm;codecs=vp9'
        });
        this.#recordedChunks = [];
        this.#mediaRecorder.ondataavailable = e => {
            if (e.data.size > 0) {
                this.#recordedChunks.push(e.data);
            }
        };
        this.#mediaRecorder.start();
    } else {
        this.dom.message.textContent = "Recording stopped";
        this.#mediaRecorder.stop();
        setTimeout(() => {
            const blob = new Blob(this.#recordedChunks, {
                type: "video/webm"
            });
            downloadBlob(blob, "recording.webm");
        }, 0);
        setTimeout(() => {
            this.dom.message.textContent = "";
        }, 3000);
    }
}

Starting and stopping the simulation

One thing that would be useful is the ability to start and stop (pause/unpause) the engine. I do this with the Esc key:

//Remember to add to onKeyDown method switch!
onEscPressed(e){
    if (this.engine.isRunning) {
        this.engine.stop();
        this.dom.message.textContent = "Paused";
    } else {
        this.engine.start();
        this.setTemporaryMessage("Started");
    }
    return;
}

On the engine side

#raf;
#isRunning = false;

start() {
    this.#isRunning = true;
    this.renderLoop();
}
stop(){
    cancelAnimationFrame(this.#raf);
    this.#isRunning = false;
}
get isRunning(){
    return this.#isRunning;
}

Now when we press Esc the engine will toggle between paused and unpaused. Note that any changes to the camera from key events still happen, they just don't render until it's running again.

OBJ Files

The last thing I want to add is .obj support so we can load some actual user created models and not just geometric shapes. We already covered most of this.

//obj=loader.js

/**
 * 
 * @param {string} txt 
 * @param {{ color?: [number, number, number, number], reverseWinding?: boolean}} options 
 * @returns 
 */
export function loadObj(txt, options = {}) {
    const positions = [];
    const normals = [];
    const uvs = [];
    const colors = [];
    const indices = [];
    const faceCombos = [];
    let positionSize = 3;
    let normalSize = 3;
    let colorSize = 4;
    let uvSize = 2;

    const lines = txt.split("\n");

    for (const line of lines) {
        const normalizedLine = line.trim();
        if (!normalizedLine || normalizedLine.startsWith("#")) continue;
        const parts = normalizedLine.split(/\s+/g);
        const values = parts.slice(1);

        switch (parts[0]) {
            case "v": {
                positions.push(values.map(x => parseFloat(x)));
                positionSize = values.length;
                break;
            }
            case "c": { //custom extension
                if (!options.color) {
                    colors.push(values.map(x => parseFloat(x)));
                    colorSize = values.length;
                }
                break;
            }
            case "vt": {
                uvs.push(values.map(x => parseFloat(x)));
                uvSize = values.length;
                break;
            }
            case "vn": {
                normals.push(values.map(x => parseFloat(x)));
                normalSize = values.length;
                break;
            }
            case "f": {
                if(values[0].includes("/")){
                    faceCombos.push(values.map(value => value.split("/").map(x => parseFloat(x) - 1)));
                } else {
                    const oneBasedIndicies = values.map(x => parseFloat(x) - 1);
                    indices.push(
                        ...(options.reverseWinding ? oneBasedIndicies.reverse() : oneBasedIndicies)
                    );
                }
                break;
            }
        }
    }

    if(faceCombos.length === 0){
        return {
            positions: positions.flat(Infinity),
            uvs: uvs.flat(Infinity),
            normals: normals.flat(Infinity),
            indices,
            colors: colors.flat(Infinity),
            length: positions.length,
            positionSize,
            uvSize,
            colorSize,
            normalSize,
        };
    }

    //For multi value faces we need to get position/uv/normal combos and put each into the pool of vertices

    const comboPositions = [];
    const comboUvs = [];
    const comboNormals = [];
    const comboIndices = [];
    let startIndex = 0;

    for(const combo of faceCombos){
        for(const attrIndex of combo){
            comboPositions.push(positions[attrIndex[0]]);
            comboUvs.push(uvs[attrIndex[1]]);
            comboNormals.push(normals[attrIndex[2]]);
        }
        if (combo.length === 3) {
            if(options.reverseWinding){
                comboIndices.push(startIndex + 2, startIndex + 1, startIndex);
            } else {
                comboIndices.push(startIndex, startIndex + 1, startIndex + 2);
            }

        } else if(combo.length === 4){
            if(options.reverseWinding){
                comboIndices.push(
                    startIndex + 2,
                    startIndex + 1,
                    startIndex,
                    startIndex + 3,
                    startIndex + 2,
                    startIndex);
            } else {
                comboIndices.push(
                    startIndex, 
                    startIndex + 1, 
                    startIndex + 2, 
                    startIndex, 
                    startIndex + 2, 
                    startIndex + 3);
            }
        }
        startIndex += combo.length;
    }

    return {
        positions: comboPositions.flat(Infinity),
        uvs: comboUvs.flat(Infinity),
        normals: comboNormals.flat(Infinity),
        indices: comboIndices.flat(Infinity),
        colors: [],
        length: comboPositions.length,
        positionSize,
        uvSize,
        colorSize: 0,
        normalSize
    };
}

Starting with the existing code this has been expanded a bit. It now properly handles faces that use multiple indices like

#square.obj
v -0.5 -0.5 0
v 0.5 -0.5 0
v 0.5 0.5 0
v -0.5 0.5 0

vt 0 0
vt 1 0
vt 1 1
vt 0 1

vn 0 0 -1

f 1/1/1 2/2/1 3/3/1 4/4/1

It can also handle models that are made with quads by doing some very simple tessellation into 2 triangles and allows for variable sizes vertices (this info will probably come in handy later if we want to autobuild pipeline vertex buffer descriptions based on the geometry). Admittedly, not the prettiest code but it works. I also made an an uploadObj helper.

//wpgu-utils.ts
/**
 * Loads an .obj file
 * @param {GPUDevice} device 
 * @param {string} url 
 * @param {{ color?: [number, number, number, number], reverseWinding?: boolean, label?: string, normalizePositions?: boolean | number }} options
 * @returns 
 */
export async function uploadObj(device, url, options = {}){
    const response = await fetch(url);
    if (!response.ok) throw new Error(`Could not fetch obj content from ${url}`);
    const objText = await response.text();
    const objContent = loadObj(objText, { color: options.color, reverseWinding: options.reverseWinding });
    const mesh = new Mesh(objContent);
    if(options.normalizePositions){
        mesh.normalizePositions(typeof options.normalizePositions === "number" ? options.normalizePositions : 1);
    }
    return {
        mesh,
        ...uploadMesh(device, mesh, { 
            positionSize: objContent.positionSize, 
            uvSize: objContent.uvSize, 
            normalSize: objContent.normalSize, 
            colorSize: objContent.colorSize,
            label: options.label ?? url
        })
    };
}

This loads the model and then shoves it through uploadMesh. It's not actually a great API in practice though because you don't have control over the Mesh. I added the ability to change the scale with normalizePositions but it probably also makes sense to have rotations too because the teapot model is Z up. This will probably be reworked into two calls to get the Mesh and then upload. But we can load the teapot and other models now which means we can work with more interesting things.

Code

https://github.com/ndesmic/geo/releases/tag/v0.5

WebGPU Engine from Scratch Part 3: Textures

ndesmic — Mon, 28 Jul 2025 19:00:00 +0000

Next let's explore textures. We'll try to recreate the globe from the WebGL chapter.

The first thing we need to do is render the sphere and that can do with a little bit of cleanup. The newly named "mesh-generator" will build a uv sphere but I've cleaned up the implementation and removed some features like centroids that we no longer care about.

Creating a texture

Creating a texture is straightforward:

async initializeTextures(){
    const image = await loadImage("./img/earth.png");
    const textureSize = {
        width: image.width,
        height: image.height,
        depthOrArrayLayers: 1
    };
    const texture = this.#device.createTexture({
        size: textureSize,
        dimension: '2d',
        format: `rgba8unorm`,
        usage: GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
    });
    this.#device.queue.copyExternalImageToTexture(
        {
            source: image
        }, 
        {
            texture: texture,
            mipLevel: 0
        },
        textureSize
    );
    this.#textures.set("earth", texture);
}

We first load in the image as an Image. We get the texture dimensions width (U), height (V) and depthOrArrayLayers (depth). Then we create a texture on the GPU with those dimensions of a 2d type with the right format and usage flags. Then we upload the texture to the GPU using copyExternalImageToTexture and tell it what mip level it represents. Then we do the normal bookkeeping thing of sticking it in a map so we can reference it later.

We also create a "sampler" which is an object that allows us to sample for the texture (eg applies filtering):

initializeSamplers(){
    const sampler = this.#device.createSampler({
        addressModeU: "repeat",
        addressModeV: "repeat",
        magFilter: "linear",
        minFilter: "nearest"
    });
    this.#samplers.set("main", sampler);
}

Once we have these two things then we can actually bind them so they appear in the shader. First we create the bindGroupLayout:

const textureBindGroupLayout = this.#device.createBindGroupLayout({
    label: "main-texture-bind-group-layout",
    entries: [
        {
            binding: 0,
            visibility: GPUShaderStage.FRAGMENT,
            sampler: {
                type: "filtering"
            }
        },
        {
            binding: 1,
            visibility: GPUShaderStage.FRAGMENT,
            texture: {
                sampleType: "float",
                viewDimension: "2d",
                multisampled: false
            }
        },
    ]
});

This is for the texture and the sampler. Note that they are only used in the fragment shader so we only need that usage. You can look up the rest of the options but they aren't important right now, these are just the basics for 2d color textures.

Make sure add it to the pipeline descriptor (not shown)!

We can reference it in the shader and sample from it using the UV coordinates:

struct VertexOut {
    @builtin(position) position : vec4<f32>,
    @location(0) uv : vec2<f32>
};
struct Uniforms {
    view_matrix: mat4x4<f32>,
    projection_matrix: mat4x4<f32>,
    model_matrix: mat4x4<f32>,
    normal_matrix: mat3x3<f32>,
    camera_position: vec3<f32>
}

@group(0) @binding(0) var<uniform> uniforms : Uniforms;
@group(1) @binding(0) var main_sampler: sampler;
@group(1) @binding(1) var earth_texture: texture_2d<f32>;
@vertex
fn vertex_main(@location(0) position: vec3<f32>, @location(1) uv: vec2<f32>) -> VertexOut
{
    var output : VertexOut;
    output.position =  uniforms.projection_matrix * uniforms.view_matrix * uniforms.model_matrix * vec4<f32>(position, 1.0);
    output.uv = uv;
    return output;
}
@fragment
fn fragment_main(fragData: VertexOut) -> @location(0) vec4<f32>
{
    return textureSample(earth_texture, main_sampler, fragData.uv);
}

Last we need to actually bind it to the render pass:

setMainTextureBindGroup(passEncoder, bindGroupLayouts){
    const textureBindGroup = this.#device.createBindGroup({
        layout: bindGroupLayouts.get("textures"),
        entries: [
            { binding: 0, resource: this.#samplers.get("main") },
            { binding: 1, resource: this.#textures.get("earth").createView() },
        ]
    });
    passEncoder.setBindGroup(1, textureBindGroup);
}

The method is called during the render pass.

Input

In order to really see the textures (especially for debug) we want to be able to move things on the screen to see them from different angles so need to hook up the drag events.

//wc-geo.js

attachEvents() {
    document.body.addEventListener("keydown", this.onKeyDown);
    this.dom.canvas.addEventListener("pointerdown", this.onPointerDown);
    this.dom.canvas.addEventListener("wheel", this.onWheel);
}
onKeyDown(e){
    if(!this.#engineReady) return;
    switch (e.code) {
        case "KeyA": {
            this.engine.cameras.get("main").panBy({ x: 0.1 });
            break;
        }
        case "KeyD": {
            this.engine.cameras.get("main").panBy({ x: -0.1 });
            break;
        }
        case "KeyW": {
            this.engine.cameras.get("main").panBy({ z: 0.1 });
            break;
        }
        case "KeyS": {
            this.engine.cameras.get("main").panBy({ z: -0.1 });
            break;
        }
        case "NumpadAdd": {
            this.engine.cameras.get("main").zoomBy(2);
            break;
        }
        case "NumpadSubtract": {
            this.engine.cameras.get("main").zoomBy(0.5);
            break;
        }
    }
    e.preventDefault();
}
onPointerDown(e){
    if (!this.#engineReady) return;
    this.#initialPointer = [e.offsetX, e.offsetY];
    this.#initialCameraPosition = this.engine.cameras.get("main").getPosition();
    this.dom.canvas.setPointerCapture(e.pointerId);
    this.dom.canvas.addEventListener("pointermove", this.onPointerMove);
    this.dom.canvas.addEventListener("pointerup", this.onPointerUp);
}
onPointerMove(e){
    const pointerDelta = [
        e.offsetX - this.#initialPointer[0],
        e.offsetY - this.#initialPointer[1]
    ];
    const radsPerWidth = (180 / DEGREES_PER_RADIAN) / this.#width;
    const xRads = pointerDelta[0] * radsPerWidth;
    const yRads = pointerDelta[1] * radsPerWidth * (this.#height / this.#width);
    this.engine.cameras.get("main").setPosition(this.#initialCameraPosition);
    this.engine.cameras.get("main").orbitBy({ long: xRads, lat: yRads });
}
onPointerUp(e){
    this.dom.canvas.removeEventListener("pointermove", this.onPointerMove);
    this.dom.canvas.removeEventListener("pointerup", this.onPointerUp);
    this.dom.canvas.releasePointerCapture(e.pointerId);
}
onWheel(e){
    if (!this.#engineReady) return;
    e.preventDefault();
    const delta = e.deltaY / 1000;
    this.engine.cameras.get("main").orbitBy({ radius: delta });

This shouldn't be surprising as it's almost entirely copied from the old engine with a few slight changes for naming. We expose the camera via the engine and move it around. This made more sense in the component because that's where the DOM stuff happens and I'd rather not push any more of that down than absolutely necessary.

Issues

The first thing we see is that things get messed up. It's much more apparent when you move the mouse around what's going on.

The back part of the sphere is bleeding through the front. Luckily we know how to solve this: backface culling. We can add this in the pipeline where the primitive is defined:

primitive: {
    topology: "triangle-list",
+   frontFace: "ccw",
+   cullMode: "back"
}

We give it the direction of front and cull the back.

Now it looks a little more consistent but it's upside down. We need to flip the texture because texture coordinates are actually opposite of the image. We can fix this where we upload the texture to the GPU as it gives us a handy flag:

this.#device.queue.copyExternalImageToTexture(
    {
        source: image,
+       flipY: true
    }, 
    {
        texture: texture,
        mipLevel: 0
    },
    textureSize
);

And that fixes it:

Cubemaps

We also want to try out cubemaps with WebGPU so lets to that. Finding a good space cubemap that's free is actually a bit hard but there's a program called spacescape that can generate simple ones, so I used that. I exported the 6 sides.

Update to Meshes

I'm going to do this in 2 passes. The first pass will draw the background and the second pass will draw the Earth. So we need a new mesh for the background which will be a screen-space quad. There's probably a smarter way to just transform the old quad but it's no too important to get clever right away. Then we add new code the initializeMeshes (the {} just lets us block scope the variables so we can reuse them, I also renamed the earth sphere mesh to "Earth" because "background" was a stupid name)

//...sphere mesh
{
    const mesh = new Mesh(screenQuad());
    const vertices = packMesh(mesh, { positions: 2 });
    const vertexBuffer = this.#device.createBuffer({
        size: vertices.byteLength,
        usage: GPUBufferUsage.VERTEX | GPUBufferUsage.COPY_DST,
    });
    this.#device.queue.writeBuffer(vertexBuffer, 0, vertices);
    const indexBuffer = this.#device.createBuffer({
        size: mesh.indices.byteLength,
        usage: GPUBufferUsage.INDEX | GPUBufferUsage.COPY_DST,
    });
    this.#device.queue.writeBuffer(indexBuffer, 0, mesh.indices);
    this.#meshes.set("background", {
        vertexBuffer,
        indexBuffer,
        mesh
    });
}

/**
 * Generates a screen space quad. For UI/backgrounds
 * @returns {Mesh}
 */
export function screenQuad() {
    return {
        positions: new Float32Array([
            -1.0, -1.0,
            1.0, -1.0,
            1.0, 1.0,
            -1.0, 1.0,
        ]),
        uvs: new Float32Array([
            0.0, 1.0,
            1.0, 1.0,
            1.0, 0.0,
            0.0, 0.0,
        ]),
        indices: [0, 1, 2, 0, 2, 3],
        length: 4
    }
}

One thing to be careful about here: the number of attributes (position/color/normals etc.) and the number indices is different for the screen quad. We have 4 points but 6 indices because we reuse them so in packMesh we need to be careful which length we use. length on the mesh object is the attribute length, the index length is just mesh.indices.length.

Updates to Textures

For the textures we add a new cubemap.

//cubemap -> [+X, -X, +Y, -Y, +Z, -Z]
const cubeSideImages = await Promise.all([
    loadImage("./img/space_right.png"),
    loadImage("./img/space_left.png"),
    loadImage("./img/space_top.png"),
    loadImage("./img/space_bottom.png"),
    loadImage("./img/space_front.png"),
    loadImage("./img/space_back.png"),
]);
const cubemapSize = {
    width: cubeSideImages[0].width,
    height: cubeSideImages[0].height,
    depthOrArrayLayers: 6
};
const cubemap = this.#device.createTexture({
    size: cubemapSize,
    dimension: "2d",
    format: `rgba8unorm`,
    usage: GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING
});
cubeSideImages.forEach((img, layer) => {
    this.#device.queue.copyExternalImageToTexture(
        {
            source: img,
            flipY: true
        },
        {
            texture: cubemap,
            origin: [0, 0, layer]
        },
        {
            width: img.width,
            height: img.height,
            depthOrArrayLayers: 1
        }
    );
});
this.#textures.set("space", cubemap);

Very similar to a normal 2d texture. We take the first images width and height because they are all the same and we create a texture with 6 layers. When copying we copy each image but we have to assign it to the right layer with the origin property. The 3rd value is the layer, the other two are the texture coordinate. Note that the depthOrArrayLayers for copying is still 1 because the image itself is just one layer and it's being copied into a multilayer texture.

Updates to Pipelines

Since we're doing 2 passes, we need to 2 pipelines so we have to define another one (bleh, there are ways around this but this is the simplest possible thing to do).

const vertexBufferDescriptor = [{
    attributes: [
        {
            shaderLocation: 0,
            offset: 0,
            format: "float32x2"
        }
    ],
    arrayStride: 8,
    stepMode: "vertex"
}];
const shaderModule = this.#device.createShaderModule({
    code: `
    struct VertexOut {
        @builtin(position) frag_position : vec4<f32>,
        @location(0) clip_position: vec4<f32>
    };
    @group(0) @binding(0) var<uniform> inverse_view_matrix: mat4x4<f32>;
    @group(1) @binding(0) var main_sampler: sampler;
    @group(1) @binding(1) var space_texture: texture_cube<f32>;
    @vertex
    fn vertex_main(@location(0) position: vec2<f32>) -> VertexOut
    {
        var output : VertexOut;
        output.frag_position =  vec4(position, 0.0, 1.0);
        output.clip_position = vec4(position, 0.0, 1.0);
        return output;
    }
    @fragment
    fn fragment_main(fragData: VertexOut) -> @location(0) vec4<f32>
    {
        var pos = inverse_view_matrix * fragData.clip_position;
        return textureSample(space_texture, main_sampler, pos.xyz);
    }
    `
});

The vertex buffer only uses float32x2s for screen space coordinates. The shader module is more interesting. If you look back at the previous chapter on the topic we used the inverse view matrix to get the cube map coordinate because we are taking a screen-space coordinate and transforming it back into the camera position. We need to pass the position value through the vertex shader. Note that the builtin "position" will automatically convert to pixel coordinates (eg 0 to 719 for 720px width) and not the clip space coordinates (-1.0 to 1.0) so we must make our own property in the struct to pass clip space coordinate through (position is still required by WGSL so we can't omit it). Once we have the clip space coordinates we can apply the inverse view to it. The resulting vector is the direction the camera is pointing at that particular fragment and we can use it to sample the cube map. Note that magnitude doesn't matter so we don't need to normalize or anything.

We're still using manual bind groups for now so let's make those.

//manually setting bind groups
const uniformBindGroupLayout = this.#device.createBindGroupLayout({
    label: "background-bind-group-layout",
    entries: [
        {
            binding: 0,
            visibility: GPUShaderStage.VERTEX | GPUShaderStage.FRAGMENT,
            buffer: {
                type: "uniform"
            }
        }
    ]
});
const textureBindGroupLayout = this.#device.createBindGroupLayout({
    label: "background-texture-bind-group-layout",
    entries: [
        {
            binding: 0,
            visibility: GPUShaderStage.FRAGMENT,
            sampler: {
                type: "filtering"
            }
        },
        {
            binding: 1,
            visibility: GPUShaderStage.FRAGMENT,
            texture: {
                sampleType: "float",
                viewDimension: "cube",
                multisampled: false
            }
        }
    ]
});

I renamed the previous bindgroups to not use the name environment since it sounded too much like "environment map" terminology. Instead all of the basic matrices are just called "uniforms" for lack of better name. We just have one uniform this time, the inverse view matrix. For textures we still need a sampler, and we'll pass the cubemap. Here we need to set the viewDimension to cube so that it knows how to use it.

The rest of the layout is the same. We don't really need the culling but it doesn't hurt.

const pipelineLayout = this.#device.createPipelineLayout({
    label: "background-pipeline-layout",
    bindGroupLayouts: [
        uniformBindGroupLayout,
        textureBindGroupLayout
    ]
});
const pipelineDescriptor = {
    label: "background-pipeline",
    vertex: {
        module: shaderModule,
        entryPoint: "vertex_main",
        buffers: vertexBufferDescriptor
    },
    fragment: {
        module: shaderModule,
        entryPoint: "fragment_main",
        targets: [
            { format: "rgba8unorm" }
        ]
    },
    primitive: {
        topology: "triangle-list",
        frontFace: "ccw",
        cullMode: "back"
    },
    layout: pipelineLayout
};
this.#pipelines.set("background", {
    pipeline: this.#device.createRenderPipeline(pipelineDescriptor),
    pipelineLayout,
    bindGroupLayouts: new Map([
        ["uniforms", uniformBindGroupLayout],
        ["textures", textureBindGroupLayout]
    ])
});

Update to BindGroups

Now the tricky part. What sucks is that bind groups are related to specific pipelines and so our methods to bind only work for the "main" pipeline. We need actual code to bind the new stuff only when the background pass happens.

First I'm going to create an association of meshes with pipelines.

initializePipelineMesh(){
    this.#pipelineMesh.set("background", ["background"]);
    this.#pipelineMesh.set("main", ["earth"]);
}

This is called from initialize like the others and just sets up a mappings. Eventually this will have to be pulled out somehow but this is the easiest way I could think while still being a bit flexible. I did pipeline -> mesh instead of mesh -> pipeline because the latter would require them to be sorted to avoid unnecessary pipeline switching between renders but maybe that would make more sense in the long run, I don't know yet.

We also need to update our pipelineContainer objects in initializePipeline to handle the binding methods since these are now specific per pipeline.

this.#pipelines.set("main", {
    pipeline: this.#device.createRenderPipeline(pipelineDescriptor),
    pipelineLayout,
    bindGroupLayouts: new Map([
        ["environment", uniformBindGroupLayout],
        ["textures", textureBindGroupLayout]
    ]),
+   bindMethod: this.setMainBindGroups.bind(this)
});

//...

this.#pipelines.set("background", {
    pipeline: this.#device.createRenderPipeline(pipelineDescriptor),
    pipelineLayout,
    bindGroupLayouts: new Map([
        ["uniform", uniformBindGroupLayout],
        ["textures", textureBindGroupLayout]
    ]),
+   bindMethod: this.setBackgroundBindGroups.bind(this) //we'll make this next
});

Render Updates

Finally we can render. Since we have an association of pipeline to mesh we can iterate through each pipeline and then each mesh and render. This avoids switching the pipeline unnecessarily.

render() {
    const commandEncoder = this.#device.createCommandEncoder({
        label: "main-command-encoder"
    });
    const passEncoder = commandEncoder.beginRenderPass({
        label: "main-render-pass",
        colorAttachments: [
            {
                storeOp: "store",
                loadOp: "load",
                view: this.#context.getCurrentTexture().createView()
            }
        ]
    });
    const camera = this.#cameras.get("main");
    for(const [pipelineName, meshNames] of this.#pipelineMesh.entries()){
        const pipelineContainer = this.#pipelines.get(pipelineName);
        passEncoder.setPipeline(pipelineContainer.pipeline);
        for(const meshName of meshNames){
            const meshContainer = this.#meshes.get(meshName);
            pipelineContainer.bindMethod(passEncoder, pipelineContainer.bindGroupLayouts, camera, meshContainer.mesh);
            passEncoder.setVertexBuffer(0, meshContainer.vertexBuffer);
            passEncoder.setIndexBuffer(meshContainer.indexBuffer, "uint16");
            passEncoder.drawIndexed(meshContainer.mesh.indices.length);
        }
        passEncoder.end();
    }
    this.#device.queue.submit([commandEncoder.finish()]);
}

Finally we get something that looks like a planet with stars that we can rotate.

Code

https://github.com/ndesmic/geo/tree/v0.3

WebGPU Engine from Scratch Part 2: Geometry

ndesmic — Mon, 14 Jul 2025 19:00:00 +0000

For the next part I wanted to improve how I generate meshes. From WebGL we generated objects with lists of positions, colors, centroids, triangles, uvs, and normals. This made sense in the WebGL world because these attributes are separate. In WebGPU they all get stuffed into a single buffer. The question then is how to pass these around in an agnostic way?

If we return the (CPU-side) buffer then we might have to carry more data than we intend. Maybe we use colors, maybe not, but if not we shouldn't waste space with them. Unless that's just parameterized into the generation function? Or maybe it comes with a description similar to the vertex buffer descriptor? If we give back a structure with lists of attributes then we have to zip and pack them back up before use. One other thing to consider is using parameterized curve like NURBS which can be rendered with ray tracing but need to be tessellated for a raster 3D engine. I'd like to support these too.

I think in this case it makes sense to split the job. If we have curves or more attributes than needed a separate process will pack them.

Geometry generation

For now we're still just generating shapes. I've added on function to generate a quad:

/**
 * Generates a quad facing negative Z, like a wall
 * @returns {Mesh}
 */
export function quad(){
    return {
        positions: new Float32Array([
            -1.0, -1.0, 0.0,
            1.0, -1.0, 0.0,
            1.0, 1.0, 0.0,

            -1.0, -1.0, 0.0,
            1.0, 1.0, 0.0,
            -1.0, 1.0, 0.0,
        ]),
        uvs: new Float32Array([
            0.0, 1.0,
            1.0, 1.0,
            1.0, 0.0,

            0.0, 1.0,
            1.0, 0.0,
            0.0, 0.0
        ]),
        length: 6
    }
}

Each attribute is it's own array and we have a final length indicating how many items there are.

To get that into something GPU can use we need to pack it based on how big each attribute is and which ones we're actually using.

/**
 * 
 * @param {Float32Array} buffer 
 * @param {Array} attributes 
 * @param {number} index 
 * @param {number} attributeOffset offset in terms of indices (not bytes, assume f32)
 * @param {number} attributeLength number of values per vertex 
 * @param {number} elementStride number of indicies per element
 * @returns 
 */
function packAttribute(buffer, attributes, index, attributeOffset, attributeLength, elementStride){
    if(!attributeLength) return;
    for (let j = 0; j < attributeLength; j++) {
        buffer[index * elementStride + attributeOffset + j] = attributes[index * attributeLength + j]
    }
}
/**
 * 
 * @param {{ positions: Float32Array, colors?: Float32Array, uvs?: Float32Array, normals?: Float32Array, indices?: Float32Array, length: number }} meshAttributes 
 * @param {{ positions: number, colors: number, indicies: boolean, uvs: number, normals: number }} options number of 32-bit elements per vertex
 */
export function packMesh(meshAttributes, options){
    const stride = (options.positions ?? 0) + (options.colors ?? 0) + (options.uvs ?? 0) + (options.normals ?? 0); //stride in terms of indices (not bytes, assume F32s)
    const buffer = new Float32Array(stride * meshAttributes.length);

    const positionOffset = 0;
    const colorOffset = options.positions ?? 0;
    const uvOffset = colorOffset + (options.colors ?? 0);
    const normalOffset = uvOffset + (options.uvs ?? 0);

    for(let i = 0; i < meshAttributes.length; i++){
        packAttribute(buffer, meshAttributes.positions, i, positionOffset, options.positions, stride);
        packAttribute(buffer, meshAttributes.colors, i, colorOffset, options.colors, stride);
        packAttribute(buffer, meshAttributes.uvs, i, uvOffset, options.uvs, stride);
        packAttribute(buffer, meshAttributes.normals, i, normalOffset, options.normals, stride);
    }

    return buffer;
}

This is a tad more hard-coded than I'd like but it'll do for now. We can use it like this:

const mesh = new Mesh(quad());
const vertices = packMesh(mesh, { positions: 3, uvs: 2 });

To pack everything in one buffer. We'll be using the same Mesh class as before for now (with a few small changes in naming and trying to use Float32Array more). This also means we'll be using vector.js from the last project but again modified to use Float32Arrays where possible. I won't show this part for brevity as it's not interesting so you can view the source if necessary.

Why Float32Arrays?

I have previous posts about this but because these are non-dynamic and allocate exactly enough they are more efficient that js arrays. Javascript numbers are 64-bit so we save half the amount of memory too. In the past I used nested arrays to maintain shape of matrices which was a bad choice that needlessly complicated things and made it less efficient. Instead, we'll pass through a shape parameter which tells us what the shape of the matrix is (another idea gleaned from the machine learning series that can easily be converted to tensors if needed). Finally, less overall conversion is necessary if we stick to one thing. Conversion gets expensive, so the fewer conversions we do internally between array types the better and all WebGPU APIs use TypedArrays. 32-bit is just a general standard. Probably not going to need more precision than that.

We can also update the vertexBufferDescriptor to take into account 3d and UVs.

initializePipelines(){
    const vertexBufferDescriptor = [{
        attributes: [
            {
                shaderLocation: 0,
                offset: 0,
                format: "float32x3"
            },
            {
                shaderLocation: 1,
                offset: 12,
                format: "float32x2"
            }
        ],
        arrayStride: 20,
        stepMode: "vertex"
    }];
    //more stuff...
}

Camera

We'll be using the same camera class as before with small changes mostly to use more Float32Arrays.

aysnc initialize()
    //stuff...
    this.initializeCamera();
}

initializeCameras(){
    this.#cameras.set("main", new Camera({
        position: [0, 0, -2],
        screenHeight: this.#canvas.height,
        screenWidth: this.#canvas.width,
        fieldOfView: 90,
        near: 0.01,
        far: 5,
        isPerspective: true
    }))
}

I'm using the same principal as before of keeping track of the objects in maps (in case we have more than one camera).

BindGroups

When we go to do perspective rendering, we need to pass in uniforms as we did in WebGL. WebGPU makes this very manual but a lot more flexible.

setBindGroups(passEncoder, bindGroupLayouts, camera, mesh){
    const viewMatrix = camera.getViewMatrix();
    const projectionMatrix = camera.getProjectionMatrix();
    const modelMatrix = mesh.getModelMatrix();
    const normalMatrix = getTranspose(getInverse(trimMatrix(multiplyMatrix(modelMatrix, [4,4], viewMatrix, [4,4]), [4,4], [3
    const cameraPosition = camera.getPosition();
    const alignment = getAlignments([
        "mat4x4f32",
        "mat4x4f32",
        "mat4x4f32",
        "mat3x3f32",
        "vec3f32"
    ]);
    const bufferSize = alignment.totalSize;
    const environmentUniformBuffer = this.#device.createBuffer({
        size: bufferSize,
        usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
        label: "environment-uniform-buffer"
    });
    const environmentUniformData = new Float32Array(bufferSize / 4);
    environmentUniformData.set(viewMatrix, alignment.offsets[0] / 4); 
    environmentUniformData.set(projectionMatrix, alignment.offsets[1] / 4),
    environmentUniformData.set(modelMatrix, alignment.offsets[2] / 4); 
    environmentUniformData.set(normalMatrix, alignment.offsets[3] / 4);
    environmentUniformData.set(cameraPosition, alignment.offsets[4]/ 4);
    this.#device.queue.writeBuffer(environmentUniformBuffer, 0, environmentUniformData);
    const environmentBindGroup = this.#device.createBindGroup({
        label: "environment-bind-group",
        layout: bindGroupLayouts[0],
        entries: [
            {
                binding: 0,
                resource: {
                    buffer: environmentUniformBuffer,
                    offset: 0,
                    size: bufferSize
                }
            }
        ]
    });
    passEncoder.setBindGroup(0, environmentBindGroup);
}

Again things have been modified slightly to use Float32Arrays. In this code we stuff all the matrices into a single buffer and upload it to the GPU and associated it with the bind group that will be defined in the shader.

Alignment

One serious note here is getAlignments. You'll probably get bitten by this but the buffers can't be any size, they need to be multiples of something (for architectural reasons). In general the buffer needs to be a multiple of 16 but not any multiple of 16 because the data inside the buffer has to be padded. For instance the vec3<f32> for the camera cannot start on a multiple of 8, only 16 because it has an alignment of 16 (see: https://www.w3.org/TR/WGSL/#alignment-and-size). Even certain types are not the size you might think, a mat4x3<f32> is 64 bytes for example, not 48. So we need to use the table in the spec to see how each value fits into the buffer.

const gpuTypeAlignSize = {
    bool: [4,4],
    i32: [4,4],
    u32: [4,4],
    f32: [4,4],
    f16: [2,2],
    atomic: [4,4],
    vec2bool: [8,8],
    vec2i32: [8,8],
    vec2u32: [8,8],
    vec2f32: [8,8],
    vec2f16: [4,4],
    vec3bool: [16,12],
    vec3i32: [16,12],
    vec3u32: [16,12],
    vec3f32: [16,12],
    vec3f16: [8,6],
    vec4bool: [16,16],
    vec4i32: [16,16],
    vec4u32: [16,16],
    vec4f32: [16,16],
    vec4f16: [8,8],
    mat2x2f32: [8,16],
    mat2x2f16: [4,8],
    mat3x2f32: [8,24],
    mat3x2f16: [4,12],
    mat4x2f32: [8,32],
    mat4x2f16: [4,16],
    mat2x3f32: [16,32],
    mat2x3f16: [8,16],
    mat3x3f32: [16,48],
    mat3x3f16: [8,24],
    mat4x3f32: [16,64],
    mat4x3f16: [8,32],
    mat2x4f32: [16,32],
    mat2x4f16: [8,16],
    mat3x4f32: [16,48],
    mat3x4f16: [8,24],
    mat4x4f32: [16,64],
    mat4x4f16: [8,32]
}

/**
 * 
 * @param {number} size 
 * @param {number} smallestUnitSize 
 * @returns
 */
export function getPaddedSize(size, smallestUnitSize){
    const remainder = size % smallestUnitSize;
    if(remainder === 0){
        return size;
    }
    return size + smallestUnitSize - remainder;
}

/**
 * @typedef {keyof gpuTypeAlignSize} GpuType
 * @param {GpuType[]} typesToPack 
 */
export function getAlignments(typesToPack){
    let offset = 0;
    let maxAlign = 0;
    const offsets = new Array(typesToPack.length);
    for(let i = 0; i < typesToPack.length; i++){
        const alignmentSize = gpuTypeAlignSize[typesToPack[i]];
        if(maxAlign < alignmentSize[0]){
            maxAlign = alignmentSize[0];
        }
        offset = getPaddedSize(offset, alignmentSize[0])
        offsets[i] = offset;
        offset += alignmentSize[1];
    }
    return {
        offsets,
        totalSize: getPaddedSize(offset, maxAlign)
    };
}

This function uses that table to fit each value and gives the offset and the total size. This should help alleviate those annoying padding issues. You can do it manually, but you will probably encounter issues eventually.

One issue I'm not solving is optimal packing. Due to the padding, depending on the order you can save space by ordering properties in the right way. For example,

f16
mat4x4<f32>
f16

Would need 2 bytes, then 14 wasted bytes of padding, and then 64 bytes and then 2 bytes aligned to 16 (because that's the max align in the struct). So 16 + 64 + 16 = 96. If instead we had done

mat4x4<f32>
f16
f16

Then we need 64 bytes, and then 2 bytes plus 2 bytes aligned to 16. So 64 + 16 = 80 saving 16 bytes. Something to consider.

BindGroup layouts

In order for the shader to understand how our data was packed into the buffer it needs the bind group layout. This can be automatic (generated from the shader code) or manual.

Auto

const pipelineDescriptor = {
    label: "pipeline",
    vertex: {
        module: shaderModule,
        entryPoint: "vertex_main",
        buffers: vertexBufferDescriptor
    },
    fragment: {
        module: shaderModule,
        entryPoint: "fragment_main",
        targets: [
            { format: "rgba8unorm" }
        ]
    },
    primitive: {
        topology: "triangle-list"
    },
    layout: "auto"
};


// construct pipleline and other stuff ...

const bindGroup = this.#device.createBindGroup({
    label: "bind-group",
    layout: pipleine.getBindGroupLayout(0)
    entries: [
        {
            binding: 0,
            resource: {
                buffer: bindGroupUniformBuffer,
                offset: 0,
                size: bufferSize
            }
        }
    ]
});

By setting "auto" in the pipeline layout you have WebGPU do this for you. It will look at the bind groups you defined in your shader code and generate it. This saves some code. However, it cannot be reused with other shaders and you need to watch out if you don't use a value in the shader because the shader compiler will tree shake it and the auto bind group won't have it anymore.

Manual

const environmentBindGroupLayout = this.#device.createBindGroupLayout({
    label: "bind-group-layout",
    entries: [
        {
            binding: 0,
            visibility: GPUShaderStage.VERTEX | GPUShaderStage.FRAGMENT,
            buffer: {
                type: "uniform"
            }
        }
    ]
});
const pipelineLayout = this.#device.createPipelineLayout({
    label: "pipeline-layout",
    bindGroupLayouts: [
        environmentBindGroupLayout
    ]
});
const pipelineDescriptor = {
    label: "pipeline",
    vertex: {
        module: shaderModule,
        entryPoint: "vertex_main",
        buffers: vertexBufferDescripto
    },
    fragment: {
        module: shaderModule,
        entryPoint: "fragment_main",
        targets: [
            { format: "rgba8unorm" }
        ]
    },
    primitive: {
        topology: "triangle-list"
    },
    layout: pipelineLayout
};

// construct pipleline and other stuff ...

const environmentBindGroup = this.#device.createBindGroup({
    label: "environment-bind-group",
    layout: bindGroupLayouts[0],
    entries: [
        {
            binding: 0,
            resource: {
                buffer: environmentUniformBuffer,
                offset: 0,
                size: bufferSize
            }
        }
    ]
});

This is the same thing, just manual. This can be reused but is more verbose.

Organizing BindGroups

Another note is to use WGSL structs. For example:

struct Environment {
    view_matrix: mat4x4<f32>,
    projection_matrix: mat4x4<f32>,
    model_matrix: mat4x4<f32>,
    camera_position: vec3<f32>
}

@group(0) @binding(0) var<uniform> environment : Environment;

@group(0) @binding(0) var<uniform> view_matix : mat4x4<f32>;
@group(0) @binding(1) var<uniform> projection_matrix : mat4x4<f32>;
@group(0) @binding(2) var<uniform> model_matrix : mat4x4<f32>;
@group(0) @binding(3) var<uniform> camera_position : vec3<f32>;

The latter will require at least 256 bytes per binding so even though a mat4x4<f32> is 64 bytes the minimum space you need to allocate is 256 * 4 because of alignment issues for uniforms in bindings. So to not waste space a struct is better and cleaner.

Pass Encoding

render() {
    const commandEncoder = this.#device.createCommandEncoder({
        label: "main-command-encoder"
    });
    const pipelineContainer = this.#pipelines.get("main");
    const camera = this.#cameras.get("main");
    const meshContainer = this.#meshes.get("background");
    const passEncoder = commandEncoder.beginRenderPass({
        label: "main-render-pass",
        colorAttachments: [
            {
                loadValue: { r: 0, g: 0, b: 0, a: 1 },
                storeOp: "store",
                loadOp: "load",
                view: this.#context.getCurrentTexture().createView()
            }
        ]
    });
    passEncoder.setPipeline(pipelineContainer.pipeline);
    this.setBindGroups(passEncoder, pipelineContainer.bindGroupLayouts, camera
    passEncoder.setVertexBuffer(0, meshContainer.vertices);
    passEncoder.draw(6); //TODO need index buffer
    passEncoder.end();
    this.#device.queue.submit([commandEncoder.finish()]);
}

For the bindgroups we need to book keep some stuff and pass it into setBindGroups which was shown above and builds out the data for the Model-View-Projection matrix. Beyond that not much has changed here.

Using indices

Like before we can be more efficient by using indices to reuse vertices. So we'll do that too. First we need to setup the indices with the geometry. Luckily this was already done. The property on the mesh object was called triangles previously, I changed it to indices to make the naming more consistent. Then we need to upload it to the GPU (probably immediately after we upload the vertices).

const indexBuffer = this.#device.createBuffer({
        size: mesh.indices.byteLength,
        usage: GPUBufferUsage.INDEX | GPUBufferUsage.COPY_DST,
});
this.#device.queue.writeBuffer(indexBuffer, 0, mesh.indices);

Only difference between vertices and indices in terms of uploading to the GPU is that indices are a UInt16Array and the have usage GPUBufferUsage.INDEX instead of GPUBufferUsage.VERTEX. Then in the passEncoder part:

passEncoder.setIndexBuffer(meshContainer.indexBuffer, "uint16");
//passEncoder.draw(...)
passEncoder.drawIndexed(meshContainer.mesh.indices.length);

Easy. Now a quad only needs to define 4 vertices instead of 6 because we can repeat them with the index buffer.

Now we should have a red quad on the screen.

Code

https://github.com/ndesmic/geo/releases/tag/v0.2