Forem: Shai Almog

Metal and Skins

Shai Almog — Fri, 15 May 2026 13:34:26 +0000

This post has a lot to cover. Before we get to any of it I want to take on the uncomfortable subject first: quality. Two incidents from the past two weeks deserve a public explanation, one was a bug that fits into our normal iteration loop and one was a serious mistake on my part. Both deserve the kind of explanation I would want if I were on the other side of the import.

What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com.

How we think about quality

Codename One is a small open source company. We are not a 200-engineer platform team with a dedicated SRE rotation and a separate QA org. We move fast, fast enough that we ship meaningful new code every week, and we put a lot of effort into making sure that speed does not come at the cost of breaking your apps.

"A lot of effort" is doing some work in that sentence, so here is what it actually looks like:

Layer	Coverage
Unit and integration tests	710 Java test files exercised on every PR.
Screenshot tests	45 tests producing 190+ golden PNGs that are diffed across the iOS simulator, Android emulator, JavaSE, and headless Chrome. Both the OpenGL and Metal backends are diffed in parallel.
ParparVM bytecode-translation suite	A separate, deeper test pass that exercises VM functionality (bytecode translation, garbage collector, native interop) beyond what the framework-level tests can reach.
Cross-platform build matrix	24 GitHub workflows build every PR against iOS, Android, JavaSE, and JavaScript.
JDK matrix	JDK 8 build, JDK 11 through 25 runtime.

That is a meaningful amount of automated coverage and it catches a lot before code ever lands. What it does not catch is brand new behaviour, because there is nothing yet to compare a brand new feature against. The first golden of a new test is also the bug, until somebody actually runs the feature and tells us so.

With that in mind, let's talk about the two specific incidents from the past two weeks.

Sticky headers were half baked, and that was by design

Last week's post introduced StickyHeaderContainer with an animated transition between section headers. Within a couple of days the issue tracker had #4849, the NONE and FADE transitions were not behaving correctly and the swap had visible jitter. We turned the fix around within hours of the report.

That round trip, ship, hear from a real user, fix, is the deal we make with the community. Our pixel-diff harness is excellent at catching regressions in code that already exists. It is structurally bad at catching the first version of a brand new component because there is nothing yet to compare against. We could have sat on StickyHeaderContainer for another two weeks polishing it in private and we would have shipped a worse component, because we would not have had Thomas's eyes on it. Iterating in public, with a tight feedback loop, is how a team our size keeps moving.

The SIMD bug, which was my mistake

PR #4842 is a different story. The SIMD code on iOS uses alloca to put working buffers on the stack for speed. That is the right call for small buffers and the wrong call for large ones, past a certain size the stack request fails outright and the process crashes. The image API uses these buffers, and on large images the buffer crossed the threshold. Result: image API crash in production.

This was not a new feature. It was a change to existing, mature code. We took the precautions we always take when changing code with a long history, ran the existing tests, every existing test passed, and the patch shipped. The bug got through anyway because the test coverage validated the SIMD path on small images and never on the large ones that actually triggered the failure. That hole in the tests was mine to spot, and I did not spot it.

Once it was reported the fix turned around in well under 24 hours. The patched version detects the over-large case and falls back to a heap allocation. Small SIMD ops keep the fast alloca path. Large ones no longer crash. New tests cover the threshold case so this specific shape of bug cannot regress.

This should not have happened, but realistically it will happen again, not this exact bug, but one like it. Tests are not perfect, mine certainly are not. So the take home is the part I want to lean on:

Codename One is the first line of defense between bugs and your end users. We are not the last line. Test your application before you release. If your app supports it, use a beta channel (TestFlight on iOS, Play Console internal or closed tracks on Android) so a bug like this hits you before it hits the people who paid for your app. That tiny extra step is the most reliable protection your users have.

We are also actively brainstorming the next generation of crash protection inside the framework. The current crash protection sits at the EDT and catches RuntimeExceptions that user code throws. The next generation needs to extend further, into native crashes, into earlier startup, and into a more useful diagnostic payload that comes back to the developer instead of just the device log. There is no PR yet, we are still working out the shape, but it is the major framework-level investment we are making to give the community a stronger floor underneath their apps.

With the quality conversation out of the way, the rest of this post is about the things that actually shipped.

Metal is here

PR #4799 is the largest single change we have landed in months: a complete Metal rendering backend for iOS. It sits next to the existing OpenGL ES 2 path, behind a single build hint, with its own CI job and its own pixel-diff goldens.

Metal is Apple's modern graphics API. OpenGL on iOS was deprecated by Apple back in iOS 12, it still runs today and we kept it running for years, but "deprecated" on Apple is a slow countdown that ends with the platform pulling support. Moving to Metal now is how we get ahead of that, and it brings real benefits to your apps:

Better rendering performance. Lower draw-call overhead, modern command-encoder batching, and pipeline state caching add up to smoother scrolling and faster transitions on the same hardware.
Less battery use. Metal's reduced CPU overhead per frame means the GPU spends less time idling and the CPU spends less time bookkeeping. Long-running, graphics-heavy apps benefit the most.
Crisper text. Glyphs go through a CoreText atlas, which produces noticeably sharper rendering at the same size, with proper kerning and correct handling of complex scripts.
Pure-GPU gradients. Linear and radial gradients render entirely on the GPU instead of round-tripping through a CGContext bitmap.
Access to modern Apple graphics features. New iOS rendering features (variable-rate shading, mesh shaders, ray tracing on Apple silicon) are Metal-only. Sticking with GL means watching that train leave without us.

To enable Metal in your project, set the build hint:

ios.metal=true

Everything else stays the same. The Java surface is unchanged, your existing code keeps working.

We plan to flip Metal to be the default within two weeks, assuming no major issues surface. The ios.metal hint will stay around (set it to false to opt back into GL), but new projects and the build server's default behaviour will move over. If you ship an iOS app, please set the hint now and put your real flows through it. We want regressions to surface against your real screens, not the day after the default changes.

The most user-visible improvement from the Metal port is text. Here is the ShowcaseTheme capture from the Metal screenshot suite:

And the SpanLabelTheme capture, which is the real test for body-copy rendering, multiple lines, variable widths, the kind of paragraphs that show up in real apps:

The Metal Dialog capture is also worth showing because the translucent surface composites correctly against the textured backdrop:

The end of the skin downloader

PR #4758 ships the Skin Designer as a JavaScript bundle, embedded into the website at /skindesigner/ the same way the Playground and Initializr are embedded. You can build a skin in the browser, save it, and use it in your simulator without installing anything.

This is bigger than a website convenience. It is how we get out of the skin business.

For the entire history of Codename One, "no skin for the iPhone 16 Pro Max" or "no skin for the iPad mini 7" has been a recurring complaint, and we have published skins as fast as we could. That model never scaled. Apple ships new device sizes faster than any of us want to maintain a parallel skin catalogue, and Android has effectively infinite device shapes. Today we are deprecating the skin downloader and moving to a generic browser-based authoring tool.

To be clear about what is changing:

Existing skins are not going anywhere. Every skin that ships today will continue to work, will continue to load in the simulator, and will continue to be supported. We are not removing them. If your team has a workflow built around an existing skin, that workflow keeps working.
We will stop issuing new skins. When the next iPhone or iPad ships, we will not publish an official skin for it. Anyone can build one in the new designer in minutes, and that "anyone" includes us, of course, but it also includes you.

The "no skin for X" problem is solved generically. If you are running a niche enterprise app on a less-common Android device, you no longer have to wait on us to produce a skin for it. Build it once, drop it into your team's shared assets, done.

How the wizard works

The Skin Designer turns a device specification (resolution, PPI, fonts, safe-area insets, cutouts) into a .skin file that the JavaSE simulator can load. It runs in your browser. There is nothing to install. The wizard is intentionally opinionated. It ships with a curated device catalog, generates the device frame procedurally, and writes a skin layout that matches the iPhoneTheme.res, iOS7Theme.res, and android_holo_light.res themes shipped with Codename One.

If you only want a skin and don't care how it is built, pick a device, accept the defaults, click Finish, then Download skin. The file is ready to load via Add in the simulator's Skins menu.

Stage 1, pick a device. The first step shows a card per device from the bundled catalog. The search box filters by name (it matches both the model and the brand) and the chips below narrow by form factor: All / Phones / Tablets / Foldables. Picking a device pulls in its resolution, PPI, screen size, default safe-area insets, and the iOS or Android system font names from the catalog, then seeds a sensible starting frame: notch, island, or hole presets are applied automatically based on the device's hardware. The catalog is large, the grid is capped to the most recent matches by default, type into the search field to find older or less-common devices.

Stage 2, pick a starting source. There are three ways to seed the skin's body image:

Pick a shape generates the device frame procedurally from a small preset library (rounded rect, notch, dynamic island, punch-hole, corner hole, classic home-button). The frame is rendered as a dark gradient with the screen rect (and any cutouts) carved into it. Best when you want a generic-looking iPhone or Android frame and don't care about exact hardware fidelity.
Upload an image opens an image picker. The wizard scales the image into the device's resolution, then carves the screen rect and cutouts on top. Use this when you have a marketing render of the specific device you are targeting.
Blank rectangle collapses the bezel and corner radius to almost nothing, drops every cutout, and turns the home indicator off. The screen fills the entire skin. Useful for desktop or web simulators where the device frame would just be visual noise.

Stage 3, the editor. The editor is split into two panes: a live preview on the left that paints the device frame, screen tint, cutouts, and home indicator, and a sidebar on the right with three tabs.

The Shape tab shows a preset grid (Rounded rect, Notch, Dynamic Island, Punch-hole, Corner hole, Classic home) and dimension fields for corner radius, bezel thickness, and a toggle for the bottom home indicator. iPhones from X onward and most modern Androids should leave the indicator on, classic devices with a hardware home button should turn it off.

The Cutouts tab lists every cutout currently on the skin. Tap a row to expand its width, height, and offset fields. The three add buttons at the bottom seed a sensible default of each type. Notch (180 x 30 viewbox px) is a physical hardware cutout drawn in the device frame above the screen rect, mirroring iPhone X / 11 / 12 / 13 hardware. Island (120 x 35) is a Dynamic Island, software-reserved space rendered as an opaque pill inside the screen rect, floating on top of the iOS status bar. Hole (28 x 28) is an Android punch-hole camera, rendered like the island. When the wizard generates the .skin, it automatically extends safePortraitTop to cover any in-screen cutouts so app content lands below the floating shape.

The Info tab is mostly read-only and shows what is about to be written into skin.properties: name, width, height, PPI, pixels-per-millimeter, and the user-editable safe-area insets. The wizard intentionally does not write smallFontSize, mediumFontSize, or largeFontSize, when those are absent the simulator auto-derives them from pixelMilliRatio, which is what you want on high-PPI screens.

Stage 4, finish and download. Clicking Finish renders the portrait skin image at the device's actual resolution with rounded corners, transparent screen, opaque cutouts, and a home indicator if enabled. It synthesises the landscape skin by 90-degree rotation, writes the skin_map.png overlays that mark the screen rectangle for the simulator's screen-position detection, bundles the appropriate native theme inside the skin zip, and writes skin.properties with the platform metadata, safe-area, PPI, and display rect. Clicking Download skin hands the file to the browser's download dialog. After the file is on disk, drop it into your simulator's skins folder (or use the Add command in the simulator's Skins menu) and your new device should appear in the picker.

A generated .skin is just a renamed zip:

Apple-iPhone-16-Pro.skin/
  skin.png            # portrait body (device frame + transparent screen + cutouts)
  skin_l.png          # 90-degree rotated portrait
  skin_map.png        # black rect = screen, white = frame, used for hit-testing
  skin_map_l.png      # rotated map
  iOS7Theme.res       # bundled native theme (or android_holo_light.res / winTheme.res)
  skin.properties     # platform metadata, safe-area, PPI, display rect

The full developer-guide chapter at Skin-Designer.asciidoc walks through every stage with annotated screenshots and documents the skin.properties keys the wizard writes (roundScreen, displayX/Y/Width/Height, safePortrait*, safeLandscape*, overrideNames, system font families, PPI, and pixel ratio).

Eating our own dog food

While we're talking about the Skin Designer, this is the right moment to point out something I think is genuinely worth highlighting. The Initializr, the Playground, and the Skin Designer are all open source Codename One apps. They are written in Java using the same Codename One UI framework you use to build your iOS and Android apps, and they are deployed to the browser through our JavaScript port.

Every interaction you have with these tools, the device picker grid, the live preview rendering the device frame and cutouts, the form-driven editor with its tabbed sidebar, the file generation that bundles a .skin zip in your browser tab, is the same Codename One code that ships in your apps. The Container, Form, BoxLayout, theming, and event-handling code is identical to what you would write for a phone build. The JavaScript port translates it into something a browser can run.

These three tools are the most direct demonstration we can give of what Codename One is capable of: real, non-trivial UIs, with state, file I/O, image generation, and complex layouts, running smoothly inside a browser tab. If you have ever wondered whether the JavaScript port is production-grade enough for a real application, the Initializr, Playground, and Skin Designer are your answer. They are also the answer to "can Codename One build apps that go beyond mobile". Same codebase, deployed to a fourth target, with no rewrite.

The source for all three lives in the same CodenameOne repository the framework itself does. If you want to see how a non-trivial Codename One app is structured, those are three good places to start reading.

iOS multi-line TextArea: Return as Done

PR #4859, driven by issue #4854, gives multi-line TextArea an opt-in flag that makes the iOS keyboard's Return key act as Done. It closes the editor and fires the Done listener instead of inserting a newline. This is the iOS Reminders-app behaviour: a growing, multi-line task-title field where Return finishes the entry.

The reason it has to be a flag is that real iOS does not expose this as a built-in primitive. Reminders implements it on a UITextView whose delegate intercepts \n in shouldChangeTextInRange:. We replicate that exactly, gated behind a client property so existing layouts are untouched:

TextArea ta = new TextArea("", 3, 30);
ta.putClientProperty("iosReturnExitsEditing", Boolean.TRUE);
ta.setDoneListener(e -> { /* Return / Done was tapped */ });

While the flag is set, the keyboard's Return key is relabelled to Done (UIReturnKeyDone). Default behaviour is unchanged: the flag defaults to off, only takes effect on multi-line TextAreas, and only intercepts an exact "\n" replacement so pasted multi-line text is unaffected.

Diagnostics for status-bar tap scroll-to-top

PR #4868, driven by issue #3589, adds three complementary diagnostics for the iOS status-bar tap path. We shipped a fix earlier (#4857) and the reporter still saw no scroll on device. Rather than another sweep in the dark, we built tools to make the path observable.

Simulator menu, Simulate > iOS Status Bar Tap. Synthesises the same (displayWidth/2, 0) tap that scrollViewShouldScrollToTop: dispatches, pops a dialog reporting the responder UIID, the build-hint state, and an OK / PROBLEM verdict, then actually fires pointerPressed and pointerReleased so any wired-up scroll-to-top is observable.
Device-side properties. Display.getProperty("cn1.iosStatusBarTap.count"), cn1.iosStatusBarTap.lastEpochMillis, cn1.iosStatusBarTap.lastX/Y, and cn1.iosStatusBarTap.proxyInstalled let you inspect the path on a real iPhone. Run your app on the device, tap the status bar, and read the property. That distinguishes "iOS never delivered the message" from "iOS delivered it but a CodenameOne component intercepted the tap".
Regression coverage. StatusBarTapDiagnosticScreenshotTest exercises the exact same code path through a 2x3 frame grid, with the visible counter rising and the scroll position alternating, so future regressions surface in CI.

Simulator: Dark / Light mode toggle

PR #4871 adds a Dark / Light Mode submenu under the simulator's Simulate menu with three options: Dark Mode, Light Mode, and Unsupported (the default).

Selecting an option flips Display.isDarkMode() (Boolean.TRUE / Boolean.FALSE / null) and calls refreshSkin(...) so themes that branch on @darkModeBool re-render immediately. The choice is persisted under the cn1.simulator.darkMode Preference so the simulator restarts in the mode you left it.

Combined with the Native Theme menu we shipped two weeks ago, you can now sit on a single skin and flip between iOS Modern, Material 3, iOS 7, and Holo Light, in light, dark, and unsupported, in seconds. The everyday win is being able to verify your own theme looks right in dark mode without restarting the simulator.

Heads-up: weekend backend maintenance

This weekend we will be doing some maintenance on our build backend servers. The work is mostly invisible from the outside but it touches enough of the infrastructure that you might see intermittent build issues during the window: slower-than-usual builds, the occasional retry, possibly a short period where new builds are queued.

We are doing it because the underlying backend needs to move forward, and the cost of putting that work off keeps compounding. We will keep the disruption as short as we can. If you have a hard release deadline that lands this weekend, please plan around it. Otherwise the impact should be small and you can build through it normally.

Warning: Android 16 will effectively disallow locking orientation

Thanks to Durank for flagging #4879. The Android 16 behavior changes include a meaningful change to how Android handles orientation, in short, on large-screen devices the platform will ignore an app's request to lock orientation. If your app calls Display.lockOrientation(...) or sets a fixed orientation in the Android manifest, that lock will be honoured on phones but effectively ignored on tablets and foldables once the device targets Android 16.

There is not much we can do about this on the framework side. It is a platform-level decision and there is no public opt-out for general apps. The realistic path forward is to design layouts that work in both orientations, and to test your app against both portrait and landscape on a tablet before Android 16 reaches your users. We will keep watching for any opt-in path Google publishes, but for the moment please plan accordingly.

Why the version jumped to 7.0.242

A small note on versioning: the current release is 7.0.242, not 7.0.238 as you might expect from the cadence. The gap is real and worth explaining. We made a fix to the Maven archetype that brings over the features we added in the Codename One Initializr to projects created from the command line. The change itself is straightforward, but it interacted badly with our release build automation and we had to delete several releases along the way to get the pipeline back on its feet. The version numbers we burned in the process are the visible scar. The bright side is that command-line mvn archetype:generate now produces projects that line up with what the Initializr generates, which is what we wanted all along.

Wrapping up

We closed 24 issues in the past week, a meaningful share of them direct beneficiaries of the Metal port. Old GL-only rasterisation diffs, font sizing on retina, polygon drawing artefacts, perspective transform issues, things that the Metal pipeline simply renders correctly out of the box. Migrating the rendering layer turned out to be the cleanest way to retire a long tail of small bugs at once. With the new Skin Designer landing in the same week, two long-running structural problems went from "we should fix this someday" to "this is fixed and shipping".

If you ship an iOS app, please flip ios.metal=true this week and run your real app through it. We want to find any remaining issues now, not the day we flip the default. Issue tracker is here, the Playground is the easiest place to poke at the new themes, the Skin Designer is live on the site.

A specific thank-you this week to Thomas (@ThomasH99) for the sticky-header transition report and the Picker centring follow-up, Francesco Galgani (@jsfan3) for the iOS Reminders-style Return RFE, and the reporter on #3589 for sticking with us through a multi-PR diagnosis on the status-bar tap. The "tests cannot catch everything" section above is also a "and that is why we need you" section. It works because you keep filing.

Liquid Glass, Material 3, And A Lot Of Plumbing

Shai Almog — Wed, 06 May 2026 13:08:18 +0000

It has been one of those weeks where the diff is bigger than the headline. The headline is short — Codename One now ships modern native themes: an iOS "liquid glass" look and an Android Material 3 look, bundled into the iOS and Android ports, on by default in the Playground, and selectable from a brand new menu in the simulator. The diff behind that headline is several thousand lines across the platform ports, the simulator, the GUI plumbing, and a small army of screenshot tests.

What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com.

The theme behind the work is simple: Codename One should look modern out of the box on every platform we ship to, and it should feel fast. Almost everything in the past week of commits is in service of one of those two goals.

Try it right now in the Playground

The easiest way to see any of this is the Playground. The Playground now defaults to iOS Modern when the device toggle is set to iPhone and Android Material 3 when it is set to Android, in both light and dark mode. No setup, no pom.xml, no build hints — just open the page, drop in any of the standard components, and the modern look is what you get. If the past releases of Codename One looked dated to you, the Playground is where to start.

The simulator is the second-easiest place. We will get to that.

The new native themes

For most of Codename One's life the iOS native theme has been the venerable iOS 7 flat theme, and the Android native theme has been Holo Light. Both still ship — backwards compatibility has always been one of our most important goals — but they are no longer where we want a brand new app to start. We spent the bulk of this week building two new themes that target current platform aesthetics:

iOS Modern — Apple system colors (accent #007aff light / #0a84ff dark, grouped-form surfaces, the system separator palette), pill borders for tabs, an iOS-Settings-style MultiButton, CHECK_CIRCLE-style checkbox glyphs, and translucent surfaces for Dialog and TabsContainer so they read as glass-frosted on top of whatever is behind them. It is not a real UIVisualEffectView backdrop — that is a port-side primitive we have not built yet — but the look is much closer to the iOS 26 vibe than anything we have shipped before.
Android Material 3 — the Material 3 baseline tonal palette (primary #6750a4 light / #d0bcff dark, surface-container tiers, elevated containers approximated tonally because real elevation drop-shadows are still on the to-do list), plus all the Material density and padding choices — Roboto-ish proportions, a top-tab bar with the underline-by-color treatment, the standard square checkbox glyph.

Each theme covers the usual ~25 UIIDs: base (Component, Form, ContentPane, Container), typography (Label, SecondaryLabel, TertiaryLabel, SpanLabel*), buttons (Button, RaisedButton, FlatButton with .pressed and .disabled), text input, selection controls, toolbar, tabs, side menu, list, MultiButton, dialog/sheet, FAB, and all the supporting separator and popup pieces. Both themes have full light and dark coverage.

The shipping CSS sources sit in the repo at native-themes/ios-modern/theme.css and native-themes/android-material/theme.css for anyone who wants to read what each UIID is doing.

iOS Modern

This is the ShowcaseTheme capture from the new screenshot suite, run on iOS in light and dark. Same Form, same components, swap Display.setDarkMode(...) and re-resolve. The form is built like this:

Container row = new Container(BoxLayout.x());
row.add(new Button("Default"));
Button raised = new Button("Raised");
raised.setUIID("RaisedButton");
row.add(raised);
form.add(row);

TextField tf = new TextField("hello@example.com");
form.add(tf);

Container toggles = new Container(BoxLayout.x());
CheckBox cb = new CheckBox("Remember me");
cb.setSelected(true);
toggles.add(cb);
RadioButton rb = new RadioButton("Agree");
rb.setSelected(true);
toggles.add(rb);
form.add(toggles);

SpanLabel body = new SpanLabel("Body copy …");
form.add(body);

That gives you the full picture in one screen:

The Default button uses the stock Button UIID. The Raised button uses RaisedButton, which cn1-derives from Button and adds a tinted pill on top of the iOS system blue — that is the iOS Modern accent in both modes.
The TextField is a single rounded-rect surface with the iOS system gray fill, the same shape Apple uses in Settings.
CheckBox and RadioButton use the new optional @checkBoxCheckedIconInt / @radioCheckedIconInt theme constants to swap to CHECK_CIRCLE / CHECK_CIRCLE_OUTLINE glyphs — Reminders-app aesthetic on iOS while Android keeps the standard square check.
The SpanLabel body uses the theme's base font and inherits transparent backgrounds so it never paints over a translucent parent.

The full screen source is DarkLightShowcaseThemeScreenshotTest.java.

Android Material 3

Same ShowcaseTheme source on Android. The Material 3 baseline palette gives Default the primary container color and Raised the elevated-surface tone, with the dark variant flipping the relationship correctly via the dark color-role mapping. Padding and font sizing follow Material density, which you can see in how compact the same Form lays out compared to iOS.

Translucent surfaces

This is the DialogTheme capture against the screenshot suite's textured diagonal-stripe backdrop. The backdrop is intentional — it lets reviewers see whether anything that is supposed to be translucent actually is. The iOS Modern Dialog uses an rgba surface fill (0.78 alpha in light, 0.95 in dark — dark needs more opacity because bright stripes bleed through) and its DialogBody, DialogTitle, ContentPane, CommandArea sub-UIIDs are transparent so the rounded corners read cleanly. The same trick is applied to TabsContainer and the iOS MultiButton.

Runtime palette overrides

The native theme is meant to be a starting point — you can layer your own palette on top without forking the theme. Above is the PaletteOverrideTheme capture: the base is iOS Modern, but the test layers a magenta palette on top at runtime via UIManager.addThemeProps(...). RaisedButton, FlatButton, the disabled tone, and the body-copy span all pick up the override in both light and dark — the override seam works at the resource-bundle layer, exactly the same mechanism a user theme uses to override the native theme on a real app.

In the simulator

Three pieces, all live:

Themes are bundled. The simulator jar-with-dependencies includes both modern themes alongside the four legacy themes (iPhoneTheme, iOS7Theme, androidTheme, android_holo_light) at the root of the jar. The simulator can pick any one of them at runtime without touching the skin repo.
A new "Native Theme" menu. Right next to the Skins menu there is now a Native Theme menu with a radio group for the six themes plus "Auto" and "Use skin's embedded theme". Selecting one writes the simulatorNativeTheme Preference, flips the simulator-reload flag, and disposes the current window so the skin reloader kicks in with the new theme. You can sit on a single skin and flip through every native theme in seconds.
Build hints know about it. The new nativeTheme, ios.themeMode, and and.themeMode build hints are registered with the simulator's Build Hints UI on launch — labels, types, value lists, descriptions, the lot. (The legacy keys cn1.nativeTheme and cn1.androidTheme are still honored for back-compat.) Set them in the Build Hints dialog, in codenameone_settings.properties, or via -D system properties; they flow through to the device build and the simulator both.

The "Auto" choice in the Native Theme menu defers to those build hints — set ios.themeMode=modern in your project's settings and "Auto" previews iOS Modern; flip the same project to ios.themeMode=ios7 and "Auto" previews iOS 7. The explicit menu entries (iOS Modern, iOS 7, etc.) override the hints regardless. -Dcn1.forceSimulatorTheme is still honored as the highest-priority override; pick "Use skin's embedded theme" to bypass the framework theme entirely and get whatever the skin shipped with.

On devices

The opt-in is the same on iOS and Android. The platform knobs follow a single naming pattern — ios.themeMode and and.themeMode — and accept modern / liquid / auto / ios7 / flat on iOS, modern / material / auto / hololight / legacy on Android. There is a single cross-platform shortcut, nativeTheme=modern, which the iOS builder consults when ios.themeMode is unset and which the Android port reads at runtime as a default for and.themeMode. The legacy aliases cn1.androidTheme and cn1.nativeTheme are still honored for back-compat, as is and.hololight=true.

The default for an existing app stays on legacy on every platform. We do not flip a 15-year-old app's look without an opt-in. New apps generated from the initializr ship with nativeTheme=modern, ios.themeMode=modern, and and.themeMode=modern already set in codenameone_settings.properties, so a brand new project starts with the modern themes preselected. The Playground does the same, and Playground project downloads carry the same defaults into the generated codenameone_settings.properties.

The HTML5 port has the runtime support for the modern themes but does not bundle them with user apps yet — that is one of the loose ends we want to close in the next round.

Sticky headers

The other piece of look-and-feel that we want to highlight is StickyHeaderContainer, which finally has a proper home in the framework. It is the iOS-contacts-list / sectioned-material-list component: scroll past a section boundary and the previous header is replaced by the next one. New this week, the swap is animated. A directional slide moves the outgoing header up on a forward scroll and down on a reverse scroll, or you can pick a cross-fade.

Above is a six-frame sweep from the screenshot test — the user scrolls through sections A, B, C, D, E and the pinned header recolors to whichever section is currently active at the top of the viewport.

The API is small. Build the container, register sections with addSection(header, content), configure the transition style and duration, and add it to a Form:

StickyHeaderContainer sticky = new StickyHeaderContainer();
sticky.setTransitionStyle(StickyHeaderContainer.TRANSITION_SLIDE);
sticky.setTransitionDurationMillis(250);
for (char c = 'A'; c <= 'Z'; c++) {
    Label header = new Label("" + c, "StickyHeader");
    Container items = new Container(BoxLayout.y());
    for (int i = 0; i < 5; i++) {
        items.add(new Label(c + " entry " + i));
    }
    sticky.addSection(header, items);
}
form.add(BorderLayout.CENTER, sticky);

TRANSITION_SLIDE is the default. TRANSITION_FADE cross-fades the outgoing header on top of the incoming one. TRANSITION_NONE keeps the prior instantaneous swap if you want it. Issue #4807 for the original request.

How we test this

Every screenshot in this post is captured by a test that runs the app on a real iOS device, an Android emulator, and headless Chrome, then diffs each capture against a stored golden image. The diff is the test — if the rendered pixels drift, the run fails.

For animations the test grabs a series of frames over a fixed-duration transition, then composites them into a single index image. That is how the dual-appearance shots end up as one side-by-side picture per test:

…and how the sticky-header animation ends up as a six-frame strip stitched into a GIF:

If you want to read the source, the suite lives at scripts/hellocodenameone/common/src/main/java/com/codenameone/examples/hellocodenameone/tests/.

Bugs and misc features from this week

The theme work was the loudest thing this week, but plenty of other commits landed alongside it:

SIMD large-allocation fallback. The SIMD path on iOS allocates its working buffers on the stack via alloca for speed. Past a certain buffer size the stack allocation simply fails — there is not enough stack to give, and the request crashes the process. The fix detects that case and falls back to a regular heap allocation when the request is too large to live on the stack. Small SIMD ops keep the fast alloca path; large ones no longer crash.
Pluggable AnimationTime clock. Motion, Timeline, MorphAnimation, Image.animate, and Label tickers now all route through a new AnimationTime class that defaults to System.currentTimeMillis() but can be overridden. Tests can drive animations deterministically frame by frame; demos can run in slow motion or fast forward; Motion.slowMotion is no longer the only lever.
POSIX character classes for non-ASCII letters. [[:alpha:]], [[:alnum:]], [[:lower:]], and [[:upper:]] silently failed to match anything outside the basic ASCII range — Greek, Cyrillic, CJK ideographs, accented letters, vulgar fractions, currency symbols. They now match the way you would expect, with five regression tests covering the failing cases from the issue.
Fail-fast on JDK < 11. The simulator and "Run as desktop app" goals fork the JVM with --add-exports=java.desktop/com.apple.eawt=ALL-UNNAMED, which JDK 8 rejects with the unhelpful "Could not create the Java Virtual Machine". Now the Maven plugin checks the runtime JDK version on entry to cn1:run and cn1:debug and aborts with a friendly message naming the detected version, JAVA_HOME, and a pointer to Adoptium. JDK 11 through 25 is the supported runtime range for the simulator, JDK 8 stays the build-time requirement for the core framework, and JDK 8 is still fully supported at runtime for shipped desktop apps — only the simulator / "Run as desktop app" Maven goals require JDK 11+.
Sheet scrolling swipe and animation. Sheet finally drags from the bottom with a real animation instead of snapping in. Issue #4825.
Picker positioning. Picker got additional button-positioning options and a small batch of coverage tests.
Playground polish. The Playground moved every Dialog.show(...) to InteractionDialog mode so user code calling Dialog.show does not blow away the editor chrome — it renders into the layered pane instead. Error messages got a substantial overhaul. The preview-resolution syntax expanded so the Playground can pick previews from a much wider set of expressions, with a new harness keeping it honest in CI.
Deeper refreshTheme(). Form.refreshTheme() has been around forever — it re-resolves the styles on a single Form. The new thing this week is UIManager.getInstance().refreshTheme(), which snapshots the current theme props and theme constants, clears the resolved-style caches, and re-applies the lot. This is what lets the screenshot suite flip dark mode mid-suite and see fresh styles, and what lets a runtime palette override take effect immediately. Most apps will never need to call it directly — palettes typically don't change at runtime, and a Display.setDarkMode(...) call already triggers the right invalidation. It is there if you do change the palette and want the change to stick on the next paint without reloading the theme from disk.

Where this is going — and a thank-you

Last week's post was about Codename One feeling faster: corrected pixel densities, principled scroll physics, SIMD on iOS, accessibility text scaling. This week is the symbiotic other half — Codename One looking like it belongs on a 2026 phone. Both halves are the same project. There is not much point in shipping a SIMD-accelerated Base64 if the surrounding UI looks like a 2014 app, and there is not much point in shipping a glass-frosted Dialog if the scroll underneath it judders.

Neither half is finished. They are both ongoing, and they both depend on community help — bug reports, RFEs, the patient back-and-forth on issue threads where somebody describes a layout problem on an iPhone you do not own. A specific thank you to the people who drove the issues that turned into this week's commits: Thomas (@ThomasH99) filed #4781 (the original "build a liquid glass example" RFE that started this whole effort), #4807 (sticky headers), #4838 (sideways tab swipe), #4841 (the POSIX regex fix), #4819 (picker buttons), and several others; Francesco Galgani (@jsfan3) filed #4825 (sheet swipe animation) and #4824 (light + dark theme by default in initializr); @ddyer0 caught #4811 (the EDT stack overflow) and #4767 (iPad restart Form size); Lucca Biagi (@LuccaPrado) filed #4817 (form creation in IntelliJ). Several of those are RFEs you would not file unless you actually use the framework day-to-day, and that is the kind of feedback that turns into shippable work.

We are sitting at 496 open issues as of this post. That is slow but steady progress — the number is moving in the right direction week over week, and the issues that close tend to ship as features or fixes you can see, not as silent triage. If you have a problem, file it. If you have an RFE, file that too. The themes you saw above started as an RFE.

You can try the new themes today by opening the Playground, by setting nativeTheme=modern (or ios.themeMode=modern / and.themeMode=modern for finer control) in your project's codenameone_settings.properties, or by picking them from the simulator's new Native Theme menu. New projects from the initializr already have them on. The shipping resources are bundled in the iOS and Android ports as of this week.

Front End Debugging Part 3: Networking

Shai Almog — Tue, 28 Jan 2025 16:46:20 +0000

Network Debugging Powerhouse
Re-Issuing and Modifying Requests
- cURL and Postman
Throttling and Debugging Race Conditions
- Why Throttling Matters:
- How to Use
Managing State with Storage Tools
- Challenges of State Management
Analyzing Request and Response Headers
Debugging in Incognito Mode: Limitations and Best Practices
Connecting the Front-End to the Database
Final Word

Debugging network communication issues is a critical skill for any front-end developer. While tools like Wireshark provide low-level insight into network traffic, modern browsers like Chrome and Firefox offer developer tools with powerful features tailored for web development. In this post we will discuss using browser-based tools to debug network communication issues effectively. This is a far better approach than using Wireshark for the vast majority of simple cases.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Network Debugging Powerhouse

Modern browsers come equipped with developer tools that rival standalone IDE debuggers in capability and convenience. Both Chrome and Firefox have robust network monitoring features that allow developers to observe/analyze requests and responses without leaving the browser.

On the basic level, which you’re probably familiar with, these tools include:

Network monitors: View all HTTP and HTTPS requests, including their headers, payloads, and responses.
Throttling controls: Simulate slower connections to test performance and debug race conditions.
Request replay functionality: Modify and resend requests directly from the browser.

While this post focuses on debugging techniques, it's worth noting that these tools are invaluable for performance optimization as well, though that topic warrants its own discussion.

Re-Issuing and Modifying Requests

One of the most powerful debugging features is the ability to re-issue requests. Instead of switching to external tools like cURL or Postman, browsers allow us to modify and resend requests directly.

This lets us quickly test variations of a failing API call to pinpoint issues without leaving the debugging environment. It’s especially useful when we have hard to reproduce issues or deep UI hierarchies.

In Firefox we can right-click any network entry in the Firefox Developer Tools and select "Resend." This opens an editable window where we can change request parameters, such as headers, payloads, or query strings, and resend the request.

Chrome provides similar functionality, though its interface for modifying and resending requests is slightly less direct than Firefox's.

cURL and Postman

Both browsers let you copy a request as a cURL command via the context menu. This is useful for reproducing issues in the terminal or sharing with back-end developers. I use this frequently as part of creating a reproducible issue.

If you prefer Postman, you can copy request headers and payloads from the browser and paste them into Postman to replicate requests.

Throttling and Debugging Race Conditions

Network throttling is a highly underrated feature that can be a game-changer for debugging specific classes of bugs. Both Chrome and Firefox allow developers to simulate various network speeds, from 2G connections to fast 4G.

Why Throttling Matters:

Some bugs only surface when requests arrive out of their expected order. Slowing down the network can help replicate and analyze these situations. Typical examples would be race conditions and related issues.

This is also very useful for simulating real-world conditions. Many users may not have fast or reliable internet connections. Throttling helps you understand how your application behaves in these scenarios.

I use this frequently when testing loading indicators which disappear too quickly when running locally. Instead of adding sleep code into the JavaScript or server code I can simulate slow-loading assets to verify that loading spinners or placeholders appear correctly.

How to Use

In Chrome we Open Developer Tools → Network tab.

We then use the "No throttling" dropdown to select pre-configured speeds or create a custom profile.

In Firefox we have similar functionality is available under the Network Monitor.

Managing State with Storage Tools

Local storage, session storage, and indexedDB often hold data critical to reproducing bugs, especially those tied to specific user states or devices.

Challenges of State Management

Even in incognito mode, state can persist if multiple private windows are open simultaneously. Persistence across sessions is a big challenge in these situations.

Understanding the exact state of a user's local storage can provide insight into seemingly random bugs. Debugging user-specific issues is problematic without control over storage.

In Firefox the dedicated Storage tab in Developer Tools makes it easy to inspect, edit, and delete data from local storage, session storage, cookies, and indexedDB.

In Chrome the Application tab consolidates all storage options, including the ability to clear specific caches or edit entries manually.

This functionality has many powerful uses for debugging:

Inject Debug Information: Tools like these let us manually add or modify storage data to simulate edge cases or specific user conditions.
Share Local State: Users can export their local storage, cookies, or indexedDB entries, allowing developers to reproduce issues locally.
Clear Cache Strategically: Clear only the relevant entries instead of a blanket cache clear, preserving useful state for debugging.

Analyzing Request and Response Headers

Request and response headers often hold the key to understanding network issues. We can use the network monitor to inspect:

Authorization headers: Check for missing or malformed tokens.
CORS headers: Verify that the server allows requests from your domain. These are some of the most painful type of http bugs. Reviewing these headers can be a lifesaver. If requests fail with CORS errors, inspect the Access-Control-Allow-Origin header in the response.
Cache-Control headers: Ensure proper caching behavior for your resources.

These tools are especially useful when debugging missing headers: Look for required headers like Content-Type or Authorization. Debugging Authentication: Use the "Copy as cURL" feature to test API calls with modified headers directly in the terminal.

Debugging in Incognito Mode: Limitations and Best Practices

Incognito mode can help isolate issues by providing a clean slate, however it has some limitations. Multiple incognito windows share the same state, which can lead to unintentional persistence of local data.

I suggest using storage management tools to manually clear or modify local data instead of relying solely on incognito mode. Keep only one incognito window open during testing to avoid unintended state sharing.

Connecting the Front-End to the Database

The front-end is often a transition point between user interaction and back-end data processing. While this post focuses on debugging the network layer, it's important to remember that:

Network issues often manifest due to back-end problems (e.g., a database error resulting in a 500 Internal Server Error).
Front-end developers should collaborate closely with back-end engineers to trace issues across the stack.

We can use custom response headers to include diagnostic information from the back end, such as query execution time or error codes. We can leverage server logs in conjunction with front-end debugging to get a complete picture of the issue.

Final Word

Browser developer tools are indispensable for debugging network communication issues, offering features like request replay, throttling, and storage management that simplify the debugging process. By mastering these tools, front-end developers can efficiently identify and resolve issues, ensuring a smoother user experience.

With the techniques and tips outlined in this post, you'll be better equipped to tackle network debugging challenges head-on. As you grow more familiar with these tools, you'll find them invaluable not only for debugging but also for improving your development workflow.

Front End Debugging Part 2: Console.log() to the Max

Shai Almog — Tue, 26 Nov 2024 17:53:11 +0000

Understanding Front-End Logging vs. Back-End Logging
Leveraging Console Log Levels
Customizing Console Output with CSS
Stack Tracing with console.trace()
Assertions for Design-by-Contract
Printing Tables for Clearer Data Visualization
Copying Objects to the Clipboard
Inspecting with console.dir() and dirxml()
Counting Function Calls
Organizing Logs with Groups
Chrome-Specific Debugging Features
Final Word

In my previous post I talked about why Console.log() isn’t the most effective debugging tool. In this installment, we will do a bit of an about-face and discuss the ways in which Console.log() is fantastic. Let’s break down some essential concepts and practices that can make your debugging life much easier and more productive.

Understanding Front-End Logging vs. Back-End Logging

Front-end logging differs significantly from back-end logging, and understanding this distinction is crucial. Unlike back-end systems, where persistent logs are vital for monitoring and debugging, the fluid nature of front-end development introduces different challenges. When debugging backends I’d often go for tracepoints which are far superior in that setting. However the front-end with its constant need to refresh, reload, contexts switch etc. is a very different beast. In the front-end relying heavily on elaborate logging mechanisms can become cumbersome.

While tracepoints remain superior to basic print statements, the continuous testing and browser reloading in front-end workflows lessen their advantage. Moreover, features like logging to a file or structured ingestion are rarely useful in the browser, diminishing the need for a comprehensive logging framework. However, using a logger is still considered best practice over the typical Console.log for long term logging… For short term logging Console.log has some tricks up its sleeve.

Leveraging Console Log Levels

One of the hidden gems of the browser console is its support for log levels, which is a significant step up from rudimentary print statements. The console provides five levels:

• log: Standard logging

• debug: Same as log but used for debugging purposes

• info: Informative messages, often rendered like log/debug

• warn: Warnings that might need attention

• error: Errors that have occurred

While log and debug can be indistinguishable, these levels allow for a more organized and filtered debugging experience. Browsers enable filtering the output based on these levels, mirroring the capabilities of server-side logging systems and allowing you to focus on relevant messages.

Customizing Console Output with CSS

Front-end development allows for creative solutions, and logging is no exception. Using CSS styles in the console can make logs more visually distinct. By utilizing %c in a console message, you can apply custom CSS:

console.customLog = function(msg) {
    console.log("%c" + msg,"color:black;background:pink;font-family:system-ui;font-size:4rem;-webkit-text-stroke: 1px black;font-weight:bold")
}
console.customLog("Dazzle")

This approach is helpful when you need to make specific logs stand out or organize output visually. You can use multiple %c substitutions to apply various styles to different parts of a log message.

Stack Tracing with console.trace()

The console.trace() method can print a stack trace at a particular location, which can sometimes be helpful for understanding the flow of your code. However, due to JavaScript’s asynchronous behavior, stack traces aren’t always as straightforward as in back-end debugging. Still, in specific scenarios, such as synchronous code segments or event handling, it can be quite valuable.

Assertions for Design-by-Contract

Assertions in front-end code allow developers to enforce expectations and promote a “fail-fast” mentality. Using Console.assert(), you can test conditions:

console.assert(x > 0, 'x must be greater than zero');

In the browser, a failed assertion appears as an error, similar to console.error. An added benefit is that assertions can be stripped from production builds, removing any performance impact. This makes assertions a great tool for enforcing design contracts during development without compromising production efficiency.

Printing Tables for Clearer Data Visualization

When working with arrays or objects, displaying data as tables can significantly enhance readability. The console.table() method allows you to output structured data easily:

console.table(["Simple Array", "With a few elements", "in line"])

This method is especially handy when debugging arrays of objects, presenting a clear, tabular view of the data and making complex data structures much easier to understand.

Copying Objects to the Clipboard

Debugging often involves inspecting objects, and the copy(object) method allows you to copy an object’s content to the clipboard for external use. This feature is useful when you need to transfer data or analyze it outside the browser.

Inspecting with console.dir() and dirxml()

The console.dir() method provides a more detailed view of objects, showing their properties as you’d see in a debugger. This is particularly helpful for inspecting DOM elements or exploring API responses. Meanwhile, console.dirxml() allows you to view objects as XML, which can be useful when debugging HTML structures.

Counting Function Calls

Keeping track of how often a function is called or a code block is executed can be crucial. The console.count() method tracks the number of times it’s invoked, helping you verify that functions are called as expected:

function myFunction() {
    console.count('myFunction called');
}

You can reset the counter using console.countReset(). This simple tool can help you catch performance issues or confirm the correct execution flow.

Organizing Logs with Groups

To prevent log clutter, use console groups to organize related messages. console.group() starts a collapsible log section, and console.groupEnd() closes it:

console.group('My Group');
console.log('Message 1');
console.log('Message 2');
console.groupEnd();

Grouping makes it easier to navigate complex logs and keeps your console clean.

Chrome-Specific Debugging Features

Monitoring Functions: Chrome’s monitor() method logs every call to a function, showing the arguments and enabling a method-tracing experience.

Monitoring Events: Using monitorEvents(), you can log events on an element. This is useful for debugging UI interactions. For example, monitorEvents(window, 'mouseout') logs only mouseout events.

Querying Object Instances: queryObjects(Constructor) lists all objects created with a specific constructor, giving you insights into memory usage and object instantiation.

Final Word

Front-end debugging tools have come a long way. These tools provide a rich set of features that go far beyond simple console.log() statements. From log levels and CSS styling to assertions and event monitoring, mastering these techniques can transform your debugging workflow.

If you read this post as part of my series you will notice a big change in my attitude toward debugging when we reached the front-end. Front-end debugging is very different when compared to backend debugging. When debugging the backend I’m vehemently against code changes for debugging (e.g. println debugging), but on the front-end this can be a reasonable hack. The change in environment justifies it. The short lifecycle, the single user use case and the risk is smaller.

While there are many transferrable skills we pick up while debugging, it’s important to remain flexible in our attitude. Next time we will discuss networking and storage debugging on the front-end.

Front End Debugging Part 1: Not just Console Log

Shai Almog — Tue, 29 Oct 2024 17:38:36 +0000

Instant Debugging with the debugger Keyword
Triggering Debugging from the Console
DOM Breakpoints: Monitoring DOM Changes
XHR Breakpoints: Uncovering Hidden Network Calls
Simulating Environments for Debugging
Debugging Layout and Style Issues
- Final Word

As a Java developer most of my focus is on the backend side of debugging. Front-end debugging poses different challenges and has sophisticated tools of its own. Unfortunately, print based debugging has become the norm in front-end. To be fair, it makes more sense there as the cycles are different and the problem is always a single user problem. But even if you choose to use Console.log, there’s a lot of nuance to pick up there.

Instant Debugging with the `debugger` Keyword

A cool yet powerful tool in JavaScript is the debugger keyword. Instead of simply printing a stack trace, we can use this keyword to launch the debugger directly at the line of interest. That is a fantastic tool that instantly brings your attention to a bug, I often use it in my debug builds of the front-end instead of just printing an error log.

How to Use It: Place the debugger keyword within your code, particularly within error-handling methods. When the code execution hits this line, it automatically pauses, allowing you to inspect the current state, step through code, and understand what's going wrong.

Notice that while this is incredibly useful during development, we must remember to remove or conditionally exclude debugger statements in production environments. A release build should not include these calls in a production site live environment.

Triggering Debugging from the Console

Modern browsers allow you to invoke debugging directly from the console, adding an additional layer of flexibility to your debugging process.

Example: By using the debug(functionName) command in the console, you can set a breakpoint at the start of the specified function. When this function is subsequently invoked, the execution halts, sending you directly into the debugger.

function hello(name) {
    Console.log("Hello " + name)
}
debug(hello)
hello("Shai")

This is particularly useful when you want to start debugging without modifying the source code, or when you need to inspect a function that’s only defined in the global scope.

DOM Breakpoints: Monitoring DOM Changes

DOM breakpoints are an advanced feature in Chrome and Firebug (Firefox plugin) that allow you to pause execution when a specific part of the DOM is altered.

To use it we can right-click on the desired DOM element, select “Break On,” and choose the specific mutation type you are interested in (e.g., subtree modifications, attribute changes, etc.).

DOM breakpoints are extremely powerful for tracking down issues where DOM manipulation causes unexpected results, such as dynamic content loading or changes in the user interface that disrupt the intended layout or functionality. Think of them like field breakpoints we discussed in the past.

These breakpoints complement traditional line and conditional breakpoints, providing a more granular approach to debugging complex front-end issues. This is a great tool to use when the DOM is manipulated by an external dependency.

XHR Breakpoints: Uncovering Hidden Network Calls

Understanding who initiates specific network requests can be challenging, especially in large applications with multiple sources contributing to a request. XHR (XMLHttpRequest) breakpoints provide a solution to this problem.

In Chrome or Firebug, set an XHR breakpoint by specifying a substring of the URI you wish to monitor. When a request matching this pattern is made, the execution stops, allowing you to investigate the source of the request.

This tool is invaluable when dealing with dynamically generated URIs or complex flows where tracking the origin of a request is not straightforward.

Notice that you should be selective with the filters you set; leaving the filter blank will cause the breakpoint to trigger on all XHR requests, which can become overwhelming.

Simulating Environments for Debugging

Sometimes, the issues you need to debug are specific to certain environments, such as mobile devices or different geographical locations. Chrome and Firefox offer several simulation tools to help you replicate these conditions on your desktop.

Simulating User Agents: Change the browser’s user agent to mimic different devices or operating systems. This can help you identify platform-specific issues or debug server-side content delivery that varies by user agent.
Geolocation Spoofing: Modify the browser’s reported location to test locale-specific features or issues. This is particularly useful for applications that deliver region-specific content or services.
Touch and Device Orientation Emulation: Simulate touch events or change the device orientation to see how your application responds to mobile-specific interactions. This is crucial for ensuring a seamless user experience across all devices.

These are things that are normally very difficult to reproduce. E.g. touch related issues are often challenging to debug on the device. By simulating them on the desktop browser we can shorten the debug cycle and use the tooling available on the desktop.

Debugging Layout and Style Issues

CSS and HTML bugs can be particularly tricky, often requiring a detailed examination of how elements are rendered and styled.

Inspect Element: The "inspect element" tool is the cornerstone of front-end debugging, allowing you to view and manipulate the DOM and CSS in real-time. As you make changes, the page updates instantly, providing immediate feedback on your tweaks.

Addressing Specificity Issues: One common problem is CSS specificity, where a more specific selector overrides the styles you intend to apply. The inspect element view highlights overridden styles, helping you identify and resolve conflicts.

Firefox vs. Chrome: While both browsers offer robust tools, they have different approaches to organizing these features. Firefox’s interface may seem more straightforward, with fewer tabs, while Chrome organizes similar tools under various tabs, which can either streamline your workflow or add complexity, depending on your preference.

Final Word

There are many front-end tools that I want to discuss in the coming posts. I hope you picked up a couple of new debugging tricks in this first part.

Front-end debugging requires deep understanding of browser tools and JavaScript capabilities. By mastering the techniques outlined in this post—instant debugging with the debugger keyword, DOM and XHR breakpoints, environment simulation, and layout inspection—you can significantly enhance your debugging efficiency and deliver more robust, error-free web applications.

The Art of Full Stack Debugging

Shai Almog — Tue, 08 Oct 2024 18:39:51 +0000

Full Stack Development, A Shifting Definition
- The Full Stack Debugging Approach
Frontend Debugging: Tools and Techniques
- It isn't "Just Console.log"
- The Power of Developer Tools
- Tackling Code Obfuscation
Debugging Across Layers
- Isolating Issues Across the Stack
- The Importance of System-Level Debugging
- Embracing Complexity
Conclusion

Full stack development is often likened to an intricate balancing act, where developers are expected to juggle multiple responsibilities across the frontend, backend, database, and beyond. As the definition of full stack development continues to evolve, so too does the approach to debugging. Full stack debugging is an essential skill for developers, as it involves tracking issues through multiple layers of an application, often navigating domains where one’s knowledge may only be cursory. In this blog post I aim to explore the nuances of full stack debugging, offering practical tips and insights for developers navigating the complex web of modern software development.

Notice that this is an introductory post focusing mostly on the front end debugging aspects, in the following posts I will dig deeper into the less familiar capabilities in front end debugging.

Full Stack Development, A Shifting Definition

The definition of full stack development is as fluid as the technology stacks themselves. Traditionally, full stack developers were defined as those who could work on both the frontend and backend of an application. However, as the industry evolves, this definition has expanded to include aspects of operations (OPS) and configuration. The modern full stack developer is expected to submit pull requests that cover all parts required to implement a feature—backend, database, frontend, and configuration. While this does not make them an expert in all these areas, it does require them to navigate across domains, often relying on domain experts for guidance.

I've heard it said that full stack developers are:

Jack of all trades, master of none.

However, the full quote probably better represents the reality:

Jack of all trades, master of none, but better than a master of one.

The Full Stack Debugging Approach

Just as full stack development involves working across various domains, full stack debugging requires a similar approach. A symptom of a bug may manifest in the frontend, but its root cause could lie deep within the backend or database layers. Full stack debugging is about tracing these issues through the layers and isolating them as quickly as possible. This is no easy task, especially when dealing with complex systems where multiple layers interact in unexpected ways. The key to successful full stack debugging lies in understanding how to track an issue through each layer of the stack and identifying common pitfalls that developers may encounter.

Frontend Debugging: Tools and Techniques

It isn't "Just Console.log"

Frontend developers are often stereotyped as relying solely on Console.log for debugging. While this method is simple and effective for basic debugging tasks, it falls short when dealing with the complex challenges of modern web development. The complexity of frontend code has increased significantly, making advanced debugging tools not just useful, but necessary. Yet, despite the availability of powerful debugging tools, many developers continue to shy away from them, clinging to old habits.

The Power of Developer Tools

Modern web browsers come equipped with robust developer tools that offer a wide range of capabilities for debugging frontend issues. These tools, available in browsers like Chrome and Firefox, allow developers to inspect elements, view and edit HTML and CSS, monitor network activity, and much more. One of the most powerful, yet underutilized, features of these tools is the JavaScript debugger.

The debugger allows developers to set breakpoints, step through code, and inspect the state of variables at different points in the execution. However, the complexity of frontend code, particularly when it has been obfuscated for performance reasons, can make debugging a challenging task.

We can launch the browser tools on Firefox using this menu:

On Chrome we can use this option:

I prefer working with Firefox, I find their developer tools more convenient but both browsers have similar capabilities. Both have fantastic debuggers (as you can see with the Firefox debugger below), unfortunately many developers limit themselves to console printing instead of exploring this powerful tool.

Tackling Code Obfuscation

Code obfuscation is a common practice in frontend development, employed to protect proprietary code and reduce file sizes for better performance. However, obfuscation also makes the code difficult to read and debug. Fortunately, both Chrome and Firefox developer tools provide a feature to de-obfuscate code, making it more readable and easier to debug. By clicking the curly brackets button in the toolbar, developers can transform a single line of obfuscated code into a well-formed, debuggable file.

Another important tool in the fight against obfuscation is the source map. Source maps are files that map obfuscated code back to its original source code, including comments. When generated and properly configured, source maps allow developers to debug the original code instead of the obfuscated version. In Chrome, this feature can be enabled by ensuring that "Enable JavaScript source maps" is checked in the developer tools settings.

You can use code like this in the JavaScript file to point at the sourcemap file:

//@sourceMappingURL=myfile.js.map

For this to work in Chrome we need to ensure that "Enable JavaScript source maps" is checked in the settings. Last I checked it was on by default but it doesn't hurt to verify:

Debugging Across Layers

Isolating Issues Across the Stack

In full stack development, issues often manifest in one layer but originate in another. For example, a frontend error might be caused by a misconfigured backend service or a database query that returns unexpected results. Isolating the root cause of these issues requires a methodical approach, starting from the symptom and working backward through the layers.

A common strategy is to reproduce the issue in a controlled environment, such as a local development setup, where each layer of the stack can be tested individually. This helps to narrow down the potential sources of the problem. Once the issue has been isolated to a specific layer, developers can use the appropriate tools and techniques to diagnose and resolve it.

The Importance of System-Level Debugging

Full stack debugging is not limited to the application code. Often, issues arise from the surrounding environment, such as network configurations, third-party services, or hardware limitations. A classic example of this that we ran into a couple of years ago was a production problem where a WebSocket connection would frequently disconnect. After extensive debugging, Steve discovered that the issue was caused by the CDN provider (CloudFlare) timing out the WebSocket after two minutes—something that could only be identified by debugging the entire system, not just the application code.

System-level debugging requires a broad understanding of how different components of the infrastructure interact with each other. It also involves using tools that can monitor and analyze the behavior of the system as a whole, such as network analyzers, logging frameworks, and performance monitoring tools.

Embracing Complexity

Full stack debugging is inherently complex, as it requires developers to navigate multiple layers of an application, often dealing with unfamiliar technologies and tools. However, this complexity also presents an opportunity for growth. By embracing the challenges of full stack debugging, developers can expand their knowledge and become more versatile in their roles.

One of the key strengths of full stack development is the ability to collaborate with domain experts. When debugging an issue that spans multiple layers, it is important to leverage the expertise of colleagues who specialize in specific areas. This collaborative approach not only helps to resolve issues more efficiently but also fosters a culture of knowledge sharing and continuous learning within the team.

As tools continue to evolve, so too do the tools and techniques available for debugging. Developers should strive to stay up-to-date with the latest advancements in debugging tools and best practices. Whether it’s learning to use new features in browser developer tools or mastering system-level debugging techniques, continuous learning is essential for success in full stack development.

Conclusion

Full stack debugging is a critical skill for modern developers, we mistakenly think it requires deep understanding of both the application and its surrounding environment. I disagree... By mastering the tools and techniques discussed in this post/upcoming posts, developers can more effectively diagnose and resolve issues that span multiple layers of the stack. Whether you’re dealing with obfuscated frontend code, misconfigured backend services, or system-level issues, the key to successful debugging lies in a methodical, collaborative approach.

You don't need to understand every part of the system, just the ability to eliminate the impossible.

Mastering Serverless Debugging

Shai Almog — Tue, 02 Jul 2024 16:26:02 +0000

Introduction to Serverless Computing
Challenges of Serverless Debugging
- Disconnected Environments
- Lack of Standardization
- Limited Debugging Tools
- Concurrency and Scale
Effective Strategies for Serverless Debugging
- Local Debugging with IDE Remote Capabilities
- Using Feature Flags for Debugging
- Staged Rollouts and Canary Deployments
- Comprehensive Logging
- Embracing Idempotency
Debugging a Lambda Application Locally with AWS SAM
- Setting Up the Local Environment
- Running the Hello World Application Locally
- Configuring Remote Debugging
- Handling Debugger Timeouts
Final Word

Serverless computing has emerged as a transformative approach to deploying and managing applications. The theory is that by abstracting away the underlying infrastructure, developers can focus solely on writing code. While the benefits are clear—scalability, cost efficiency, and performance—debugging serverless applications presents unique challenges. This post explores effective strategies for debugging serverless applications, particularly focusing on AWS Lambda.

Before I proceed I think it's important to disclose a bias: I am personally not a huge fan of Serverless or PaaS after I was burned badly by PaaS in the past. However, some smart people like Adam swear by it so I should keep an open mind.

Introduction to Serverless Computing

Serverless computing, often referred to as Function as a Service (FaaS), allows developers to build and run applications without managing servers. In this model, cloud providers automatically handle the infrastructure, scaling, and management tasks, enabling developers to focus purely on writing and deploying code. Popular serverless platforms include AWS Lambda, Azure Functions, and Google Cloud Functions.

In contrast, Platform as a Service (PaaS) offers a more managed environment where developers can deploy applications but still need to configure and manage some aspects of the infrastructure. PaaS solutions, such as Heroku and Google App Engine, provide a higher level of abstraction than Infrastructure as a Service (IaaS) but still require some server management.

Kubernetes, which we recently discussed, is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. While Kubernetes offers powerful capabilities for managing complex, multi-container applications, it requires significant expertise to set up and maintain. Serverless computing simplifies this by removing the need for container orchestration and management altogether.

The "catch" is two fold:

Serverless programming removes the need to understand the servers but also removes the ability to rely on them resulting in more complex architectures.
Pricing starts off cheap. Practically free. It can quickly escalate especially in case of an attack or misconfiguration.

Challenges of Serverless Debugging

While serverless architectures offer some benefits, they also introduce unique debugging challenges. The primary issues stem from the inherent complexity and distributed nature of serverless environments. Here are some of the most pressing challenges.

Disconnected Environments

One of the major hurdles in serverless debugging is the lack of consistency between development, staging, and production environments. While traditional development practices rely on these separate environments to test and validate code changes, serverless architectures often complicate this process. The differences in configuration and scale between these environments can lead to bugs that only appear in production, making them difficult to reproduce and fix.

Lack of Standardization

The serverless ecosystem is highly fragmented, with various vendors offering different tools and frameworks. This lack of standardization can make it challenging to adopt a unified debugging approach. Each platform has its own set of practices and tools, requiring developers to learn and adapt to multiple environments.

This is slowly evolving with some platforms gaining traction, but since this is a vendor driven industry there are many edge cases.

Limited Debugging Tools

Traditional debugging tools, such as step-through debugging and breakpoints, are often unavailable in serverless environments. The managed and controlled nature of serverless functions restricts access to these tools, forcing developers to rely on alternative methods, such as logging and remote debugging.

Concurrency and Scale

Serverless functions are designed to handle high concurrency and scale seamlessly. However, this can introduce issues that are hard to reproduce in a local development environment. Bugs that manifest only under specific concurrency conditions or high load are particularly challenging to debug.

Notice that when I discuss concurrency here I'm often referring to race conditions between separate services.

Effective Strategies for Serverless Debugging

Despite these challenges, several strategies can help make serverless debugging more manageable. By leveraging a combination of local debugging, feature flags, staged rollouts, logging, idempotency, and Infrastructure as Code (IaC), developers can effectively diagnose and fix issues in serverless applications.

Local Debugging with IDE Remote Capabilities

While serverless functions run in the cloud, you can simulate their execution locally using tools like AWS SAM (Serverless Application Model). This involves setting up a local server that mimics the cloud environment, allowing you to run tests and perform basic trial-and-error debugging.

To get started, you need to install Docker or Docker Desktop, create an AWS account, and set up the AWS SAM CLI. Deploy your serverless application locally using the SAM CLI, which enables you to run the application and simulate Lambda functions on your local machine. Configure your IDE for remote debugging, launching the application in debug mode, and connecting your debugger to the local host. Set breakpoints to step through the code and identify issues.

Using Feature Flags for Debugging

Feature flags allow you to enable or disable parts of your application without deploying new code. This can be invaluable for isolating issues in a live environment. By toggling specific features on or off, you can narrow down the problematic areas and observe the application’s behavior under different configurations.

Implementing feature flags involves adding conditional checks in your code that control the execution of specific features based on the flag’s status. Monitoring the application with different flag settings helps identify the source of bugs and allows you to test fixes without affecting the entire user base.

This is essentially "debugging in production". Working on a new feature?

Wrap it in a feature flag which is effectively akin to wrapping the entire feature (client and server) in if statements. You can then enable it conditionally globally or on a per user basis. This means you can test the feature, enable or disable it based on configuration without redeploying the application.

Staged Rollouts and Canary Deployments

Deploying changes incrementally can help catch bugs before they affect all users. Staged rollouts involve gradually rolling out updates to a small percentage of users before a full deployment. This allows you to monitor the performance and error logs of the new version in a controlled manner, catching issues early.

Canary deployments take this a step further by deploying new changes to a small subset of instances (canaries) while the rest of the system runs the stable version. If issues are detected in the canaries, you can roll back the changes without impacting the majority of users. This method limits the impact of potential bugs and provides a safer way to introduce updates. This isn't great as in some cases some demographics might be more reluctant to report errors. However, for server side issues this might make sense as you can see the impact based on server logs and metrics.

Comprehensive Logging

Logging is one of the most common and essential tools for debugging serverless applications. I wrote and spoke a lot about logging in the past. By logging all relevant data points, including inputs and outputs of your functions, you can trace the flow of execution and identify where things go wrong.

However, excessive logging can increase costs, as serverless billing is often based on execution time and resources used. It’s important to strike a balance between sufficient logging and cost efficiency. Implementing log levels and selectively enabling detailed logs only when necessary can help manage costs while providing the information needed for debugging.

I talk about striking the delicate balance between debuggable code, performance and cost with logs in the following video. Notice that this is a general best practice and not specific to serverless.

Embracing Idempotency

Idempotency, a key concept from functional programming, ensures that functions produce the same result given the same inputs, regardless of the number of times they are executed. This simplifies debugging and testing by ensuring consistent and predictable behavior.

Designing your serverless functions to be idempotent involves ensuring that they do not have side effects that could alter the outcome when executed multiple times. For example, including timestamps or unique identifiers in your requests can help maintain consistency. Regularly testing your functions to verify idempotency can make it easier to pinpoint discrepancies and debug issues.

Testing is always important but in serverless and complex deployments it becomes critical. Awareness and embrace of idempotency allows for more testable code and easier to reproduce bugs.

Debugging a Lambda Application Locally with AWS SAM

Debugging serverless applications, particularly AWS Lambda functions, can be challenging due to their distributed nature and the limitations of traditional debugging tools. However, AWS SAM (Serverless Application Model) provides a way to simulate Lambda functions locally, enabling developers to test and debug their applications more effectively. I will use it as a sample to explore the process of setting up a local debugging environment, running a sample application, and configuring remote debugging.

Setting Up the Local Environment

Before diving into the debugging process, it's crucial to set up a local environment that can simulate the AWS Lambda environment. This involves a few key steps:

Install Docker: Docker is required to run the local simulation of the Lambda environment. You can download Docker or Docker Desktop from the official Docker website.
Create an AWS Account: If you don't already have an AWS account, you need to create one. Follow the instructions on the AWS account creation page.
Set Up AWS SAM CLI: The AWS SAM CLI is essential for building and running serverless applications locally. You can install it by following the AWS SAM installation guide.

Running the Hello World Application Locally

To illustrate the debugging process, let's use a simple "Hello World" application. The code for this application can be found in the AWS Hello World tutorial.

Deploy Locally: Use the SAM CLI to deploy the Hello World application locally. This can be done with the following command:
```
sam local start-api
```
This command starts a local server that simulates the AWS Lambda cloud environment.
Trigger the Endpoint: Once the local server is running, you can trigger the endpoint using a curl command:
```
curl http://localhost:3000/hello
```
This command sends a request to the local server, allowing you to test the function's response.

Configuring Remote Debugging

While running tests locally is a valuable step, it doesn't provide full debugging capabilities. To debug the application, you need to configure remote debugging. This involves several steps.

First we need to start the application in debug mode using the following SAM command:

sam local invoke -d 5858

This command pauses the application and waits for a debugger to connect.

Next we need to configure the IDE for remote debugging. We start by setting up the IDE to connect to the local host for remote debugging. This typically involves creating a new run configuration that matches the remote debugging settings.

We can now set breakpoints in the code where we want the execution to pause. This allows us to step through the code and inspect variables and application state just like in any other local application.

We can test this by invoking the endpoint e.g. using curl. With the debugger connected we would stop on the breakpoint like any other tool:

curl http://localhost:3000/hello

The application will pause at the breakpoints you set, allowing you to step through the code.

Handling Debugger Timeouts

One significant challenge when debugging Lambda functions is the quick timeout setting. Lambda functions are designed to execute quickly, and if they take too long, the costs can become prohibitive. By default, the timeout is set to a short duration, but you can configure this in the template.yaml file e.g.:

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: app.lambdaHandler
      Timeout: 60  # timeout in seconds

After updating the timeout value, re-issue the sam build command to apply the changes.

In some cases, running the application locally might not be enough. You may need to simulate running on the actual AWS stack to get more accurate debugging information. Solutions like SST (Serverless Stack) or MerLoc can help achieve this, though they are specific to AWS and relatively niche.

Final Word

Serverless debugging requires a combination of strategies to effectively identify and resolve issues. While traditional debugging methods may not always apply, leveraging local debugging, feature flags, staged rollouts, comprehensive logging, idempotency, and IaC can significantly improve your ability to debug serverless applications. As the serverless ecosystem continues to evolve, staying adaptable and continuously updating your debugging techniques will be key to success.

Debugging serverless applications, particularly AWS Lambda functions, can be complex due to their distributed nature and the constraints of traditional debugging tools. However, by leveraging tools like AWS SAM, you can simulate the Lambda environment locally and use remote debugging to step through your code. Adjusting timeout settings and considering advanced simulation tools can further enhance your debugging capabilities.

Debugging Kubernetes - Troubleshooting Guide

Shai Almog — Tue, 18 Jun 2024 14:26:44 +0000

Identifying Configuration Issues
- Common Causes and Solutions
- Detailed Investigation Steps
Dealing with Image Pull Errors
- Troubleshooting Steps
Handling Node Issues
- Preventive Measures
Managing Missing Configuration Keys or Secrets
Utilizing Buildg for Interactive Debugging
Conclusion

As Kubernetes continues to revolutionize the way we manage and deploy applications, understanding its intricacies becomes essential for developers and operations teams alike. If you don't have a dedicated DevOps team you probably shouldn't be working with Kubernetes. Despite that, in some cases a DevOps engineer might not be available while we're debugging an issue. For these situations and for general familiarity we should still familiarize ourselves with common Kubernetes issues to bridge the gap between development and operations. I think this also provides an important skill that helps us understand the work of DevOps better, with that understanding we can improve as a cohesive team. This guide explores prevalent Kubernetes errors and provides troubleshooting tips to help developers navigate the complex landscape of container orchestration.

Identifying Configuration Issues

When you encounter configuration issues in Kubernetes, the first place to check is the status column using the kubectl get pods command. Common errors manifest here, requiring further inspection with kubectl describe pod.

$ kubectl get pods
NAME                     READY    STATUS     RESTARTS   AGE 
my-first-pod-id-xxxx      1/1     Running    0          13s
my-second-pod-id-xxxx     1/1     Running    0          13s

Common Causes and Solutions

Insufficient Resources: Notice that this means resources for the POD itself and not resources within the container. It means the hardware or surrounding VM is hitting a limit.

Symptom: Pods fail to schedule due to resource constraints.

Solution: Scale up the cluster by adding more nodes to accommodate the resource requirements.

Volume Mounting Failures:

Symptom: Pods cannot mount volumes correctly.

Solution: Ensure storage is defined accurately in the pod specification and check the storage class and Persistent Volume (PV) configurations.

Detailed Investigation Steps

We can use kubectl describe pod: This command provides a detailed description of the pod, including events that have occurred. By examining these events, we can pinpoint the exact cause of the issue.

Another important step is resource quota analysis. Sometimes, resource constraints are due to namespace-level resource quotas. Use kubectl get resourcequotas to check if quotas are limiting pod creation.

Dealing with Image Pull Errors

Errors like ErrImagePull or ImagePullBackOff indicate issues with fetching container images. These errors are typically related to image availability or access permissions.

Troubleshooting Steps

The first step is checking the image name which we can do with the following command:

docker pull <image-name>

We then need to verify the image name for typos or invalid characters. I pipe the command through grep to verify the name is 100% identical, some typos are just notoriously hard to spot.

Credentials can also be a major pitfall. E.g. an authorization failure when pulling images from private repositories.

We must ensure that Docker registry credentials are correctly configured in Kubernetes secrets.

Network configuration should also be reviewed. Ensure that the Kubernetes nodes have network access to the Docker registry. Network policies or firewall rules might block access.

There are quite a few additional pitfalls such as problems with image tags. Ensure you are using the correct image tags. Latest tags might not always point to the expected image version.

If you're using a private registry you might be experiencing access issues. Make sure your credentials are up-to-date and the registry is accessible from all nodes in all regions.

Handling Node Issues

Node-related errors often point to physical or virtual machine issues. These issues can disrupt the normal operation of the Kubernetes cluster and need prompt attention.

To check node status use the command:

kubectl get nodes

We can then identify problematic nodes in the resulting output.

It's a cliché but sometimes rebooting nodes is the best solution to some problems. We can reboot the affected machine or VM. Kubernetes should attempt to "self-heal" and recover within a few minutes.

To investigate node conditions we can use the command:

kubectl describe node <node-name>

We should look for conditions such as MemoryPressure, DiskPressure, or NetworkUnavailable. These conditions provide clues about the underlying issue we should address in the node.

Preventive Measures

Node monitoring should be used to with tools such as Prometheus, Grafana to keep an eye on node health and performance. These work great for the low level Kubernetes related issues, we can also use them for high level application issues.

There are some automated healing tools such as the Kubernetes Cluster Autoscaler that we can leverage to automatically manage the number of nodes in your cluster based on workload demands. Personally, I'm not a huge fan as I'm afraid of a cascading failure that would trigger additional resource consumption.

Managing Missing Configuration Keys or Secrets

Missing configuration keys or secrets are common issues that disrupt Kubernetes deployments. Proper management of these elements is crucial for smooth operation.

We need to use ConfigMaps and secrets. These let us store configuration values and sensitive information securely. To avoid that we need to ensure that ConfigMaps and Secrets are correctly referenced in your pod specifications.

Inspect pod descriptions using the command:

kubectl describe pod <pod-name>

Review the output and look for missing configuration details. Rectify any misconfigurations.

ConfigMap and secret creation can be verified using the command:

kubectl get configmaps

and:

kubectl get secrets

Ensure that the required ConfigMaps and Secrets exist in the namespace and contain the expected data.

It's best to keep non-sensitive parts of ConfigMaps in version control while excluding Secrets for security. Furthermore, you should use different ConfigMaps and Secrets for different environments (development, staging, production) to avoid configuration leaks.

Utilizing Buildg for Interactive Debugging

Buildg is a relatively new tool that enhances the debugging process for Docker configurations by allowing interactive debugging.

It provides Interactive Debugging for configuration issues in a way that's similar to a standard debugging. It lets us step through the Dockerfile stages and set breakpoints. Buildg is compatible with VSCode and other IDEs via the Debug Adapter Protocol (DAP).

Buildg lets us inspect container state at each stage of the build process to identify issues early.

To install buildg follow the instructions on the Buildg GitHub page.

Conclusion

Debugging Kubernetes can be challenging, but with the right knowledge and tools, developers can effectively identify and resolve common issues. By understanding configuration problems, image pull errors, node issues, and the importance of ConfigMaps and Secrets, developers can contribute to more robust and reliable Kubernetes deployments. Tools like Buildg offer promising advancements in interactive debugging, further bridging the gap between development and operations.

As Kubernetes continues to evolve, staying informed about new tools and best practices will be essential for successful application management and deployment. By proactively addressing these common issues, developers can ensure smoother, more efficient Kubernetes operations, ultimately leading to more resilient and scalable applications.

Why is Kubernetes Debugging so Problematic?

Shai Almog — Tue, 04 Jun 2024 15:15:04 +0000

The Immutable Nature of Containers
The Limitations of kubectl exec
Avoiding Direct Modifications
Enter Ephemeral Containers
- Using kubectl debug
Practical Application of Ephemeral Containers
Security Considerations
Interlude: The Role of Observability
Command Line Debugging
Connecting a Standard IDE for Remote Debugging
Conclusion

Debugging application issues in a Kubernetes cluster can often feel like navigating a labyrinth. Containers are ephemeral by design, intended to be immutable once deployed. This presents a unique challenge when something goes wrong and we need to dig into the issue. Before diving into the debugging tools and techniques, it's essential to grasp the core problem: why modifying container instances directly is a bad idea. This blog post will walk you through the intricacies of Kubernetes debugging, offering insights and practical tips to effectively troubleshoot your Kubernetes environment.

The Immutable Nature of Containers

One of the fundamental principles of Kubernetes is the immutability of container instances. This means that once a container is running, it shouldn't be altered. Modifying containers on the fly can lead to inconsistencies and unpredictable behavior, especially as Kubernetes orchestrates the lifecycle of these containers, replacing them as needed. Imagine trying to diagnose an issue only to realize that the container you’re investigating has been modified, making it difficult to reproduce the problem consistently.

The idea behind this immutability is to ensure that every instance of a container is identical to any other instance. This consistency is crucial for achieving reliable, scalable applications. If you start modifying containers, you undermine this consistency, leading to a situation where one container behaves differently from another, even though they are supposed to be identical.

The Limitations of `kubectl exec`

We often start our journey in Kubernetes with commands such as:

$ kubectl -- exec -ti <pod-name>

This logs into a container and feels like accessing a traditional server with SSH. However, this approach has significant limitations. Containers often lack basic diagnostic tools—no vim, no traceroute, sometimes not even a shell. This can be a rude awakening for those accustomed to a full-featured Linux environment. Additionally, if a container crashes, kubectl exec becomes useless as there's no running instance to connect to. This tool is insufficient for thorough debugging, especially in production environments.

Consider the frustration of logging into a container only to find out that you can't even open a simple text editor to check configuration files. This lack of basic tools means that you are often left with very few options for diagnosing problems. Moreover, the minimalistic nature of many container images, designed to reduce their attack surface and footprint, exacerbates this issue.

Avoiding Direct Modifications

While it might be tempting to install missing tools on-the-fly using commands like apt-get install vim, this practice violates the principle of container immutability. In production, installing packages dynamically can introduce new dependencies, potentially causing application failures. The risks are high, and it's crucial to maintain the integrity of your deployment manifests, ensuring that all configurations are predefined and reproducible.

Imagine a scenario where a quick fix in production involves installing a missing package. This might solve the immediate problem but could lead to unforeseen consequences. Dependencies introduced by the new package might conflict with existing ones, leading to application instability. Moreover, this approach makes it challenging to reproduce the exact environment, which is vital for debugging and scaling your application.

Enter Ephemeral Containers

The solution to the aforementioned problems lies in ephemeral containers. Kubernetes allows the creation of these temporary containers within the same pod as the application container you need to debug. These ephemeral containers are isolated from the main application, ensuring that any modifications or tools installed do not impact the running application.

Ephemeral containers provide a way to bypass the limitations of kubectl exec without violating the principles of immutability and consistency. By launching a separate container within the same pod, you can inspect and diagnose the application container without altering its state. This approach preserves the integrity of the production environment while giving you the tools you need to debug effectively.

Using `kubectl debug`

The kubectl debug command is a powerful tool that simplifies the creation of ephemeral containers. Unlike kubectl exec, which logs into the existing container, kubectl debug creates a new container within the same namespace. This container can run a different OS, mount the application container’s filesystem, and provide all necessary debugging tools without altering the application’s state. This method ensures you can inspect and diagnose issues even if the original container is not operational.

For example, let’s consider a scenario where we’re debugging a container using an ephemeral Ubuntu container:

kubectl debug <myapp> -it <pod-name> --image=ubuntu --share-process --copy-to=<myapp-debug>

This command launches a new Ubuntu-based container within the same pod, providing a full-fledged environment to diagnose the application container. Even if the original container lacks a shell or crashes, the ephemeral container remains operational, allowing you to perform necessary checks and install tools as needed. It relies on the fact that we can have multiple containers in the same pod, that way we can inspect the filesystem of the debugged container without physically entering that container.

Practical Application of Ephemeral Containers

To illustrate, let’s delve deeper into how ephemeral containers can be used in real-world scenarios. Suppose you have a container that consistently crashes due to a mysterious issue. By deploying an ephemeral container with a comprehensive set of debugging tools, you can monitor the logs, inspect the filesystem, and trace processes without worrying about the constraints of the original container environment.

For instance, you might encounter a situation where an application container crashes due to an unhandled exception. By using kubectl debug, you can create an ephemeral container that shares the same network namespace as the original container. This allows you to capture network traffic and analyze it to understand if there are any issues related to connectivity or data corruption.

Security Considerations

While ephemeral containers reduce the risk of impacting the production environment, they still pose security risks. It’s critical to restrict access to debugging tools and ensure that only authorized personnel can deploy ephemeral containers. Treat access to these systems with the same caution as handing over the keys to your infrastructure.

Ephemeral containers, by their nature, can access sensitive information within the pod. Therefore, it is essential to enforce strict access controls and audit logs to track who is deploying these containers and what actions are being taken. This ensures that the debugging process does not introduce new vulnerabilities or expose sensitive data.

Interlude: The Role of Observability

While tools like kubectl exec and kubectl debug are invaluable for troubleshooting, they are not replacements for comprehensive observability solutions. Observability allows you to monitor, trace, and log the behavior of your applications in real-time, providing deeper insights into issues without the need for intrusive debugging sessions.

These tools aren't meant for everyday debugging, that role should be occupied by various observability tools. I will discuss observability in more detail in an upcoming post.

Command Line Debugging

While tools like kubectl exec and kubectl debug are invaluable, there are times when you need to dive deep into the application code itself. This is where we can use command line debuggers. Command line debuggers allow you to inspect the state of your application at a very granular level, stepping through code, setting breakpoints, and examining variable states. Personally, I don't use them much

For instance, Java developers can use jdb, the Java Debugger, which is analogous to gdb for C/C++ programs. Here’s a basic rundown of how you might use jdb in a Kubernetes environment:

Set Up Debugging: First, you need to start your Java application with debugging enabled. This typically involves adding a debug flag to your Java command. However, as discussed in my post here, there's an even more powerful way that doesn't require a restart:

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar myapp.jar

Port Forwarding: Since the debugger needs to connect to the application, you’ll set up port forwarding to expose the debug port of your pod to your local machine. This is important as JDWP is dangerous:

kubectl port-forward <pod-name> 5005:5005

Connecting the Debugger: With port forwarding in place, you can now connect jdb to the remote application:

jdb -attach localhost:5005

From here, you can use jdb commands to set breakpoints, step through code, and inspect variables. This process allows you to debug issues within the code itself, which can be invaluable for diagnosing complex problems that aren’t immediately apparent through logs or superficial inspection.

Connecting a Standard IDE for Remote Debugging

I prefer IDE debugging by far. I never used JDB for anything other than a demo. Modern IDEs support remote debugging, and by leveraging Kubernetes port forwarding, you can connect your IDE directly to a running application inside a pod.

To set up remote debugging we start with the same steps as the command line debugging. Configuring the application and setting up the port forwarding.

Configure the IDE: In your IDE (e.g., IntelliJ IDEA, Eclipse), set up a remote debugging configuration. Specify the host as localhost and the port as 5005.
Start Debugging: Launch the remote debugging session in your IDE. You can now set breakpoints, step through code, and inspect variables directly within the IDE, just as if you were debugging a local application.

I show how to do it in IntelliJ/IDEA here.

Conclusion

Debugging Kubernetes environments requires a blend of traditional techniques and modern tools designed for container orchestration. Understanding the limitations of kubectl exec and the benefits of ephemeral containers can significantly enhance your troubleshooting process. However, the ultimate goal should be to build robust observability into your applications, reducing the need for ad-hoc debugging and enabling proactive issue detection and resolution.

By following these guidelines and leveraging the right tools, you can navigate the complexities of Kubernetes debugging with confidence and precision. In the next installment of this series, we’ll delve into common configuration issues in Kubernetes and how to address them effectively.

Debugging Kubernetes Part 1: An Introduction

Shai Almog — Tue, 21 May 2024 14:56:00 +0000

Introduction to Kubernetes and Distributed Systems
- The Evolution of Deployment Technologies
- Enter Virtualization
- Rise of Containers
- Rise of Orchestration
Enter Kubernetes
- Why Kubernetes Stands Out
Kubernetes For Developers
- Kubernetes Basics In Practice
Final Word

While debugging in an IDE or using simple command line tools is relatively straightforward, the real challenge lies in production debugging. Modern production environments have enabled sophisticated self-healing deployments, yet they have also made troubleshooting more complex. Kubernetes (aka k8s) is probably the most well known orchestration production environment. To effectively teach debugging in Kubernetes, it's essential to first introduce its fundamental principles.

This part of the debugging series is designed for developers looking to effectively tackle application issues within Kubernetes environments, without delving deeply into the complex DevOps aspects typically associated with its operations. Kubernetes is a big subject, it took me two videos just to explain the basic concepts and background.

Introduction to Kubernetes and Distributed Systems

Kubernetes, while often discussed in the context of cloud computing and large-scale operations, is not just a tool for managing containers. Its principles apply broadly to all large-scale distributed systems. In this post I want to explore Kubernetes from the ground up, emphasizing its role in solving real-world problems faced by developers in production environments.

The Evolution of Deployment Technologies

Before Kubernetes, the deployment landscape was markedly different. Understanding this evolution helps appreciate the challenges Kubernetes aims to solve. The image below represents the road to Kubernetes and the technologies we passed along the way.

In the image we can see that initially, applications were deployed directly onto physical servers. This process was manual, error-prone, and difficult to replicate across multiple environments. For instance, if a company needed to scale its application, it involved procuring new hardware, installing operating systems, and configuring the application from scratch. This could take weeks or even months, leading to significant downtime and operational inefficiencies.

Imagine a retail company preparing for the holiday season surge. Each time they needed to handle increased traffic, they would manually set up additional servers. This was not only time-consuming but also prone to human error. Scaling down after the peak period was equally cumbersome, leading to wasted resources.

Enter Virtualization

Virtualization technology introduced a layer that emulated the hardware, allowing for easier replication and migration of environments but at the cost of performance. However, fast virtualization enabled the cloud revolution. It let companies like Amazon lease its servers at scale without compromising their own workloads.

Virtualization involves running multiple operating systems on a single physical hardware host. Each virtual machine (VM) includes a full copy of an operating system, the application, necessary binaries, and libraries—taking up tens of GBs. VMs are managed via a hypervisor, such as VMware's ESXi or Microsoft's Hyper-V, which sits between the hardware and the operating system and is responsible for distributing hardware resources among the VMs. This layer adds additional overhead and can lead to decreased performance due to the need to emulate hardware.

Note that virtualization is often referred to as "virtual machines", I chose to avoid that terminology due to the focus of this blog on Java and the JVM where a virtual machine is typically a reference to the Java Virtual Machine (JVM).

Rise of Containers

Containers emerged as a lightweight alternative to full virtualization. Tools like Docker standardized container formats, making it easier to create and manage containers without the overhead associated with traditional virtual machines. Containers encapsulate an application’s runtime environment, making them portable and efficient.

Unlike virtualization, containerization encapsulates an application in a container with its own operating environment, but it shares the host system’s kernel with other containers. Containers are thus much more lightweight, as they do not require a full OS instance; instead, they include only the application and its dependencies, such as libraries and binaries. This setup reduces the size of each container and improves boot times and performance by removing the hypervisor layer.

Containers operate using several key Linux kernel features:

Namespaces: Containers use namespaces to provide isolation for global system resources between independent containers. This includes aspects of the system like process IDs, networking interfaces, and file system mounts. Each container has its own isolated namespace, which gives it a private view of the operating system with access only to its resources.
Control Groups (cgroups): Cgroups further enhance the functionality of containers by limiting and prioritizing the hardware resources a container can use. This includes parameters such as CPU time, system memory, network bandwidth, or combinations of these resources. By controlling resource allocation, cgroups ensure that containers do not interfere with each other’s performance and maintain the efficiency of the underlying server.
Union File Systems: Containers use union file systems, such as OverlayFS, to layer files and directories in a lightweight and efficient manner. This system allows containers to appear as though they are running on their own operating system and file system, while they are actually sharing the host system’s kernel and base OS image.

Rise of Orchestration

As containers began to replace virtualization due to their efficiency and speed, developers and organizations rapidly adopted them for a wide range of applications. However, this surge in container usage brought with it a new set of challenges, primarily related to managing large numbers of containers at scale.

While containers are incredibly efficient and portable, they introduce complexities when used extensively, particularly in large-scale, dynamic environments:

Management Overhead: Manually managing hundreds or even thousands of containers quickly becomes unfeasible. This includes deployment, networking, scaling, and ensuring availability and security.
Resource Allocation: Containers must be efficiently scheduled and managed to optimally use physical resources, avoiding underutilization or overloading of host machines.
Service Discovery and Load Balancing: As the number of containers grows, keeping track of which container offers which service and how to balance the load between them becomes critical.
Updates and Rollbacks: Implementing rolling updates, managing version control, and handling rollbacks in a containerized environment require robust automation tools.

To address these challenges, the concept of container orchestration was developed. Orchestration automates the scheduling, deployment, scaling, networking, and lifecycle management of containers, which are often organized into microservices. Efficient orchestration tools help ensure that the entire container ecosystem is healthy and that applications are running as expected.

Enter Kubernetes

Among the orchestration tools, Kubernetes emerged as a frontrunner due to its robust capabilities, flexibility, and strong community support. Kubernetes offers several features that address the core challenges of managing containers:

Automated Scheduling: Kubernetes intelligently schedules containers on the cluster’s nodes, taking into account the resource requirements and other constraints, optimizing for efficiency and fault tolerance.
Self-Healing Capabilities: It automatically replaces or restarts containers that fail, ensuring high availability of services.
Horizontal Scaling: Kubernetes can automatically scale applications up and down based on demand, which is essential for handling varying loads efficiently.
Service Discovery and Load Balancing: Kubernetes can expose a container using the DNS name or using its own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic so that the deployment is stable.
Automated Rollouts and Rollbacks: Kubernetes allows you to describe the desired state for your deployed containers using declarative configuration, and can change the actual state to the desired state at a controlled rate, such as to roll out a new version of an application.

Why Kubernetes Stands Out

Kubernetes not only solves practical, operational problems associated with running containers but also integrates with the broader technology ecosystem, supporting continuous integration and continuous deployment (CI/CD) practices. It is backed by the Cloud Native Computing Foundation (CNCF), ensuring it remains cutting-edge and community-focused.

There used to be a site called "doyouneedkubernetes.com" when you visited that site it said "No". Most of us don't need Kubernetes and it is often a symptom of Resume Driven Design (RDD). However, even when we don't need its scaling capabilities the advantages of its standardization are tremendous. Kubernetes became the de-facto standard and created a cottage industry of tools around it. Features such as, observability and security can be plugged in easily. Cloud migration becomes arguably easier. Kubernetes is now the "lingua franca" of production environments.

Kubernetes For Developers

Understanding Kubernetes architecture is crucial for debugging and troubleshooting. The following image shows the high level view of a Kubernetes deployment. There are far more details in most tutorials geared towards DevOps engineers, but for a developer the point that matters is just "Your Code": that tiny corner at the edge.

In the image above we can see:

Master Node (represented by the blue Kubernetes logo on the left): The control plane of Kubernetes, responsible for managing the state of the cluster, scheduling applications, and handling replication.
Worker Nodes: These nodes contain the pods that run the containerized applications. Each worker node is managed by the master.
Pods: The smallest deployable units created and managed by Kubernetes, usually containing one or more containers that need to work together.

These components work together to ensure that an application runs smoothly and efficiently across the cluster.

Kubernetes Basics In Practice

Up until now this post has been theory heavy, let's review some commands we can use to work with a Kubernetes cluster. First we would want to list the pods we have within the cluster which we can do using the get pods command as such:

$ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
  my-first-pod-id-xxxx     1/1    Running   0          13s
  my-second-pod-id-xxxx    1/1    Running   0          13s

A command such as kubectl describe pod returns high level description of the pod such as its name, parent node, etc. Many problems in production pods can be solved by looking at the system log, this can be accomplished by invoking the logs command:

$ kubectl logs -f <pod>
[2022-11-29 04:12:17,262] INFO log data
...

Most typical large scale applications logs are ingested by tools such as Elastic, Loki etc. As such, the logs command isn't as useful in production except for debugging edge cases.

Final Word

This introduction to Kubernetes has set the stage for deeper exploration into specific debugging and troubleshooting techniques, which we will cover in the upcoming posts. The complexity of Kubernetes makes is much harder to debug, but there are facilities in place to workaround some of that complexity.

While this article (and its followups) focus on Kubernetes, future posts will delve into observability and related tools, which are crucial for effective debugging in production environments.

Failure is Required: Understanding Fail-Safe and Fail-Fast Strategies

Shai Almog — Tue, 07 May 2024 16:59:55 +0000

Fail-Fast
Fail-Safe
Choosing Between Fail-Fast and Fail-Safe
- Balancing Both
- Consistent Layer Behavior
- Retry's Are not Fail-Safe
Redefining Failure
- Data Corruption
- Don't Fix the Bug
Debugging Failure
- Avoiding Cascading Failure
Hybrid in Production
Final Word

Failures in software systems are inevitable. How these failures are handled can significantly impact system performance, reliability, and the business’s bottom line. In this post I want to discuss the upside of failure. Why you should seek failure, why failure is good and why avoiding failure can reduce the reliability of your application. We will start with the discussion of fail-fast vs. fail-safe, this will take us to the second discussion about failures in general.

Fail-Fast

Fail-fast systems are designed to immediately stop functioning upon encountering an unexpected condition. This immediate failure helps to catch errors early, making debugging more straightforward.

The fail-fast approach ensures that errors are caught immediately. For example, in the world of programming languages, Java embodies this approach by producing a NullPointerException instantly when encountering a null value, stopping the system and making the error clear. This immediate response helps developers identify and address issues quickly, preventing them from becoming more serious.

By catching and stopping errors early, fail-fast systems reduce the risk of cascading failures, where one error leads to others. This makes it easier to contain and resolve issues before they spread through the system, preserving overall stability.

It is easy to write unit and integration tests for fail-fast systems. This advantage is even more pronounced when we need to understand the test failure. Fail-fast systems usually point directly at the problem in the error stack trace.

However, fail-fast systems carry their own risks, particularly in production environments:

Production Disruptions: If a bug reaches production, it can cause immediate and significant disruptions, potentially impacting both system performance and the business’s operations.
Risk Appetite: Fail-fast systems require a level of risk tolerance from both engineers and executives. They need to be prepared to handle and address failures quickly, often balancing this with potential business impacts.

Fail-Safe

Fail-safe systems take a different approach, aiming to recover and continue even in the face of unexpected conditions. This makes them particularly suited for uncertain or volatile environments.

Microservices are a prime example of fail-safe systems, embracing resiliency through their architecture. Circuit breakers, both physical and software-based, disconnect failing functionality to prevent cascading failures, helping the system continue operating.

Fail-safe systems ensure that systems can survive even harsh production environments, reducing the risk of catastrophic failure. This makes them particularly suited for mission-critical applications, such as in hardware devices or aerospace systems, where smooth recovery from errors is crucial.

However, fail-safe systems have downsides:

Hidden Errors: By attempting to recover from errors, fail-safe systems can delay the detection of issues, making them harder to trace and potentially leading to more severe cascading failures.
Debugging Challenges: This delayed nature of errors can complicate debugging, requiring more time and effort to find and resolve issues.

Choosing Between Fail-Fast and Fail-Safe

It's challenging to determine which approach is better, as both have their merits. Fail-fast systems offer immediate debugging, lower risk of cascading failures, and quicker detection and resolution of bugs. This helps catch and fix issues early, preventing them from spreading.

Fail-safe systems handle errors gracefully, making them better suited for mission-critical systems and volatile environments, where catastrophic failures can be devastating.

Balancing Both

To leverage the strengths of each approach, a balanced strategy can be effective:

Fail-Fast for Local Services: When invoking local services like databases, fail-fast can catch errors early, preventing cascading failures.
Fail-Safe for Remote Resources: When relying on remote resources, such as external web services, fail-safe can prevent disruptions from external failures.

A balanced approach also requires clear and consistent implementation throughout coding, reviews, tooling, and testing processes, ensuring it is integrated seamlessly. Fail-fast can integrate well with orchestration and observability. Effectively, this moves the fail-safe aspect to a different layer of OPS instead of into the developer layer.

Consistent Layer Behavior

This is where things get interesting. It isn't about choosing between fail-safe and fail-fast. It's about choosing the right layer for them. E.g. if an error is handled in a deep layer using a fail-safe approach, it won't be noticed. This might be OK, but if that error has adverse impact (performance, garbage data, corruption, security, etc.) then we will have a problem later on and won't have a clue.

The right solution is to handle all errors in a single layer, in modern systems the top layer is the OPS layer and it makes the most sense. It can report the error to the engineers who are most qualified to deal with the error. But they can also provide immediate mitigation such as restarting a service, allocating additional resources or reverting a version.

Retry's Are not Fail-Safe

Recently I was at a lecture where the speakers listed their updated cloud architecture. They chose to take a shortcut to microservices by using a framework that allows them to retry in the case of failure. Unfortunately, failure doesn't behave the way we would like. You can't eliminate it completely through testing alone. Retry, isn't fail-safe. In-fact: it can mean catastrophe.

They tested their system and "it works", even in production. But lets assume that a catastrophic situation does occur, their retry mechanism can operate as a denial of service attack against their own servers. The number of ways in which ad-hoc architectures such as this can fail are mind-boggling.

This is especially important once we redefine failures.

Redefining Failure

Failures in software systems aren't just about crashes. A crash can be seen as a simple and immediate failure, but there are more complex issues to consider. In fact, crashes in the age of containers are probably the best failures. A system restarts seamlessly with barely an interruption.

Data Corruption

Data corruption is far more severe and insidious than a crash. It carries with it long-term consequences. Corrupted data can lead to security and reliability problems that are challenging to fix, requiring extensive reworking and potentially unrecoverable data.

Cloud computing has led to defensive programming techniques, like circuit breakers and retries, emphasizing comprehensive testing and logging to catch and handle failures gracefully. In a way, this environment sent us back in terms of quality.

A fail-fast system at the data level could stop this from happening. Addressing a bug goes beyond a simple fix. It requires understanding its root cause and preventing reoccurrence, extending into comprehensive logging, testing, and process improvements. This ensures that the bug is fully addressed, reducing the chances of it reoccurring.

Don't Fix the Bug

If it's a bug in production you should probably revert, if you can't instantly revert production. This should always be possible and if it isn't this is something you should work on.

Failures must be fully understood before a fix is undertaken. In my own companies I often skipped that step due to pressure, in a small startup that is forgivable. In larger companies we need to understand the root cause. A culture of debrief for bugs and production issues is essential. The fix should also include process mitigation that prevents similar issues from reaching production.

Debugging Failure

Fail-fast systems are much easier to debug. They have inherently simpler architecture and it is easier to pinpoint an issue to a specific area. It is crucial to throw exceptions even for minor violations (e.g. validations). This prevents cascading types of bugs that prevail in loose systems.

This should be further enforced by unit tests that verify the limits we define and verify proper exceptions are thrown. Retries should be avoided in the code as they make debugging exceptionally difficult and their proper place is in the OPS layer. To facilitate that further, timeouts should be short by default.

Avoiding Cascading Failure

Failure isn't something we can avoid, predict or fully test against. The only thing we can do is soften the blow when a failure occurs. Often this "softening" is achieved by using long running tests meant to replicate extreme conditions as much as possible with the goal of finding our applications weak spots. This is rarely enough, robust systems need to revise these tests often based on real production failures.

A great example of fail-safe would be a cache of REST responses that lets us keep working even when a service is down. Unfortunately, this can lead to complex niche issues such as cache poisoning or a situation in which a banned user still had access due to cache.

Hybrid in Production

Fail-safe is best applied only in production/staging and in the OPS layer. This reduces the amount of changes between production and dev, we want them to be as similar as possible, yet it's still a change which can negatively impact production. But the benefits are tremendous as observability can get a clear picture of system failures.

The discussion here is a bit colored by my more recent experience of building observable cloud architectures. However, the same principle applies to any type of software whether embedded or in the cloud. In such cases we often choose to implement fail-safe in the code, in this case I would suggest implementing it consistently and consciously in a specific layer.

There's also a special case of libraries/frameworks that often provide inconsistent and badly documented behaviors in these situations. I myself am guilty of such inconsistency in some of my work. It's an easy mistake to make.

Final Word

This is my last post on the theory of debugging series that's part of my book/course on debugging. We often think of debugging as the action we take when something fails, it isn't. Debugging starts the moment we write the first line of code. We make decisions that will impact the debugging process as we code, often we're just unaware of these decisions until we get a failure.

I hope this post and series will help you write code that is prepared for the unknown. Debugging, by its nature, deals with the unexpected. Tests can't help. But as I illustrated in my previous posts, there are many simple practices we can undertake that would make it easier to prepare. This isn't a one time process, it's an iterative process that requires re-evaluation of decisions made as we encounter failure.

Software Testing as a Debugging Tool

Shai Almog — Tue, 23 Apr 2024 15:00:00 +0000

The Intersection of Debugging and Testing
- Unit Tests
- Integration Tests
- Coverage
The Debug-Fix Cycle
Composing Tests with Debuggers
Test-Driven Development
Final Word

Debugging is not just about identifying errors—it's about instituting a reliable process for ensuring software health and longevity. In this post we discuss the role of software testing in debugging, including foundational concepts and how they converge to improve software quality.

The Intersection of Debugging and Testing

Debugging and testing play distinct roles in software development. Debugging is the targeted process of identifying and fixing known bugs. Testing, on the other hand, encompasses a adjacent scope, identifying unknown issues by validating expected software behavior across a variety of scenarios.

Both are a part of the debug fix cycle which is a core concept in debugging. Before we cover the cycle we should first make sure we're aligned on the basic terminology.

Unit Tests

Unit tests are tightly linked to debugging efforts, focusing on isolated parts of the application—typically individual functions or methods. Their purpose is to validate that each unit operates correctly in isolation, making them a swift and efficient tool in the debugging arsenal. These tests are characterized by their speed and consistency, enabling developers to run them frequently, sometimes even automatically as code is written within the IDE.

Since software is so tightly bound it is nearly impossible to compose unit tests without extensive mocking. Mocking involves substituting a genuine component with a stand-in that returns predefined results, thus a test method can simulate scenarios without relying on the actual object. This is a powerful yet controversial tool. By using mocking we're in-effect creating a synthetic environment that might misrepresent the real world. We're reducing the scope of the test and might perpetuate some bugs.

Integration Tests

Opposite to unit tests, integration tests examine the interactions between multiple units, providing a more comprehensive picture of the system's health. While they cover broader scenarios, their setup can be more complex due to the interactions involved. However, they are crucial in catching bugs that arise from the interplay between different software components.

In general mocking can be used in integration tests but it is discouraged. They take longer to run and are sometimes harder to set up. However, many developers (myself included) would argue that they are the only benchmark for quality. Most bugs express themselves in the seams between the modules and integration tests are better at detecting that.

Since they are far more important some developers would argue that unit tests are unnecessary. This isn't true, unit test failures are much easier to read and understand. Since they are faster we can run them during development, even while typing. In that sense the balance between the two approaches is the important part.

Coverage

Coverage is a metric that helps quantify the effectiveness of testing by indicating the proportion of code exercised by tests. It helps identify potential areas of the code that have not been tested, which could harbor undetected bugs. However, striving for 100% coverage can be a case of diminishing returns; the focus should remain on the quality and relevance of the tests rather than the metric itself. In my experience, chasing high coverage numbers often results in bad test practices that persist problems.

It is my opinion that unit tests should be excluded from coverage metrics due to the importance of integration tests to overall quality. To get a sense of quality coverage should focus on integration and end to end tests.

The Debug-Fix Cycle

The debug-fix cycle is a structured approach that integrates testing into the debugging process. The stages include identifying the bug, creating a test that reproduces the bug, fixing the bug, verifying the fix with the test, and finally, running the application to ensure the fix works in the live environment. This cycle emphasizes the importance of testing in not only identifying but also in preventing the recurrence of bugs.

Notice that this is a simplified version of the cycle with a focus on the testing aspect only. The full cycle includes discussion of the issue tracking and versioning as part of the whole process. I discuss this more in-depth in other posts in the series and my book.

Composing Tests with Debuggers

A powerful feature of using debuggers in test composition is their ability to "jump to line" or "set value." Developers can effectively reset the execution to a point before the test and rerun it with different conditions, without recompiling or rerunning the entire suite. This iterative process is invaluable for achieving desired test constraints and improves the quality of unit tests by refining the input parameters and expected outcomes.

Increasing test coverage is about more than hitting a percentage; it's about ensuring that tests are meaningful and that they contribute to software quality. A debugger can significantly assist in this by identifying untested paths. When a test coverage tool highlights lines or conditions not reached by current tests, the debugger can be used to force execution down those paths. This helps in crafting additional tests that cover missed scenarios, ensuring that the coverage metric is not just a number but a true reflection of the software's tested state.

In this case you will notice that the next line in the body is a rejectValue call which will throw an exception. I don’t want an exception thrown as I still want to test all the permutations of the method. I can drag the execution pointer (arrow on the left) and place it back at the start of the method.

Test-Driven Development

How does all of this fit with disciplines like Test-Driven Development (TDD)?

It doesn't fit well. Before we get into that let's revisit the basics of TDD. Weak TDD typically means just writing tests before writing the code. Strong TDD involves a red-green-refactor cycle:

Red: Write a test that fails because the feature it tests isn't implemented yet.
Green: Write the minimum amount of code necessary to make the test pass.
Refactor: Clean up the code while ensuring that tests continue to pass.

This rigorous cycle guarantees that new code is continually tested and refactored, reducing the likelihood of complex bugs. It also means that when bugs do appear, they are often easier to isolate and fix due to the modular and well-tested nature of the codebase. At least, that's the theory.

TDD can be especially advantageous for scripting and loosely typed languages. In environments lacking the rigid structure of compilers and linters, TDD steps in to provide the necessary checks that would otherwise be performed during compilation in statically typed languages. It becomes a crucial substitute for compiler/linter checks, ensuring that type and logic errors are caught early.

In real-world application development, TDD's utility is nuanced. While it encourages thorough testing and upfront design, it can sometimes hinder the natural flow of development, especially in complex systems that evolve through numerous iterations. The requirement for 100% test coverage can lead to an unnecessary focus on fulfilling metrics rather than writing meaningful tests.

The biggest problem in TDD is its focus on unit testing. TDD is impractical with integration tests as the process would take too long. But as we determined in the start of this post, integration tests are the true benchmark for quality. In that test TDD is a methodology that provides great quality for arbitrary tests, but not necessarily great quality for the final product. You might have the best cog in the world, but if doesn't fit well into the machine then it isn't great.

Final Word

Debugging is a tool that not only fixes bugs but also actively aids in crafting tests that bolster software quality. By utilizing debuggers in test composition and increasing coverage, developers can create a suite of tests that not only identifies existing issues but also guards against future ones, thus ensuring the delivery of reliable, high-quality software.

Debugging lets us increase coverage and verify edge cases effectively. It's part of a standardized process for issue resolution that's critical for reliability and prevents regressions.

Forem: Shai Almog

Metal and Skins

How we think about quality

Sticky headers were half baked, and that was by design

The SIMD bug, which was my mistake

Metal is here

The end of the skin downloader

How the wizard works

Eating our own dog food

iOS multi-line TextArea: Return as Done

Diagnostics for status-bar tap scroll-to-top

Simulator: Dark / Light mode toggle

Heads-up: weekend backend maintenance

Warning: Android 16 will effectively disallow locking orientation

Why the version jumped to 7.0.242

Wrapping up

Liquid Glass, Material 3, And A Lot Of Plumbing

Try it right now in the Playground

The new native themes

iOS Modern

Android Material 3

Translucent surfaces

Runtime palette overrides

In the simulator

On devices

Sticky headers

How we test this

Bugs and misc features from this week

Where this is going — and a thank-you

Front End Debugging Part 3: Networking

Network Debugging Powerhouse

Re-Issuing and Modifying Requests

cURL and Postman

Throttling and Debugging Race Conditions

Why Throttling Matters:

How to Use

Managing State with Storage Tools

Challenges of State Management

Analyzing Request and Response Headers

Debugging in Incognito Mode: Limitations and Best Practices

Connecting the Front-End to the Database

Final Word

Front End Debugging Part 2: Console.log() to the Max

Understanding Front-End Logging vs. Back-End Logging

Leveraging Console Log Levels

Customizing Console Output with CSS

Stack Tracing with console.trace()

Assertions for Design-by-Contract

Printing Tables for Clearer Data Visualization

Copying Objects to the Clipboard

Inspecting with console.dir() and dirxml()

Counting Function Calls

Organizing Logs with Groups

Chrome-Specific Debugging Features

Final Word

Front End Debugging Part 1: Not just Console Log

Instant Debugging with the debugger Keyword

Triggering Debugging from the Console

DOM Breakpoints: Monitoring DOM Changes

XHR Breakpoints: Uncovering Hidden Network Calls

Simulating Environments for Debugging

Debugging Layout and Style Issues

Final Word

The Art of Full Stack Debugging

Full Stack Development, A Shifting Definition

The Full Stack Debugging Approach

Frontend Debugging: Tools and Techniques

It isn't "Just Console.log"

The Power of Developer Tools

Tackling Code Obfuscation

Debugging Across Layers

Isolating Issues Across the Stack

The Importance of System-Level Debugging

Embracing Complexity

Conclusion

Mastering Serverless Debugging

Introduction to Serverless Computing

Challenges of Serverless Debugging

Disconnected Environments

Lack of Standardization

Instant Debugging with the `debugger` Keyword

The Limitations of `kubectl exec`

Using `kubectl debug`