Forem: Shai Almog

Front End Debugging Part 3: Networking

Shai Almog — Tue, 28 Jan 2025 16:46:20 +0000

Network Debugging Powerhouse
Re-Issuing and Modifying Requests
- cURL and Postman
Throttling and Debugging Race Conditions
- Why Throttling Matters:
- How to Use
Managing State with Storage Tools
- Challenges of State Management
Analyzing Request and Response Headers
Debugging in Incognito Mode: Limitations and Best Practices
Connecting the Front-End to the Database
Final Word

Debugging network communication issues is a critical skill for any front-end developer. While tools like Wireshark provide low-level insight into network traffic, modern browsers like Chrome and Firefox offer developer tools with powerful features tailored for web development. In this post we will discuss using browser-based tools to debug network communication issues effectively. This is a far better approach than using Wireshark for the vast majority of simple cases.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Network Debugging Powerhouse

Modern browsers come equipped with developer tools that rival standalone IDE debuggers in capability and convenience. Both Chrome and Firefox have robust network monitoring features that allow developers to observe/analyze requests and responses without leaving the browser.

On the basic level, which you’re probably familiar with, these tools include:

Network monitors: View all HTTP and HTTPS requests, including their headers, payloads, and responses.
Throttling controls: Simulate slower connections to test performance and debug race conditions.
Request replay functionality: Modify and resend requests directly from the browser.

While this post focuses on debugging techniques, it's worth noting that these tools are invaluable for performance optimization as well, though that topic warrants its own discussion.

Re-Issuing and Modifying Requests

One of the most powerful debugging features is the ability to re-issue requests. Instead of switching to external tools like cURL or Postman, browsers allow us to modify and resend requests directly.

This lets us quickly test variations of a failing API call to pinpoint issues without leaving the debugging environment. It’s especially useful when we have hard to reproduce issues or deep UI hierarchies.

In Firefox we can right-click any network entry in the Firefox Developer Tools and select "Resend." This opens an editable window where we can change request parameters, such as headers, payloads, or query strings, and resend the request.

Chrome provides similar functionality, though its interface for modifying and resending requests is slightly less direct than Firefox's.

cURL and Postman

Both browsers let you copy a request as a cURL command via the context menu. This is useful for reproducing issues in the terminal or sharing with back-end developers. I use this frequently as part of creating a reproducible issue.

If you prefer Postman, you can copy request headers and payloads from the browser and paste them into Postman to replicate requests.

Throttling and Debugging Race Conditions

Network throttling is a highly underrated feature that can be a game-changer for debugging specific classes of bugs. Both Chrome and Firefox allow developers to simulate various network speeds, from 2G connections to fast 4G.

Why Throttling Matters:

Some bugs only surface when requests arrive out of their expected order. Slowing down the network can help replicate and analyze these situations. Typical examples would be race conditions and related issues.

This is also very useful for simulating real-world conditions. Many users may not have fast or reliable internet connections. Throttling helps you understand how your application behaves in these scenarios.

I use this frequently when testing loading indicators which disappear too quickly when running locally. Instead of adding sleep code into the JavaScript or server code I can simulate slow-loading assets to verify that loading spinners or placeholders appear correctly.

How to Use

In Chrome we Open Developer Tools → Network tab.

We then use the "No throttling" dropdown to select pre-configured speeds or create a custom profile.

In Firefox we have similar functionality is available under the Network Monitor.

Managing State with Storage Tools

Local storage, session storage, and indexedDB often hold data critical to reproducing bugs, especially those tied to specific user states or devices.

Challenges of State Management

Even in incognito mode, state can persist if multiple private windows are open simultaneously. Persistence across sessions is a big challenge in these situations.

Understanding the exact state of a user's local storage can provide insight into seemingly random bugs. Debugging user-specific issues is problematic without control over storage.

In Firefox the dedicated Storage tab in Developer Tools makes it easy to inspect, edit, and delete data from local storage, session storage, cookies, and indexedDB.

In Chrome the Application tab consolidates all storage options, including the ability to clear specific caches or edit entries manually.

This functionality has many powerful uses for debugging:

Inject Debug Information: Tools like these let us manually add or modify storage data to simulate edge cases or specific user conditions.
Share Local State: Users can export their local storage, cookies, or indexedDB entries, allowing developers to reproduce issues locally.
Clear Cache Strategically: Clear only the relevant entries instead of a blanket cache clear, preserving useful state for debugging.

Analyzing Request and Response Headers

Request and response headers often hold the key to understanding network issues. We can use the network monitor to inspect:

Authorization headers: Check for missing or malformed tokens.
CORS headers: Verify that the server allows requests from your domain. These are some of the most painful type of http bugs. Reviewing these headers can be a lifesaver. If requests fail with CORS errors, inspect the Access-Control-Allow-Origin header in the response.
Cache-Control headers: Ensure proper caching behavior for your resources.

These tools are especially useful when debugging missing headers: Look for required headers like Content-Type or Authorization. Debugging Authentication: Use the "Copy as cURL" feature to test API calls with modified headers directly in the terminal.

Debugging in Incognito Mode: Limitations and Best Practices

Incognito mode can help isolate issues by providing a clean slate, however it has some limitations. Multiple incognito windows share the same state, which can lead to unintentional persistence of local data.

I suggest using storage management tools to manually clear or modify local data instead of relying solely on incognito mode. Keep only one incognito window open during testing to avoid unintended state sharing.

Connecting the Front-End to the Database

The front-end is often a transition point between user interaction and back-end data processing. While this post focuses on debugging the network layer, it's important to remember that:

Network issues often manifest due to back-end problems (e.g., a database error resulting in a 500 Internal Server Error).
Front-end developers should collaborate closely with back-end engineers to trace issues across the stack.

We can use custom response headers to include diagnostic information from the back end, such as query execution time or error codes. We can leverage server logs in conjunction with front-end debugging to get a complete picture of the issue.

Final Word

Browser developer tools are indispensable for debugging network communication issues, offering features like request replay, throttling, and storage management that simplify the debugging process. By mastering these tools, front-end developers can efficiently identify and resolve issues, ensuring a smoother user experience.

With the techniques and tips outlined in this post, you'll be better equipped to tackle network debugging challenges head-on. As you grow more familiar with these tools, you'll find them invaluable not only for debugging but also for improving your development workflow.

Front End Debugging Part 2: Console.log() to the Max

Shai Almog — Tue, 26 Nov 2024 17:53:11 +0000

Understanding Front-End Logging vs. Back-End Logging
Leveraging Console Log Levels
Customizing Console Output with CSS
Stack Tracing with console.trace()
Assertions for Design-by-Contract
Printing Tables for Clearer Data Visualization
Copying Objects to the Clipboard
Inspecting with console.dir() and dirxml()
Counting Function Calls
Organizing Logs with Groups
Chrome-Specific Debugging Features
Final Word

In my previous post I talked about why Console.log() isn’t the most effective debugging tool. In this installment, we will do a bit of an about-face and discuss the ways in which Console.log() is fantastic. Let’s break down some essential concepts and practices that can make your debugging life much easier and more productive.

Understanding Front-End Logging vs. Back-End Logging

Front-end logging differs significantly from back-end logging, and understanding this distinction is crucial. Unlike back-end systems, where persistent logs are vital for monitoring and debugging, the fluid nature of front-end development introduces different challenges. When debugging backends I’d often go for tracepoints which are far superior in that setting. However the front-end with its constant need to refresh, reload, contexts switch etc. is a very different beast. In the front-end relying heavily on elaborate logging mechanisms can become cumbersome.

While tracepoints remain superior to basic print statements, the continuous testing and browser reloading in front-end workflows lessen their advantage. Moreover, features like logging to a file or structured ingestion are rarely useful in the browser, diminishing the need for a comprehensive logging framework. However, using a logger is still considered best practice over the typical Console.log for long term logging… For short term logging Console.log has some tricks up its sleeve.

Leveraging Console Log Levels

One of the hidden gems of the browser console is its support for log levels, which is a significant step up from rudimentary print statements. The console provides five levels:

• log: Standard logging

• debug: Same as log but used for debugging purposes

• info: Informative messages, often rendered like log/debug

• warn: Warnings that might need attention

• error: Errors that have occurred

While log and debug can be indistinguishable, these levels allow for a more organized and filtered debugging experience. Browsers enable filtering the output based on these levels, mirroring the capabilities of server-side logging systems and allowing you to focus on relevant messages.

Customizing Console Output with CSS

Front-end development allows for creative solutions, and logging is no exception. Using CSS styles in the console can make logs more visually distinct. By utilizing %c in a console message, you can apply custom CSS:

console.customLog = function(msg) {
    console.log("%c" + msg,"color:black;background:pink;font-family:system-ui;font-size:4rem;-webkit-text-stroke: 1px black;font-weight:bold")
}
console.customLog("Dazzle")

This approach is helpful when you need to make specific logs stand out or organize output visually. You can use multiple %c substitutions to apply various styles to different parts of a log message.

Stack Tracing with console.trace()

The console.trace() method can print a stack trace at a particular location, which can sometimes be helpful for understanding the flow of your code. However, due to JavaScript’s asynchronous behavior, stack traces aren’t always as straightforward as in back-end debugging. Still, in specific scenarios, such as synchronous code segments or event handling, it can be quite valuable.

Assertions for Design-by-Contract

Assertions in front-end code allow developers to enforce expectations and promote a “fail-fast” mentality. Using Console.assert(), you can test conditions:

console.assert(x > 0, 'x must be greater than zero');

In the browser, a failed assertion appears as an error, similar to console.error. An added benefit is that assertions can be stripped from production builds, removing any performance impact. This makes assertions a great tool for enforcing design contracts during development without compromising production efficiency.

Printing Tables for Clearer Data Visualization

When working with arrays or objects, displaying data as tables can significantly enhance readability. The console.table() method allows you to output structured data easily:

console.table(["Simple Array", "With a few elements", "in line"])

This method is especially handy when debugging arrays of objects, presenting a clear, tabular view of the data and making complex data structures much easier to understand.

Copying Objects to the Clipboard

Debugging often involves inspecting objects, and the copy(object) method allows you to copy an object’s content to the clipboard for external use. This feature is useful when you need to transfer data or analyze it outside the browser.

Inspecting with console.dir() and dirxml()

The console.dir() method provides a more detailed view of objects, showing their properties as you’d see in a debugger. This is particularly helpful for inspecting DOM elements or exploring API responses. Meanwhile, console.dirxml() allows you to view objects as XML, which can be useful when debugging HTML structures.

Counting Function Calls

Keeping track of how often a function is called or a code block is executed can be crucial. The console.count() method tracks the number of times it’s invoked, helping you verify that functions are called as expected:

function myFunction() {
    console.count('myFunction called');
}

You can reset the counter using console.countReset(). This simple tool can help you catch performance issues or confirm the correct execution flow.

Organizing Logs with Groups

To prevent log clutter, use console groups to organize related messages. console.group() starts a collapsible log section, and console.groupEnd() closes it:

console.group('My Group');
console.log('Message 1');
console.log('Message 2');
console.groupEnd();

Grouping makes it easier to navigate complex logs and keeps your console clean.

Chrome-Specific Debugging Features

Monitoring Functions: Chrome’s monitor() method logs every call to a function, showing the arguments and enabling a method-tracing experience.

Monitoring Events: Using monitorEvents(), you can log events on an element. This is useful for debugging UI interactions. For example, monitorEvents(window, 'mouseout') logs only mouseout events.

Querying Object Instances: queryObjects(Constructor) lists all objects created with a specific constructor, giving you insights into memory usage and object instantiation.

Final Word

Front-end debugging tools have come a long way. These tools provide a rich set of features that go far beyond simple console.log() statements. From log levels and CSS styling to assertions and event monitoring, mastering these techniques can transform your debugging workflow.

If you read this post as part of my series you will notice a big change in my attitude toward debugging when we reached the front-end. Front-end debugging is very different when compared to backend debugging. When debugging the backend I’m vehemently against code changes for debugging (e.g. println debugging), but on the front-end this can be a reasonable hack. The change in environment justifies it. The short lifecycle, the single user use case and the risk is smaller.

While there are many transferrable skills we pick up while debugging, it’s important to remain flexible in our attitude. Next time we will discuss networking and storage debugging on the front-end.

Front End Debugging Part 1: Not just Console Log

Shai Almog — Tue, 29 Oct 2024 17:38:36 +0000

Instant Debugging with the debugger Keyword
Triggering Debugging from the Console
DOM Breakpoints: Monitoring DOM Changes
XHR Breakpoints: Uncovering Hidden Network Calls
Simulating Environments for Debugging
Debugging Layout and Style Issues
- Final Word

As a Java developer most of my focus is on the backend side of debugging. Front-end debugging poses different challenges and has sophisticated tools of its own. Unfortunately, print based debugging has become the norm in front-end. To be fair, it makes more sense there as the cycles are different and the problem is always a single user problem. But even if you choose to use Console.log, there’s a lot of nuance to pick up there.

Instant Debugging with the `debugger` Keyword

A cool yet powerful tool in JavaScript is the debugger keyword. Instead of simply printing a stack trace, we can use this keyword to launch the debugger directly at the line of interest. That is a fantastic tool that instantly brings your attention to a bug, I often use it in my debug builds of the front-end instead of just printing an error log.

How to Use It: Place the debugger keyword within your code, particularly within error-handling methods. When the code execution hits this line, it automatically pauses, allowing you to inspect the current state, step through code, and understand what's going wrong.

Notice that while this is incredibly useful during development, we must remember to remove or conditionally exclude debugger statements in production environments. A release build should not include these calls in a production site live environment.

Triggering Debugging from the Console

Modern browsers allow you to invoke debugging directly from the console, adding an additional layer of flexibility to your debugging process.

Example: By using the debug(functionName) command in the console, you can set a breakpoint at the start of the specified function. When this function is subsequently invoked, the execution halts, sending you directly into the debugger.

function hello(name) {
    Console.log("Hello " + name)
}
debug(hello)
hello("Shai")

This is particularly useful when you want to start debugging without modifying the source code, or when you need to inspect a function that’s only defined in the global scope.

DOM Breakpoints: Monitoring DOM Changes

DOM breakpoints are an advanced feature in Chrome and Firebug (Firefox plugin) that allow you to pause execution when a specific part of the DOM is altered.

To use it we can right-click on the desired DOM element, select “Break On,” and choose the specific mutation type you are interested in (e.g., subtree modifications, attribute changes, etc.).

DOM breakpoints are extremely powerful for tracking down issues where DOM manipulation causes unexpected results, such as dynamic content loading or changes in the user interface that disrupt the intended layout or functionality. Think of them like field breakpoints we discussed in the past.

These breakpoints complement traditional line and conditional breakpoints, providing a more granular approach to debugging complex front-end issues. This is a great tool to use when the DOM is manipulated by an external dependency.

XHR Breakpoints: Uncovering Hidden Network Calls

Understanding who initiates specific network requests can be challenging, especially in large applications with multiple sources contributing to a request. XHR (XMLHttpRequest) breakpoints provide a solution to this problem.

In Chrome or Firebug, set an XHR breakpoint by specifying a substring of the URI you wish to monitor. When a request matching this pattern is made, the execution stops, allowing you to investigate the source of the request.

This tool is invaluable when dealing with dynamically generated URIs or complex flows where tracking the origin of a request is not straightforward.

Notice that you should be selective with the filters you set; leaving the filter blank will cause the breakpoint to trigger on all XHR requests, which can become overwhelming.

Simulating Environments for Debugging

Sometimes, the issues you need to debug are specific to certain environments, such as mobile devices or different geographical locations. Chrome and Firefox offer several simulation tools to help you replicate these conditions on your desktop.

Simulating User Agents: Change the browser’s user agent to mimic different devices or operating systems. This can help you identify platform-specific issues or debug server-side content delivery that varies by user agent.
Geolocation Spoofing: Modify the browser’s reported location to test locale-specific features or issues. This is particularly useful for applications that deliver region-specific content or services.
Touch and Device Orientation Emulation: Simulate touch events or change the device orientation to see how your application responds to mobile-specific interactions. This is crucial for ensuring a seamless user experience across all devices.

These are things that are normally very difficult to reproduce. E.g. touch related issues are often challenging to debug on the device. By simulating them on the desktop browser we can shorten the debug cycle and use the tooling available on the desktop.

Debugging Layout and Style Issues

CSS and HTML bugs can be particularly tricky, often requiring a detailed examination of how elements are rendered and styled.

Inspect Element: The "inspect element" tool is the cornerstone of front-end debugging, allowing you to view and manipulate the DOM and CSS in real-time. As you make changes, the page updates instantly, providing immediate feedback on your tweaks.

Addressing Specificity Issues: One common problem is CSS specificity, where a more specific selector overrides the styles you intend to apply. The inspect element view highlights overridden styles, helping you identify and resolve conflicts.

Firefox vs. Chrome: While both browsers offer robust tools, they have different approaches to organizing these features. Firefox’s interface may seem more straightforward, with fewer tabs, while Chrome organizes similar tools under various tabs, which can either streamline your workflow or add complexity, depending on your preference.

Final Word

There are many front-end tools that I want to discuss in the coming posts. I hope you picked up a couple of new debugging tricks in this first part.

Front-end debugging requires deep understanding of browser tools and JavaScript capabilities. By mastering the techniques outlined in this post—instant debugging with the debugger keyword, DOM and XHR breakpoints, environment simulation, and layout inspection—you can significantly enhance your debugging efficiency and deliver more robust, error-free web applications.

The Art of Full Stack Debugging

Shai Almog — Tue, 08 Oct 2024 18:39:51 +0000

Full Stack Development, A Shifting Definition
- The Full Stack Debugging Approach
Frontend Debugging: Tools and Techniques
- It isn't "Just Console.log"
- The Power of Developer Tools
- Tackling Code Obfuscation
Debugging Across Layers
- Isolating Issues Across the Stack
- The Importance of System-Level Debugging
- Embracing Complexity
Conclusion

Full stack development is often likened to an intricate balancing act, where developers are expected to juggle multiple responsibilities across the frontend, backend, database, and beyond. As the definition of full stack development continues to evolve, so too does the approach to debugging. Full stack debugging is an essential skill for developers, as it involves tracking issues through multiple layers of an application, often navigating domains where one’s knowledge may only be cursory. In this blog post I aim to explore the nuances of full stack debugging, offering practical tips and insights for developers navigating the complex web of modern software development.

Notice that this is an introductory post focusing mostly on the front end debugging aspects, in the following posts I will dig deeper into the less familiar capabilities in front end debugging.

Full Stack Development, A Shifting Definition

The definition of full stack development is as fluid as the technology stacks themselves. Traditionally, full stack developers were defined as those who could work on both the frontend and backend of an application. However, as the industry evolves, this definition has expanded to include aspects of operations (OPS) and configuration. The modern full stack developer is expected to submit pull requests that cover all parts required to implement a feature—backend, database, frontend, and configuration. While this does not make them an expert in all these areas, it does require them to navigate across domains, often relying on domain experts for guidance.

I've heard it said that full stack developers are:

Jack of all trades, master of none.

However, the full quote probably better represents the reality:

Jack of all trades, master of none, but better than a master of one.

The Full Stack Debugging Approach

Just as full stack development involves working across various domains, full stack debugging requires a similar approach. A symptom of a bug may manifest in the frontend, but its root cause could lie deep within the backend or database layers. Full stack debugging is about tracing these issues through the layers and isolating them as quickly as possible. This is no easy task, especially when dealing with complex systems where multiple layers interact in unexpected ways. The key to successful full stack debugging lies in understanding how to track an issue through each layer of the stack and identifying common pitfalls that developers may encounter.

Frontend Debugging: Tools and Techniques

It isn't "Just Console.log"

Frontend developers are often stereotyped as relying solely on Console.log for debugging. While this method is simple and effective for basic debugging tasks, it falls short when dealing with the complex challenges of modern web development. The complexity of frontend code has increased significantly, making advanced debugging tools not just useful, but necessary. Yet, despite the availability of powerful debugging tools, many developers continue to shy away from them, clinging to old habits.

The Power of Developer Tools

Modern web browsers come equipped with robust developer tools that offer a wide range of capabilities for debugging frontend issues. These tools, available in browsers like Chrome and Firefox, allow developers to inspect elements, view and edit HTML and CSS, monitor network activity, and much more. One of the most powerful, yet underutilized, features of these tools is the JavaScript debugger.

The debugger allows developers to set breakpoints, step through code, and inspect the state of variables at different points in the execution. However, the complexity of frontend code, particularly when it has been obfuscated for performance reasons, can make debugging a challenging task.

We can launch the browser tools on Firefox using this menu:

On Chrome we can use this option:

I prefer working with Firefox, I find their developer tools more convenient but both browsers have similar capabilities. Both have fantastic debuggers (as you can see with the Firefox debugger below), unfortunately many developers limit themselves to console printing instead of exploring this powerful tool.

Tackling Code Obfuscation

Code obfuscation is a common practice in frontend development, employed to protect proprietary code and reduce file sizes for better performance. However, obfuscation also makes the code difficult to read and debug. Fortunately, both Chrome and Firefox developer tools provide a feature to de-obfuscate code, making it more readable and easier to debug. By clicking the curly brackets button in the toolbar, developers can transform a single line of obfuscated code into a well-formed, debuggable file.

Another important tool in the fight against obfuscation is the source map. Source maps are files that map obfuscated code back to its original source code, including comments. When generated and properly configured, source maps allow developers to debug the original code instead of the obfuscated version. In Chrome, this feature can be enabled by ensuring that "Enable JavaScript source maps" is checked in the developer tools settings.

You can use code like this in the JavaScript file to point at the sourcemap file:

//@sourceMappingURL=myfile.js.map

For this to work in Chrome we need to ensure that "Enable JavaScript source maps" is checked in the settings. Last I checked it was on by default but it doesn't hurt to verify:

Debugging Across Layers

Isolating Issues Across the Stack

In full stack development, issues often manifest in one layer but originate in another. For example, a frontend error might be caused by a misconfigured backend service or a database query that returns unexpected results. Isolating the root cause of these issues requires a methodical approach, starting from the symptom and working backward through the layers.

A common strategy is to reproduce the issue in a controlled environment, such as a local development setup, where each layer of the stack can be tested individually. This helps to narrow down the potential sources of the problem. Once the issue has been isolated to a specific layer, developers can use the appropriate tools and techniques to diagnose and resolve it.

The Importance of System-Level Debugging

Full stack debugging is not limited to the application code. Often, issues arise from the surrounding environment, such as network configurations, third-party services, or hardware limitations. A classic example of this that we ran into a couple of years ago was a production problem where a WebSocket connection would frequently disconnect. After extensive debugging, Steve discovered that the issue was caused by the CDN provider (CloudFlare) timing out the WebSocket after two minutes—something that could only be identified by debugging the entire system, not just the application code.

System-level debugging requires a broad understanding of how different components of the infrastructure interact with each other. It also involves using tools that can monitor and analyze the behavior of the system as a whole, such as network analyzers, logging frameworks, and performance monitoring tools.

Embracing Complexity

Full stack debugging is inherently complex, as it requires developers to navigate multiple layers of an application, often dealing with unfamiliar technologies and tools. However, this complexity also presents an opportunity for growth. By embracing the challenges of full stack debugging, developers can expand their knowledge and become more versatile in their roles.

One of the key strengths of full stack development is the ability to collaborate with domain experts. When debugging an issue that spans multiple layers, it is important to leverage the expertise of colleagues who specialize in specific areas. This collaborative approach not only helps to resolve issues more efficiently but also fosters a culture of knowledge sharing and continuous learning within the team.

As tools continue to evolve, so too do the tools and techniques available for debugging. Developers should strive to stay up-to-date with the latest advancements in debugging tools and best practices. Whether it’s learning to use new features in browser developer tools or mastering system-level debugging techniques, continuous learning is essential for success in full stack development.

Conclusion

Full stack debugging is a critical skill for modern developers, we mistakenly think it requires deep understanding of both the application and its surrounding environment. I disagree... By mastering the tools and techniques discussed in this post/upcoming posts, developers can more effectively diagnose and resolve issues that span multiple layers of the stack. Whether you’re dealing with obfuscated frontend code, misconfigured backend services, or system-level issues, the key to successful debugging lies in a methodical, collaborative approach.

You don't need to understand every part of the system, just the ability to eliminate the impossible.

Mastering Serverless Debugging

Shai Almog — Tue, 02 Jul 2024 16:26:02 +0000

Introduction to Serverless Computing
Challenges of Serverless Debugging
- Disconnected Environments
- Lack of Standardization
- Limited Debugging Tools
- Concurrency and Scale
Effective Strategies for Serverless Debugging
- Local Debugging with IDE Remote Capabilities
- Using Feature Flags for Debugging
- Staged Rollouts and Canary Deployments
- Comprehensive Logging
- Embracing Idempotency
Debugging a Lambda Application Locally with AWS SAM
- Setting Up the Local Environment
- Running the Hello World Application Locally
- Configuring Remote Debugging
- Handling Debugger Timeouts
Final Word

Serverless computing has emerged as a transformative approach to deploying and managing applications. The theory is that by abstracting away the underlying infrastructure, developers can focus solely on writing code. While the benefits are clear—scalability, cost efficiency, and performance—debugging serverless applications presents unique challenges. This post explores effective strategies for debugging serverless applications, particularly focusing on AWS Lambda.

Before I proceed I think it's important to disclose a bias: I am personally not a huge fan of Serverless or PaaS after I was burned badly by PaaS in the past. However, some smart people like Adam swear by it so I should keep an open mind.

Introduction to Serverless Computing

Serverless computing, often referred to as Function as a Service (FaaS), allows developers to build and run applications without managing servers. In this model, cloud providers automatically handle the infrastructure, scaling, and management tasks, enabling developers to focus purely on writing and deploying code. Popular serverless platforms include AWS Lambda, Azure Functions, and Google Cloud Functions.

In contrast, Platform as a Service (PaaS) offers a more managed environment where developers can deploy applications but still need to configure and manage some aspects of the infrastructure. PaaS solutions, such as Heroku and Google App Engine, provide a higher level of abstraction than Infrastructure as a Service (IaaS) but still require some server management.

Kubernetes, which we recently discussed, is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. While Kubernetes offers powerful capabilities for managing complex, multi-container applications, it requires significant expertise to set up and maintain. Serverless computing simplifies this by removing the need for container orchestration and management altogether.

The "catch" is two fold:

Serverless programming removes the need to understand the servers but also removes the ability to rely on them resulting in more complex architectures.
Pricing starts off cheap. Practically free. It can quickly escalate especially in case of an attack or misconfiguration.

Challenges of Serverless Debugging

While serverless architectures offer some benefits, they also introduce unique debugging challenges. The primary issues stem from the inherent complexity and distributed nature of serverless environments. Here are some of the most pressing challenges.

Disconnected Environments

One of the major hurdles in serverless debugging is the lack of consistency between development, staging, and production environments. While traditional development practices rely on these separate environments to test and validate code changes, serverless architectures often complicate this process. The differences in configuration and scale between these environments can lead to bugs that only appear in production, making them difficult to reproduce and fix.

Lack of Standardization

The serverless ecosystem is highly fragmented, with various vendors offering different tools and frameworks. This lack of standardization can make it challenging to adopt a unified debugging approach. Each platform has its own set of practices and tools, requiring developers to learn and adapt to multiple environments.

This is slowly evolving with some platforms gaining traction, but since this is a vendor driven industry there are many edge cases.

Limited Debugging Tools

Traditional debugging tools, such as step-through debugging and breakpoints, are often unavailable in serverless environments. The managed and controlled nature of serverless functions restricts access to these tools, forcing developers to rely on alternative methods, such as logging and remote debugging.

Concurrency and Scale

Serverless functions are designed to handle high concurrency and scale seamlessly. However, this can introduce issues that are hard to reproduce in a local development environment. Bugs that manifest only under specific concurrency conditions or high load are particularly challenging to debug.

Notice that when I discuss concurrency here I'm often referring to race conditions between separate services.

Effective Strategies for Serverless Debugging

Despite these challenges, several strategies can help make serverless debugging more manageable. By leveraging a combination of local debugging, feature flags, staged rollouts, logging, idempotency, and Infrastructure as Code (IaC), developers can effectively diagnose and fix issues in serverless applications.

Local Debugging with IDE Remote Capabilities

While serverless functions run in the cloud, you can simulate their execution locally using tools like AWS SAM (Serverless Application Model). This involves setting up a local server that mimics the cloud environment, allowing you to run tests and perform basic trial-and-error debugging.

To get started, you need to install Docker or Docker Desktop, create an AWS account, and set up the AWS SAM CLI. Deploy your serverless application locally using the SAM CLI, which enables you to run the application and simulate Lambda functions on your local machine. Configure your IDE for remote debugging, launching the application in debug mode, and connecting your debugger to the local host. Set breakpoints to step through the code and identify issues.

Using Feature Flags for Debugging

Feature flags allow you to enable or disable parts of your application without deploying new code. This can be invaluable for isolating issues in a live environment. By toggling specific features on or off, you can narrow down the problematic areas and observe the application’s behavior under different configurations.

Implementing feature flags involves adding conditional checks in your code that control the execution of specific features based on the flag’s status. Monitoring the application with different flag settings helps identify the source of bugs and allows you to test fixes without affecting the entire user base.

This is essentially "debugging in production". Working on a new feature?

Wrap it in a feature flag which is effectively akin to wrapping the entire feature (client and server) in if statements. You can then enable it conditionally globally or on a per user basis. This means you can test the feature, enable or disable it based on configuration without redeploying the application.

Staged Rollouts and Canary Deployments

Deploying changes incrementally can help catch bugs before they affect all users. Staged rollouts involve gradually rolling out updates to a small percentage of users before a full deployment. This allows you to monitor the performance and error logs of the new version in a controlled manner, catching issues early.

Canary deployments take this a step further by deploying new changes to a small subset of instances (canaries) while the rest of the system runs the stable version. If issues are detected in the canaries, you can roll back the changes without impacting the majority of users. This method limits the impact of potential bugs and provides a safer way to introduce updates. This isn't great as in some cases some demographics might be more reluctant to report errors. However, for server side issues this might make sense as you can see the impact based on server logs and metrics.

Comprehensive Logging

Logging is one of the most common and essential tools for debugging serverless applications. I wrote and spoke a lot about logging in the past. By logging all relevant data points, including inputs and outputs of your functions, you can trace the flow of execution and identify where things go wrong.

However, excessive logging can increase costs, as serverless billing is often based on execution time and resources used. It’s important to strike a balance between sufficient logging and cost efficiency. Implementing log levels and selectively enabling detailed logs only when necessary can help manage costs while providing the information needed for debugging.

I talk about striking the delicate balance between debuggable code, performance and cost with logs in the following video. Notice that this is a general best practice and not specific to serverless.

Embracing Idempotency

Idempotency, a key concept from functional programming, ensures that functions produce the same result given the same inputs, regardless of the number of times they are executed. This simplifies debugging and testing by ensuring consistent and predictable behavior.

Designing your serverless functions to be idempotent involves ensuring that they do not have side effects that could alter the outcome when executed multiple times. For example, including timestamps or unique identifiers in your requests can help maintain consistency. Regularly testing your functions to verify idempotency can make it easier to pinpoint discrepancies and debug issues.

Testing is always important but in serverless and complex deployments it becomes critical. Awareness and embrace of idempotency allows for more testable code and easier to reproduce bugs.

Debugging a Lambda Application Locally with AWS SAM

Debugging serverless applications, particularly AWS Lambda functions, can be challenging due to their distributed nature and the limitations of traditional debugging tools. However, AWS SAM (Serverless Application Model) provides a way to simulate Lambda functions locally, enabling developers to test and debug their applications more effectively. I will use it as a sample to explore the process of setting up a local debugging environment, running a sample application, and configuring remote debugging.

Setting Up the Local Environment

Before diving into the debugging process, it's crucial to set up a local environment that can simulate the AWS Lambda environment. This involves a few key steps:

Install Docker: Docker is required to run the local simulation of the Lambda environment. You can download Docker or Docker Desktop from the official Docker website.
Create an AWS Account: If you don't already have an AWS account, you need to create one. Follow the instructions on the AWS account creation page.
Set Up AWS SAM CLI: The AWS SAM CLI is essential for building and running serverless applications locally. You can install it by following the AWS SAM installation guide.

Running the Hello World Application Locally

To illustrate the debugging process, let's use a simple "Hello World" application. The code for this application can be found in the AWS Hello World tutorial.

Deploy Locally: Use the SAM CLI to deploy the Hello World application locally. This can be done with the following command:
```
sam local start-api
```
This command starts a local server that simulates the AWS Lambda cloud environment.
Trigger the Endpoint: Once the local server is running, you can trigger the endpoint using a curl command:
```
curl http://localhost:3000/hello
```
This command sends a request to the local server, allowing you to test the function's response.

Configuring Remote Debugging

While running tests locally is a valuable step, it doesn't provide full debugging capabilities. To debug the application, you need to configure remote debugging. This involves several steps.

First we need to start the application in debug mode using the following SAM command:

sam local invoke -d 5858

This command pauses the application and waits for a debugger to connect.

Next we need to configure the IDE for remote debugging. We start by setting up the IDE to connect to the local host for remote debugging. This typically involves creating a new run configuration that matches the remote debugging settings.

We can now set breakpoints in the code where we want the execution to pause. This allows us to step through the code and inspect variables and application state just like in any other local application.

We can test this by invoking the endpoint e.g. using curl. With the debugger connected we would stop on the breakpoint like any other tool:

curl http://localhost:3000/hello

The application will pause at the breakpoints you set, allowing you to step through the code.

Handling Debugger Timeouts

One significant challenge when debugging Lambda functions is the quick timeout setting. Lambda functions are designed to execute quickly, and if they take too long, the costs can become prohibitive. By default, the timeout is set to a short duration, but you can configure this in the template.yaml file e.g.:

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: app.lambdaHandler
      Timeout: 60  # timeout in seconds

After updating the timeout value, re-issue the sam build command to apply the changes.

In some cases, running the application locally might not be enough. You may need to simulate running on the actual AWS stack to get more accurate debugging information. Solutions like SST (Serverless Stack) or MerLoc can help achieve this, though they are specific to AWS and relatively niche.

Final Word

Serverless debugging requires a combination of strategies to effectively identify and resolve issues. While traditional debugging methods may not always apply, leveraging local debugging, feature flags, staged rollouts, comprehensive logging, idempotency, and IaC can significantly improve your ability to debug serverless applications. As the serverless ecosystem continues to evolve, staying adaptable and continuously updating your debugging techniques will be key to success.

Debugging serverless applications, particularly AWS Lambda functions, can be complex due to their distributed nature and the constraints of traditional debugging tools. However, by leveraging tools like AWS SAM, you can simulate the Lambda environment locally and use remote debugging to step through your code. Adjusting timeout settings and considering advanced simulation tools can further enhance your debugging capabilities.

Debugging Kubernetes - Troubleshooting Guide

Shai Almog — Tue, 18 Jun 2024 14:26:44 +0000

Identifying Configuration Issues
- Common Causes and Solutions
- Detailed Investigation Steps
Dealing with Image Pull Errors
- Troubleshooting Steps
Handling Node Issues
- Preventive Measures
Managing Missing Configuration Keys or Secrets
Utilizing Buildg for Interactive Debugging
Conclusion

As Kubernetes continues to revolutionize the way we manage and deploy applications, understanding its intricacies becomes essential for developers and operations teams alike. If you don't have a dedicated DevOps team you probably shouldn't be working with Kubernetes. Despite that, in some cases a DevOps engineer might not be available while we're debugging an issue. For these situations and for general familiarity we should still familiarize ourselves with common Kubernetes issues to bridge the gap between development and operations. I think this also provides an important skill that helps us understand the work of DevOps better, with that understanding we can improve as a cohesive team. This guide explores prevalent Kubernetes errors and provides troubleshooting tips to help developers navigate the complex landscape of container orchestration.

Identifying Configuration Issues

When you encounter configuration issues in Kubernetes, the first place to check is the status column using the kubectl get pods command. Common errors manifest here, requiring further inspection with kubectl describe pod.

$ kubectl get pods
NAME                     READY    STATUS     RESTARTS   AGE 
my-first-pod-id-xxxx      1/1     Running    0          13s
my-second-pod-id-xxxx     1/1     Running    0          13s

Common Causes and Solutions

Insufficient Resources: Notice that this means resources for the POD itself and not resources within the container. It means the hardware or surrounding VM is hitting a limit.

Symptom: Pods fail to schedule due to resource constraints.

Solution: Scale up the cluster by adding more nodes to accommodate the resource requirements.

Volume Mounting Failures:

Symptom: Pods cannot mount volumes correctly.

Solution: Ensure storage is defined accurately in the pod specification and check the storage class and Persistent Volume (PV) configurations.

Detailed Investigation Steps

We can use kubectl describe pod: This command provides a detailed description of the pod, including events that have occurred. By examining these events, we can pinpoint the exact cause of the issue.

Another important step is resource quota analysis. Sometimes, resource constraints are due to namespace-level resource quotas. Use kubectl get resourcequotas to check if quotas are limiting pod creation.

Dealing with Image Pull Errors

Errors like ErrImagePull or ImagePullBackOff indicate issues with fetching container images. These errors are typically related to image availability or access permissions.

Troubleshooting Steps

The first step is checking the image name which we can do with the following command:

docker pull <image-name>

We then need to verify the image name for typos or invalid characters. I pipe the command through grep to verify the name is 100% identical, some typos are just notoriously hard to spot.

Credentials can also be a major pitfall. E.g. an authorization failure when pulling images from private repositories.

We must ensure that Docker registry credentials are correctly configured in Kubernetes secrets.

Network configuration should also be reviewed. Ensure that the Kubernetes nodes have network access to the Docker registry. Network policies or firewall rules might block access.

There are quite a few additional pitfalls such as problems with image tags. Ensure you are using the correct image tags. Latest tags might not always point to the expected image version.

If you're using a private registry you might be experiencing access issues. Make sure your credentials are up-to-date and the registry is accessible from all nodes in all regions.

Handling Node Issues

Node-related errors often point to physical or virtual machine issues. These issues can disrupt the normal operation of the Kubernetes cluster and need prompt attention.

To check node status use the command:

kubectl get nodes

We can then identify problematic nodes in the resulting output.

It's a cliché but sometimes rebooting nodes is the best solution to some problems. We can reboot the affected machine or VM. Kubernetes should attempt to "self-heal" and recover within a few minutes.

To investigate node conditions we can use the command:

kubectl describe node <node-name>

We should look for conditions such as MemoryPressure, DiskPressure, or NetworkUnavailable. These conditions provide clues about the underlying issue we should address in the node.

Preventive Measures

Node monitoring should be used to with tools such as Prometheus, Grafana to keep an eye on node health and performance. These work great for the low level Kubernetes related issues, we can also use them for high level application issues.

There are some automated healing tools such as the Kubernetes Cluster Autoscaler that we can leverage to automatically manage the number of nodes in your cluster based on workload demands. Personally, I'm not a huge fan as I'm afraid of a cascading failure that would trigger additional resource consumption.

Managing Missing Configuration Keys or Secrets

Missing configuration keys or secrets are common issues that disrupt Kubernetes deployments. Proper management of these elements is crucial for smooth operation.

We need to use ConfigMaps and secrets. These let us store configuration values and sensitive information securely. To avoid that we need to ensure that ConfigMaps and Secrets are correctly referenced in your pod specifications.

Inspect pod descriptions using the command:

kubectl describe pod <pod-name>

Review the output and look for missing configuration details. Rectify any misconfigurations.

ConfigMap and secret creation can be verified using the command:

kubectl get configmaps

and:

kubectl get secrets

Ensure that the required ConfigMaps and Secrets exist in the namespace and contain the expected data.

It's best to keep non-sensitive parts of ConfigMaps in version control while excluding Secrets for security. Furthermore, you should use different ConfigMaps and Secrets for different environments (development, staging, production) to avoid configuration leaks.

Utilizing Buildg for Interactive Debugging

Buildg is a relatively new tool that enhances the debugging process for Docker configurations by allowing interactive debugging.

It provides Interactive Debugging for configuration issues in a way that's similar to a standard debugging. It lets us step through the Dockerfile stages and set breakpoints. Buildg is compatible with VSCode and other IDEs via the Debug Adapter Protocol (DAP).

Buildg lets us inspect container state at each stage of the build process to identify issues early.

To install buildg follow the instructions on the Buildg GitHub page.

Conclusion

Debugging Kubernetes can be challenging, but with the right knowledge and tools, developers can effectively identify and resolve common issues. By understanding configuration problems, image pull errors, node issues, and the importance of ConfigMaps and Secrets, developers can contribute to more robust and reliable Kubernetes deployments. Tools like Buildg offer promising advancements in interactive debugging, further bridging the gap between development and operations.

As Kubernetes continues to evolve, staying informed about new tools and best practices will be essential for successful application management and deployment. By proactively addressing these common issues, developers can ensure smoother, more efficient Kubernetes operations, ultimately leading to more resilient and scalable applications.

Why is Kubernetes Debugging so Problematic?

Shai Almog — Tue, 04 Jun 2024 15:15:04 +0000

The Immutable Nature of Containers
The Limitations of kubectl exec
Avoiding Direct Modifications
Enter Ephemeral Containers
- Using kubectl debug
Practical Application of Ephemeral Containers
Security Considerations
Interlude: The Role of Observability
Command Line Debugging
Connecting a Standard IDE for Remote Debugging
Conclusion

Debugging application issues in a Kubernetes cluster can often feel like navigating a labyrinth. Containers are ephemeral by design, intended to be immutable once deployed. This presents a unique challenge when something goes wrong and we need to dig into the issue. Before diving into the debugging tools and techniques, it's essential to grasp the core problem: why modifying container instances directly is a bad idea. This blog post will walk you through the intricacies of Kubernetes debugging, offering insights and practical tips to effectively troubleshoot your Kubernetes environment.

The Immutable Nature of Containers

One of the fundamental principles of Kubernetes is the immutability of container instances. This means that once a container is running, it shouldn't be altered. Modifying containers on the fly can lead to inconsistencies and unpredictable behavior, especially as Kubernetes orchestrates the lifecycle of these containers, replacing them as needed. Imagine trying to diagnose an issue only to realize that the container you’re investigating has been modified, making it difficult to reproduce the problem consistently.

The idea behind this immutability is to ensure that every instance of a container is identical to any other instance. This consistency is crucial for achieving reliable, scalable applications. If you start modifying containers, you undermine this consistency, leading to a situation where one container behaves differently from another, even though they are supposed to be identical.

The Limitations of `kubectl exec`

We often start our journey in Kubernetes with commands such as:

$ kubectl -- exec -ti <pod-name>

This logs into a container and feels like accessing a traditional server with SSH. However, this approach has significant limitations. Containers often lack basic diagnostic tools—no vim, no traceroute, sometimes not even a shell. This can be a rude awakening for those accustomed to a full-featured Linux environment. Additionally, if a container crashes, kubectl exec becomes useless as there's no running instance to connect to. This tool is insufficient for thorough debugging, especially in production environments.

Consider the frustration of logging into a container only to find out that you can't even open a simple text editor to check configuration files. This lack of basic tools means that you are often left with very few options for diagnosing problems. Moreover, the minimalistic nature of many container images, designed to reduce their attack surface and footprint, exacerbates this issue.

Avoiding Direct Modifications

While it might be tempting to install missing tools on-the-fly using commands like apt-get install vim, this practice violates the principle of container immutability. In production, installing packages dynamically can introduce new dependencies, potentially causing application failures. The risks are high, and it's crucial to maintain the integrity of your deployment manifests, ensuring that all configurations are predefined and reproducible.

Imagine a scenario where a quick fix in production involves installing a missing package. This might solve the immediate problem but could lead to unforeseen consequences. Dependencies introduced by the new package might conflict with existing ones, leading to application instability. Moreover, this approach makes it challenging to reproduce the exact environment, which is vital for debugging and scaling your application.

Enter Ephemeral Containers

The solution to the aforementioned problems lies in ephemeral containers. Kubernetes allows the creation of these temporary containers within the same pod as the application container you need to debug. These ephemeral containers are isolated from the main application, ensuring that any modifications or tools installed do not impact the running application.

Ephemeral containers provide a way to bypass the limitations of kubectl exec without violating the principles of immutability and consistency. By launching a separate container within the same pod, you can inspect and diagnose the application container without altering its state. This approach preserves the integrity of the production environment while giving you the tools you need to debug effectively.

Using `kubectl debug`

The kubectl debug command is a powerful tool that simplifies the creation of ephemeral containers. Unlike kubectl exec, which logs into the existing container, kubectl debug creates a new container within the same namespace. This container can run a different OS, mount the application container’s filesystem, and provide all necessary debugging tools without altering the application’s state. This method ensures you can inspect and diagnose issues even if the original container is not operational.

For example, let’s consider a scenario where we’re debugging a container using an ephemeral Ubuntu container:

kubectl debug <myapp> -it <pod-name> --image=ubuntu --share-process --copy-to=<myapp-debug>

This command launches a new Ubuntu-based container within the same pod, providing a full-fledged environment to diagnose the application container. Even if the original container lacks a shell or crashes, the ephemeral container remains operational, allowing you to perform necessary checks and install tools as needed. It relies on the fact that we can have multiple containers in the same pod, that way we can inspect the filesystem of the debugged container without physically entering that container.

Practical Application of Ephemeral Containers

To illustrate, let’s delve deeper into how ephemeral containers can be used in real-world scenarios. Suppose you have a container that consistently crashes due to a mysterious issue. By deploying an ephemeral container with a comprehensive set of debugging tools, you can monitor the logs, inspect the filesystem, and trace processes without worrying about the constraints of the original container environment.

For instance, you might encounter a situation where an application container crashes due to an unhandled exception. By using kubectl debug, you can create an ephemeral container that shares the same network namespace as the original container. This allows you to capture network traffic and analyze it to understand if there are any issues related to connectivity or data corruption.

Security Considerations

While ephemeral containers reduce the risk of impacting the production environment, they still pose security risks. It’s critical to restrict access to debugging tools and ensure that only authorized personnel can deploy ephemeral containers. Treat access to these systems with the same caution as handing over the keys to your infrastructure.

Ephemeral containers, by their nature, can access sensitive information within the pod. Therefore, it is essential to enforce strict access controls and audit logs to track who is deploying these containers and what actions are being taken. This ensures that the debugging process does not introduce new vulnerabilities or expose sensitive data.

Interlude: The Role of Observability

While tools like kubectl exec and kubectl debug are invaluable for troubleshooting, they are not replacements for comprehensive observability solutions. Observability allows you to monitor, trace, and log the behavior of your applications in real-time, providing deeper insights into issues without the need for intrusive debugging sessions.

These tools aren't meant for everyday debugging, that role should be occupied by various observability tools. I will discuss observability in more detail in an upcoming post.

Command Line Debugging

While tools like kubectl exec and kubectl debug are invaluable, there are times when you need to dive deep into the application code itself. This is where we can use command line debuggers. Command line debuggers allow you to inspect the state of your application at a very granular level, stepping through code, setting breakpoints, and examining variable states. Personally, I don't use them much

For instance, Java developers can use jdb, the Java Debugger, which is analogous to gdb for C/C++ programs. Here’s a basic rundown of how you might use jdb in a Kubernetes environment:

Set Up Debugging: First, you need to start your Java application with debugging enabled. This typically involves adding a debug flag to your Java command. However, as discussed in my post here, there's an even more powerful way that doesn't require a restart:

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar myapp.jar

Port Forwarding: Since the debugger needs to connect to the application, you’ll set up port forwarding to expose the debug port of your pod to your local machine. This is important as JDWP is dangerous:

kubectl port-forward <pod-name> 5005:5005

Connecting the Debugger: With port forwarding in place, you can now connect jdb to the remote application:

jdb -attach localhost:5005

From here, you can use jdb commands to set breakpoints, step through code, and inspect variables. This process allows you to debug issues within the code itself, which can be invaluable for diagnosing complex problems that aren’t immediately apparent through logs or superficial inspection.

Connecting a Standard IDE for Remote Debugging

I prefer IDE debugging by far. I never used JDB for anything other than a demo. Modern IDEs support remote debugging, and by leveraging Kubernetes port forwarding, you can connect your IDE directly to a running application inside a pod.

To set up remote debugging we start with the same steps as the command line debugging. Configuring the application and setting up the port forwarding.

Configure the IDE: In your IDE (e.g., IntelliJ IDEA, Eclipse), set up a remote debugging configuration. Specify the host as localhost and the port as 5005.
Start Debugging: Launch the remote debugging session in your IDE. You can now set breakpoints, step through code, and inspect variables directly within the IDE, just as if you were debugging a local application.

I show how to do it in IntelliJ/IDEA here.

Conclusion

Debugging Kubernetes environments requires a blend of traditional techniques and modern tools designed for container orchestration. Understanding the limitations of kubectl exec and the benefits of ephemeral containers can significantly enhance your troubleshooting process. However, the ultimate goal should be to build robust observability into your applications, reducing the need for ad-hoc debugging and enabling proactive issue detection and resolution.

By following these guidelines and leveraging the right tools, you can navigate the complexities of Kubernetes debugging with confidence and precision. In the next installment of this series, we’ll delve into common configuration issues in Kubernetes and how to address them effectively.

Debugging Kubernetes Part 1: An Introduction

Shai Almog — Tue, 21 May 2024 14:56:00 +0000

Introduction to Kubernetes and Distributed Systems
- The Evolution of Deployment Technologies
- Enter Virtualization
- Rise of Containers
- Rise of Orchestration
Enter Kubernetes
- Why Kubernetes Stands Out
Kubernetes For Developers
- Kubernetes Basics In Practice
Final Word

While debugging in an IDE or using simple command line tools is relatively straightforward, the real challenge lies in production debugging. Modern production environments have enabled sophisticated self-healing deployments, yet they have also made troubleshooting more complex. Kubernetes (aka k8s) is probably the most well known orchestration production environment. To effectively teach debugging in Kubernetes, it's essential to first introduce its fundamental principles.

This part of the debugging series is designed for developers looking to effectively tackle application issues within Kubernetes environments, without delving deeply into the complex DevOps aspects typically associated with its operations. Kubernetes is a big subject, it took me two videos just to explain the basic concepts and background.

Introduction to Kubernetes and Distributed Systems

Kubernetes, while often discussed in the context of cloud computing and large-scale operations, is not just a tool for managing containers. Its principles apply broadly to all large-scale distributed systems. In this post I want to explore Kubernetes from the ground up, emphasizing its role in solving real-world problems faced by developers in production environments.

The Evolution of Deployment Technologies

Before Kubernetes, the deployment landscape was markedly different. Understanding this evolution helps appreciate the challenges Kubernetes aims to solve. The image below represents the road to Kubernetes and the technologies we passed along the way.

In the image we can see that initially, applications were deployed directly onto physical servers. This process was manual, error-prone, and difficult to replicate across multiple environments. For instance, if a company needed to scale its application, it involved procuring new hardware, installing operating systems, and configuring the application from scratch. This could take weeks or even months, leading to significant downtime and operational inefficiencies.

Imagine a retail company preparing for the holiday season surge. Each time they needed to handle increased traffic, they would manually set up additional servers. This was not only time-consuming but also prone to human error. Scaling down after the peak period was equally cumbersome, leading to wasted resources.

Enter Virtualization

Virtualization technology introduced a layer that emulated the hardware, allowing for easier replication and migration of environments but at the cost of performance. However, fast virtualization enabled the cloud revolution. It let companies like Amazon lease its servers at scale without compromising their own workloads.

Virtualization involves running multiple operating systems on a single physical hardware host. Each virtual machine (VM) includes a full copy of an operating system, the application, necessary binaries, and libraries—taking up tens of GBs. VMs are managed via a hypervisor, such as VMware's ESXi or Microsoft's Hyper-V, which sits between the hardware and the operating system and is responsible for distributing hardware resources among the VMs. This layer adds additional overhead and can lead to decreased performance due to the need to emulate hardware.

Note that virtualization is often referred to as "virtual machines", I chose to avoid that terminology due to the focus of this blog on Java and the JVM where a virtual machine is typically a reference to the Java Virtual Machine (JVM).

Rise of Containers

Containers emerged as a lightweight alternative to full virtualization. Tools like Docker standardized container formats, making it easier to create and manage containers without the overhead associated with traditional virtual machines. Containers encapsulate an application’s runtime environment, making them portable and efficient.

Unlike virtualization, containerization encapsulates an application in a container with its own operating environment, but it shares the host system’s kernel with other containers. Containers are thus much more lightweight, as they do not require a full OS instance; instead, they include only the application and its dependencies, such as libraries and binaries. This setup reduces the size of each container and improves boot times and performance by removing the hypervisor layer.

Containers operate using several key Linux kernel features:

Namespaces: Containers use namespaces to provide isolation for global system resources between independent containers. This includes aspects of the system like process IDs, networking interfaces, and file system mounts. Each container has its own isolated namespace, which gives it a private view of the operating system with access only to its resources.
Control Groups (cgroups): Cgroups further enhance the functionality of containers by limiting and prioritizing the hardware resources a container can use. This includes parameters such as CPU time, system memory, network bandwidth, or combinations of these resources. By controlling resource allocation, cgroups ensure that containers do not interfere with each other’s performance and maintain the efficiency of the underlying server.
Union File Systems: Containers use union file systems, such as OverlayFS, to layer files and directories in a lightweight and efficient manner. This system allows containers to appear as though they are running on their own operating system and file system, while they are actually sharing the host system’s kernel and base OS image.

Rise of Orchestration

As containers began to replace virtualization due to their efficiency and speed, developers and organizations rapidly adopted them for a wide range of applications. However, this surge in container usage brought with it a new set of challenges, primarily related to managing large numbers of containers at scale.

While containers are incredibly efficient and portable, they introduce complexities when used extensively, particularly in large-scale, dynamic environments:

Management Overhead: Manually managing hundreds or even thousands of containers quickly becomes unfeasible. This includes deployment, networking, scaling, and ensuring availability and security.
Resource Allocation: Containers must be efficiently scheduled and managed to optimally use physical resources, avoiding underutilization or overloading of host machines.
Service Discovery and Load Balancing: As the number of containers grows, keeping track of which container offers which service and how to balance the load between them becomes critical.
Updates and Rollbacks: Implementing rolling updates, managing version control, and handling rollbacks in a containerized environment require robust automation tools.

To address these challenges, the concept of container orchestration was developed. Orchestration automates the scheduling, deployment, scaling, networking, and lifecycle management of containers, which are often organized into microservices. Efficient orchestration tools help ensure that the entire container ecosystem is healthy and that applications are running as expected.

Enter Kubernetes

Among the orchestration tools, Kubernetes emerged as a frontrunner due to its robust capabilities, flexibility, and strong community support. Kubernetes offers several features that address the core challenges of managing containers:

Automated Scheduling: Kubernetes intelligently schedules containers on the cluster’s nodes, taking into account the resource requirements and other constraints, optimizing for efficiency and fault tolerance.
Self-Healing Capabilities: It automatically replaces or restarts containers that fail, ensuring high availability of services.
Horizontal Scaling: Kubernetes can automatically scale applications up and down based on demand, which is essential for handling varying loads efficiently.
Service Discovery and Load Balancing: Kubernetes can expose a container using the DNS name or using its own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic so that the deployment is stable.
Automated Rollouts and Rollbacks: Kubernetes allows you to describe the desired state for your deployed containers using declarative configuration, and can change the actual state to the desired state at a controlled rate, such as to roll out a new version of an application.

Why Kubernetes Stands Out

Kubernetes not only solves practical, operational problems associated with running containers but also integrates with the broader technology ecosystem, supporting continuous integration and continuous deployment (CI/CD) practices. It is backed by the Cloud Native Computing Foundation (CNCF), ensuring it remains cutting-edge and community-focused.

There used to be a site called "doyouneedkubernetes.com" when you visited that site it said "No". Most of us don't need Kubernetes and it is often a symptom of Resume Driven Design (RDD). However, even when we don't need its scaling capabilities the advantages of its standardization are tremendous. Kubernetes became the de-facto standard and created a cottage industry of tools around it. Features such as, observability and security can be plugged in easily. Cloud migration becomes arguably easier. Kubernetes is now the "lingua franca" of production environments.

Kubernetes For Developers

Understanding Kubernetes architecture is crucial for debugging and troubleshooting. The following image shows the high level view of a Kubernetes deployment. There are far more details in most tutorials geared towards DevOps engineers, but for a developer the point that matters is just "Your Code": that tiny corner at the edge.

In the image above we can see:

Master Node (represented by the blue Kubernetes logo on the left): The control plane of Kubernetes, responsible for managing the state of the cluster, scheduling applications, and handling replication.
Worker Nodes: These nodes contain the pods that run the containerized applications. Each worker node is managed by the master.
Pods: The smallest deployable units created and managed by Kubernetes, usually containing one or more containers that need to work together.

These components work together to ensure that an application runs smoothly and efficiently across the cluster.

Kubernetes Basics In Practice

Up until now this post has been theory heavy, let's review some commands we can use to work with a Kubernetes cluster. First we would want to list the pods we have within the cluster which we can do using the get pods command as such:

$ kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
  my-first-pod-id-xxxx     1/1    Running   0          13s
  my-second-pod-id-xxxx    1/1    Running   0          13s

A command such as kubectl describe pod returns high level description of the pod such as its name, parent node, etc. Many problems in production pods can be solved by looking at the system log, this can be accomplished by invoking the logs command:

$ kubectl logs -f <pod>
[2022-11-29 04:12:17,262] INFO log data
...

Most typical large scale applications logs are ingested by tools such as Elastic, Loki etc. As such, the logs command isn't as useful in production except for debugging edge cases.

Final Word

This introduction to Kubernetes has set the stage for deeper exploration into specific debugging and troubleshooting techniques, which we will cover in the upcoming posts. The complexity of Kubernetes makes is much harder to debug, but there are facilities in place to workaround some of that complexity.

While this article (and its followups) focus on Kubernetes, future posts will delve into observability and related tools, which are crucial for effective debugging in production environments.

Failure is Required: Understanding Fail-Safe and Fail-Fast Strategies

Shai Almog — Tue, 07 May 2024 16:59:55 +0000

Fail-Fast
Fail-Safe
Choosing Between Fail-Fast and Fail-Safe
- Balancing Both
- Consistent Layer Behavior
- Retry's Are not Fail-Safe
Redefining Failure
- Data Corruption
- Don't Fix the Bug
Debugging Failure
- Avoiding Cascading Failure
Hybrid in Production
Final Word

Failures in software systems are inevitable. How these failures are handled can significantly impact system performance, reliability, and the business’s bottom line. In this post I want to discuss the upside of failure. Why you should seek failure, why failure is good and why avoiding failure can reduce the reliability of your application. We will start with the discussion of fail-fast vs. fail-safe, this will take us to the second discussion about failures in general.

Fail-Fast

Fail-fast systems are designed to immediately stop functioning upon encountering an unexpected condition. This immediate failure helps to catch errors early, making debugging more straightforward.

The fail-fast approach ensures that errors are caught immediately. For example, in the world of programming languages, Java embodies this approach by producing a NullPointerException instantly when encountering a null value, stopping the system and making the error clear. This immediate response helps developers identify and address issues quickly, preventing them from becoming more serious.

By catching and stopping errors early, fail-fast systems reduce the risk of cascading failures, where one error leads to others. This makes it easier to contain and resolve issues before they spread through the system, preserving overall stability.

It is easy to write unit and integration tests for fail-fast systems. This advantage is even more pronounced when we need to understand the test failure. Fail-fast systems usually point directly at the problem in the error stack trace.

However, fail-fast systems carry their own risks, particularly in production environments:

Production Disruptions: If a bug reaches production, it can cause immediate and significant disruptions, potentially impacting both system performance and the business’s operations.
Risk Appetite: Fail-fast systems require a level of risk tolerance from both engineers and executives. They need to be prepared to handle and address failures quickly, often balancing this with potential business impacts.

Fail-Safe

Fail-safe systems take a different approach, aiming to recover and continue even in the face of unexpected conditions. This makes them particularly suited for uncertain or volatile environments.

Microservices are a prime example of fail-safe systems, embracing resiliency through their architecture. Circuit breakers, both physical and software-based, disconnect failing functionality to prevent cascading failures, helping the system continue operating.

Fail-safe systems ensure that systems can survive even harsh production environments, reducing the risk of catastrophic failure. This makes them particularly suited for mission-critical applications, such as in hardware devices or aerospace systems, where smooth recovery from errors is crucial.

However, fail-safe systems have downsides:

Hidden Errors: By attempting to recover from errors, fail-safe systems can delay the detection of issues, making them harder to trace and potentially leading to more severe cascading failures.
Debugging Challenges: This delayed nature of errors can complicate debugging, requiring more time and effort to find and resolve issues.

Choosing Between Fail-Fast and Fail-Safe

It's challenging to determine which approach is better, as both have their merits. Fail-fast systems offer immediate debugging, lower risk of cascading failures, and quicker detection and resolution of bugs. This helps catch and fix issues early, preventing them from spreading.

Fail-safe systems handle errors gracefully, making them better suited for mission-critical systems and volatile environments, where catastrophic failures can be devastating.

Balancing Both

To leverage the strengths of each approach, a balanced strategy can be effective:

Fail-Fast for Local Services: When invoking local services like databases, fail-fast can catch errors early, preventing cascading failures.
Fail-Safe for Remote Resources: When relying on remote resources, such as external web services, fail-safe can prevent disruptions from external failures.

A balanced approach also requires clear and consistent implementation throughout coding, reviews, tooling, and testing processes, ensuring it is integrated seamlessly. Fail-fast can integrate well with orchestration and observability. Effectively, this moves the fail-safe aspect to a different layer of OPS instead of into the developer layer.

Consistent Layer Behavior

This is where things get interesting. It isn't about choosing between fail-safe and fail-fast. It's about choosing the right layer for them. E.g. if an error is handled in a deep layer using a fail-safe approach, it won't be noticed. This might be OK, but if that error has adverse impact (performance, garbage data, corruption, security, etc.) then we will have a problem later on and won't have a clue.

The right solution is to handle all errors in a single layer, in modern systems the top layer is the OPS layer and it makes the most sense. It can report the error to the engineers who are most qualified to deal with the error. But they can also provide immediate mitigation such as restarting a service, allocating additional resources or reverting a version.

Retry's Are not Fail-Safe

Recently I was at a lecture where the speakers listed their updated cloud architecture. They chose to take a shortcut to microservices by using a framework that allows them to retry in the case of failure. Unfortunately, failure doesn't behave the way we would like. You can't eliminate it completely through testing alone. Retry, isn't fail-safe. In-fact: it can mean catastrophe.

They tested their system and "it works", even in production. But lets assume that a catastrophic situation does occur, their retry mechanism can operate as a denial of service attack against their own servers. The number of ways in which ad-hoc architectures such as this can fail are mind-boggling.

This is especially important once we redefine failures.

Redefining Failure

Failures in software systems aren't just about crashes. A crash can be seen as a simple and immediate failure, but there are more complex issues to consider. In fact, crashes in the age of containers are probably the best failures. A system restarts seamlessly with barely an interruption.

Data Corruption

Data corruption is far more severe and insidious than a crash. It carries with it long-term consequences. Corrupted data can lead to security and reliability problems that are challenging to fix, requiring extensive reworking and potentially unrecoverable data.

Cloud computing has led to defensive programming techniques, like circuit breakers and retries, emphasizing comprehensive testing and logging to catch and handle failures gracefully. In a way, this environment sent us back in terms of quality.

A fail-fast system at the data level could stop this from happening. Addressing a bug goes beyond a simple fix. It requires understanding its root cause and preventing reoccurrence, extending into comprehensive logging, testing, and process improvements. This ensures that the bug is fully addressed, reducing the chances of it reoccurring.

Don't Fix the Bug

If it's a bug in production you should probably revert, if you can't instantly revert production. This should always be possible and if it isn't this is something you should work on.

Failures must be fully understood before a fix is undertaken. In my own companies I often skipped that step due to pressure, in a small startup that is forgivable. In larger companies we need to understand the root cause. A culture of debrief for bugs and production issues is essential. The fix should also include process mitigation that prevents similar issues from reaching production.

Debugging Failure

Fail-fast systems are much easier to debug. They have inherently simpler architecture and it is easier to pinpoint an issue to a specific area. It is crucial to throw exceptions even for minor violations (e.g. validations). This prevents cascading types of bugs that prevail in loose systems.

This should be further enforced by unit tests that verify the limits we define and verify proper exceptions are thrown. Retries should be avoided in the code as they make debugging exceptionally difficult and their proper place is in the OPS layer. To facilitate that further, timeouts should be short by default.

Avoiding Cascading Failure

Failure isn't something we can avoid, predict or fully test against. The only thing we can do is soften the blow when a failure occurs. Often this "softening" is achieved by using long running tests meant to replicate extreme conditions as much as possible with the goal of finding our applications weak spots. This is rarely enough, robust systems need to revise these tests often based on real production failures.

A great example of fail-safe would be a cache of REST responses that lets us keep working even when a service is down. Unfortunately, this can lead to complex niche issues such as cache poisoning or a situation in which a banned user still had access due to cache.

Hybrid in Production

Fail-safe is best applied only in production/staging and in the OPS layer. This reduces the amount of changes between production and dev, we want them to be as similar as possible, yet it's still a change which can negatively impact production. But the benefits are tremendous as observability can get a clear picture of system failures.

The discussion here is a bit colored by my more recent experience of building observable cloud architectures. However, the same principle applies to any type of software whether embedded or in the cloud. In such cases we often choose to implement fail-safe in the code, in this case I would suggest implementing it consistently and consciously in a specific layer.

There's also a special case of libraries/frameworks that often provide inconsistent and badly documented behaviors in these situations. I myself am guilty of such inconsistency in some of my work. It's an easy mistake to make.

Final Word

This is my last post on the theory of debugging series that's part of my book/course on debugging. We often think of debugging as the action we take when something fails, it isn't. Debugging starts the moment we write the first line of code. We make decisions that will impact the debugging process as we code, often we're just unaware of these decisions until we get a failure.

I hope this post and series will help you write code that is prepared for the unknown. Debugging, by its nature, deals with the unexpected. Tests can't help. But as I illustrated in my previous posts, there are many simple practices we can undertake that would make it easier to prepare. This isn't a one time process, it's an iterative process that requires re-evaluation of decisions made as we encounter failure.

Software Testing as a Debugging Tool

Shai Almog — Tue, 23 Apr 2024 15:00:00 +0000

The Intersection of Debugging and Testing
- Unit Tests
- Integration Tests
- Coverage
The Debug-Fix Cycle
Composing Tests with Debuggers
Test-Driven Development
Final Word

Debugging is not just about identifying errors—it's about instituting a reliable process for ensuring software health and longevity. In this post we discuss the role of software testing in debugging, including foundational concepts and how they converge to improve software quality.

The Intersection of Debugging and Testing

Debugging and testing play distinct roles in software development. Debugging is the targeted process of identifying and fixing known bugs. Testing, on the other hand, encompasses a adjacent scope, identifying unknown issues by validating expected software behavior across a variety of scenarios.

Both are a part of the debug fix cycle which is a core concept in debugging. Before we cover the cycle we should first make sure we're aligned on the basic terminology.

Unit Tests

Unit tests are tightly linked to debugging efforts, focusing on isolated parts of the application—typically individual functions or methods. Their purpose is to validate that each unit operates correctly in isolation, making them a swift and efficient tool in the debugging arsenal. These tests are characterized by their speed and consistency, enabling developers to run them frequently, sometimes even automatically as code is written within the IDE.

Since software is so tightly bound it is nearly impossible to compose unit tests without extensive mocking. Mocking involves substituting a genuine component with a stand-in that returns predefined results, thus a test method can simulate scenarios without relying on the actual object. This is a powerful yet controversial tool. By using mocking we're in-effect creating a synthetic environment that might misrepresent the real world. We're reducing the scope of the test and might perpetuate some bugs.

Integration Tests

Opposite to unit tests, integration tests examine the interactions between multiple units, providing a more comprehensive picture of the system's health. While they cover broader scenarios, their setup can be more complex due to the interactions involved. However, they are crucial in catching bugs that arise from the interplay between different software components.

In general mocking can be used in integration tests but it is discouraged. They take longer to run and are sometimes harder to set up. However, many developers (myself included) would argue that they are the only benchmark for quality. Most bugs express themselves in the seams between the modules and integration tests are better at detecting that.

Since they are far more important some developers would argue that unit tests are unnecessary. This isn't true, unit test failures are much easier to read and understand. Since they are faster we can run them during development, even while typing. In that sense the balance between the two approaches is the important part.

Coverage

Coverage is a metric that helps quantify the effectiveness of testing by indicating the proportion of code exercised by tests. It helps identify potential areas of the code that have not been tested, which could harbor undetected bugs. However, striving for 100% coverage can be a case of diminishing returns; the focus should remain on the quality and relevance of the tests rather than the metric itself. In my experience, chasing high coverage numbers often results in bad test practices that persist problems.

It is my opinion that unit tests should be excluded from coverage metrics due to the importance of integration tests to overall quality. To get a sense of quality coverage should focus on integration and end to end tests.

The Debug-Fix Cycle

The debug-fix cycle is a structured approach that integrates testing into the debugging process. The stages include identifying the bug, creating a test that reproduces the bug, fixing the bug, verifying the fix with the test, and finally, running the application to ensure the fix works in the live environment. This cycle emphasizes the importance of testing in not only identifying but also in preventing the recurrence of bugs.

Notice that this is a simplified version of the cycle with a focus on the testing aspect only. The full cycle includes discussion of the issue tracking and versioning as part of the whole process. I discuss this more in-depth in other posts in the series and my book.

Composing Tests with Debuggers

A powerful feature of using debuggers in test composition is their ability to "jump to line" or "set value." Developers can effectively reset the execution to a point before the test and rerun it with different conditions, without recompiling or rerunning the entire suite. This iterative process is invaluable for achieving desired test constraints and improves the quality of unit tests by refining the input parameters and expected outcomes.

Increasing test coverage is about more than hitting a percentage; it's about ensuring that tests are meaningful and that they contribute to software quality. A debugger can significantly assist in this by identifying untested paths. When a test coverage tool highlights lines or conditions not reached by current tests, the debugger can be used to force execution down those paths. This helps in crafting additional tests that cover missed scenarios, ensuring that the coverage metric is not just a number but a true reflection of the software's tested state.

In this case you will notice that the next line in the body is a rejectValue call which will throw an exception. I don’t want an exception thrown as I still want to test all the permutations of the method. I can drag the execution pointer (arrow on the left) and place it back at the start of the method.

Test-Driven Development

How does all of this fit with disciplines like Test-Driven Development (TDD)?

It doesn't fit well. Before we get into that let's revisit the basics of TDD. Weak TDD typically means just writing tests before writing the code. Strong TDD involves a red-green-refactor cycle:

Red: Write a test that fails because the feature it tests isn't implemented yet.
Green: Write the minimum amount of code necessary to make the test pass.
Refactor: Clean up the code while ensuring that tests continue to pass.

This rigorous cycle guarantees that new code is continually tested and refactored, reducing the likelihood of complex bugs. It also means that when bugs do appear, they are often easier to isolate and fix due to the modular and well-tested nature of the codebase. At least, that's the theory.

TDD can be especially advantageous for scripting and loosely typed languages. In environments lacking the rigid structure of compilers and linters, TDD steps in to provide the necessary checks that would otherwise be performed during compilation in statically typed languages. It becomes a crucial substitute for compiler/linter checks, ensuring that type and logic errors are caught early.

In real-world application development, TDD's utility is nuanced. While it encourages thorough testing and upfront design, it can sometimes hinder the natural flow of development, especially in complex systems that evolve through numerous iterations. The requirement for 100% test coverage can lead to an unnecessary focus on fulfilling metrics rather than writing meaningful tests.

The biggest problem in TDD is its focus on unit testing. TDD is impractical with integration tests as the process would take too long. But as we determined in the start of this post, integration tests are the true benchmark for quality. In that test TDD is a methodology that provides great quality for arbitrary tests, but not necessarily great quality for the final product. You might have the best cog in the world, but if doesn't fit well into the machine then it isn't great.

Final Word

Debugging is a tool that not only fixes bugs but also actively aids in crafting tests that bolster software quality. By utilizing debuggers in test composition and increasing coverage, developers can create a suite of tests that not only identifies existing issues but also guards against future ones, thus ensuring the delivery of reliable, high-quality software.

Debugging lets us increase coverage and verify edge cases effectively. It's part of a standardized process for issue resolution that's critical for reliability and prevents regressions.

Wireshark & tcpdump: A Debugging Power Couple

Shai Almog — Tue, 09 Apr 2024 13:58:49 +0000

Introduction to Wireshark
- Browser Network Monitors
Installation and Getting Started
Navigating Through Noise with Filters
Deep Dive into Data Analysis
Beyond Basic Usage
- The Basics of HTTPS Encryption
- Methods for Decrypting HTTPS in Wireshark
  - Pre-Master Secret Key Logging
  - Using a Proxy
Integrating tcpdump with Wireshark for Enhanced Network Analysis
The Role of tcpdump in Network Troubleshooting
- Key Scenarios for tcpdump Usage:
  - Using tcpdump Effectively
  - Challenges and Considerations
Final Word

Wireshark, the free open-source packet sniffer and network protocol analyzer, has cemented itself as an indispensable tool in network troubleshooting, analysis, and security (on both sides). This blog post delves into the features, uses, and practical tips for harnessing the full potential of Wireshark, expanding on aspects that may have been glossed over in discussions or demonstrations. Whether you're a developer, security expert, or just curious about network operations, this guide will enhance your understanding of Wireshark and its applications.

Introduction to Wireshark

Wireshark was initially developed by Eric Rescorla and Gerald Combs, designed to capture and analyze network packets in real-time. Its capabilities extend across various network interfaces and protocols, making it a versatile tool for anyone involved in networking. Unlike its command-line counterpart, tcpdump, Wireshark's graphical interface simplifies the analysis process, presenting data in a user-friendly "proto view" that organizes packets in a hierarchical structure. This facilitates quick identification of protocols, ports, and data flows.

The key features of Wireshark are:

Graphical User Interface (GUI): Eases the analysis of network packets compared to command-line tools.
Proto View: Displays packet data in a tree structure, simplifying protocol and port identification.
Compatibility: Supports a wide range of network interfaces and protocols.

Browser Network Monitors

FireFox and Chrome contain a far superior network monitor tool built into them. It is superior because it is simpler to use and works with secure websites out of the box. If you can use the browser to debug the network traffic you should do that.

In the cases where your traffic requires low level protocol information or is outside of the browser, Wireshark is the next best thing.

Installation and Getting Started

To begin with Wireshark, visit their official website for the download. The installation process is straightforward, but attention should be paid to the installation of command-line tools, which may require separate steps. Upon launching Wireshark, users are greeted with a selection of network interfaces as seen below. Choosing the correct interface, such as the loopback for local server debugging, is crucial for capturing relevant data.

When debugging a Local Server (localhost) use the loopback interface. Remote servers will probably fit with the en0 network adapter. You can use the activity graph next to the network adapter to identify active interfaces for capture.

Navigating Through Noise with Filters

One of the challenges of using Wireshark is the overwhelming amount of data captured, including irrelevant "background noise" as seen in the following image.

Wireshark addresses this with powerful display filters, allowing users to hone in on specific ports, protocols, or data types. For instance, filtering TCP traffic on port 8080 can significantly reduce unrelated data, making it easier to debug specific issues.

Notice that the there is a completion widget on top of the Wireshark UI that lets you find out the values more easily.

In this case we filter by port tcp.port == 8080 which is the port used typically in Java servers (e.g. Spring Boot/tomcat).

But this isn't enough as HTTP is more concise. We can filter by protocol by adding http to the filter which narrows the view to HTTP requests and responses as shown in the following image.

Deep Dive into Data Analysis

Wireshark excels in its ability to dissect and present network data in an accessible manner. For example, HTTP responses carrying JSON data are automatically parsed and displayed in a readable tree structure as seen below. This feature is invaluable for developers and analysts, providing insights into the data exchanged between clients and servers without manual decoding.

Wireshark parses and displays JSON data within the packet analysis pane. It offers both hexadecimal and ASCII views for raw packet data.

Beyond Basic Usage

While Wireshark's basic functionalities cater to a wide range of networking tasks, its true strength lies in advanced features such as ethernet network analysis, HTTPS decryption, and debugging across devices. These tasks, however, may involve complex configuration steps and a deeper understanding of network protocols and security measures.

There are two big challenges when working with Wireshark:

HTTPS Decryption: Decrypting HTTPS traffic requires additional configuration but offers visibility into secure communications.
Device Debugging: Wireshark can be used to troubleshoot network issues on various devices, requiring specific knowledge of network configurations.

The Basics of HTTPS Encryption

HTTPS uses the Transport Layer Security (TLS) or its predecessor, Secure Sockets Layer (SSL), to encrypt data. This encryption mechanism ensures that any data transferred between the web server and the browser remains confidential and untouched. The process involves a series of steps including handshake, data encryption, and data integrity checks.

Decrypting HTTPS traffic is often necessary for developers and network administrators to troubleshoot communication errors, analyze application performance, or ensure that sensitive data is correctly encrypted before transmission. It's a powerful capability in diagnosing complex issues that cannot be resolved by simply inspecting unencrypted traffic or server logs.

Methods for Decrypting HTTPS in Wireshark

Important: Decrypting HTTPS traffic should only be done on networks and systems you own or have explicit permission to analyze. Unauthorized decryption of network traffic can violate privacy laws and ethical standards.

Pre-Master Secret Key Logging

One common method involves using the pre-master secret key to decrypt HTTPS traffic. Browsers like Firefox and Chrome can log the pre-master secret keys to a file when configured to do so. Wireshark can then use this file to decrypt the traffic:

Configure the Browser: Set an environment variable (SSLKEYLOGFILE) to specify a file where the browser will save the encryption keys.
Capture Traffic: Use Wireshark to capture the traffic as usual.
Decrypt the Traffic: Point Wireshark to the file with the pre-master secret keys (through Wireshark's preferences) to decrypt the captured HTTPS traffic.

Using a Proxy

Another approach involves routing traffic through a proxy server that decrypts HTTPS traffic and then re-encrypts it before sending it to the destination. This method might require setting up a dedicated decryption proxy that can handle the TLS encryption/decryption:

Set Up a Decryption Proxy: Tools like Mitmproxy or Burp Suite can act as an intermediary that decrypts and logs HTTPS traffic.
Configure Network to Route Through Proxy: Ensure the client's network settings route traffic through the proxy.
Inspect Traffic: Use the proxy's tools to inspect the decrypted traffic directly.

Integrating tcpdump with Wireshark for Enhanced Network Analysis

While Wireshark offers a graphical interface for analyzing network packets, there are scenarios where using it directly may not be feasible due to security policies or operational constraints. tcpdump, a powerful command-line packet analyzer, becomes invaluable in these situations, providing a flexible and less intrusive means of capturing network traffic.

The Role of tcpdump in Network Troubleshooting

tcpdump allows for the capture of network packets without a graphical user interface, making it ideal for use in environments with strict security requirements or limited resources. It operates under the principle of capturing network traffic to a file, which can then be analyzed at a later time or on a different machine using Wireshark.

Key Scenarios for tcpdump Usage:

High-security Environments: In places like banks or government institutions where running network sniffers might pose a security risk, tcpdump offers a less intrusive alternative.
Remote Servers: Debugging issues on a cloud server can be challenging with Wireshark due to the graphical interface; tcpdump captures can be transferred and analyzed locally.
Security-conscious Customers: Customers may be hesitant to allow third-party tools to run on their systems; tcpdump's command-line operation is often more palatable.

Using tcpdump Effectively

Capturing traffic with tcpdump involves specifying the network interface and an output file for the capture. This process is straightforward but powerful, allowing for detailed analysis of network interactions:

Command Syntax: The basic command structure for initiating a capture involves specifying the network interface (e.g., en0 for wireless connections) and the output file name.
Execution: Once the command is run, tcpdump silently captures network packets. The capture continues until it's manually stopped, at which point the captured data can be saved to the specified file.
Opening Captures in Wireshark: The file generated by tcpdump can be opened in Wireshark for detailed analysis, utilizing Wireshark's advanced features for dissecting and understanding network traffic.

The following shows the tcpdump command and its output:

$ sudo tcpdump -i en0 -w output
Password:
tcpdump: listening on en, link-type EN10MB (Ethernet), capture size 262144 bytes
^C3845 packets captured
4189 packets received by filter
0 packets dropped by kernel

Challenges and Considerations

Identifying the correct network interface for capture on remote systems might require additional steps, such as using the ifconfig command to list available interfaces. This step is crucial for ensuring that relevant traffic is captured for analysis.

Final Word

Wireshark stands out as a powerful tool for network analysis, offering deep insights into network traffic and protocols. Whether it's for low-level networking work, security analysis, or application development, Wireshark's features and capabilities make it an essential tool in the tech arsenal. With practice and exploration, users can leverage Wireshark to uncover detailed information about their networks, troubleshoot complex issues, and secure their environments more effectively.

Wireshark's blend of ease of use with profound analytical depth ensures it remains a go-to solution for networking professionals across the spectrum. Its continuous development and wide-ranging applicability underscore its position as a cornerstone in the field of network analysis.

Combining tcpdump's capabilities for capturing network traffic with Wireshark's analytical prowess offers a comprehensive solution for network troubleshooting and analysis. This combination is particularly useful in environments where direct use of Wireshark is not possible or ideal. While both tools possess a steep learning curve due to their powerful and complex features, they collectively form an indispensable toolkit for network administrators, security professionals, and developers alike.

This integrated approach not only addresses the challenges of capturing and analyzing network traffic in various operational contexts but also highlights the versatility and depth of tools available for understanding and securing modern networks.

Debugging Streams with Peek

Shai Almog — Mon, 18 Mar 2024 22:00:00 +0000

Understanding Java Streams
- A Simple Stream Example
What is the peek() Method?
- Debugging with peek()
Uncovering Common Bugs with peek()
- Filtering Issues
- Large Data Sets
  - Addressing Side Effects
Limitations and Pitfalls
- Potential for Misuse in Production Code
- Performance Overhead
- Side Effects and Functional Purity
- The Right Tool for the Job
- Navigating the Pitfalls
Final Thoughts

I blogged about Java Stream debugging in the past but I skipped an important method that's worthy of a post of its own: peek. This blog post delves into the practicalities of using peek() to debug Java streams, complete with code samples and common pitfalls.

Understanding Java Streams

Java Streams represent a significant shift in how Java developers work with collections and data processing, introducing a functional approach to handling sequences of elements. Streams facilitate declarative processing of collections, enabling operations such as filter, map, reduce, and more in a fluent style. This not only makes the code more readable but also more concise compared to traditional iterative approaches.

A Simple Stream Example

To illustrate, consider the task of filtering a list of names to only include those that start with the letter "J" and then transforming each name into uppercase. Using the traditional approach, this might involve a loop and some if statements. However, with streams, this can be accomplished in a few lines:

List<String> names = Arrays.asList("John", "Jacob", "Edward", "Emily");
// Convert list to stream
List<String> filteredNames = names.stream()       
                  // Filter names that start with "J"
                  .filter(name -> name.startsWith("J"))  
                  // Convert each name to uppercase
                  .map(String::toUpperCase)              
                  // Collect results into a new list
                  .collect(Collectors.toList());         
System.out.println(filteredNames);

Output:

[JOHN, JACOB]

This example demonstrates the power of Java streams: by chaining operations together, we can achieve complex data transformations and filtering with minimal, readable code. It showcases the declarative nature of streams, where we describe what we want to achieve rather than detailing the steps to get there.

What is the `peek()` Method?

At its core, peek() is a method provided by the Stream interface, allowing developers a glance into the elements of a stream without disrupting the flow of its operations. The signature of peek() is as follows:

Stream<T> peek(Consumer<? super T> action)

It accepts a Consumer functional interface, which means it performs an action on each element of the stream without altering them. The most common use case for peek() is logging the elements of a stream to understand the state of data at various points in the stream pipeline. To understand peek lets look at a sample similar to the previous one:

List<String> collected = Stream.of("apple", "banana", "cherry")
                               .filter(s -> s.startsWith("a"))
                               .collect(Collectors.toList());
System.out.println(collected);

This code filters a list of strings, keeping only the ones that start with "a". While it's straightforward, understanding what happens during the filter operation is not visible.

Debugging with `peek()`

Now, let's incorporate peek() to gain visibility into the stream:

List<String> collected = Stream.of("apple", "banana", "cherry")
                               .peek(System.out::println) // Logs all elements
                               .filter(s -> s.startsWith("a"))
                               .peek(System.out::println) // Logs filtered elements
                               .collect(Collectors.toList());
System.out.println(collected);

By adding peek() both before and after the filter operation, we can see which elements are processed and how the filter impacts the stream. This visibility is invaluable for debugging, especially when the logic within the stream operations becomes complex.

We can't step over stream operations with the debugger, but peek() provides a glance into the code that is normally obscured from us.

Uncovering Common Bugs with `peek()`

Filtering Issues

Consider a scenario where a filter condition is not working as expected:

List<String> collected = Stream.of("apple", "banana", "cherry", "Avocado")
                               .filter(s -> s.startsWith("a"))
                               .collect(Collectors.toList());
System.out.println(collected);

Expected output might be ["apple"], but let's say we also wanted "Avocado" due to a misunderstanding of the startsWith method's behavior. Since "Avocado" is spelled with an upper case "A" this code will return false: Avocado".startsWith("a"). Using peek(), we can observe the elements that pass the filter:

List<String> debugged = Stream.of("apple", "banana", "cherry", "Avocado")
                              .peek(System.out::println)
                              .filter(s -> s.startsWith("a"))
                              .peek(System.out::println)
                              .collect(Collectors.toList());
System.out.println(debugged);

Large Data Sets

In scenarios involving large datasets, directly printing every element in the stream to the console for debugging can quickly become impractical. It can clutter the console and make it hard to spot the relevant information. Instead, we can use peek() in a more sophisticated way to selectively collect and analyze data without causing side effects that could alter the behavior of the stream.

Consider a scenario where we're processing a large dataset of transactions, and we want to debug issues related to transactions exceeding a certain threshold:

class Transaction {
    private String id;
    private double amount;

    // Constructor, getters, and setters omitted for brevity
}

List<Transaction> transactions = // Imagine a large list of transactions

// A placeholder for debugging information
List<Transaction> highValueTransactions = new ArrayList<>();

List<Transaction> processedTransactions = transactions.stream()
    // Filter transactions above a threshold
    .filter(t -> t.getAmount() > 5000) 
    .peek(t -> {
        if (t.getAmount() > 10000) {
            // Collect only high-value transactions for debugging
            highValueTransactions.add(t);
        }
     })
     .collect(Collectors.toList());

// Now, we can analyze high-value transactions separately, without overloading the console
System.out.println("High-value transactions count: " + 
       highValueTransactions.size());

In this approach, peek() is used to inspect elements within the stream conditionally. High-value transactions that meet a specific criterion (e.g., amount > 10,000) are collected into a separate list for further analysis. This technique allows for targeted debugging without printing every element to the console, thereby avoiding performance degradation and clutter.

Addressing Side Effects

While streams shouldn't have side effects. In fact, such side effects would break the stream debugger in IntelliJ which I discussed in the past. It's crucial to note that while collecting data for debugging within peek() avoids cluttering the console, it does introduce a side effect to the stream operation, which goes against the recommended use of streams. Streams are designed to be side-effect-free to ensure predictability and reliability, especially in parallel operations.

Therefore, while the above example demonstrates a practical use of peek() for debugging, it's important to use such techniques judiciously. Ideally, this debugging strategy should be temporary and removed once the debugging session is completed to maintain the integrity of the stream's functional paradigm.

Limitations and Pitfalls

While peek() is undeniably a useful tool for debugging Java streams, it comes with its own set of limitations and pitfalls that developers should be aware of. Understanding these can help avoid common traps and ensure that peek() is used effectively and appropriately.

Potential for Misuse in Production Code

One of the primary risks associated with peek() is its potential for misuse in production code. Because peek() is intended for debugging purposes, using it to alter state or perform operations that affect the outcome of the stream can lead to unpredictable behavior. This is especially true in parallel stream operations, where the order of element processing is not guaranteed. Misusing peek() in such contexts can introduce hard-to-find bugs and undermine the declarative nature of stream processing.

Performance Overhead

Another consideration is the performance impact of using peek(). While it might seem innocuous, peek() can introduce a significant overhead, particularly in large or complex streams. This is because every action within peek() is executed for each element in the stream, potentially slowing down the entire pipeline. When used excessively or with complex operations, peek() can degrade performance, making it crucial to use this method judiciously and remove any peek() calls from production code after debugging is complete.

Side Effects and Functional Purity

As highlighted in the enhanced debugging example, peek() can be used to collect data for debugging purposes, but this introduces side effects to what should ideally be a side-effect-free operation. The functional programming paradigm, which streams are a part of, emphasizes purity and immutability. Operations should not alter state outside their scope. By using peek() to modify external state (even for debugging), you're temporarily stepping away from these principles. While this can be acceptable for short-term debugging, it's important to ensure that such uses of peek() do not find their way into production code, as they can compromise the predictability and reliability of your application.

The Right Tool for the Job

Finally, it's essential to recognize that peek() is not always the right tool for every debugging scenario. In some cases, other techniques such as logging within the operations themselves, using breakpoints and inspecting variables in an IDE, or writing unit tests to assert the behavior of stream operations might be more appropriate and effective. Developers should consider peek() as one tool in a broader debugging toolkit, employing it when it makes sense and opting for other strategies when they offer a clearer or more efficient path to identifying and resolving issues.

Navigating the Pitfalls

To navigate these pitfalls effectively:

Reserve peek() strictly for temporary debugging purposes. If you have a linter as part of your CI tools it might make sense to add a rule that block code from invoking peek().
Always remove peek() calls from your code before committing it to your codebase, especially for production deployments.
Be mindful of performance implications and the potential introduction of side effects.
Consider alternative debugging techniques that might be more suited to your specific needs or the particular issue you're investigating.

By understanding and respecting these limitations and pitfalls, developers can leverage peek() to enhance their debugging practices without falling into common traps or inadvertently introducing problems into their codebases.

Final Thoughts

The peek() method offers a simple yet effective way to gain insights into Java stream operations, making it a valuable tool for debugging complex stream pipelines. By understanding how to use peek() effectively, developers can avoid common pitfalls and ensure their stream operations perform as intended. As with any powerful tool, the key is to use it wisely and in moderation.

The true value of peek() is in debugging massive data sets, these elements are very hard to analyze even with dedicated tools. By using peek() we can dig into said data set and understand the source of the issue programmatically.

Forem: Shai Almog

Front End Debugging Part 3: Networking

Network Debugging Powerhouse

Re-Issuing and Modifying Requests

cURL and Postman

Throttling and Debugging Race Conditions

Why Throttling Matters:

How to Use

Managing State with Storage Tools

Challenges of State Management

Analyzing Request and Response Headers

Debugging in Incognito Mode: Limitations and Best Practices

Connecting the Front-End to the Database

Final Word

Front End Debugging Part 2: Console.log() to the Max

Understanding Front-End Logging vs. Back-End Logging

Leveraging Console Log Levels

Customizing Console Output with CSS

Stack Tracing with console.trace()

Assertions for Design-by-Contract

Printing Tables for Clearer Data Visualization

Copying Objects to the Clipboard

Inspecting with console.dir() and dirxml()

Counting Function Calls

Organizing Logs with Groups

Chrome-Specific Debugging Features

Final Word

Front End Debugging Part 1: Not just Console Log

Instant Debugging with the debugger Keyword

Triggering Debugging from the Console

DOM Breakpoints: Monitoring DOM Changes

XHR Breakpoints: Uncovering Hidden Network Calls

Simulating Environments for Debugging

Debugging Layout and Style Issues

Final Word

The Art of Full Stack Debugging

Full Stack Development, A Shifting Definition

The Full Stack Debugging Approach

Frontend Debugging: Tools and Techniques

It isn't "Just Console.log"

The Power of Developer Tools

Tackling Code Obfuscation

Debugging Across Layers

Isolating Issues Across the Stack

The Importance of System-Level Debugging

Embracing Complexity

Conclusion

Mastering Serverless Debugging

Introduction to Serverless Computing

Challenges of Serverless Debugging

Disconnected Environments

Lack of Standardization

Limited Debugging Tools

Concurrency and Scale

Effective Strategies for Serverless Debugging

Local Debugging with IDE Remote Capabilities

Using Feature Flags for Debugging

Staged Rollouts and Canary Deployments

Comprehensive Logging

Embracing Idempotency

Debugging a Lambda Application Locally with AWS SAM

Setting Up the Local Environment

Running the Hello World Application Locally

Configuring Remote Debugging

Handling Debugger Timeouts

Final Word

Debugging Kubernetes - Troubleshooting Guide

Identifying Configuration Issues

Common Causes and Solutions

Detailed Investigation Steps

Dealing with Image Pull Errors

Troubleshooting Steps

Handling Node Issues

Preventive Measures

Managing Missing Configuration Keys or Secrets

Utilizing Buildg for Interactive Debugging

Conclusion

Why is Kubernetes Debugging so Problematic?

The Immutable Nature of Containers

The Limitations of kubectl exec

Instant Debugging with the `debugger` Keyword

The Limitations of `kubectl exec`

Using `kubectl debug`

What is the `peek()` Method?

Debugging with `peek()`

Uncovering Common Bugs with `peek()`