Forem: Em De Barros

406, 404 or 400? Investigating the Fine Line Between API Status Codes.

Em De Barros — Mon, 13 Mar 2023 15:47:34 +0000

Introduction

Hello, debuggers! I'm a junior developer who's just started my second year of engineering, and I'm currently working on building internal APIs with my team. As we all know, having a solid understanding of HTTP status codes and client-server communication is crucial when creating web services, no matter the scale. But, even experienced developers can face challenges when it comes to the nuances of these status codes and troubleshooting them.

For my first debugging writeathon, I'm excited to share a particular challenge I faced recently: investigating the fine line between the 406, 404, and 400 status codes for invalid requests. Whether you're a programming wizard or a junior like me, I hope you’ll find some helpful insights and tips in this article. So, let's dive in and explore how I troubleshooted this issue!

💡 The code examples I'll give in this article are written in C#, on top of ASP.NET Core.

The Problem

I recently was assigned the task of investigating an unexpected issue that we encountered while testing one of our GET endpoints. We discovered that, whenever the id query parameter was of the incorrect data type (in this case, not a GUID), our API was returning a 406 status code to the client. This was concerning as we were expecting the response to be 400 - Bad Request, indicating a client error due to incorrect input.

To better illustrate the problem, consider the following sample code snippet:

// implementation details omitted for brevity

[HttpGet("{id:guid}/discovery", Name = nameof (GetResource))]
public ActionResult<ResponseDto> GetResource(Guid id)
{
    return new ResponseDto();
}

So, to reiterate, when an invalid value type is passed in the path parameter of our /GetResource endpoint, the client receives a misleading error code, i.e. 406 - Not Acceptable. Based on our technical requirements, we needed the response to be 400 - Bad Request with an appropriate error message.

Can you spot that we have two problems? If not, it’s ok, I didn’t at first too! Let’s jump to the investigation to understand what’s going on.

Investigation and Diagnosis

The first step I took to investigate the issue was to review the meaning of HTTP status codes in the 400 realm. I wanted to make sure that I fully understood the difference between a 400 - Bad Request status code and a 406 - Not Acceptable status code. After refreshing my knowledge on the meaning of these HTTP status codes, I suspected that the issue might be related to the way our API routes were defined (or constrained), so I immediately turned my attention to our use of route constraints, which is a concept from the .Net framework.

After digging a little deeper, I discovered that the default .NET behaviour for this use case is to return a 404 - Not Found status code, not 406 - Not Acceptable.

Phase 1

Let’s clarify the meaning of each status code involved in the context and make sure that the 400 status code is really what we want our API to return.

I’ve formatted in bold the relevant information for our use case.

400 - Bad Request (expected)
The server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).

400 errors indicate an invalid request, which means either that mandatory parameters are missing, or that syntactically invalid parameter values have been detected (for example an expected URL being text only)

404 - Not Found (default .Net behaviour when using route constraints)
The server cannot find the requested resource. In the browser, this means the URL is not recognized. In an API, this can also mean that the endpoint is valid but the resource itself does not exist.

406 - Not Acceptable (improper API response)
This response is sent when the web server, after performing server-driven content negotiation, doesn't find any content that conforms to the criteria given by the user agent (calling service).

This status code is often bound to Accept/Content-Type Header use cases, e.g. if you set Accept-Header to text/plain for the client and the backend can only produce application/json Content-Type.

These status code descriptions come from MDN web docs, IBM API status codes reference and GitHub issue.

At this point, here’s what we know for sure:

The 406 status code is returned AFTER the server has performed content negotiation and is related to the user agent request header specifications. Currently, the server responds with a 406 status code if the id query parameter is of the incorrect data type, which is inaccurate.
It should be clear to the client whether they have a typo (or an invalid type) in their request or whether the requested resource is missing. Based on the MDN web docs description, the 404 status code can be returned when the request is valid but the resource is missing (which is how .Net treats this use case, by default). In our specific use case, however, the request would simply be invalid as the GUID data type is enforced at the controller level with validation. \Hint: Here’s our second problem, and it appears to be the underlying cause of the first problem we encountered.*
Based on the MDN documentation, the 400 Bad Request status code is indeed what should be returned for malformed request syntax.
Whether the status code would be a 406 or a 404, this is not the behaviour we want. This tells us that the issue is coming from how we validate the route parameter of our endpoint. Let's take a closer look and see how we can fix it!

Phase 2

The second phase of my investigation aimed at digging deeper into the inner workings of the framework's route constraints mechanism.

After reading more on the subject of enforcing types directly in route templates, I found this:

Don't use constraints for input validation. If constraints are used for input validation, invalid input results in a 404 Not Found response. Invalid input should produce a 400 Bad Request with an appropriate error message. Route constraints are used to disambiguate similar routes, not to validate the inputs for a particular route.
Route constraints generally inspect the route value associated via the route template and make a true or false decision about whether the value is acceptable.
Source: official Microsoft documentation.

This validated my suspicion that our route parameter validation implementation was incorrect. Route constraints should only be used to differentiate between similar action methods/paths and NOT for input validation.

[HttpGet("{id:guid}/discovery", Name = nameof (GetResource))]

Examples of similar routes that need route constraints for the routing middleware to map the request correctly:

[HttpGet("{bookId}")
public ActionResult GetBookDetails(int bookId) {}

[HttpGet("{bookTitle}")
public ActionResult GetBookDetails(string bookTitle) {}

In this sample code, the routing middleware gets confused and doesn't know what action method to invoke. The solution is to add route constraints:

[HttpGet("{bookId:int}")
public ActionResult GetBookDetails(int bookId) {}

[HttpGet("{bookTitle:string}")
public ActionResult GetBookDetails(string bookTitle) {}

Basic type validation is performed in the background by .Net, by validating the type against the type declaration from the method parameter definition, like so:

public ActionResult<ResponseDto> GetResource(Guid id){}

❗️ Note that this is only basic type validation. In this specific case, it does not check for empty (default) GUID value. Further validation should be performed at the DTO level with custom attributes.

Solution

With these clarifications out of the way, my first step toward fixing the problem was simply to remove the GUID route constraint. This action immediately resolved the issue.

[HttpGet("{id}/discovery", Name = nameof (GetResource))]
public ActionResult<ResponseDto> GetResource(Guid id)
{
    return new ResponseDto();
}

With this implementation, a 400 - Bad Request status code is now properly sent back to the client in the event of an incorrect type being passed in the id path parameter.

💡This endpoint was tested with thorough unit test cases.
However, at this point, one issue remains: If we wanted to properly use route constraints to differentiate similar action methods, the client would still receive a 406 - Not Acceptable response if no match is found for a route. As we saw previously, the proper response for this use case should be the default .Net Core response of sending back a 404 - Not Found status code.

Based on our technical requirements and priorities, I didn't get to continue the investigations as this next issue was out of scope. However, to ensure that it doesn't go unnoticed in the future, I took a proactive approach and added it to our technical debt list. While I did not have a definite answer on the root cause of the issue, I made an educated guess and documented it for future reference. It's crucial to understand that an API's behaviour is shaped by various factors, including the implementation and design choices of the project's dependencies, and there may be valid reasons in this case for overriding the default .Net behaviour and sending back a 406 status code. It was important to me to make sure that we were all aware of this potential issue and have it documented so that we can address it together as a team if it comes up in the future.

Lessons Learned

During this debugging experience, I encountered several roadblocks and learned many valuable lessons along the way.

One of the most valuable lessons I learned was to not make assumptions and thoroughly investigate the problem before jumping to conclusions. This can also mean always questioning what you know first! In the case of the 406/400 issue, it would have been easy to assume that the bug was related to a typo or some other external factor, without taking the time to fully investigate the issue. So, even though I deal with status codes every day, my first thought was to review their meaning anyway to make sure I was not missing anything important in the definition subtleties. With this approach, I was able to discover the root cause of the issue and avoid wasting time and resources on unnecessary debugging or troubleshooting. This, I think, is an important lesson for any developer to learn, as it can help prevent a lot of frustrating and wasted effort down the road.

In the same train of thought, I learned when to ask for help and seek out resources when faced with so many possibilities to explore. It's easy to go down rabbit holes because, more often than not, one problem will uncover many other underlying problems you didn't even know existed in the first place. Presenting to your teammates your possible avenues of investigation and asking for some guidance early on and frequently is a great skill to leverage.

Finally, this debugging experience has taught me the importance of recognizing when to stop and asking myself “Should I continue down this path?” As you progress in your career, you'll realize that the most effective programmers don't always have a one-size-fits-all magical answer for when to stop investigating a problem. Instead, they understand that the right moment to stop will vary depending on the situation. The key skill to develop here is the ability to question often whether it's worth investing more time and resources in something, based on the business and technical requirements. As demonstrated in the solution section, although I was unable to resolve every issue, I made a judgment call on when to stop and move on to other tasks.

Conclusion

Every debugging challenge I face is an opportunity for me to improve my problem-solving skills and shape my problem-solving framework. Over time, this framework becomes more refined and easier to apply when tackling new problems. By repeatedly going through the steps of gathering information, being methodical and always questioning what I know first (even for seemingly simple issues), discussing the possible avenues of investigation with my team, and questioning often whether it's time to stop investigating, I have developed a reusable framework of important habits and steps to take when faced with a coding problem. Not only this framework helps me to solve problems more efficiently, but it also allows me to approach new problems with greater confidence.

To sum up, debugging skills are an essential part of software development, and it's important to continue sharpening these skills to become a better problem-solver. I encourage all readers to share their own debugging stories and engage with the community to learn new strategies and approaches to solving coding problems. Every challenge can be an opportunity for growth and development, so just remember to always be curious, stay focused, and never give up when faced with a problem.

Happy debugging!

Conquer Your Fear of Documenting in 9 Simple Steps

Em De Barros — Sun, 13 Mar 2022 06:00:10 +0000

Conquer Your Fear of Documenting in 9 Simple Steps

A team without strong communication skills is like a car without wheels. You won't get far, if at all, anywhere. You can walk, sure, but needless to say that it will take you longer to get to your destination.

Accordingly, it is crucial for any organization to make sure that there are set and clear communication guidelines, platforms, and mediums to help teams achieve their goals faster and more efficiently. Even more so now that the sanitary crisis has completely changed the way we communicate by forcing countless organizations across the globe to work remotely.

While many platforms like Zoom, Teams, and Slack ensure great verbal communication, it is important to not underestimate the power of written communication within your organization. Sure, this can be ensured with instant messages and e-mails, but there is an underestimated superpower that many teams overlook and that is documenting your work.

Documenting your work does much more than just helping your team getting $#!t done, and that's what we'll explore in this post.

Why Should You Even Care?

It is no secret that one of the most important and efficient strategy for getting through a technical interview is to THINK OUT LOUD.

You need to share your thought process out loud with your interviewer/recruiter when trying to solve a given problem. Doing so, more often than not, is more significant than solving the actual problem, as it lets your recruiter evaluate your communication skills and your abilities to collaborate.

Unfortunately, this is not your average college exam where you toss in the trash all your efforts after you hand in your paper to the teacher, and leave the classroom to go meet your buddies for a night out. If you passed the interview, that's great. Now, it's time to keep that job. Therefore, thinking out loud shouldn't be something that you leave behind after the interview. You should, unquestionably, continue to leverage and perfect that skill throughout your entire career. This is the skill that got you the job, after all.

Does That Mean More Meetings?!

Hell, no.

But it does mean more writing. Ugh, I know, yet another thing to add to your team's to-do list. However, it offers loads of advantages that make it well worth the time and effort, trust me.

As you might have realized already, developing a whole service, a component or just a simple feature are all results of team members working together. In fact, one of the main purposes for PR's (Pull Requests) is to set the stage for open communication within a team. PRs not only prevent broken or bad code from getting into production environments, but they also help us improve our design and problem-solving skills, expose us to different point of views, and can even generate new ideas.

Just like PR's, documenting your work encourages open communication between team members; it is the continuity of thinking out loud. That's why documenting your work goes way beyond just a simple formality. I like to think of this process as the extension of a PR.

Communication and collaboration should be at the core of your team's values and so, by documenting your work, you are contributing to maintaining a strong and healthy team structure. This process also prevents your team to enter what I like to call the unintentional information hoarding hell, scary I know.

As the author of the Documentation Best Practices article best put it: "Your mission as a Code Health-conscious engineer is to write for humans first, computers second. Documentation is an important part of this skill". Your documentation is the story of your code. It's an open book for your thought process, where your team members and yourself can find helpful resources related to your project and what you are currently working on.

What Are the Benefits?

Documenting your work has two categories of benefits. Those two categories are the immediate benefits and the long-term benefits.

Immediate benefits:

Immediate benefits are benefits that you and your team immediately benefit from when documenting your work, duh. Here are a few examples:

It sets the stage for open communication on every aspect of the building process, as you're letting everyone in on the ideas that lead to architectural and design decisions.
It helps you and your teammates understand the scope of your project better as you get a more visual and concrete representation of your thought process.
It helps your team members rapidly understand what you're working on; therefore better contribute to the PR reviews.
Furthermore, it helps prevent future bugs down the road as writing down your ideas is helpful for spotting bottlenecks, weak architectural design, overkills, and bloated workflows in the early stages of the development process.

Long-Term benefits:

Long-term benefits are benefits that you, your team and other colleagues will continue to benefit from, even long after a project is finished. Here are a few examples:

If other team members need to step in on something, they are able to contribute to the project sooner and more easily, as they won't have to spend hours tracking down details and directions to start coding.
The latter applies to yourself as well. If you go back to an old project of yours, you'll be able to pick up where you left from sooner and more confidently.
It adds value and quality to your work by mirroring all the engagement behind every aspect of the building process and by educating the readers.
As team members come and go, it's crucial that knowledge is not kept to one employee only, but rather shared within the team. It facilitates change inside the company, makes hiring and onboarding much easier, and prevents information hoarding.
It creates consistency by ensuring that the same information, processes, and plans are being consumed by everyone in the team.
Good documentation not only mirrors your thought process, but also collects all the necessary information about a task and/or a project in a centralized and organized place. This ensures that your teammates don't go down a rabbit hole trying to understand what you are doing, therefore cuts down on duplicative work. You already did the hard work, save them the googling.

💡 Did you know that the average worker spends about two and a half hours per day searching for the information they need?

Tips & Tricks

Hopefully, by now, I've convinced you that documenting your work should be an integral part of your daily workflow. In this section, I'll discuss some tips & tricks on how to get started. If documenting your work is already something you're familiar with, perhaps you'll still pickup something helpful. Keep reading!

Just like creating a mockup and a wireframe when you're designing a website, you should start documenting your work at the very beginning of your project. You can even document the thought process that led to the project idea in the first place! There are many ways to go about this and, as you get used to documenting your work regularly, you'll develop your own strategies. However, to make the most of your documentation efforts, there are (you guessed it), a few common guidelines that you can follow to structure your documentation.

Follow this structure: Intro (Opening Remarks, Build-Up) → Conclusion (Design architecture and decisions) → Thought Process (Explanation, Research) → Extra (User Cases, Resources)
If the documentation is long, add a table of content at the beginning of your documentation
Say what you mean simply and directly.
Keep it short and useful. As written by the author of the Documentation Best Practices article: "Docs work best when they are alive but frequently trimmed, like a bonsai tree".
Get into the habit of updating your documentation frequently
To avoid scope creep, try to keep your documentation linked to a single User Story
Leverage the tools provided by your organization's writing platform to add visual resources, links, etc.
Get into the habit of reviewing other documentation and contributing to them as well; this will help you get better at writing!
Set an environment in your team's wiki platform for each project, and keep every doc under one main project directory. Break down that structure as necessary.

Don't worry about your documentation being perfect. News flash: docs are never perfect - they are messy and should be. The purpose of keeping a project wiki is to jot down your thought process, set the stage for open communication within your team, and encourage the process of building in public.
Focus on keeping it clean and practical. You'll get better with time.

During my internship, I had the chance to collaborate with the former system architect of my team, who generously and gladly shared with me all those tips & tricks to build strong and useful project documentations. These simple guidelines really helped me grow as a developer, and I hope that you can now benefit from them as well.

What Next?

As projects grow, wikis grow. Sometimes, it's good to revisit old documentation and clean up a little. This habit ensures a good and healthy wiki environment for everyone in the team. A good way to go about this is to start by deleting what you are certain is wrong and, then, including your team members in the process.

Conclusion

As stated in the beginning of this article, documenting your work goes way beyond just a simple formality. Prioritizing documentation within your organization means you and your team will build strong resources that you'll be able to rely on.

From covering the unexpected absence of a team member to tackling an unfamiliar project, you and your team will be able to overcome these challenges more easily and confidently. The process of documenting your work will increase transparency within your team and encourage a more collaborative and strategic culture. You'll all be able to make smarter decisions as information won't be locked away in one person's head only, therefore avoiding the unintentional information hoarding hell.

I really hope this post was useful for you and that you learned something. I encourage you to share the above advantages to your team members and find ways to incorporate this process in your day-to-day tasks.

Thank you for reading!

P.S.: To thank you for reading this far, I've provided you with a free template to get your feet wet with documenting your work. Head over to the page below 👇🏻
Project Wiki Template
💡 Follow me on Medium, Twitter, and LinkedIn!