Forem: Heitor Tashiro Sergent

How to Debug Common API Errors

Heitor Tashiro Sergent — Fri, 01 Mar 2019 00:33:39 +0000

Introduction

Debugging and troubleshooting APIs is something that any developer that works with APIs has to go through at some point. In an ideal world, APIs would always return a 200 OK with just the right data that we need, or in case of a failure, it would return us the perfect status code and error message allowing us to easily understand what went wrong.

In reality, APIs don't always work like that. API developers might have constraints that do not allow them to implement the most informative status code or error message, API errors can be triggered by real-world conditions that are hard to account or test for, and sometimes ourselves, as API users, can make requests with typos or mistakes that APIs just don't know how to handle.

In this post, we're going to focus on API users and what they can do to debug common API errors they might encounter when testing and working with APIs, whether these APIs are public or private.

Status Codes

When working with APIs, every HTTP request that is sent to the server will be responded to with an HTTP status code. HTTP status codes can be grouped into five different categories, by looking at their first digit:

1xx - Informational
2xx - Success
3xx - Redirection
4xx - Client Error
5xx - Server Error

We're going to be focusing on the 4xx client error codes in this post, as they are the more common errors we might see when working with APIs, and which can usually be solved by the API user itself. We'll also talk a little bit about the 5xx error category at the end.

General Tips

Before we dive into the specifics of a few HTTP status codes, there are certain tips that we can apply to any API debugging attempt.

Use an API Testing Tool

Use a tool that allows you to make, edit, replay, and inspect API calls. There are quite a few that I can recommend, such as Runscope itself, Postman, Paw, Insomnia, cURL, HTTPie. Most of these tools are free or have trials, and they can save you a ton of time when debugging errors.

(Runscope example)

(curl example)

Another advantage of using a tool that allows you to make and replay API requests is they can help you isolate an issue due to a bad API call, or something in your app environment (a configuration issue, firewall, etc.) preventing the API call from returning a 200 status code.

I remember when I first started developing mobile apps, I would sometimes run a Windows Phone app in debug mode and go through several steps to get to an API call I was trying to debug. Don't be like me.

Read the Docs

I can't stress this enough, and not only because I enjoy writing documentation, but please read the docs.

I have been guilty of not following this advice as well but, reading the docs can be the difference between being stuck with an issue for 1 hour, or making a successful API call and moving on to your next task.

API documentation, when thorough and well-written, will give you all the answers you might need to successfully work with an API. How to authenticate, how to get an access token, what methods and endpoints are supported, and even what errors you might encounter and how to fix them.

Be Careful with Copy+Pasting

Copy and pasting code snippets from API documentation, or StackOverflow answers, is really easy, but be careful with those.

Examples can be outdated and contain general errors or use an old API version that's not supported anymore. Copy+pasting code has an even trickier issue where certain characters might completely change how your API call works.

I had an error once where I copy+pasted a piece of text as a query parameter to an API request that should have included a ' symbol. Except that the text editor I was using converted the ' symbol to a ’ symbol (it might be hard to see, but they are different). When I pasted the text to my API call, that caused it to keep returning a 400 error, and it probably took me 30 minutes to debug.

Now, let's get to the less general advice, and dig in into specific status codes!

400 Bad Request

A 400 status code means that the server could not process an API request due to invalid syntax. A few possibilities why this might happen are:

A typo or mistake while building out the request manually, such as mistyping the API endpoint, a header name or value, or a query parameter. This can also be caused if the request is missing a required query parameter, or the query parameter value is invalid or missing
A malformed JSON body, for example, missing a set of double-quotes, or a comma. If you need to make an API request with a JSON body, I highly recommend writing it using a linter, whether you use a web editor or code editor plugin
The request is missing authentication information, or the Authorization header provided could not be validated

The main advice when debugging a 400 Bad Request error is to review every piece of text. Make sure there are no typos in the endpoint, headers (name and values), and body. If you copied and pasted any part of your API request, pay extra attention that they don't include any mistakes or random characters that could be causing an issue.

Our last bullet point above (missing authentication information) is a good example of something you might see when working with multiple APIs. One API might send you a 400 status code for a request that has invalid authentication credentials, while another one might send you a 401 unauthorized status code, which can be used specifically for that purpose.

The 400 bad request status code sometimes is used as a catch-all for multiple types of errors. So if you have checked your API request for any invalid syntax and haven't found any errors, try to see if any of the other 4xx status code solutions could help you with debugging.

401 Unauthorized

The 401 Unauthorized status code is returned when the API request is missing authentication credentials or the credentials provided are invalid.

APIs can be fickle, and that's especially true when creating and formatting the Authorization header. OAuth 2 tokens, for example, need to be prepended with "Bearer" for them to work:

Authorization: Bearer your_api_token

It's also important when using HTTP Basic authentication to pay close attention to the syntax of the header value. The form is as follows:

Authorization: Basic base64_encode(username:password)

Another thing to be mindful of, for cases where you only have a username, or only a password, make sure to include the colon (":") before encoding. Otherwise, the authentication will fail.

403 Forbidden

The 403 Forbidden status code looks similar to the 401 code, but they shouldn't be confused with each other.

Usually, when you see the 403 status code, the API request that is being made includes some form of authorization. But, different from the 401 status code, the server does recognize the authorization credentials and accepts is as valid. The issue this time is that the authenticated user cannot access the resource for that endpoint. For example, the user might be trying to access account-level information that's only viewable by the account administrator, and the credentials being passed on the API request are for a regular user account.

Another reason this status code might be returned is in case the user did not request an API access token with the correct permissions.

To fix the API call for those two situations, make sure that the credentials you are using have the access-level required by the endpoint, or that the access token has the correct permissions.

A less common reason we might see this error is if we're not explicit about the Accept header value. Some APIs require you to include those headers on requests. That way, the server knows what information the client is sending, and also what format they expect to receive in return.

404 Not Found

404 Not Found one of those status codes that we don't have to be working with APIs to see. But, specifically for APIs, it can usually mean a few different things:

The endpoint does not exist
The endpoint exists, but the resource cannot be found
The endpoint exists and the resource can be found, but the user does not have the required permissions to access it, and so the server returns a 404 status code

For the first two cases, there's not much we can do as an API user, except double-check that the endpoint really does exist in the documentation, or double-check that there are no misspellings or typos.

For the third case, the advice is similar to what we covered in the previous status code, 403. Make sure that the authorization credentials you are using actually have access to that endpoint, as some APIs might return a 404 error instead of 403.

429 Too Many Requests

The 429 Too Many Requests is one of the longer ones, and it's also self-explanatory. We're calling the API too often.

It's common for public and 3rd-party APIs to include some form of rate-limiting. That is a way to limit how many requests the user can make over a period of time. This serves to protect the API provider from having a user of making too many API calls, which can take up too many resources and potentially cause API slowdowns or even crashes for all users.

If you run into this error, there are several things that you can try to fix it:

A Retry-After header might be included, indicating the Date or time in seconds when the user can retry the request
Check the API documentation for the specifics of their rate-limiting. Sometimes, users might be able to give additional forms of authentication, upgrade their accounts, or reach out to the API support team to increase the rate limits
Caching API responses is another strategy that can help mitigate this error. If you're working with an API and you don't rely on always up-to-date information, caching the response every x amount of seconds/minutes/hours/days and displaying the saved API response can you help you avoid limits or surcharges

5xx Error Codes

5xx status codes generally mean that the server has encountered an error, and can't resolve the API request being made. Some of the more common status codes you might encounter are:

500 Internal Server Error
502 Bad Gateway
503 Service Unavailable
504 Gateway Timeout

A 5xx status usually means that the error is on the server side, not on the user side of things. So the general advice is to:

Try again later. The server might be under too much load at the moment
Check the API status page, if one is available, for more information if there's maintenance going on, or the API is down

Conclusion

We've covered just a few of the most common HTTP status codes you might encounter when debugging or troubleshooting APIs. Hopefully, with this information, you will feel more comfortable when encountering those issues and know about the tips and tricks you can use to find the solution.

If you have any other issues you would like to see added here, or any additional tips that you might have for fixing API errors, we'd love to hear from you!

6 Common API Errors

Heitor Tashiro Sergent — Mon, 06 Feb 2017 17:59:49 +0000

Have you ever used an API that returned an HTML error page instead of the JSON you expected, causing your code to blow up? What about receiving a 200 OK status code with a cryptic error message in your response?

Building an API can be as quick as serving fast food. Frameworks like Express, Flask, and Sinatra combined with Heroku or zeit's now help any developer have an API up and running in a few minutes.

However, building a truly secure, sturdy, hearty API, can take a little more work, just as a chef takes more time when crafting a great meal. You need great docs, clear and concise error messages, and to meet developers' expectations of how your API should work.

On the other side of the table, we have developers interacting with these APIs. And we, as developers, sometimes make mistakes. We can make false assumptions about how an endpoint should work, not read the docs closely enough, or just not have enough coffee that morning to parse an error message. That's where Runscope comes in.

Our testing and monitoring tools can help you uncover issues that would otherwise stay hidden by a lack of integration tests, or real-world use case scenarios. Working with thousands of developers to resolve their API problems has given us unique insight into issues they often see when integrating and interacting with APIs.

Here's our list of 6 common mistakes that can catch you off guard, why they happen, and how you can avoid them:

1. Using http:// instead of https://

Forgetting a single "s" can get you in a lot of trouble when testing an API. Some APIs may only support HTTPS, while others may support HTTP for some endpoints and not others.

Even when an API supports both, you might still run into some errors. For example, some APIs redirect HTTP traffic to their HTTPS counterpart, but not all frameworks will be configured to follow a 302 status code. Node.js request module, for example, will follow GET redirects by default, but you have to explicitly set followAllRedirects to true if you want to follow redirects to POST and other methods.

APIs may also stop supporting HTTP, so it's important to stay up-to-date with any changes. Good API providers will let users know beforehand via email and any social media channels they have. Another step you can take is to use a tool like Hitch, which lets you follow certain APIs and be notified if anything changes.

If you're asking yourself if your API should support HTTPS, then the answer is yes. The process for getting certificates used to be a hassle, but with solutions like Let's Encrypt and Cloudflare, there's no excuse to not support HTTPS. If you're unsure why you should do it, or don't think you should because you're not transmitting any sensitive data, I highly recommend reading "Why HTTPS for Everything?" from CIO.gov.

2. Unexpected error codes

A good API error message will allow developers to quickly find why, and how, they can fix a failed call. A bad API error message will cause an increase in blood pressure, along with a high number of support tickets and wasted time.

I ran into this issue a couple of weeks ago while trying to retrieve an API's access token. The code grant flow would return an error message saying that my request was invalid, but it wouldn't give me any more details. After an hour banging my head against the wall, I realized I hadn't paid attention to the docs and forgot to include an Authorization header with a base64 encoded string of my application's client_id and client_secret.

Good usage of HTTP status code and clear error messages may not be sexy, but it can be the difference between a developer evangelizing your API and an angry tweet.

Steve Marx had this to say in "How many HTTP status codes should your API use?": "...developers will have an easier time learning and understanding an API if it follows the same conventions as other APIs they're familiar with." As an API provider, you don't have to implement 70+ different status codes. Another great advice by Steve is:

"Following this pragmatic approach, APIs should probably use at least 3 status codes (e.g. 200, 400, 500) and should augment with status codes that have specific, actionable meaning across multiple APIs. Beyond that, keep your particular developer audience in mind and try to meet their expectations."

Twilio is a great example of best practices for status code and error messages. They go the extra mile and include links in their responses, so the error message is concise while still providing the developer with more information in case they need it.

{
  "code": 21211,
  "message": "The 'To' number 5551234567 is not a valid phone number.",
  "more_info": "https://www.twilio.com/docs/errors/21211",
  "status": 400
}

As API consumers, we need to be careful and not assume that an API 200 status code means the request made a successful call and returned the information we want. Some APIs, like Facebook's Graph API, always return a 200 status code, with the error being included in the response data. So, when testing and monitoring APIs, always be careful and don't automatically assume that a 200 means everything is ok™.

Another great resource about response handling is Mike Stowe's blog post on "API Best Practices: Response Handling."

3. Using the incorrect method

This is an easy one, but surprisingly common. A lot of times this can be blamed on poor documentation. Maybe the endpoints do not explicitly say what methods are supported between GET/POST/PUT etc., or they have the wrong verb.

Tools can also play tricks on you if you're not careful. For example, let's say you want to make a GET request with a request-body (not a great practice, but it happens). If you make a curl request using the -d option, and don't use the -XGET flag, it will automatically default to POST and include the Content-Type: application/x-www-form-urlencoded header.

This post by Daniel Stenberg (author and maintainer of curl) on "Unnecessary use of curl -X" also illustrates another possibility you might run into this issue when dealing with redirects:

"One of most obvious problems is that if you also tell curl to follow HTTP redirects (using -L or –location), the -X option will also be used on the redirected-to requests which may not at all be what the server asks for and the user expected."

Other times, we might fall into past assumptions and just use the wrong method. For example, the Runscope API uses POST when creating new resources, such as test steps or environments, and PUT when modifying them. But Stripe's API uses POST methods when creating and updating objects.

Both approaches are valid, and Stormpath has a great blog post talking about their differences, and how to handle them as an API provider. No matter which one you choose, just be consistent throughout your API and make sure to have correct and up-to-date docs, so your users don't run into this error.

4. Sending invalid authorization credentials

APIs that implement OAuth 2, such as PayPal, usually require the developer to include an Authorization header for each request. It's common to confuse that with Authentication instead (I did exactly that while making the GIFs for our last blog post.), so if your request is failing, make sure you're using the correct word.

Another issue that pops up with Authorization headers is actually constructing it correctly. OAuth 2 tokens need to be prepended with "Bearer" for them to work:

Authorization: Bearer your_api_token

It's also important when using HTTP Basic authentication to pay close attention to the syntax of the header value. The form is as follows:

Authorization: Basic base64_encode(username:password)

Common mistakes include forgetting the 'Basic ' (note the space) prefix, not encoding the username and password or forgetting the colon between them. If an API provider only requires a username without a password (like Stripe, where your API key is the username), you'll need that pesky colon after the username, even if there's no password.

5. Not specifying Content-Type or Accept header

Accept and Content-Type headers negotiate the type of information that will be sent or received between a client and server. Some APIs will accept requests that don't contain any of those headers, and just default to a common format like JSON or XML.

Other APIs are a little more strict. Some might return a 403 error if you're not explicit about the Accept header value, and require you to include those headers on requests. That way, the server knows what information the client is sending, and also what format they expect to receive in return.

This issue can also be cause some confusion if you are testing your API with different tools. curl, for example, along with other popular testing tools, will automatically include an Accept header for any MIME type: */* with every request. We, at Runscope, don't add a default Accept header, so this can get you different results when testing the same endpoint.

6. APIs returning invalid content type when there is an error

I can say that this is one of my pet peeves with APIs. Seeing that <!DOCTYPE HTML> line in a response makes my blood pressure go sky high.

Well, sometimes that's my fault. If you forget to send an Accept header with your request, the API can't be sure what response format you're expecting.

For API providers, some frameworks and web servers default to HTML. For example, Symfony, a PHP framework, defaults to returning a 500 HTML error. So if you're creating an API that has no business returning HTML, make sure to check the defaults error response.

Another reason this might happen may not have to do with your API, but with the routing mesh or load balancer that sits in front of your API. For example, if you have a nginx instance fronting your API and it encounters a request timeout or other error, it may return an HTML error before your API instances even have a chance to know what's going on.

Conclusion

These are some of the most common mistakes we have seen across multiple APIs. This list could go on for much longer, so if there's some other error you came across and managed to fix it, please share it with us @Runscope.

As API providers and API consumers, these mistakes can sometimes go unnoticed and waste thousands of debugging hours. To get more visibility into your APIs and avoid getting stuck in bad redirects or unexpected error codes, check out our API testing and monitoring tools with a free trial account.