Understanding HTTP Request Headers: Classification, Behavior, and How Servers React

Demystifying HTTP for Web Developers — Part 3

HTTP Headers: The Real Drivers of Behavior

In Part 2 of this series, we dissected the structure of an HTTP request — from the request line to headers and message body. We saw how a minimal request could be valid, but real-world HTTP traffic rarely resembles those minimal forms. In practice, requests are filled with headers that instruct servers, inform intermediaries, and enable negotiation.

Request headers are not mere accessories. They are the instruments through which clients express identity, context, preference, and security credentials. They control everything from whether a server compresses a response to whether the request is authorized. Without headers, HTTP cannot fulfill its role as a stateless, flexible, and interoperable protocol.

In this article, we shift our focus from structure to semantics. We’ll analyze headers from three angles:

• RFC-defined syntax and classification
• Real client behavior using Postman
• Real server behavior using ASP.NET Core MVC

The goal is not just to understand what headers exist — but to see how they influence behavior and how a modern web framework processes them in real time.

What Are HTTP Headers?

HTTP headers are textual metadata fields included in request and response messages. Each header is a key-value pair, defined in the format:

field-name: token

This structure is formally defined in RFC 9110 §5.1. Field names are case-insensitive, and whitespace around the colon is ignored. Values can be simple strings or structured lists, depending on the header.

Header field semantics are defined individually across the specification, depending on their purpose. There is no dedicated section in RFC 9110 for general request header semantics; instead, each header is defined in context. For example:

• Accept
• Accept-Encoding
• Authorization
• User-Agent
• Content-Type
• Content-Length

Headers are placed between the request line and the optional body. They are terminated by a blank line (CRLF) that signals the start of the body. In ASP.NET Core, headers are accessible via the Request.Headers dictionary, which automatically handles normalization and case insensitivity.

Here is an example of a request as seen in Postman’s “Console” tab:

GET /api/home HTTP/1.1
User-Agent: PostmanRuntime/7.43.0
Accept: */*
Postman-Token: 865051eb-6af4-4900-b82c-a96172193517
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive

ASP.NET Core allows you to inspect them directly:

[HttpGet]
[Route("Inspect")
public IActionResult Inspect()
{
    var userAgent = Request.Headers["User-Agent"].ToString();
    var accept = Request.Headers["Accept"].ToString();
    var correlationId = Request.Headers["X-Correlation-ID"].FirstOrDefault();

    return Ok(new
    {
        UserAgent = userAgent,
        Accept = accept,
        CorrelationId = correlationId
    });
}

Header Classification and Behavior

To reason about headers systematically, it’s useful to classify them based on their function and scope.

1. General Headers

These apply to both requests and responses, and often relate to message handling rather than payload semantics.

• Cache-Control: controls caching behavior
• Connection: indicates connection persistence (e.g., keep-alive)

2. Request Headers

Used only in client-to-server communication. These inform the server about the client’s capabilities, preferences, or context.

• Accept: indicates preferred response formats
• Accept-Encoding: lists supported compression algorithms
• User-Agent: identifies client software
• Authorization: supplies credentials (e.g., Bearer tokens)
• Referer, Origin: supply context for cross-origin requests

3. Entity Headers

These describe the content of the message body.

• Content-Type: media type of the payload
• Content-Length: size in bytes
• Content-Encoding: compression applied to the body

4. Custom Headers

These are not defined by the core specification but widely used in real-world applications.

• X-Correlation-ID: used to trace requests across distributed systems
• X-Requested-With: often used in AJAX requests to differentiate browser types

Observing Header Behavior in ASP.NET Core MVC

Let’s now see how headers affect behavior using Postman and how ASP.NET Core interprets them.

Accept Header and Format Negotiation

Scenario: A client sends Accept: application/json expecting a JSON response.

Postman Request:

GET /api/home HTTP/1.1
Accept: application/json
User-Agent: PostmanRuntime/7.43.0
Postman-Token: 012874d8-ddcd-4f65-9e36-6abd3f97a722
Host: localhost:5000

ASP.NET Core Behavior: By default, ASP.NET Core uses registered output formatters to serialize the response. JSON is the default. If you request text/html, ASP.NET Core may still return JSON unless strict negotiation is enabled.

Without strict negotiation:

With strict negotation:

builder.Services.AddControllers(options =>
{
    options.ReturnHttpNotAcceptable = true;
});

With this option enabled, unsupported formats will result in:

406 Not Acceptable

Content-Type and Model Binding

Scenario: A POST request includes JSON data but does not specify Content-Type.

Postman Body: Raw → JSON

{
    "email": "user@example.com",
    "password": "123"
}

Missing Header: Content-Type: application/json

ASP.NET Core Behavior: Model binding fails. The action may receive a null model or return:

415 Unsupported Media Type

Model binding depends on Content-Type matching a registered input formatter.

Authorization Header

Scenario: A client sends an access token using Authorization: Bearer .

Postman Setup: Use the “Authorization” tab → Bearer Token.

ASP.NET Core Behavior: If JWT authentication is configured, [Authorize] attributes will automatically enforce security. No manual parsing of headers is required.

[Authorize]
[HttpGet("secure")]
public IActionResult SecureEndpoint() => Ok("Access granted.");

Invalid or missing tokens return:

401 Unauthorized

Accept-Encoding and Compression

Scenario: The client requests compressed content.

Header: Accept-Encoding: gzip

ASP.NET Core Setup:

services.AddResponseCompression();
app.UseResponseCompression();

ASP.NET Core automatically compresses responses if the client supports it. You’ll see Content-Encoding: gzip in the response headers and reduced payload size.

X-Correlation-ID and Middleware Handling

Scenario: You want to enforce that a custom header is present on all requests.

Middleware Implementation:

public class CorrelationIdMiddleware
{
    public async Task Invoke(HttpContext context, RequestDelegate next)
    {
        if (!context.Request.Headers.ContainsKey("X-Correlation-ID"))
        {
            context.Response.StatusCode = 400;
            await context.Response.WriteAsync("Missing X-Correlation-ID");
            return;
        }

        await next(context);
    }
}

public class CorrelationIdMiddleware
{
    public async Task Invoke(HttpContext context, RequestDelegate next)
    {
        if (!context.Request.Headers.ContainsKey("X-Correlation-ID"))
        {
            context.Response.StatusCode = 400;
            await context.Response.WriteAsync("Missing X-Correlation-ID");
            return;
        }

        await next(context);
    }
}

This pattern is commonly used for enforcing traceability in distributed environments.

Common Pitfalls and Clarifications

• Accept is not enforced by default. ASP.NET Core responds with the best available formatter unless explicitly configured.
• Content-Type must match the request body. Otherwise, model binding fails silently or throws a 415.
• Custom headers are not validated. You must implement your own enforcement logic.
• Some headers can be duplicated (e.g., Accept). Others cannot (e.g., Content-Length). Duplicate Content-Length can result in request rejection.
• Case insensitivity is handled for you. Request.Headers["accept"] and Request.Headers["Accept"] return the same result.

Conclusion

HTTP request headers are not ornamental — they are essential. They determine how content is negotiated, how credentials are passed, how bodies are interpreted, and how APIs are secured. In ASP.NET Core, headers are not only accessible but also actionable through middleware, model binding, and authentication frameworks.

By learning how to inspect and react to headers with precision, you move from merely calling endpoints to truly mastering the protocol beneath your applications.

In Part 4, we’ll shift to the body of the request — exploring payloads, semantics, media types, and how to handle them securely and effectively in modern APIs.