Forem: Neil Madden

XSS doesn’t have to be game over

Neil Madden — Thu, 10 Dec 2020 13:01:46 +0000

A message I’m very used to seeing – but does XSS have to mean game over for web security?

There’s a persistent belief among web security people that cross-site scripting (XSS) is a “game over” event for defence: there is no effective way to recover if an attacker can inject code into your site. Brian Campbell refers to this as “XSS Nihilism”, which is a great description. But is this bleak assessment actually true? For the most part yes, but in this post I want to talk about a faint glimmer on the horizon that might just be a ray of sunshine after all.

Why is XSS so catastrophic anyway?

A naïve view of the dangers of XSS is that the attacker primarily wants to steal your authentication tokens or cookies so that they can use them from their own machine to perform malicious actions in their own time. This used to be a very common attack pattern, but it has a lot of drawbacks for the attacker:

It is easily defeated by simple measures such as using HttpOnly cookies, which stop the attacker’s script being able to steal your session cookie in the first place.
If the web app in question is only accessible from a corporate network or VPN, then the attacker won’t be able to connect to it from their own machine even if they have your cookie.
By using their own machine (or a compromised machine they have access to already) they make it easier to detect the attack and block their access. There will be clues given away by the change of IP address, geo-location, browser version, and so on. This is by no means guaranteed, but it certainly increases the risk of the attacker being spotted.

Attackers are well aware of these issues and have developed a solution to all of them. Rather than stealing your session cookie and using it from their own machine, they will instead use the XSS attack to proxy their requests through your web browser. This is similar to how a Cross-Site Request Forgery (CSRF) attack occurs, but with XSS the requests come from the same site (same origin) as legitimate requests and so almost all CSRF defences can be bypassed, as shown in the image below from chapter 5 of my book. SameSite cookies do not protect against this attack, and nor do typical anti-CSRF tokens because the attacker’s script running in the same origin as the legitimate code can extract these from the DOM or local storage.

An attacker can use an XSS vulnerability to proxy their requests through the victim’s web browser. The browser will add the victim’s cookies to those requests and they will appear to be legitimate requests from the user.

This technique of proxying requests through the victim’s browser can even defeat more advanced protection measures such as the in-development DPoP method for securing OAuth tokens against token theft or misuse. There are various advanced tricks you can try, such as using a Web Worker to control access to tokens and keys, but that is only a partial defence as handily summarised by Philippe De Ryck in this post to the OAuth mailing list.

This is why web security experts are often so gloomy about XSS and describe it as Game Over. This is especially distressing given that XSS is still one of the most prevalent vulnerabilities in web applications, with OWASP claiming that around two-thirds of applications have an XSS vulnerability. Although better JavaScript frameworks and technologies such as CSP and Trusted Types should reduce this over time, it seems likely that we’ll have to deal with XSS for a long time to come, so it would be nice if things weren’t quite so bleak.

One bad solution

Thankfully, there are some possible solutions. One quite poor solution would be to simply confirm each request with the user before allowing it to proceed. After all, if we can’t distinguish between legitimate actions performed by a user and malicious ones injected by the attacker, why not simply ask the user? “Do you really want to email all your photos to hacker@example.com?” This assumes that the application has access to some kind of trusted UI which it can use to confirm requests with the user, and which can’t be interfered with by the attacker. An example would be to send a push authorization request to an app on the user’s phone, asking them to confirm each request.

This kind of solution can work for occasional high-value transactions, and is often used in banking for exactly that use-case. But it’s not a general solution that could be used for every request made by your app. Users would quickly get tired of manually approving requests every time they click on a link or added an item to their shopping cart. Anyone who’s ever installed Little Snitch will know this feeling. Also many legitimate requests made by an app are not made in direct response to a user action or are not meaningful to users, so it would be hard for them to even know whether it was something they wanted to do or not.

So this specific solution is not very practical, but it does illustrate that solutions are possible: XSS is not necessarily Game Over. But it probably is within the mental models we’re used to using to think about web security.

A ray of sunlight

If we can’t ask the user to confirm every request, is there another way that we can distinguish between legitimate and malicious requests that can’t be easily defeated? The key is to notice that proxying requests through a web browser is an example of a Confused Deputy attack, just like CSRF. The attacker instructs the web browser (or JavaScript app) to make requests of their choosing, and the browser/app happily adds the user’s authority to those requests in the form of a cookie, access token, DPoP proof, or whatever. This type of attack can only occur because an attacker is able to construct a valid request independently of having the authority to perform it, and can then trick another person/process (the “deputy”) to perform it for them.

A systematic solution to confused deputy problems is provided by capability security, which I discuss in some detail in chapter 9 of my book and also in an older post on this blog. A fundamental principle of capability security is to combine designation with authority: it shouldn’t be possible even name a resource that you don’t have access to, much less craft a legitimate request to access it. For a web application, the idea is that rather than having a single cookie or token that provides access to everything, you would instead have lots of individual tokens that provide access to specific objects—one particular photo, for example—and that you encode these individual tokens directly into the URLs that are used to access these objects. The only way to have a legitimate URL is to be given one, and it’s impossible for any user to create one from scratch. (In the book I go into more detail about how to make this secure and convenient, which I won’t repeat here).

If access to a website was driven by capability URLs rather than cookies or all-powerful access tokens, then the attacker’s job after exploiting an XSS vulnerability is much harder. They cannot simply proxy requests through the victim’s web browser because, without access to any capability URLs, they cannot even begin to create those requests. Instead they must try to steal capability URLs from the app or intercept them in use, and hope that the ones they capture correspond to objects they want to access or manipulate. By storing capability URLs inside closures or using other security boundaries, an app can make it very hard for an attacker to intercept these URLs. I also believe that browser vendors could provide further protection by supporting a special URL scheme for capability URLs, but I’ll write about that another time.

I believe that such an approach can be made very secure against XSS attacks, while also being immune to CSRF attacks. But it’s very different to how most web apps are written today, and would require a fundamental change in security architecture. I have some ideas about how future versions of OAuth could incorporate some of these ideas, and with a bit of work you could retrofit it using techniques such as macaroons to create many individual tokens from a single all-powerful one (and then throw that one away). I believe that the security advantages are worth it, and I further believe that a capability-security model is the only viable long-term approach for securing the web. The same-origin policy has never been very effective, as the continued impact of XSS shows, and things like CSP and SameSite cookies are at best a poorly-fitting sticking plaster. At some point we need to rip it off and adopt a more systematic approach.

Insert coin to continue playing.

Macaroon access tokens for OAuth: Part 2 – transactional auth

Neil Madden — Wed, 09 Sep 2020 15:49:55 +0000

In part 1, I showed how Macaroon access tokens in ForgeRock Access Management 7.0 can be used as a lightweight and easy-to-deploy alternative to proof of possession (PoP) schemes for securing tokens in browser-based apps. The same techniques can be adapted to secure tokens in microservice architectures and IoT applications, and I hope to expand on some of the patterns they enable in future blog posts. But in this post, I want to look at third-party caveats and their application to transactional authorization.

What is transactional authorization?

In a normal OAuth flow, a user is granting some part of their authority to an app or service for ongoing access. For example, you might grant an app read-only access to files in your Dropbox or Google Drive. In these cases the scope of access is usually well defined, allowing the client to easily determine what scope to ask for (eg, read-all-files), and the grant of authority creates a long-lived relationship between the user and the client. If the user stops using the app, or otherwise decides to end this relationship, they can typically go to a page on the authorization server and revoke access from that client.

In transactional authorization the setup is somewhat different. The paradigmatic example is of making a high-value bank transfer. Suppose you want use an independently developed app to send $500 to your friend Alice. In a normal OAuth flow you’d grant the app some static scope like bank-transfer, and then it would be authorized to make any bank transfer on your behalf. But this is a bit risky, unless you really really trust that app, because it could then make lots of bank transfers that you haven’t asked for. The solution is to require authorization for each individual transfer. When you click the button to send $500 to Alice, your bank actively confirms this with you outside of the app to make sure that you really intended it.

For this to work, the user needs to be told the details of the transaction she is approving: how much money is being transferred, from which account, and to who? It seems intuitive that you could use OAuth to approve these individual transactions. Unfortunately, OAuth is not very good at details. The set of scopes understood by an authorization server is usually fixed and independent of any particular request, so there is no easy way to encode the specific details of a transaction. To get around this, the real Open Banking specifications (in the UK at least) require that the payment details are registered with the bank first to obtain a ConsentId that is then included in the OAuth authorization request by (ab)using the OpenID Connect claims parameter. My employer, ForgeRock, have a good explainer of this process.

This is all a little awkward, so there are proposals to add native support for transactions to OAuth 2, or to develop a new protocol that supports this from the start (gnow a separate IETF working group).

Macaroons and transactions

You may wonder how macaroons fit into all of this, seeing as this is a post about macaroons. Well, it turns out that they are a very good fit for transactional authorization, due to something known as third-party caveats. I’ll get to what those are in a moment, but first let’s consider some aspects of using OAuth for this. Suppose you had an OAuth provider that let you send a richly structured description of the transaction instead of a simple scope string. This is what the Rich Authorization Requests (RAR) proposal for OAuth 2 allows. The process would look something like the following:

To initiate a payment, the client (app) would send an authorization request to the authorization server (bank) with the details of the transfer.
The bank would then authenticate the user and ask for consent to approve this specific transaction.
If approved, the client would get back an access token with the scope being the details of the transaction.
The client uses the access token to call an API on the bank to actually perform the transfer. As part of this, the bank would check that the approved detailed scope matches the actual transfer being requested.

In principle this all works fine, but it lacks some safeguards. Every transaction is effectively a brand new interaction with the client and the access grant is only valid for that one specific transaction. This has some negative consequences:

There is no permanent relationship between the user and the client app stored at the AS. This means that if the user uninstalls the app or otherwise decides she no longer wants to use it, there is no way for her to revoke the ability for it to initiate payment consent requests.
Any registered client with the bank can suddenly decide to initiate a payment, even if I have no pre-existing relationship with them. As the continued success of phishing attacks shows, if you can pop up a consent screen on a user’s device then the battle is half won. In a highly regulated ecosystem like Open Banking you can perhaps rely on legal controls to prevent clients acting unethically, but as a user I’d prefer that only clients that I have explicitly approved can ask me to initiate transactions.

You can read more detailed discussion of these issues in the email thread I linked above. Not everyone agrees with my position, so make your own mind up.

We can use macaroon access tokens to overcome these issues. The basic idea is that the user would approve a long-lived access token (and optional refresh token) when she first installs an app or signs up for a service. This access token would have a normal static OAuth scope like initiate_payments. Then individual transfers can be authorized by appending a caveat that ties the token to a specific transaction, either by referencing a unique transaction ID (like the ConsentId of Open Banking) or directly including the structured transaction details in the caveat. This provides the best of both worlds. The user has a single long-lived authorization grant with the client, which she can revoke whenever she wants, and we can derive unique per-transaction tokens from that to approve individual requests.

Third-party caveats

The major downside of the scheme I just described is that the client can just approve its own transactions by appending a suitable caveat to its access token. No need to involve the user at all! What we want is some kind of caveat attached to the access token that forces the client to obtain authorization from the user for each transaction. This is exactly what macaroon third-party caveats allow.

The caveats we saw in part 1are all examples of first-party caveats: simple restrictions on how a token can be used that can be checked at the point of use (by the resource server). A third-party caveat can’t be checked locally by the RS, but instead requires the client to visit another service (the third party) and obtain proof that it satisfies some condition. For example, you might have an access token that allows placing orders for alcohol, but only if the client gets a proof from a government service that the user is over 18. The proof is in the form of a discharge macaroon that is cryptographically tied to the original caveat. The client presents the original macaroon and the discharge macaroon to the API to gain access.

(A cool feature is that discharge macaroons can themselves contain caveats, including more third-party caveats, requiring the client to get more proofs before they can access a protected resource).

We can use this property of third-party caveats to implement transactional authorization. The original long-lived access token issued to the client will contain a third-party caveat that requires the client to obtain a discharge macaroon from a separate transaction authorization service in order to initiate a payment. This transaction authorization service will authenticate the user and approve the transaction and then issue a discharge macaroon that approves that one transaction (by adding appropriate first-party caveats to bind it to the transaction ID or details). The client then presents its original long-lived access token and the discharge macaroon to a payment service to action the transfer.

This has some very nice properties:

The user gets to manage their relationship with the client over time exactly as they would for any other OAuth client.
The permission to ask to initiate transactions is itself properly represented as a separate scope that the user can consent to in an informed manner, rather than just letting any client initiate transactions.
Discharge macaroons are completely stateless and so consume no resources on the authorization server.
The AS need not be involved in individual transaction authorizations, which is nice if you are outsourcing the AS to a cloud service but don’t want them tracking individual user activity at that fine grained level. (Although you could use features of the AS if you wished, such as ForgeRock’s rich existing support for transactional authorization).

3rd-party caveats in AM 7.0

An idea is one thing, but it would be nice if you could actually implement this. Well, we thought so too, so we built some experimental support for 3rd-party caveats into AM 7.0 alongside the macaroon access token support I described in part 1.

Adding the 3rd-party caveat is simple using AM’s support for access token modification scripts. This allows an administrator to configure a JavaScript or Groovy script to make modifications to access tokens before they are issued. If you turn on macaroon access tokens then you can also add caveats to an access token as shown in the screenshot below:

In this example, we’re adding a third-party caveat to the access token to force the client to go to the transactional authorization service at https://txauth.example whenever it wants to use the access token. The third-party caveat consists of three parts:

A hint for where the client has to go to obtain a discharge macaroon
A unique random secret key that will be used to sign the discharge macaroon (known as the caveat key). This gets encrypted and encoded into the macaroon so that only the verifier can recover it.
An identifier to give to the third party service so that it knows what condition needs to be checked and how to recover the caveat key. In this example, I’ve encrypted the caveat key using a shared secret between the AS and the transactional authorization service, but you could also use public key encryption or pre-register the caveat key with the service and include a unique identifier instead.

Once the 3rd-party caveat is attached, the access token will be considered invalid unless it is accompanied by a discharge macaroon signed with the caveat key. The transaction authorization service can easily create the discharge macaroon using a macaroon library, as shown in this example code I wrote to test the idea using our internal macaroon library:

// Parse the macaroon access token and find the 3rd-party caveat // that requires authorizationMacaroon accessTokenMacaroon = Macaroon.deserialize(accessToken);Caveat txCaveat = accessTokenMacaroon.getThirdPartyCaveats() .filter(caveat -> SELF.equals(caveat.getLocationHint())) .findAny() .orElseThrow(() -> new IllegalArgumentException());// The key to create the discharge macaroon is encrypted in the 3rd-party // caveat, so decrypt it with the shared secret to recover it.Key caveatKey = decryptCaveatKey(txCaveat);// Now generate a discharge macaroon that satisfies the 3rd-party caveat.// We add additional caveats to this discharge macaroon that tie it to // this specific transaction and limiting the time it can be used for. String dischargeMacaroon = Macaroon.create(caveatKey, SELF, txCaveat.getIdentifier()) .addFirstPartyCaveat(new JsonCaveatSet() .expiresAt(now().plus(1, MINUTES)) .audience("https://payments.bank.example.com") .put("tx", json.toMap())) .serialize();

To verify the 3rd-party caveat has been satisfied, we added a new (non-standard!) header to the OAuth introspection endpoint that allows passing the discharge macaroon: X-Discharge-Macaroon. This is a header rather than a body parameter partly to give a hint that this is a non-standard extension and partly because there may be multiple discharge macaroons, but introspection request parameters can’t be repeated.

In the spirit of making this as transparent as possible for the RS (apart from the extra header of course), the introspection response combines the caveats of the original access token and any attached to the discharge macaroon. For example, if the discharge macaroon has an earlier expiry time than the access token then this will be reflected in the expiry time in the response. The same occurs for scope and audience restrictions on the discharge macaroon, making it easy for the transaction authorization service to limit what that client can do in the context of this one specific transaction.

Any unrecognized caveats are returned in a new “caveats” element on the introspection response, allowing the transaction details associated with the discharge macaroon to be returned:

{ "active": true", ... "caveats": { "tx": { "type": "payment\_initiation", "locations": ["https://example.com/payments"], "instructedAmount": { "currency": "EUR", "amount": "123.50" }, "creditorName": "Merchant123", "creditorAccount": { "iban": "DE02100100109307118603" }, ... }}

The payment service can then check these details match the transaction being processed (for example, using the scripting features of ForgeRock Identity Gateway 7.0 which support extracting and checking individual fields in the introspection response). Importantly, the use of the introspection endpoint ensures that the access token signing key never has to leave the AS.

Summary

In my opinion, third-party caveats present a really interesting alternative way of thinking about transactional authorization in OAuth. They have a lot of advantages compared to other approaches, and you can put together a working solution with very little code. It took me less than 2 hours to prototype a fully working version of the solution presented here using the existing features of AM I’d already added, before I’d even considered that they might be useful for transactional authZ. To me this is one of the real benefits of macaroon access tokens: they enable an extraordinary level of flexibility in the solution design space.

I’m sure there are plenty of downsides to this approach too. It relies pretty heavily on token introspection, which isn’t always a suitable solution. It also requires the RS to handle two (or more) tokens rather than just a single one. (Perhaps it would be better to “staple” them together?) The support for 3rd-party caveats in AM 7.0 is most definitely experimental, but I believe it points the way to some really interesting future capabilities. I’d love to hear feedback on this feature, and I’d love to eventually move towards standardisation of macaroon access tokens for OAuth. Let me know what you think in the comments.

PS – I’ll try and publish the demo code I have so that you can try it out if anyone is interested.

Least privilege with less effort: Macaroon access tokens for OAuth

Neil Madden — Wed, 29 Jul 2020 17:20:58 +0000

Part 1: Lightweight PoP

One of the major changes between OAuth 1 and OAuth 2 was the decision to drop the requirement that requests had to be signed using a secret key associated with each access token. This signing process was notoriously hard to get right and dropping it from OAuth 2 enormously simplified the life of client and resource server developers. In OAuth 2, access tokens are pure bearer tokens, a bit like passwords: if you know the value of the token you can use it, without needing any other credentials. This is very easy for developers, but is definitely less secure. The rise of single-page apps (SPAs) and other browser-based clients leaves access tokens more exposed to theft or accidental leakage compared to traditional server-side clients.

Drinking the PoP

Signed requests in OAuth 1 are an example of proof-of-possession (PoP) tokens rather than bearer tokens. If bearer tokens are like cash, PoP tokens are like Chip-and-PIN. Even if you steal an OAuth 1 token you can’t use it without the associated secret key used to sign requests. The only problem was it turned out that even if you did have the secret key you often couldn’t use the token either, because it was just too hard to get request signing to work reliably. Some brave souls periodically try and revive this idea.

To get around the problems with request signing, various alternative schemes have been proposed. Brian Campbell gave a nice overview of the history of OAuth PoP schemes at Identiverse recently, called The Burden of Proof. It’s a good talk and worth watching. The latest such scheme is DPoP, which tries to keep things as simple as possible: rather than signing the whole request, you just sign enough basic details of the request (the URI and HTTP method) to be able to stop the most likely attacks and rely on TLS to protect the content of the request.

DPoP is good, and for some use-cases I think it’s really good, but it still has some downsides:

Although the signature process itself is simplified, the client still has to generate and manage this secret key securely, which adds complexity that many client developers will avoid unless they have to.
The client, authorization server, and resource servers all have to know about DPoP. It’s quite hard to incrementally deploy.
It relies on fresh public key signatures on every request. Public key crypto is slow and expensive, so you generally want to do it as little as possible. I could imagine a rollout of DPoP requiring a significant increase in front-line servers to handle the extra load.
DPoP is based on JWS, which supports a wide range of signature algorithms, with no guidance as to which one to use. This means the resource servers have to potentially support them all (including some really expensive ones like ES512), or risk rejecting some clients. Clients will also have to guess which algorithms the server supports when they generate the private key. In practice this means that DPoP will only be used in closed deployments, or else everyone will gravitate to a lowest common denominator option like 2048-bit RSA with PKCS#1 v1.5 padding.
Public key signature algorithms are fragile. The point of a signature is that it can be verified by anybody, but in the case of an OAuth request the DPoP token only needs to be verified by exactly one party: the server that you’re sending the request to. There are much simpler and safer ways to achieve request authentication in this case.

Taking the biscuit

OK, so if DPoP is still too complex, what would a better solution look like? The risk that PoP solutions are trying to prevent is an access token being stolen from one request and used to authorize completely different requests that the client didn’t intend. A different way to solve this problem would be to have individual access tokens for every request that just authorizes that one specific request and no others. For example, it could have an audience restriction limiting it to only be used for requests to one specific server, and a single scope that authorizes just the specific type of API call that is being made. The expiry time could be limited to just a few seconds in the future, and so on. In current OAuth implementations it’s pretty hard to restrict an access token to exactly one request, but you can get pretty close.

Of course, this all sounds like a nightmare. The client can’t keep going back to the user to authorize access tokens for every little request!

It turns out there is a way to get lots of little access tokens for individual requests. It’s easy for developers, relatively simple to deploy, and cheap. (Much cheaper than digital signatures). It’s also quite delicious.

Enter macaroons

Apart from being a tasty treat, macaroons are also a new token format invented by the boffins at Google. At first glance, a macaroon is an authenticated token a bit like a JWT using the HS256 algorithm. But the additional yummy goodness that macaroons add is the ability to add caveats. A caveat is a restriction on how the token can be used. For example, you might limit the scope of a token, or reduce its expiry time. They always reduce, never expand, the authority of the token. The really cool thing is that adding a caveat gives you a new token, leaving the original token unchanged. It’s also really cheap: a few microseconds typically. This means you can have a single powerful access token and then derive multiple more restricted access tokens from it. This lets us have our cake and eat it (sorry, my metaphors are all over the place today): you can get a single token approved by the user but then derive individual access tokens for every single API call you make, with just the perfect amount of privilege for that one call.

This gives us the benefits of having lots of tiny individual access tokens for each request, but we can generate these tokens on the fly from the original token. This is what the original macaroons paper refers to as contextual caveats: specific details of the context in which an API call is taking place that we add to a token to restrict its use. That way, if the token does get stolen or misused, the scope of what you can do with it is severely limited by the caveats attached to it. The genuine client still has the original access token though, so they can still do whatever they like within the scope of the original grant.

If we restrict the tokens enough then they become pretty hard to misuse. We get much of the same benefits of a PoP approach but with much less complexity. Where you might have had code like the following to call an API:

fetch('[https://api.example.com/v1/kittens](https://api.example.com/v1/kittens)', {
    method: 'POST',
    headers: {
        'Authorization': 'Bearer ' + token
    },
    body: JSON.stringify(kittenDetails)
});

The new macaroon-enhanced version becomes something as simple as the following with an appropriate library:

fetch('[https://api.example.com/v1/kittens](https://api.example.com/v1/kittens)', {
    method: 'POST',
    headers: {
        'Authorization': 'Bearer ' + **token.restrict({**  
**            aud: 'api.example.com',**  
**            exp: nowPlus5Seconds(),**  
**            scope: 'upload\_cute\_kitten\_picture'**  
**        })**
    },
    body: JSON.stringify(kittenDetails)
});

Macaroon OAuth tokens in ForgeRock AM 7.0

At ForgeRock we love OAuth and we also love macaroons. We think they make a great combination. OAuth benefits from the simple but flexible approach to least privilege that macaroons enable, and macaroons benefit from the framework and tooling that exists around OAuth. I strongly believe that macaroons provide about 90% of the benefit of PoP for about 20% of the effort.

That’s why we’ve included support for issuing OAuth access and refresh tokens as macaroons in the upcoming 7.0 release of ForgeRock Access Management. So how does it work? Well, mostly you just go into the OAuth2 Provider Settings and turn it on:

Now when you complete an OAuth flow you get issued with a macaroon access token and optional refresh token. They look just like normal access tokens, but are a little bit longer:

You can call the standard token introspection endpoint to see the details of the token, just like you could with a normal access token:

You can then add a bunch of caveats just like in the example earlier, to reduce the scope, expiry time, and audience and introspect it again to see the difference:

Repeating the token introspection request a few seconds later, the restricted access token has expired:

But the original access token is still valid. In fact, all the details of the original token in the database (AM’s Core Token Service) are exactly as they were when the token was first issued.

Everything is handled transparently through the token introspection endpoint, making it really easy to deploy this incrementally:

In step 1, you simply turn on macaroon access tokens at the Authorization Server. So long as your resource servers are using token introspection, everything can carry on as normal. Clients and resource servers need no changes.
In step 2, your clients start adding caveats to tokens using a macaroon library. Resource servers carry on introspecting the tokens, but now the introspection responses take into account the caveats on the tokens.
There is no step 3.

That’s a little taster of macaroon access tokens in ForgeRock AM 7.0 and why I think they will be a game changer. Least privilege with less effort. In part 2, I’ll show you some more exotic things you can do with macaroon access tokens using 3rd-party caveats.

What Cache-Control headers do I need?

Neil Madden — Tue, 19 May 2020 06:38:54 +0000

Understanding caching is an important part of web development. From a performance point of view, you don’t want a client to be downloading the same large file again and again. But from a security point of view, you don’t want sensitive data, such as users’ personal details, cached by shared web proxies and served to other users.

There are many aspects to HTTP caching, but the one that’s always confused me the most is the Cache-Control header. There are so many options! When should I use each one? Why does no-cache not mean “disable caching”??!

To help myself remember what all the options do I made this little flowchart. Hopefully you’ll find it useful too. If you spot an error, please contribute on GitHub. And if somebody wants to make a prettier version, please do! (Just credit me).

PDF version

Java KeyStores—the gory details

Neil Madden — Sat, 18 Nov 2017 17:54:22 +0000

Originally posted on my blog

Java KeyStores are used to store key material and associated certificates in an encrypted and integrity protected fashion. Like all things Java, this mechanism is pluggable and so there exist a variety of different options. There are lots of articles out there that describe the different types and how you can initialise them, load keys and certificates, etc. However, there is a lack of detailed technical information about exactly how these keystores store and protect your key material. This post attempts to gather those important details in one place for the most common KeyStores.

Each key store has an overall password used to protect the entire store, and can optionally have per-entry passwords for each secret- or private-key entry (if your backend supports it).

Java Key Store (JKS)

The original Sun JKS (Java Key Store) format is a proprietary binary format file that can only store asymmetric private keys and associated X.509 certificates.

Individual private key entries are protected with a simple home-spun stream cipher—basically the password is salted (160-bits) and hashed with SHA-1 in a trivial chained construction until it has generated enough output bytes to XOR into the private key. It then stores a simple authenticator tag consisting of SHA-1(password + private key bytes) — that's the unencrypted private key bytes. In other words, this is an Encrypt-and-MAC scheme with homespun constructions both based on simple prefix-keyed SHA-1. (This scheme has OID 1.3.6.1.4.1.42.2.17.1.1).

The whole archive is again integrity protected by a home-spun prefix keyed hash construction, consisting of the SHA1 hash of the UTF-16 bytes of the raw keystore password, followed by the UTF-8 bytes of the phrase "Mighty Aphrodite" (I'm not kidding) followed by the bytes of the encoded key store entries.

If every part of this description has not got you screaming at your screen in equal parts terror and bemusement, then you probably haven't fully grasped how awful this is. Don't use it, even for just storing certificates — it's tampering resistance is if anything even worse than the encryption.

JCE Key Store (JCEKS)

Sun later updated the cryptographic capabilities of the JVM with the Java Cryptography Extensions (JCE). With this they also introduced a new proprietary key store format: JCEKS.

JCEKS uses "PBEWithMD5AndTripleDES" to encrypt individual key entries, with a 64-bit random salt and 200,000 iterations of PBKDF1 to derive the key. TripleDES is used with 3 keys ("encrypt-decrypt-encrypt") in CBC mode. There is no separate integrity protection of individual keys, which is fine if the archive as a whole is integrity protected, but it means that access control is effectively at the level of the whole keystore. This is not terrible from a crypto point of view, but can definitely be improved—neither MD5 nor TripleDES are considered secure any more, and it's been a long time since anyone recommended them for new projects. However, it would also not be a trivial effort to break it.

JCEKS uses the same ridiculous "Mighty Aphrodite" prefix-keyed hash as JKS for integrity protection of the entire archive. It is probably best to assume that there is no serious integrity protection of either of these key stores.

PKCS#12

Apart from these proprietary key stores, Java also supports "standard" PKCS#12 format key stores. The reason for the scare quotes around "standard" is that while it is indeed a standard format, it is a very flexible one, and so in practice there are significant differences between what "key bag" formats and encryption algorithms are supported by different software. For instance, when you store symmetric SecretKey objects in a PKCS#12 key store from Java, then OpenSSL cannot read them as they use a bag type ("secretBag" - OID 1.2.840.113549.1.12.10.1.5) that it does not understand.

Java uses version 3 of the PKCS#12 standard format. It stores secret keys in the aforementioned "secretBag" format, and asymmetric private keys in "PKCS#8 Shrouded Key Bag" format (OID 1.2.840.113549.1.12.10.1.2). This just dictates the format of bytes on the disk. In both cases the actual key material is encrypted using some form of password-based encryption (PBE) mode. By default this is "PBEWithSHA1AndDESede" — "DESede" is another name for TripleDES in encrypt-decrypt-encrypt mode, so this is pretty similar to the mode used by JCEKS apart from using a slightly better (but still deprecated) hash in the form of SHA-1. By default this uses a 160-bit salt and 50,000 iterations.

But, there is an important improvement in the PKCS#12 implementation—you get to choose the encryption algorithm! By passing in a PasswordProtection parameter (from Java 8 onwards) when saving a key you can specify a particular (password-based) cipher to use. I haven't checked exactly what ciphers are allowed, but you can at least specify a stronger PBE mode, such as "PBEWithHmacSHA512AndAES_256", which will derive a 256-bit AES key using salted PBKDF2 and then encrypt the stored key using AES/CBC/PKCS5Padding with that key. You can also increase the number of iterations of PBKDF2 used. For example:

import java.io.FileOutputStream;
import java.security.KeyStore;
import java.security.KeyStore.PasswordProtection;
import java.security.KeyStore.SecretKeyEntry;
import java.security.SecureRandom;

import javax.crypto.SecretKey;
import javax.crypto.spec.PBEParameterSpec;
import javax.crypto.spec.SecretKeySpec;

public class scratch {
    public static void main(String... args) throws Exception {
        KeyStore keyStore = KeyStore.getInstance("PKCS12");
        keyStore.load(null, null); // Initialize a blank keystore

        SecretKey key = new SecretKeySpec(new byte[32], "AES");

        char[] password = "changeit".toCharArray();
        byte[] salt = new byte[20];
        new SecureRandom().nextBytes(salt);
        keyStore.setEntry("test", new SecretKeyEntry(key),
            new PasswordProtection(password,
                "PBEWithHmacSHA512AndAES_128",
                new PBEParameterSpec(salt, 100_000)));

        keyStore.store(new FileOutputStream("/tmp/keystore.p12"), password);
    }
}

Note that despite the inclusion of "HmacSHA512" in the above PBE mode that only applies to the key derivation from the password. There is no integrity protection at the level of individual entries.

It is also worth noting that the keystore and individual key passwords should be the same. I don't think this is a fundamental limitation of PKCS#12 in Java, but certainly standard Java tools like the command line "keytool" utility will fail to handle PKCS#12 keystores with different passwords used for the store vs individual keys. If you don't need to use those tools then you might be able to get away with different passwords for each key.

In contrast to the previous entries, the PKCS#12 key store format does actually encrypt certificates too. It does this with a hard-coded algorithm "PBEWithSHA1AndRC2_40". This uses 50,000 rounds of salted PBKDF1 to derive a 40-bit key for RC2 encryption. RC2 is an old stream cipher that I certainly wouldn't recommend. The 40-bit key is far too small to provide any serious security. It makes me wonder why bother applying 50,000 rounds of PBKDF1 to protect the password while generating a key that is itself vulnerable to brute-force. It is probably actually faster to brute force the derived key than the original password. I can only assume it is maintaining compatibility with some decision taken way back in the depths of time that everyone involved now deeply regrets.

The integrity of the overall PKCS#12 key store is protected with "HmacPBESHA1". This is HMAC-SHA1 using a key derived from the store password using 100,000 iterations of salted PBKDF2-HMAC-SHA1. This is all hard-coded so cannot be changed. This is an ok choice, although it would be nice to be able to use something other than SHA-1 here, as it appears that PKCS#12 allows other MACs to be used. For HMAC usage, SHA-1 is still just about ok for now, but it would be better to remove it. It would also be nice to be able to tune the iteration count.

Overall, the PKCS#12 key store is considerably better than either of the Sun-designed proprietary options. If you specify your own PasswordProtection instances with AES and SHA2 and use high iteration counts and good random salts, then it's actually a pretty solid design even by modern standards. The only really ugly part is the 40-bit RC2 encryption of trusted certificates, but if you do not care about the confidentiality of certificates then we can overlook that detail and just consider them lightly obfuscated. At least the use of HMAC-SHA1 is a decent integrity protection at last.

PKCS#11

There's not much to say about PKCS#11. It is a standard interface, intended for use with hardware security tokens of various kinds: in particular Hardware Security Modules (HSMs). These range from 50 Euro USB sticks up to network-attached behemoths that cost tens or hundreds of thousands of dollars. The hardware is usually proprietary and closed, so it's hard to say exactly how your keys will be stored. Generally, though, there are significant protections against access to keys from either remote attackers or even those with physical access to the hardware and a lot of time on their hands. This isn't a guarantee of security, as there are lots of ways that keys might accidentally leak from the hardware, as the recent ROCA vulnerability in Infineon hardware demonstrated. Still, a well-tested HSM is probably a pretty secure option for high-value keys.

I won't go into the details of how to set up a PKCS#11 key store, as it really varies from vendor to vendor. As for PKCS#12, while the interface is standardised there is enormous room for variation within that standard. In most cases you would let the HSM generate keys in the secure hardware and never export the private key material (except perhaps for backup).

Summary

Use a HSM or a PKCS#12 keystore, and specify manual PasswordProtection arguments when storing keys. Avoid the proprietary key stores.

Alternatively, farm out key management to somebody else and use a Key Management System (KMS) like Hashicorp Vault.

7 Best Practices for JSON Web Tokens

Neil Madden — Wed, 25 Jan 2017 13:56:34 +0000

Update 15th March 2017: Given recent criticism of the JWT and JOSE specifications, I've written some notes on should you use JWT/JOSE?.

JSON Web Tokens (JWTs, pronounced "jots") are gaining in popularity as a way to securely transmit small packets of information, such as session tokens, proof of identity, and network protocol messages. While there are a great many libraries available that make constructing and verifying JWTs straightforward, it is also easy to misuse them and end up compromising the very security properties you wanted to achieve. In this article, I outline some best practices to ensure that your information stays secure.

1. Take time to learn about the underlying security properties

It is easy to think of cryptography libraries as a black box. You pick a library, read the docs, and before long you have sprinkled a little digital signature or encryption magic dust on your application and moved on to the next task. This is a mistake. Even with the most carefully designed and implemented library, a lack of knowledge of the security properties that it provides and how it provides them can lead to fatal flaws when used incorrectly.

JWT libraries provide a lot of confusing options:

Symmetric Message Authentication Codes (MACs) based on HMAC with SHA-256, SHA-384 or SHA-512
Digital signatures based on RSA or Elliptic Curve DSA.
Symmetric authenticated encryption based on AES in CBC mode or AES in GCM mode.
Public key encryption using RSA with various padding modes (PKCS#1.5, OAEP).
Key agreement protocols such as ECDH or password-based, and various key-wrapping modes.
Compression of the payload.

These options can be bewildering at first, and you need to pick carefully to get the security properties you desire. For instance, to protect session tokens issued and consumed by the same application, it may be reasonable to use a symmetric encryption mode such as AES-GCM. This provides both confidentiality of the content of the session token and integrity and authenticity to prevent forgeries. However, a public key encryption scheme such as RSA would be wholly inappropriate in this case, as anybody with the public key can then create a valid session token.

Taking some time to learn about the security properties and how the underlying cryptographic primitives provide them will pay dividends in the long run. The Coursera Crypto I course will give you a good start.

2. Don't go overboard

With the plethora of modes described in the previous point, it can be tempting to go overboard and use a bit of everything. This is often unnecessary. For instance, if you want to ensure confidentiality and authenticity of your own JWTs then it is is sufficient to use one of the symmetric encryption modes, as these are already authenticated. You can either directly encrypt the JWTs with a shared key, or use one of the key-wrapping modes. It is certainly not necessary to also then apply a separate HMAC, unless you are very paranoid about security margins. (The better idea is to rotate your keys frequently, see below). The more options you use, the more complex your system becomes to reason about and the more credentials you potentially have to manage.

3. Plan for how you will manage your keys

The weakest part of any system's security is often the key management. If a key is compromised then the security properties are no longer guaranteed: messages may be decrypted, security tokens forged, and other parts of your system breached as a result. You should follow best practices for managing your keys:

Use secure storage for all keys that is not easily accessed in case of a breach. Consider using a Hardware Security Module (HSM) or a dedicated key management service running on a separate host such as Vault or Amazon KMS. Make sure appropriate access and audit controls are in place.
Rotate keys often, such as once a day or week or month, depending on usage volumes.
Use a high-quality source of random data for generating keys. Usually this means /dev/random or /dev/urandom on UNIX-like systems, or a dedicated hardware random data generator. If you must use passwords, then use a high-quality password-based key derivation algorithm such as bcrypt, scrypt, Argon2 or PBKDF2. Use high iteration counts (e.g. NIST recommends at least 10,000 iterations for PBKDF2) and unique per-key salts when applicable.
Immediately rotate any keys suspected of compromise and stop trusting any JWTs using that key.

4. Consider using "headless" JWTs

A JWT consists of a protected payload together with a plaintext "header" section. This can contain various bits of information such as the algorithms used to sign or encrypt the payload or application-specific information to be used by intermediaries on the network, e.g. for message routing. In a lot of cases, this information is redundant and it is downright dangerous to trust its contents anyway. If you do not need to interoperate with third parties that expect standard JWTs, you can save some space and eliminate a whole class of vulnerabilities by simply stripping off the header section when producing a JWT and then recreate it from known data before parsing. I call these "headless JWTs" and recommend you use them wherever you can.

Stripping the header is easy: just remove everything up to the first "." character in the encoded JWT. To reconstruct the JWT, just base64url-encode a fixed header identifying the known algorithm and parameters and prepend it to the headless JWT.

5. Be careful when using encryption and compression together

Combining encryption with compression can cause information leaks if you are not careful. This is not just a theoretical vulnerability as practical attacks have been shown. If an attacker is able to control any information in the same encrypted block as some sensitive information, then compression may leak information about the secret parts. Compressing a JWT can be a big win when space is at a premium (e.g. for HTTP cookies), but it should be weighted up against the potential risks. If you need confidentiality and really want compression too, some options to consider include:

Padding the compressed content to some fixed size that you are happy with. You may still get a decent compression ratio with this.
Segregate your data into secret information controlled by your application, and non-secret information controlled by user input. Place the secret information into the encrypted & compressed payload and the user-supplied information into the uncompressed (and unencrypted) JWT header. This way the user-controlled information cannot alterÂ the compression of the secret information. If the user-supplied information should also be encrypted, then consider using two separate JWTs.

6. Consider JWT lifetimes and revocation

Where JWTs are used to convey authority, best practice is to limit the period of time that the JWT is valid for. If JWTs must be long-lived, then investigate options for revoking JWTs to prevent further use:

Blacklisting JWTs based on a unique ID can be made scalable and efficient, but defaults to JWTs being trusted in the absence of information.
Have short-lived JWTs and require them to be re-issued regularly (e.g. via an OAuth2 refresh token flow). This can be a suitable compromise when the clients are programmatic and so can automatically refresh tokens periodically.
Use JWTs only to identify resources and some minimal state, while storing the majority of data on the server. The database stores state as normal with a unique identifier (e.g. UUID). The JWT is constructed by the application layer from this unique ID and some minimal metadata. This may seem like the worst of all worlds, but is actually quite a sensible default for a number of reasons: 1. State is kept on the server and can easily be managed, audited and revoked (deleted). 2. Protections in the JWT (e.g. HMAC) allow for a quick check of validity before hitting a central database resource. 3. If tokens are stored in the database but signed/verified by the application layer on-the-fly using a key stored separately, then an attacker must compromise both systems in order to forge or steal tokens. 4. State can be moved from the JWT toÂ the database and vice versa over time as requirements change. 5. Larger amounts of state can be stored in the database than can typically be toleratedÂ in a JWT, and it is much easier to update it.

7. Read the Security Considerations!

Like most RFCs, the JWT specs contain Security Considerations sections that detail common threats and advice on how to avoid them. You should read all of these and make sure you understand them before deploying a JWT-based solution:

You should also read common criticisms of JWT-based solutions. While I believe that article overplays the risks (there are reasonable mitigations available), it makes some valid points that you should consider before betting the farm on JWTs.

Conclusions

I hope this article has given you some solid advice on building robust and secure JWT-based solutions. When used carefully and with plenty of planning, JWTs can form the basis of highly scalable and reliable systems. By following the best practice in this article, you should be able to put appropriate protections in place to ensure that your data stays secure.

If you liked this article, you'll love my book: API Security in Action

Hi, I'm Neil Madden

Neil Madden — Sun, 15 Jan 2017 23:09:53 +0000

I have been coding for ~25 years (18 years professionally).

You can find me on GitHub as NeilMadden

I live in the UK.

I am a freelance software engineer and application security consultant.

I mostly program in these languages: Java, C, Tcl, Prolog, Scheme and a little ML or Haskell every now and then.

I am currently learning more about Erlang/Elixir and Go.

Nice to meet you.