Forem: Shyamala

The case of disappearing metrics in Kubernetes

Shyamala — Sun, 24 May 2020 15:46:30 +0000

🧐 Context 🧐

My team hosts the kubernetes platform running on EKS for several teams. When EKS announced the support for Kubernetes version 1.16 we upgraded our infrastructure. This is pretty routine process for us. We use Prometheus and grafana dashboard to observe and monitor our infrastructure.

After we upgraded, our metrics dashboard stopped showing RAM and CPU utilization of pods. I set out to find why?

⚠️ Disclaimer ⚠️

Before we move ahead,

I am a newbie to Kubernetes learning the tricks on the job
This is my journey on how I debugged the issue, so what is obvious to a Kubernetes expert is not obvious to me.

TL;DR

If you already know what caused this issue, jump to final section

Debugging Step 1:

So the dashboard is not showing metrics, normally Kubectl has commands that shows the CPU and RAM utilization of the Nodes and Pods. So I checked if those worked.

 > kubectl top nodes
NAME                                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ip-xx-xx-xxx-xxx.ec2.internal   8xxm         5%     2xxxxMi         23%
ip-xx-xx-xxx-xxx.ec2.internal   2xxxxm       15%    3xxxxMi         29%
ip-....

> kubectl top pods
NAME         CPU(cores)   MEMORY(bytes)
xxxx-wnpgj   xxm          xxMi
xxxx-r4k85   xxm          xxxMi
...

🤔 What does this mean? 🤔

Kubernetes is able to fetch the metrics. These are the metrics that have to be used by the HPA to scale the Pods based on the resource utilization.

Debugging Step 2:

Since the dashboard does not show the resource utilization metrics, but the kubectl top command works, my next thought was may be the scaling does not work properly anymore.

According to Resource usage monitoring

components such as the Horizontal Pod Autoscaler controller, as well as the kubectl top utility. These metrics are collected by the lightweight, short-term, in-memory metrics-server and are exposed via the metrics.k8s.io API

So now I wanted to see what is the status with one of our teams.

> kubectl get hpa -n xxx 
NAME         REFERENCE               TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
xxx-worker   Deployment/xxx-worker   82/150, 5%/100%   1         6         1          4d19h

Since the Targets look fine in the above output. I thought let me see if at any point the scaling actually happened.

> kubectl describe hpa -n xxx 

Type     Reason                   Age                      From                       Message
----     ------                   ----                     ----                       -------
Warning  FailedGetResourceMetric  25m (x24 over 2d23h)     horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API
Normal   SuccessfulRescale        9m56s (x233 over 2d23h)  horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
Normal   SuccessfulRescale        4m9s (x233 over 2d22h)   horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

🤔 What does this mean? 🤔

As we can see at some point fetching the resource metric failed but we can also see that the system recovered and was able to scale successfully.

Since there is a FailedGetResourceMetric at some point, I thought this could lead me to why metrics do not appear in the dashboard.

Debugging Step 3:

As we have seen from several documentation mentioned above and also from the error no metrics returned from resource metrics API. There is some component in Kubernetes that is responsible for the metrics. So I cast a wide net

> kubectl get all -n metrics | grep metrics | grep pod
pod/metrics-server-xxxxxxx-xxxxx                        1/1     Running            0          4d21h
pod/prometheus-kube-state-metrics-xxxx-xxxxx            1/1     Running            0          4d21h

This sound promising. So metrics-server sounds like some server dealing with metrics. Naturally I am interested in logs

> k logs -n metrics pod/metrics-server-xxxxxxx-xxxxx
I0524 11:17:54.371856       1 manager.go:120] Querying source: kubelet_summary:ip-xx-xx-xxx-xxx.ec2.internal
I0524 11:17:54.388840       1 manager.go:120] Querying source: kubelet_summary:ip-xx-xx-xxx-xxx.ec2.internal
I0524 11:17:54.495829       1 manager.go:148] ScrapeMetrics: time: 233.232388ms, nodes: x, pods: xxx
E0524 11:18:18.542051       1 reststorage.go:160] unable to fetch pod metrics for pod my-namespace/team-worker-xxxxx-xxxx: no metrics known for pod

🤔 What does this mean? 🤔

As soon as I saw the words ScrapeMetrics (my brain translated it to Prometheus related) and the error unable to fetch pod metrics for pod my-namespace/team-worker-xxxxx-xxxx: no metrics known for pod
I thought I have found the issue.

Debugging Step 4:

As any seasoned developer would do, I took the error message and googled it. Which resulted in some links and on a quick scan, I found couple of interesting Links

After quickly scanning through the links, every one seem to suggest this following change in metrics-server

command:
- /metrics-server
- --kubelet-insecure-tls #<-- 🤓
- --kubelet-preferred-address-types=InternalIP #<-- 🤓

--kubelet-insecure-tls seems like something you would do in a test setup. So I wanted to understand a bit more what this flags do.
From the documentation, it is clear that the above flag is intended for testing purposes only and in the logs of metrics server I did not find any error pertaining to certificates. So I did not add the flag.

--kubelet-preferred-address-types this configuration is to determine which address type to use to connect to the nodes for getting the metrics.
So before adding this config I wanted to verify if the metrics server api was reachable or not.

> kubectl get apiservices | grep metrics-server
v1beta1.metrics.k8s.io                 kube-system/metrics-server       True        4d23h

since the status of this service is available, those two flags would not make any sense. I wanted to see if I can get some raw metrics by querying this API.

> kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq '.items | .[] | select(.metadata.namespace == "xxx")'
{
  "metadata": {
    "name": "xxx-worker-6c5bd59857-xxxxx",
    "namespace": "xxx",
    "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/xxx/pods/xxx-worker-6c5bd59857-xxxxx",
    "creationTimestamp": "2020-05-24T12:06:45Z"
  },
  "timestamp": "2020-05-24T12:05:48Z",
  "window": "30s",
  "containers": [
    {
      "name": "xxx-worker",
      "usage": {
        "cpu": "xxxxxn",
        "memory": "xxxxxxxKi"
      }
    }
  ]
}

🤔 What does this mean? 🤔

So everything seems to function well. I do not know why the metrics I am clearly able to query via Kubectl is not available for the dashboard.

Debugging Step 5:

Now I want to see the source code of metrics-server to figure out under what circumstance this error happens

    podMetrics, err := m.getPodMetrics(pod)
    if err == nil && len(podMetrics) == 0 {
        err = fmt.Errorf("no metrics known for pod \"%s/%s\"", pod.Namespace, pod.Name)
    }
    if err != nil {
        klog.Errorf("unable to fetch pod metrics for pod %s/%s: %v", pod.Namespace, pod.Name, err)
        return nil, errors.NewNotFound(m.groupResource, fmt.Sprintf("%v/%v", namespace, name))
    }

It is very clear from the code that the Pod exists but metrics is not available. So I tailed the logs and got the pod for which metrics is not found

 > kubectl get po -n xxx
NAME                          READY   STATUS    RESTARTS   AGE
xxx-worker-6c5bd59857-xxxxx   1/1     Running   0          1m5s

🤔 What does this mean? 🤔

It is very clear from above that all these pods where metrics are not available are pods that have been recently created
and since metrics server scrapes for metrics periodically, it's just not available at the moment.

In my opinion it is really not an ERROR rather an expected behaviour. A WARN or INFO would be reasonable. It is good to see am not alone in this

Remove/propose different "no metrics known for pod" log #349

serathius posted on Nov 05, 2019

There are a lot of issues where users are seeing metrics-server reporting error "no metrics known for pod" and asking for help.

To my understanding this error is expected to occur in normal healthy metrics-server. Metrics Server periodically scrapes all nodes to gather metrics and populate it's internal cache. When there is a request to Metrics API, metrics-server reaches to this cache and looks for existing value for pod. If there is no value for existing pod in k8s, metrics server reports error "no metrics known for pod". This means this error can happen in situation when:

fresh metrics-server is deployed with clean cache
query is about fresh pod/node that was not yet scraped

Providing better information to users would greatly reduce throughput of tickets.

View on GitHub

I have been chasing a wrong lead after all.

Debugging Step 5:

I have eliminated that Metrics server is not the reason for the bug. This got me to thinking, I should find how Prometheus is scraping the metrics.

After a lot of reading I figured out Prometheus gets the metrics exposed by the kube-state-metrics. It is very important to understand the difference between metrics-server and kube-state-metrics

Comment for #55

metalmatze commented on May 07, 2018

kube-state-metrics focuses on

...on the health of the various objects inside, such as deployments, nodes and pods.

metrics-server on the other hand implements the Resource Metrics API

To summarize in a sentence: kube-state-metrics exposes metrics for all sorts of Kubernetes objects. metrics-server only exposes very few metrics to Kubernetes itself (not scrapable directly with Prometheus), like node & pod utilization.

View on GitHub

Looking at the logs of prometheus server and kube-state-metrics nothing jumped out of ordinary.

Finally

I asked for help, I explained what I have done so far and my team mate suggested the obvious, did you check if the query works!!

so I went to prometheus query explorer and executed

container_cpu_usage_seconds_total{namespace="xxx", pod_name="xxx-worker-6c5bd59857-xxxxx"}

Returned no results

container_cpu_usage_seconds_total{namespace="xxx"}

Returned results

container_cpu_usage_seconds_total{namespace="xxx", pod="xxx-worker-6c5bd59857-xxxxx"}

Returned results

🤔 What does this mean? 🤔

So some where during the EKS version upgrade the support for label pod_name is dropped.

After fixing the queries in grafana, I could see the metrics finally.

Lessons Learned

Learn the components and its purpose before assuming
Always read the release notes when upgrading even minor versions

Removed cadvisor metric labels pod_name and container_name to match instrumentation guidelines. Any Prometheus queries that match pod_name and container_name labels (e.g. cadvisor or kubelet probe metrics) must be updated to use pod and container instead. (#80376, @ehashman)

The solution is always the obvious solution

I welcome all constructive feedback and comments

Microservices - Why wait for your Integration tests to fail when you have Pact?

Shyamala — Tue, 02 Apr 2019 19:08:22 +0000

With more and more software moving from monolith to microservices, we are swamped in the world of more APIs than ever. How can we write APIs that are meaningful, precise , testable and maintainable? Once an API is out there we are obligated not to break them or alternatively we have to know before hand that we have breaking changes.

In this post I try to put together my experience on how Pact (https://docs.pact.io) helps us in writing better APIs in our microservices architecture.

What is a Pact?

Pact (noun):
A formal agreement between individuals or parties.

According to Pact Foundation

Pact is a contract testing tool. Contract testing is a way to ensure that services (such as an API provider and a consumer) can communicate with each other. Without contract testing, the only way to know that services can communicate is by using expensive and brittle integration tests.

Parties

Any API(Application Programming Interface) consists of two important parties:

Provider: One who exposes one or more API(s)

Consumer(s): One or more client(s) who uses the API(s) exposed by the Provider

How does a Pact work?

Consumer captures an expectation as a separate pact, and provider agrees to it.

Advantages of using Pacts

Isolation:

The Ability to test the provider and consumer in Isolation but in conjunction, i.e. Pacts gives us the sophistication of

testing a consumer with provider's API from the comfort of a developer's machine Without having to make an actual call.
testing a provider's API changes against one or more consumers to make sure that the changes do not accidentally break the existing consumers

Quick Feedback

We do not have to wait for our End to End tests to fail, the same feedback can be unit tested.

User Centred API design

Pacts are essentially Consumer Driven Contracts, so the consumers are laying out the expectation which leads to better usable APIs

Overview of existing APIs

Pacts provides Network Graph of dependant services
Pacts helps us understand, if one or more APIs are similar
Pacts can also show unused APIs, when the consumers are not using them anymore.

Language independent

Pact is a specification, which makes it perfect for microservices testing. You can find Pact consumer and provider libraries implemented in many programming languages here

Pact in Action:

An Example Pact

Let us take the most simple use case of authentication as a service.

Consumer: I need to log the user in, I have user credentials
Provider: I can authenticate a user given credentials

For demonstration, we use the same pact as described above. Complete implementation can be found here

Consumer Pact in Go:

    import (
        "fmt"
        "github.com/pact-foundation/pact-go/dsl"
        "net/http"
        "testing"
    )

    func TestClient_AuthenticateUser(t *testing.T) {

        var username = "alice"
        var password = "s3cr3t"

        t.Run("user exists", func(t *testing.T) {
            pact := &dsl.Pact{
            Consumer: "Quoki",
            Provider: "UserManager",
          PactDir:  "../pacts",
          LogDir:   "../pacts/logs",
        }

      defer pact.Teardown()

            pact.
                AddInteraction().
                Given("user exists").
                UponReceiving("a request to authenticate").
                WithRequest(dsl.Request{
                    Method:  "POST",
                    Path:    dsl.String(fmt.Sprintf("/users/%s/authentication", username)),
                    Headers: dsl.MapMatcher{"Content-Type": dsl.Like("application/x-www-form-urlencoded")},
                    Body: dsl.MapMatcher{
                    "password": dsl.Like(password),
                    },
                }).
                WillRespondWith(dsl.Response{
                    Status: http.StatusNoContent,
                })

        pact.Verify(func() error {
                subject := New(fmt.Sprintf("http://localhost:%d", pact.Server.Port))
                ok := subject.AuthenticateUser(username, password)

                if !ok {
                    t.Fail()
                }

                return nil
            })

        })
    }

Running this successfully will generate,

    {
      "consumer": {
        "name": "Quoki"
      },
      "provider": {
        "name": "UserManager"
      },
      "interactions": [
        {
          "description": "a request to authenticate",
          "providerState": "user exists",
          "request": {
            "method": "POST",
            "path": "/users/alice/authentication",
            "headers": {
              "Content-Type": "application/x-www-form-urlencoded"
            },
            "body": "password=s3cr3t",
            "matchingRules": {
              "$.headers.Content-Type": {
                "match": "type"
              },
              "$.body.password": {
                "match": "type"
              }
            }
          },
          "response": {
            "status": 204,
            "headers": {
            }
          }
        }
      ],
      "metadata": {
        "pactSpecification": {
          "version": "2.0.0"
        }
      }
    }

Provider verification of Pact in Kotlin

The Provider then takes this as requirement, then verifies by running this test

    package com.shyamz.provider.authenticate

    import au.com.dius.pact.provider.junit.Consumer
    import au.com.dius.pact.provider.junit.Provider
    import au.com.dius.pact.provider.junit.State
    import au.com.dius.pact.provider.junit.loader.PactFolder
    import au.com.dius.pact.provider.junit.target.HttpTarget
    import au.com.dius.pact.provider.junit.target.Target
    import au.com.dius.pact.provider.junit.target.TestTarget
    import au.com.dius.pact.provider.spring.SpringRestPactRunner
    import org.junit.Before
    import org.junit.runner.RunWith
    import org.springframework.beans.factory.annotation.Autowired
    import org.springframework.boot.test.context.SpringBootTest


    @RunWith(SpringRestPactRunner::class)
    @Provider("UserManager")
    @Consumer("Quoki")
    @PactFolder("../consumer/src/consumer/http/pacts")
    @SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT,
            properties = ["server.port=8601"])
    class QuokiUserAuthenticatePactItTest {

        @Suppress("unused")
        @JvmField
        @TestTarget
        final val target: Target = HttpTarget(port=8601)

        @Autowired
        private lateinit var userRepository: UserRepository

        @Before
        fun setUp() {
            userRepository.deleteAll()
        }

        @State("user exists")
        fun userExists() {
            userRepository.save(QuokiUser(userName = "alice", password = "s3cr3t"))
        }
    }

Running this will provide the result

    Verifying a pact between Quoki and UserManager
    Given user exists
        a request to authenticate
    returns a response which
        has status code 204 (OK)
        has a matching body (OK)

Best Practices

A Bad Pact

A bad pact, dictates what it considers as a precondition for an invalid username and tightly couples the exact error message that is expected.

This is then unit testing the provider's implementation detail rather than the contract itself. So the provider cannot change the validation rule or the error message without breaking the consumer.

Given : Alice does not exist

Upon receiving: A request to create user, with user name that has special characters other than underscores

Response: Is 400 Bad Request

Response Body: {"error": "username cannot contain special characters"}

A Good Pact

A Good pact from a consumer hides the provider's implementation details. It must only capture expectation from consumer point of view. In the below example a consumer shall not dictate what it considers an invalid username.

Given : Alice does not exist

Upon receiving: A request to create user with invalid username

Response: Is 400 Bad Request

Response Body: {"error": "[a non empty string]"}

Conclusion

If you have more questions or if you need more information, you can find all the information here

Spring boot + Spring Security 5 + OAuth2/OIDC Client - Deep Dive

Shyamala — Mon, 25 Mar 2019 21:51:04 +0000

In my previous post we saw how easy it is to protect your application with Google Login.

Now let us see what are all the components responsible for this to work.

The Login process

Request to access the protected Endpoint and Google Authentication process starts

Prerequisites

application.yml is configured with client and provider values
Provider name in the property spring.security.oauth2.client.registration.provider is set to google

When access to http://localhost:8080/me is requested, If you have only one identity provider configured then, Spring redirects you automatically to http://localhost:8080/oauth2/authorization/google. OAuth2AuthorizationRequestRedirectFilter which is registered to the url pattern /oauth2/authorization/* will load the respective configuration and redirect to the Identity Provider. In our case Google.

If you want your users to choose between multiple providers, then configure your application.yml for multiple providers, but you need a login page where you can have multiple links for the users to choose from.

On Successful authentication google redirects to the app's redirect url

Once the user authenticates with Google successfully, Google now redirects to the app's redirect url configured in Google's developer console. In our example we chose to have a particular url http://localhost:8080/login/oauth2/code/google. This is because the Authentication Processing filter for OAuth2 OAuth2LoginAuthenticationFilter is registered to listen to /login/oauth2/code/*.

OAuth2LoginAuthenticationFilter delegates authentication to OidcAuthorizationCodeAuthenticationProvider which does 3 things:

Exchanges Code for token
Validates id_token
Populates User Info by calling the User Info endpoint, from Google's well known configuration

Now you might ask what if I have registered a different redirect URI, and want OAuth2LoginAuthenticationFilter to listen to this. It is pretty simple all you need to do is have the following Security Configuration

@Configuration
@EnableWebSecurity
class SecurityConfiguration: WebSecurityConfigurerAdapter() {
    override fun configure(http: HttpSecurity) {
        http
                .authorizeRequests()
                .anyRequest()
                .authenticated()
                .and()
                .oauth2Login()
                .redirectionEndpoint()
                .baseUri("/oauth/callback/*")

    }
}

What Next

As a result of successful authentication, you will get an Authentication object of type OAuth2AuthenticationToken. This token will contain all the necessary information from id_token and the user info endpoint.

You can access all data about the logged in user by

SecurityContextHolder.getContext().authentication as OAuth2AuthenticationToken

 @GetMapping("/me")
 fun hello(currentUser: OAuth2AuthenticationToken): ResponseEntity<OAuth2AuthenticationToken> {
        return ResponseEntity.ok(currentUser)
}

But what about Access and Refresh Tokens?

As a result of successful OpenID Connect flow, a client application receives three tokens, access_token, refresh_token and id_token. We might want to use this access token to access some protected resource from a resource server like tasks API of google. The OAuth2AuthorizedClientService keeps track of the tokens associated with the user.

 val currentUser = SecurityContextHolder.getContext().authentication as OAuth2AuthenticationToken
 val currentUserClientConfig = oAuth2AuthorizedClientService.loadAuthorizedClient(
                authorizedClientRegistrationId,
                currentUser.name)
  println("AccessToken: ${currentUserClientConfig.accessToken.tokenValue}")
  println("RefreshToken: ${currentUserClientConfig.refreshToken.tokenValue}")

But Access Tokens can expire

When access tokens expire, the resource server like like tasks API of google will return 401 HTTP status, the simplest solution is to throw an OAuth2AuthorizationException which is a type of AuthenticationException that will trigger the login flow again.

But we can also use Refresh Tokens to automatically refresh our tokens, by customizing RestTemplate with a request interceptor that will refresh the tokens on expiry

class BearerTokenInterceptor(private val oAuth2AuthorizedClientService: OAuth2AuthorizedClientService) : ClientHttpRequestInterceptor {

    companion object {
        val log: Logger = LoggerFactory.getLogger(BearerTokenInterceptor::class.java)
    }

    private var accessTokenExpiresSkew = Duration.ofMinutes(1)
    private val clock = Clock.systemUTC()

    override fun intercept(request: HttpRequest, body: ByteArray, execution: ClientHttpRequestExecution): ClientHttpResponse {
        val currentUser = SecurityContextHolder.getContext().authentication as OAuth2AuthenticationToken
        val currentUserClientConfig = currentUser.clientConfig()

        if (isExpired(accessToken = currentUserClientConfig.accessToken)) {
            log.info("AccessToken expired, refreshing automatically")
            refreshToken(currentUserClientConfig, currentUser)
        }

        request.headers[AUTHORIZATION] = "Bearer ${currentUserClientConfig.accessToken.tokenValue}"

        return execution.execute(request, body)
    }

    private fun OAuth2AuthenticationToken.clientConfig(): OAuth2AuthorizedClient {
        return oAuth2AuthorizedClientService.loadAuthorizedClient(
                authorizedClientRegistrationId,
                name) ?: throw CredentialsExpiredException("could not load client config for $name, reauthenticate")
    }

    private fun refreshToken(currentClient: OAuth2AuthorizedClient, currentUser: OAuth2AuthenticationToken) {
        val atr = refreshTokenClient(currentClient)
        if (atr == null || atr.accessToken == null) {
            log.info("Failed to refresh token for ${currentUser.name}")
            return
        }

        val refreshToken = atr.refreshToken ?: currentClient.refreshToken
        val updatedClient = OAuth2AuthorizedClient(
                currentClient.clientRegistration,
                currentClient.principalName,
                atr.accessToken,
                refreshToken
        )

        oAuth2AuthorizedClientService.saveAuthorizedClient(updatedClient, currentUser)
    }

    private fun refreshTokenClient(currentClient: OAuth2AuthorizedClient): OAuth2AccessTokenResponse? {

        val formParameters = LinkedMultiValueMap<String, String>()
        formParameters.add(OAuth2ParameterNames.GRANT_TYPE, AuthorizationGrantType.REFRESH_TOKEN.value)
        formParameters.add(OAuth2ParameterNames.REFRESH_TOKEN, currentClient.refreshToken?.tokenValue)
        formParameters.add(OAuth2ParameterNames.REDIRECT_URI, currentClient.clientRegistration.redirectUriTemplate)


        val requestEntity = RequestEntity
                .post(URI.create(currentClient.clientRegistration.providerDetails.tokenUri))
                .header(CONTENT_TYPE, APPLICATION_FORM_URLENCODED_VALUE)
                .body(formParameters)

        return try {
            val r = restTemplate(currentClient.clientRegistration.clientId, currentClient.clientRegistration.clientSecret)
            val responseEntity = r.exchange(requestEntity, OAuth2AccessTokenResponse::class.java)
            responseEntity.body
        } catch (e: OAuth2AuthorizationException) {
            log.error("Unable to refresh token ${e.error.errorCode}")
            throw OAuth2AuthenticationException(e.error, e)
        }
    }

    private fun isExpired(accessToken: OAuth2AccessToken): Boolean {
        val now = this.clock.instant()
        val expiresAt = accessToken.expiresAt ?: return false
        return now.isAfter(expiresAt.minus(this.accessTokenExpiresSkew))
    }

    private fun restTemplate(clientId: String, clientSecret: String): RestTemplate {
        return RestTemplateBuilder()
                .additionalMessageConverters(
                        FormHttpMessageConverter(),
                        OAuth2AccessTokenResponseHttpMessageConverter())
                .errorHandler(OAuth2ErrorResponseErrorHandler())
                .basicAuthentication(clientId, clientSecret)
                .build()
    }

}

So far I have not found that the oauth2-client can automatically refresh tokens within the user session, Let me know if this is the case :)

Conclusion

I tried to put together all pieces involved, Please give me feedback if I missed something :)

Spring boot + Spring Security 5 + OAuth2/OIDC Client - Basics

Shyamala — Sun, 24 Mar 2019 17:45:04 +0000

Since a long time I wanted to integrate an OpenID Connect provider using Spring Security, The last time I tried, I felt it was very complicated and wrote my own library. Since Spring Security 5 has native support for OAuth2 Client and extended its use for OpenID connect, I wanted to see how easy it is to integrate.

For this example we are going to build a simple app, the redirects to google when we try to access a protected endpoint

Step 1:

Create a spring boot project from https://start.spring.io with following dependencies

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-oauth2-client'
    implementation 'org.springframework.boot:spring-boot-starter-security'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'com.fasterxml.jackson.module:jackson-module-kotlin'
    implementation 'org.jetbrains.kotlin:kotlin-reflect'
    implementation 'org.jetbrains.kotlin:kotlin-stdlib-jdk8'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    testImplementation 'org.springframework.security:spring-security-test'
}

Step 2:

Create an endpoint that will show current user's authentication data

@RestController
class HelloController {

    @GetMapping("/me")
    fun hello(currentUser: OAuth2AuthenticationToken): ResponseEntity<OAuth2AuthenticationToken> {
        return ResponseEntity.ok(currentUser)
    }

}

Step 3:

Configure OAuth2 Client information in application.yml. In google's developer console configure the app's redirect uri as http://localhost:8080/login/oauth2/code/google

# @see https://console.developers.google.com/apis/ to create your client credentials
logging.level.org.springframework: INFO
spring:
  security:
    oauth2:
      client:
        registration:
          google:
            provider: google
            client-id: <<your-client-id>>
            client-secret: <<your-client-secret>> 
            client-authentication-method: basic
            authorization-grant-type: authorization_code
            scope:
              - openid
              - email
              - profile
              - https://www.googleapis.com/auth/tasks.readonly
        provider:
          google:
            issuer-uri: https://accounts.google.com

Step 4:

Run the application, goto http://localhost:8080/me , complete the login process and you will see this.

{
"authorities": [
{
"authority": "ROLE_USER",
"attributes": {
"at_hash": "28AV0o6xKM8f3UQlljlGuw",
"sub": "10080000000000000",
"email_verified": true,
"iss": "https://accounts.google.com",
"given_name": "Syamala",
"locale": "en",
"picture": "https://lh6.googleusercontent.com/photo.jpg",
"aud": [
"client-id"
],
"azp": "client-id",
"name": "Syamala Umamaheswaran",
"exp": "2019-03-24T18:27:19Z",
"family_name": "Umamaheswaran",
"iat": "2019-03-24T17:27:19Z",
"email": "xxxx@gmail.com"
},
"idToken": {...},
"userInfo": null
}
],
"details": null,
"authenticated": true,
"principal": {},
"authorizedClientRegistrationId": "google",
"credentials": "",
"name": "10080000000000000"
}

Mind Blown:

As much as it blows my mind that without writing any code for security we are able to integrate with an OpenID Connect provider, I needed to know how this is working so easily. The Devil is in the details, Stay tuned for my next blog post where I explain the behind the scenes and How to access a protected resource and how to refresh tokens automatically.

Complete Source Code @ https://github.com/shyamz-22/oidc-spring-security-5

Beyond the login screen - Part II

Shyamala — Mon, 12 Mar 2018 03:03:45 +0000

In Previous post, we reasoned around why transparency while collecting user data is important. In this part let us dig deeper into one of the Open Standards OpenID Connect and see how the standard gives flexibility and transparency around user data management.

The Familiar OAuth2 dance

OpenID Connect is built on top of OAuth2,

Before jumping on to technicalities of OpenID Connect, let us brush our memory on how we (The Users) are used to signing in with Google (The Identity Provider) to Meetup (The App).

Move 1: User goes to Meetup app and clicks on log in with Google

Move 2: Meetup app then asks Google to authenticate the user and share the basic profile of the User

Move 3: Google then asks the user to login

Move 4: On successful authentication of the user, Google now asks User if it is ok for them to share their basic profile with Meetup and User consents to Google to share their data to Meetup

Move 5: Google now shares an ephemeral Key to Meetup that can unlock the User's consented data

Move 6: Meetup uses the key to get Basic profile data.

What User Data?

At Move 2, Meetup knows that it only needs basic profile data like Name, email and profile picture to complete its user profile.

Why User Data?

At Move 4, Google asks the User's consent to share the data to Meetup explaining that this will be used as a profile.

Flexibility and Transparency

Now replace Google with your own Identity Provider, then you have achieved a new level of flexibility towards transparent data management.

Assume Meetup needs this new feature, that can show meetups nearby, So now Meetup app needs to know the postal code of the user, all Meetup app now needs to do is request for a new scope pincode along with basic_profile and now the Identity Provider would ask consent for the newly added scope to user.

Behind the scenes

If you know the dance, you can skip to Conclusion. If you want to learn the moves, continue reading

Move 1

User goes to Meetup app and clicks on log in with Google

In order for this to be possible,The App should register itself with the Identity Provider, by providing minimum the Name it wants its Users to see in Move 4 and a url for the Identity Provider to communicate the result to.

This is Client Registration. As a result of registration now the App gets an Id and a Secret.

Move 2

Meetup app then asks Google to authenticate the user and share the basic profile of the User

Now The app forms a request with its Id and registered Url and requests for a scope e.g. Basic Profile using response_type one of the three flows (Basic, Implicit or Hybrid) to receive the user's data

This is an Authentication Request.

Move 3

Google then asks the user to login

At this point The Identity Provider checks for the validity of client (id and registered url), and asks the user to login.

This is an Identity Provider doing its identity thing.

Move 4

Google now asks User if it is ok for them to share their basic profile with Meetup

On successful authentication, now Identity Provider shows the consent screen to the user describing the
what data the app needs and why.

This is Getting the user's consent

How user gets authenticated or consents is out of scope for the OpenID Connect standard.

Move 5

Google now shares an ephemeral Key to Meetup that can unlock the User's consented data

On successful authentication and receiving user's consent, Now Identity Provider has to share this with the App. How this information is shared to the app is decided by the response_type value provided by the app in Move 1

The end result of any flow (Basic, Implicit, Hybrid) is generating one or more of access_token refresh_token and id_token.

This is successful token response from token endpoint for Basic and Hybrid flows or authorize endpoint for Implicit flow.

Choosing flows

Move 6

Meetup uses the key to get Basic profile data.

Now the App can use this short lived access_token to get information about user.

This is User Info endpoint

The Questions we need to ask,

So Identity Provider can validate App and User, But How does the App know

It is indeed The Identity Provider that authenticated the user and responded to it?
How long since the user was last authenticated? Given all the SSO convenience going on?
With which level of security (1FA or 2FA) the user was authenticated? Given the App deals with of course money, we need to have iron clad security don't we?

The Answer is `id_token`

This is what makes OpenID Connect different about OAuth2, It provides an Identity layer on top of the OAuth2 standard. It gives feedback to the App about the authenticated user.

You can think of id_token as digital secure identity card of the user. It is a JWT token signed by the Identity Provider's private key. The Public key is shared through keys endpoint

Conclusion

By abstracting away authentication and authorization data of the user we can achieve flexibility on how and when to collect the data there by building up a User Profile. The Identity Provider can define scopes, that allows an App to not only get user data but also to write user data through Scope Management.

And the standard gives you enough trust and confidence that the data is being exchanged between the App, User and Identity Provider in a secure and transparent manner.

Beyond the login screen - Part I

Shyamala — Fri, 09 Mar 2018 11:49:15 +0000

Recently I came across this tweet,

██ Oliver █ L██████

@eey0re

If you're collecting personal data, "how should I protect this?" is actually your third question.

"Should I collect this?" is only the second question.

The first question is "what would the worst people do if they got hold of this?"

03:49 AM - 04 Mar 2018

This tweet got me thinking, Personal data even how trivial it is can be dangerous in the hands of a wrong person. But can we stop collecting data about the user? Can we say with certainty that the system we are building is impenetrable?

We as part of a product team has responsibility to be transparent to our users about What data of the user our product needs and Why the user has to share it, So the user is conscious of the data being shared and there by implications.

Let us not be very greedy to ask for all data at once from the user.

I think we can also apply lean principle in collecting the data from the user, by asking for user's data only when we need them and of course explaining the Whats and Whys.

We can craft a unique solution for each of our suite of products ourselves or we can abstract away the method of collecting some of the most common user data that we collect in a safe, secure and trusted way.

There are a number of Open standards for exchanging the authentication and authorization data between users and services/products. Most popular among them are SAML, OAuth2 and OpenId Connect. These standards gives us the trust that a user data is exchanged in a secure manner that is transparent to the user.

Let us bring a change in thought process that authentication or authorization is not mere a login screen. Let us think about these aspects early in our product design phase.

So may be it is time for us to think about these Federated Authentication and Authorization standards, that gives users the right to grant and revoke access to their data at any point?

In the next part , I will go deeper in to one of these Open Standards OpenID Connect..

Hi, I'm Shyamala

Shyamala — Tue, 14 Mar 2017 08:00:41 +0000

I have been coding for 6 years.

You can find me on Twitter as @shyamala_u

I mostly program in these language: Java.

I am currently learning more about functional programming

Nice to meet you.

Forem: Shyamala

The case of disappearing metrics in Kubernetes

🧐 Context 🧐

⚠️ Disclaimer ⚠️

TL;DR

Debugging Step 1:

🤔 What does this mean? 🤔

Debugging Step 2:

🤔 What does this mean? 🤔

Debugging Step 3:

🤔 What does this mean? 🤔

Debugging Step 4:

🤔 What does this mean? 🤔

Debugging Step 5:

🤔 What does this mean? 🤔

Remove/propose different "no metrics known for pod" log #349

Debugging Step 5:

Comment for #55

Finally

🤔 What does this mean? 🤔

Lessons Learned

Microservices - Why wait for your Integration tests to fail when you have Pact?

What is a Pact?

Parties

How does a Pact work?

Advantages of using Pacts

Isolation:

Quick Feedback

User Centred API design

Overview of existing APIs

Language independent

Pact in Action:

An Example Pact

Consumer Pact in Go:

Provider verification of Pact in Kotlin

Best Practices

A Bad Pact

A Good Pact

Conclusion

Spring boot + Spring Security 5 + OAuth2/OIDC Client - Deep Dive

The Login process

Request to access the protected Endpoint and Google Authentication process starts

On Successful authentication google redirects to the app's redirect url

What Next

But what about Access and Refresh Tokens?

But Access Tokens can expire

Conclusion

Spring boot + Spring Security 5 + OAuth2/OIDC Client - Basics

Step 1:

Step 2:

Step 3:

Step 4:

Mind Blown:

Beyond the login screen - Part II

The Familiar OAuth2 dance

What User Data?

Why User Data?

Flexibility and Transparency

Behind the scenes

Move 1

Move 2

Move 3

Move 4

Move 5

Choosing flows

Move 6

The Questions we need to ask,

The Answer is id_token

Conclusion

Beyond the login screen - Part I

Hi, I'm Shyamala

The Answer is `id_token`