Forem: Vikas Garg

Using PingAccess and PingFederate for Authentication and Authorization

Vikas Garg — Thu, 30 Sep 2021 19:49:04 +0000

Recently, I came across PingIdentity's solution for authorization and authentication. There are multiple services provided by them, two of those are PingFederate and PingAccess. PingFederate, in simple terms, is the token provider. PingAccess is the gateway/proxy to your underlying services or APIs. More information on these two can be found here.

In this post, I will focus more on what needs to happen in the services right from configuration and any code changes.

Don't do that. Don't give me hope

PingAccess and PingFederate
Services
Getting the Token
Call the APIs
Decode the Token
Conclusion

PingAccess and PingFederate

First things first, you will need access to PingAccess and PingFederate. You can install them on your local machine as standalone services OR can have them running inside docker container. Ping provides pre-built docker images for each of their services. More info for the docker images can be found here.

For this post, I have these running inside docker containers. These containers need ping credentials to fetch the license information. You can get the credentials by registering on devops.pingidentity.com. Once you have registered, ping will send you your credentials which then needs to be passed to PingAccess and PingFederate containers either via environment variables (PING_IDENTITY_DEVOPS_USER and PING_IDENTITY_DEVOPS_KEY). You can either add these to the docker-compose OR use a .env file which is passed to docker-compose up command OR use ping-devops utility. Sample .env file will look like this,



PING_IDENTITY_DEVOPS_USER=<user name>
PING_IDENTITY_DEVOPS_KEY=<user key>

and then you will start your containers like, docker-compose --env-file .env up -d.

Once the containers are started, you will need to orchestrate both PingAccess and PingFederate with the information related to your service including the host and port the services will be available, client-id/secret for token, signing certificate etc. All the setup required for this post can be found here.
Who wants Infinity Stones?

Services

For this post, I am using 3 Spring Boot services but you are not restricted to using the same but can use services in other languages like nodeJS. The 3 services are,

bff-service -- This service acts as a gateway to the other 2 API services by fetching and using the auth token from PingFederate.
todo-service -- This is a simple service providing APIs to fetch and add todos. The service is behind the PingAccess proxy layer.
tweet-service -- This is a simple service providing APIs to fetch and add tweets (not making call to twitter). The service is behind the PingAccess proxy layer.

Getting the Token

In order to make calls to any of todo-service or tweet-service, we will need to get the token from PingFederate. For that to happen, we will need to the following pieces, clientId, clientSecret, grantType and audience. PingFederate provides an API to get the token after providing the required parameters in the request body.

Let's setup bff-service to get the token. In application.yml,



auth:
  tokenEndpoint: https://pingfederate:9031/as/token.oauth2
  grantType: client_credentials

todo:
  serviceUrl: https://todo.pa.local:3000
  audience: https://todo.pa.local:3000
  clientId: todo_client
  clientSecret: toomanysecrets

tweet:
  serviceUrl: https://tweet.pa.local:3000
  audience: https://tweet.pa.local:3000
  clientId: tweet_client
  clientSecret: toomanysecrets

tokenEndpoint - This is the PingFederate API endpoint to get the JWT token.
serviceUrl - This is the base url for your API service. This should be of the pattern: <virtual-host-on-pingaccess>:<application-port-for-pingaccess>. This url is same as what you set under Applications on PingAccess.

The clientId and secret for individual services are the same ones you have setup on PingFederate, whereas the audience should match to what has been setup in the Access Token Manager on PingFederate.

To fetch the token, you will need to add something like this in your code, add a Token class,



public class Token {
    @JsonProperty(value = "access_token")
    private final String accessToken;

    @JsonProperty(value = "token_type")
    private final String tokenType;

    @JsonProperty(value = "expires_in")
    private final int expiresIn;
    private final long expireAt;
}

Then add this method somewhere in your code,



public String getToken(String clientId, String clientSecret, String audience, String grantType) {
    HttpHeaders httpHeaders = new HttpHeaders();
httpHeaders.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
    MultiValueMap<String, String> map = new LinkedMultiValueMap<>();
    map.add("client_id", clientId);
    map.add("client_secret", clientSecret);
    map.add("audience", audience);
    map.add("grant_type", grantType);
    ResponseEntity<Token> response =
            restTemplate.postForEntity(tokenEndpoint, new HttpEntity<>(map, httpHeaders), Token.class);
    return response.getBody().getAccessToken();
}

Now before making any request to any of the todo or tweet service, we will first fetch the token and then pass it along.

Call the APIs

Now we have seen how we can get the token. Next we will see how to use the token to make the API requests. In your code, you can create your request like,



public HttpEntity<String> getRequest(String clientId, String clientSecret, String audience) {
    String accessToken = tokenUtil.getToken( clientId, clientSecret, audience, grantType);
    HttpHeaders httpHeaders = new HttpHeaders();
    httpHeaders.add(HttpHeaders.AUTHORIZATION, "Bearer " + accessToken);
    httpHeaders.setContentType(MediaType.APPLICATION_JSON);
    return new HttpEntity<>(httpHeaders);
}

That's it. Once you have this setup, you can pass the request to any GET API (for POST APIs you will need to add the request body as well) to get the requested data.

Decode the Token

Now, since the request has been made from bff-service to your API services via PingAccess using the token provided by PingFederate, there are 2 things you can do on the API Services.

You can ignore the token altogether, given that the request is coming via PingAccess which has already established that the token is valid and only then it has let the request pass through. This is a BIG security loophole, as anyone who can get the knowledge of your API can exploit it by bypassing the PingAccess altogether by directly making the API requests.
Check and verify the token. This will involve decoding the token and then performing any rule checks like audience matches, expiration and any other scope checks. Let's take a look at the config and code changes required on the individual service.



spring:
  security:
    oauth2:
      resourceserver:
        jwt:
          jwk-set-uri: https://pingfederate:9031/ext/todoauthtoken/jwks
          issuer-uri: https://pingfederate:9031

auth:
  audience: todo.pa.local:8082

jwk-set-uri - This is the jwks url where the keys for your token is published on PingFederate.
issuer-uri - This is the token issuer. In our case it is PingFederate
auth.audience - This is the audience we are expecting in the token payload.



@Override
protected void configure(HttpSecurity http) throws Exception {        http.authorizeRequests().mvcMatchers("/todos/**").authenticated().and().oauth2ResourceServer().jwt();
}

@Bean
public JwtDecoder jwtDecoder() {
    NimbusJwtDecoder jwtDecoder = NimbusJwtDecoder.withJwkSetUri(jwkSetUri).build();

    OAuth2TokenValidator<Jwt> audienceValidator = new AudienceValidator(audience);
    OAuth2TokenValidator<Jwt> withIssuer = JwtValidators.createDefaultWithIssuer(issuer);
    OAuth2TokenValidator<Jwt> withAudience = new DelegatingOAuth2TokenValidator<>(withIssuer, audienceValidator);

    jwtDecoder.setJwtValidator(withAudience);

    return jwtDecoder;
}

Add the above bean in your spring config. The configure method is telling Spring that the given APIs needs to be authenticated with a JWT else the request will be rejected. The jwtDecoder defined here will make use of the provided jwkSetUri, issuer and the custom validator (AudienceValidator) to validate the incoming token.

Doing just this will secure your APIs with JWT and will reject the requests with no Authorization or invalid JWTs.

Conclusion

Above approach is a sort of generic approach followed to secure your APIs using a Token Provider and Resource Server. Here I have used PingFederate but it can be replaced with any other available Token Providers like auth0, Okta etc.

The full working code used in this post can be found here:

vikasgarghb / blog-ping-id

Application integration with PingAccess and PingFederate

I see this as an absolute win

Feature Flags with Spring Boot

Vikas Garg — Tue, 07 Sep 2021 13:57:14 +0000

Feature Flags is a very broad topic and have been coming back up regularly irrespective of what language you use. There are multiple platforms available which provide extensive support for managing your features external to the code or deployed as part of the code. Some of the examples are,

LaunchDarkly
Togglz
Property (File) based and many more.

Problem at hand
Project Setup
Dependencies
Feature Loading
Enhancement
Remarks

Problem at hand

This article specifically targets a case where you can add a toggle check before any method you want before actually getting into the method execution. Be it at a controller level to prevent an API to be executed OR a service method OR a repository method. This pattern can be extended with any feature toggle implementation you use.

Code for the blog can be found here.

vikasgarghb / blog-feature-flags-spring

Feature Flag Annotation usage with Spring Boot.

Project Setup

To begin with you will need,

Java 11+
Gradle 7+
IDE of your choice
Spring Boot 2.5.4+

Dependencies

Apart from the other spring dependencies, you will need to include spring-aop and aspectjweaver. There are much documentation available for Aspect based programming with Spring. Won't go into much details about it. But one thing which this article focuses on is how aspect can be used to call a method before executing actual method.

AspectJ's @Before annotation allows us to accomplish this. As long as correct pointcut expression is provided, the method decorated with this annotation will be called before the actual method.

For eg,



@Before("execution (* blog.vgarg.features..*(..)) && @annotation(checkFeatureFlag)")
public void checkFeatureFlag(JoinPoint joinPoint, CheckFeatureFlag checkFeatureFlag) {
    String flag = checkFeatureFlag.flag();
    if (!featureFlagService.isFeatureFlagSet(flag)) {
        throw new FeatureNotEnabledException();
    }
}



@CheckFeatureFlag(flag = "flags.goodmorning")
@GetMapping("/good-morning")
public ResponseEntity<String> getGoodMorning() {
    return new ResponseEntity<>(greetingService.getGoodMorning(), HttpStatus.OK);
}

checkFeatureFlag method will be executed by spring before getGoodMorning method gets executed.

Feature Loading

So, for this post, I am using file based features but the approach can be extended to use with any other feature provider.



public interface FeatureFlagService {
    boolean isFeatureFlagSet(String flag);
}

I have defined an interface with a simple single method to check the status of any given feature flag. As mentioned for this post the implementation reads the toggle from a file named, featureflags.properties residing in src/main/resources (can be anywhere as long as it is on classpath).

Enhancement

As mentioned before this can be extended for other feature support providers. For eg, with launch darkly the implementation will involve using the SDK provided by them. As long as the LD setup is done correct way, using their SDK is pretty simple. More can be read here.

Remarks

This is very bare minimum implementation of the functionality and has the capability of being extended into more. This only applies when the feature check needs to be applied before the actual method. For eg, if a new API needs to be deployed but not released, this check can come in handy.

But as any other solution this is not one size fits all solution. Would be happy to hear from the community what solutions have they tried.

Namespaced Stacks via SAM

Vikas Garg — Thu, 19 Nov 2020 21:22:22 +0000

AWS SAM is a nice tool when it comes to orchestrating a new serverless stack. It has its own quirks and shenanigans which needs to be worked around. One such was the lack of support for namespaced stacks, which could come in handy where you have multiple developers deploying in the same environment during development phase.

There can be only one!!

But thanks to --config-file samcli option, this has become possible. Here is my take on how we can establish this.

Create Sam Configuration File

You need to create a new configuration file, which will have the same contents as existing samconfig.toml file apart from some minor differences as highlighted below.

samconfig.toml

stack_name = "blog-sam-custom-namespace-dev"
s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-s7lq825ncn5b"
s3_prefix = "blog-sam-custom-namespace-dev"
region = "us-east-1"
capabilities = "CAPABILITY_NAMED_IAM"
parameter_overrides = """
  Environment=dev
  NodeEnv=dev
"""

samconfig-jdoe.toml

stack_name = "blog-sam-custom-namespace-jdoe"
s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-s7lq825ncn5b"
s3_prefix = "blog-sam-custom-namespace-jdoe"
region = "us-east-1"
capabilities = "CAPABILITY_NAMED_IAM"
parameter_overrides = """
  Environment=jdoe
  NodeEnv=jdoe
"""

Update your template

Update your template.yaml(or yml) like following,

S3UploaderFunction:
  Type: AWS::Serverless::Function
  Properties:
    CodeUri: build/s3-uploader
    Handler: index.handler
    Role: !Sub ${BlogSamCustomNamespaceRole.Arn}
    Environment:
      Variables:
        ENV: !Ref Environment
        NODE_ENV: !Ref NodeEnv

Note: If you have multiple functions, you can also move the environment variable declaration to Globals.

Handle or use environment in the code

Now, if you want to use the passed in environment, you can use it in following way (not limited to),

const S#_BUCKET = `blog-sam-custom-namespace-bucket-${process.env.ENV}`;

Deploy

Given you already have your AWS profile set and project built, you can run the following command to deploy,

sam deploy --config-file samconfig-jdoe.toml

If everything goes well, you will get the stack deployed ending with your provided namespace.

We are all the same

Full code can be found here.

RDS Proxy via SAM

Vikas Garg — Fri, 17 Jul 2020 20:21:27 +0000

Databases, Serverless, Connection Pooling all these terms are pretty cool in isolation but can be intimidating when tried to use together. And to add to that managing the orchestration of these via SAM or CDK makes it more.
Phoo, Phoo, Phoo

AWS introduced RDS Proxy to handle some of the issues related to Connection Pooling. In this post, I will try to orchestrate my application to use RDS Proxy via SAM.

Secret
IAM Role 1
RDS Proxy
Database Connection
IAM Role 2
Alternative to IAM Role 2

Secret

In order for accessing the database, you need to create a secret in AWS SecretsManager. The secret will be using the username and password for the user you want to use to connect the database. You will need to encrypt the secret. You can use it by using the default encryption key provided by AWS OR can create one yourself.
Can you keep the secret?

IAM Role 1

Now you have the secret, but your user will need to have permissions to access the secret. For that you will need to add the following policy to your existing role OR create a new one.

"Statement": [
  {
    "Sid": "GetSecretValue",
    "Action": [
      "secretsmanager:GetSecretValue"
    ],
    "Effect": "Allow",
    "Resource": [
      "arn:aws:secretsmanager:us-east-1:123456789012:secret:<secretId>"
    ]
  },
  {
    "Sid": "DecryptSecretValue",
    "Action": [
      "kms:Decrypt"
    ],
    "Effect": "Allow",
    "Resource": [
      "arn:aws:kms:us-east-1:123456789012:key/<keyId>"
    ],
    "Condition": {
      "StringEquals": {
        "kms:ViaService": "secretsmanager.<region>.amazonaws.com"
      }
    }
  }
]

What this will do is get your role permissions to access the secret value using the encryption key.

RDS Proxy

Now we need to setup the rds proxy. You can do this via Console OR also SAM, if you want to have separate proxy for each of your application (There is a limitation of 20 proxies per AWS Account, so might want to consider that before going this route). Anyways, here is how you can do it via SAM,

Resources:
 DBProxy:
   Type: AWS::RDS::DBProxy
   Properties:
     DBProxyName: rds-proxy-test
     EngineFamily: MYSQL
     RoleArn: arn:aws:iam::123456789012:role/<role-with-permissions-to-access-secret>
     RequireTLS: true
     Auth:
       - {AuthScheme: SECRETS, SecretArn: arn:aws:secretsmanager:us-east-1:123456789012:secret:secret-you-created-earlier, IAMAuth: ENABLED}
     VpcSubnetIds: <Comma-separated list of vpc ids setup for your DB> # Out of scope of this post

 ProxyTargetGroup: 
   Type: AWS::RDS::DBProxyTargetGroup
   Properties:
     DbProxyName: !Ref DBProxy
     TargetGroupName: default
     InstanceIdentifiers: 
       - Fn::ImportValue: DBInstanceName # Name of your RDS instance
   DependsOn: DBProxy

That's how you create it!!

Database Connection

Now that you have your database, user setup and proxy created, you would like to make use of it in your code. Here is what you need to do (I used typeorm and mysql2 for managing the connections and types),

const connectionOptions = {
  type: 'mysql',
  host: config.database.host, // This is the RDS Proxy host and NOT the db host
  port: 3306,
  username: config.database.username, // This needs to be the same user you have stored in secretsmanager 
  database: config.database.name,
  entities: [], // List of your entities
};

const getDBConfig = () => {
  // Since we have IAM enabled on our proxy, we will make use of the token to authenticate and connect to the db.
  const signer = new RDS.Signer({
    region: config.region,
    username: connectionOptions.username,
    hostname: connectionOptions.host,
    port: connectionOptions.port,
  });

  const token = signer.getAuthToken({
    username: connectionOptions.username,
  });

  return {
    ...connectionOptions,
    password: token,
    extra: {
      authPlugins: {
        mysql_clear_password: () => (): string => `${token}\0`,
      },
    },
    // Drop minimum TLS version supported to 1.0.
    ssl: {
      ...sslProfiles['Amazon RDS'],
      minVersion: 'TLSv1',
    },
  };
};

export const getDBConnection = async (): Promise<Connection> => {
  const CONNECTION_NAME = 'default';
  const connectionManager = getConnectionManager();
  let connection: Connection;

  if (connectionManager.has(CONNECTION_NAME)) {
    connection = connectionManager.get(CONNECTION_NAME);
    if (!connection.isConnected) {
      connection = await connection.connect();
    }
  } else {
    connection = await createConnection(getDBConfig());
  }

  return connection;
};

Not too bad. Seems pretty straightforward from configurations aspect.
Don't you say that...

Lambda

Your configuration is in place, now you just need to make your lambda do some db operations. It can be done simply like this,

export default const handler = async () => {
  try {
    const connection = await getConnection();
    // Perform db operations using connection
    ...
  } finally {
    await (await getDBConnection()).close();
  }
};

Simple!!! Yes, it is. But you will ask why are we explicitly closing the connection, shouldn't lambda take care of that for us? Well, the answer is Yes and No both.
I am listening!!!

So, if you let AWS take care of closing the proxy connection, that's fine but AWS will keep the connection open for some unknown time and that connection, if reused will run into expired token problem, given the IAM token is very short-lived. This will probably happen in a frequently requested service.
But on the other hand, if you close the proxy connection after each lambda execution explicitly, each new lambda execution will create a new connection to proxy and you will get a new token to use.
BAM

IAM Role 2

Next up is your lambda execution role. This role can be the same as the one you used earlier OR a different one. Whichever you choose to go with, just make sure you have this policy attached to it,

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "rds-db:connect"
      ],
      "Resource": [
        "arn:aws:rds-db:us-east-1:123456789012:dbuser:prx-123456789/db-user"
      ] 
    }
  ]
}

Thing to note here is that you will be using the proxy resource id and NOT the RDS database resource id. DB user should be the same as what you stored in the secretsmanager.

Alternative to IAM Role 2

If you don't want to update the execution role, you can also specify the policy directly on your lambda function. You can do this via SAM like this,

SomeLambdaFunction:
  Type: AWS::Serverless::Function
  Properties:
    Description: Database Operations Function
    FunctionName: db-function
    Handler: db.handler
    Role: !Sub ${SomeRole.Arn}
    Policies:
      - Version: '2012-10-17' 
        Statement:
          - Effect: Allow
            Action:
              - rds-db:connect
            Resource:
              - !Sub arn:aws:rds-db:us-east-1:123456789012:dbuser:prx-123456789/db-user

That's it. RDS Proxy can be a great tool to use in a system where you need to do db heavy operations like multiple reads in parallel OR you have a system with a state machine fanning out multiple requests. This takes the connection pooling overhead out of the equation and lets you focus on the business side of things.

Bazingaaa

ASL Expressions with Step Functions

Vikas Garg — Tue, 16 Jun 2020 18:33:01 +0000

Step Functions are a great tool to coordinate multiple AWS services into serverless workflows. Your workflows can be linear OR conditionals OR parallel OR mapped. Each step can be a different service and inputs are passed from each step to another. More on this can be read here.
Which is the odd one out?

So, this post is not about how step functions work and what can they do but in this post I would like to focus on how you can pass and process inputs. There are multiple ways to pass inputs between steps and process them before they are passed. By process, I mean validating any primitives OR clean any arrays etc.
filter it out!!!

ASL
Input Processing
Operations

ASL

Step functions definition can be defined using Amazon States Language. More about it can be found here. It is a JSON-based structured language which defines various states and tasks performed by those states. An example state machine definition can be:

{
  "Comment": "This is a simple state machine",
  "StartAt": "InitialState",
  "States": {
    "InitialState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:FUNCTION_NAME",
      "Next": "MiddleState"
    },
    "MiddleState": {
      "Type" : "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:FUNCTION_NAME2",
      "Next": "EndState"
    },
    "EndState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:FUNCTION_NAME3",
      "End": true
    }
  }
}

Input Processing

Each state can accept InputPath, OutputPath, Parameters and ResultPath parameters. More info on this can be found here. There are 2 types of input here:

One which is passed to the state machine. Eg,

{
  "comment": "Input comment.",
  "data": {
    "val1": 23,
    "val2": 17
  }
}

Other which passed to each step. Eg,

"MiddleState": {
  "Type" : "Task",
  "InputPath": "$.data",
  "Resource": "arn:aws:lambda:us-east-1:123456789012:function:FUNCTION_NAME2",
  "Next": "EndState"
}

{
  "val1": 23,
  "val2": 17
}

In the reminder of the post, we will talk about the 2nd type of input. $ is a referential operator to refer to original input. . allows you to access any property in the input. For eg, $.data refers to the data property in your input object.

Operations

As I mentioned earlier, you can pass inputs to each state in multiple ways. You can pass the output from previous step as input to next. You can skip the output from previous step and pass in the same input as previous step to next step. You can combine the output and input from previous step and then pass it entirely to next step. You can create your own input by picking properties from both input and output from previous step.
Confused!!!

Won't go into solving that confusion, it can be read here.

For the inputs validation or filtering, you can either handle that in your function, pass in the inputs as is and then in your code apply the validations.
Easy

But what if I don't want to do this in any function but want to handle it in the step definition itself. Yes, you can do that. ASL spec uses JSONPath to determine the inputs. Using JSONPath, you can accomplish some of the validations or filtering in the definition itself.

Subset of List

Sample Input

{
  "list": [1, 2, 3, 4, 5]
}

"InputPath": "$.list[1, 2]", will result into [2, 3] to be passed to the step.

Nested Property

{
  "person": {
    "name": "it",
    "email": "email",
    "phone": "phone"
  }
}

"InputPath": "$.person.email", will result into "email" to be passed to the step.

Nested Property for an item in the list

{
  "persons": [{
    "name": "it1",
    "email": "email1",
    "phone": "phone1"
  }, {
    "name": "it2",
    "email": "email2",
    "phone": "phone2"
  }, {
    "name": "it3",
    "email": "email3",
    "phone": "phone3"
  }]
}

"InputPath": "$.persons[0].email", will result into "email1" to be passed to the step.
"InputPath": "$.persons[*].email", will result into ["email1", "email2", "email3"] to be passed to the step.

Logical Filtering

{
  "scores": [
    {
      "math": 10,
      "english": 20
    },
    {
      "math": 15,
      "english": 20
    },
    {
      "math": 5,
      "english": 20
    },
    null,
    {
      "math": 20,
      "english": 20
    },
    null
  ]
}

"InputPath": "$.scores[(@.math > 10)]", will result into [{ "math": 15, "english": 20 }, { "math": 20, "english": 20 }] to be passed to the step.
"InputPath": "$.scores[(@ != null)]", will result into the same list without nulls to be passed to the step.

Existence Check

{
  "result": {
      "item": 12345,
    }
}

"InputPath": "$.[?(@.result)].result.item", will result into [12345]. Basically this checks the existence of property in the root object.

These are few of the examples which you can follow. A full list of supported operators can be found here.

You wish!!!

Streaming Files to S3 using Axios

Vikas Garg — Tue, 21 Apr 2020 14:35:36 +0000

Recently, ran into a simple yet interesting problem (well not that interesting) about streaming file from one S3 location to another S3 location. You will argue that why can't you just use the aws-sdk's copyObject method. Well, that method takes in source bucket/key and destination bucket/key which is fine but not available all the time. Like when file is coming from an external vendor.

Consider a scenario, where you just have the url of the file. It can be anything like an end-point to an html page, OR a pre-signed url to another S3 location etc. How to copy that file? Couple of approaches could be,

Manually download the file and then upload it to your S3 bucket.
Have a UI interface to download the file and then upload it via some backend service call.

How about we write a simple lambda function which takes in your url and destination S3 location and copies the file directly? Let's see how we can achieve that using available resources.

const downloadFile = async (downloadUrl: string): Promise<any> => {
  return axios.get(downloadUrl, {
    responseType: 'stream',
  });
};

const uploadFromStream = (fileResponse: AxiosResponse, fileName: string, bucket: string): PassThrough => {
  const s3 = new S3();
  const passThrough = new PassThrough();
  s3.upload(
    { Bucket: bucket, Key: fileName, ContentType: fileResponse.headers['content-type'], Body: passThrough },
    (err: Error, data: S3.ManagedUpload.SendData) => {
      if (err) {
        console.log(err);
        throw err;
      }
    },
  );
  return passThrough;
};

export const handler = async (event: CopyEvent) => {
  const response = await downloadFile(event.fileUrl);
  response.data.pipe(uploadFromStream(response, event.fileName, 'test-bucket'));

  return event.fileName;
};

So, now when you run this function, the function will end successfully. But when you go to your S3 bucket and check it, the file is not there.

Where is my file?

So, what went wrong? We didn't await on the s3.upload. The lambda function returned before the upload could finish. Let's add await and see what happens.

Wrong Solution 2

const uploadFromStream = async (fileResponse: AxiosResponse, fileName: string, bucket: string): Promise<PassThrough> => {
  const s3 = new S3();
  const passThrough = new PassThrough();
  await s3
    .upload({ Bucket: bucket, Key: fileName, ContentType: fileResponse.headers['content-type'], Body: passThrough })
    .promise();
  return passThrough;
};

export const handler = async (event: MovePdfEvent): Promise<string> => {
  const response = await downloadFile(event.fileUrl);
  const stream = await uploadFromStream(response, event.fileName, 'test-bucket');
  response.data.pipe(stream);
  return event.fileName;
};

So, now when you run this function, the function will end successfully. But when you go to your S3 bucket and check it, the file is still not there.

Where is my file, again?

So, what went wrong, again? This time we had await. Even after having await, we resolved the upload stream function before we passed it to stream here: response.data.pipe. So, how do we solve it? Thanks to @elthrasher , for figuring out the missing pieces. Let's see what they are.

Correct Solution

const uploadFromStream = (
  fileResponse: AxiosResponse,
  fileName: string,
  bucket: string,
): { passThrough: PassThrough; promise: Promise<S3.ManagedUpload.SendData> } => {
  const s3 = new S3();
  const passThrough = new PassThrough();
  const promise = s3
    .upload({
      Bucket: bucket,
      Key: fileName,
      ContentType: fileResponse.headers['content-type'],
      ContentLength: fileResponse.headers['content-length'],
      Body: passThrough,
    })
    .promise();
  return { passThrough, promise };
};

export const handler = async (event: CopyFileEvent): Promise<string> => {
  const responseStream = await downloadFile(event.fileUrl);

  const { passThrough, promise } = uploadFromStream(responseStream, event.fileName, 'test-bucket');

  responseStream.data.pipe(passThrough);

  return promise
    .then((result) => {
      return result.Location;
    })
    .catch((e) => {
      throw e;
    });
};

So, now when you run this function, the function will end successfully. But now when you go to your S3 bucket and check it, the file is there.

Yay!!!

So, what did we change? This time we returned the upload promise and also the pass through stream from our function and then piped that stream to response stream. And in the function we returned the upload promise, so that way the lambda function will wait for the promise to be either resolved or rejected before quitting.

Conclusion

This looks simple but as you saw it was a bit tricky coz async/await are great but can become hairy and a dev's nightmare real soon. But anyways, correct usage of these helped us solve our problem at hand.

Full code can be found here.

Step Functions with Localstack

Vikas Garg — Mon, 30 Mar 2020 21:27:21 +0000

AWS State Machines are a great tool to co-ordinate the components of a distributed applications and microservices using visual workflows. Using the aws console to build and deploy step functions is great and easy. But it makes local development somewhat slow. So, as usual, I invested some time in trying to make this whole setup work with localstack.

Tools
Functions
State Machine
Run everything

Tools

For this you will need to have,

Docker
Node12.X
awscli-local

Functions

We are not talking calculus
By functions, we are talking lambda functions. For our case, we will have 2 functions: createMatches and calculateRunRate. Both are pretty simple functions. createMatches takes in the following input



{
  teamA: string; // Team playing the match
  teamB: string; // 2nd team playing the match
  numMatches: number; // How many matches they are playing?
  matchType: MatchType; // Is it a T20 or ODI?
}

and it outputs an array of match results in the form,



{
  matchType: MatchType; // Is it a T20 or ODI?
  teamA: MatchScore; // How many runs scored, wickets lost, overs played?
  teamB: MatchScore;
}

calculateRunRate then takes in the output of createMatches function as input and calculates the run rate for each team and returns it as,



{
  teamA: number; // signed floating number
  teamB: number; // signed floating number
}

Run rates can be heart breaking

State Machine

Now that we have our lambda functions in place, lets create the state machine. This is what our machine will look like,



{
  "Comment": "State Machine to calculate net run rates for given teams.",
  "StartAt": "CreateMatches",
  "States": {
    "CreateMatches": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:CreateMatchesFunction",
      "Next": "CalculateRunRate"
    },
    "CalculateRunRate": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:CalculateRunRateFunction",
      "End": true
    }
  }
}

This is a pretty straight-forward json file. The specs for each property can be read here. The state machine needs to specify a starting point, in this case it is the CreateMatches step. And then it also needs to be specify the end step.
The bowling machine

docker-compose

Now we will write the docker-compose.yml. This is pretty straight-forward,



version: '3.7'

services:
  localstack:
    image: localstack/localstack
    environment:
      - DATA_DIR=/tmp/localstack/data
      - SERVICES=lambda
      - TZ=America/New_York
      - LAMBDA_EXECUTOR=docker
    ports:
      - '4574:4574'
    volumes:
      - /tmp/localstack:/tmp/localstack
      - /var/run/docker.sock:/var/run/docker.sock

  stepfunctions:
    container_name: stepfunctions
    image: amazon/aws-stepfunctions-local:latest
    environment:
      - LAMBDA_ENDPOINT=http://localstack:4574
    ports:
      - '8083:8083'

Things to note here that we are starting lambda service on localstack. More information on localstack can be found here. The service will be available on port 4574. Also pay attention to the LAMBDA_EXECUTOR and shared volumes. We need to do that coz with localstack the lambda executor needs docker to run. Next we have the stepfunctions service, which is built using the amazon/aws-stepfunctions-local:latest docker image. Here we need to specify the lambda endpoint. This is endpoint where the step functions will look for lambda functions to execute.
Where is the ball?

Run everything

In order to run this, we will have to run a few commands which can be put together is a script. To deploy the lambda functions to localstack, we will first need to zip them. You can do that by any zipping tool. I did it via zip command line utility.



zip -r ./createMatches.zip ./createMatchesHandler.js
zip -r ./calculateRunRate.zip ./calculateRunRateHandler.js

Next start the containers by running docker-compose up -d. Once the containers are up, you can deploy the lambda functions to localstack like this:



awslocal lambda create-function \
  --function-name CreateMatchesFunction \
  --runtime nodejs12.x \
  --handler createMatchesHandler.handler \
  --role arn:aws:iam::012345678901:role/DummyRole \
  --zip-file fileb://dist/createMatches.zip

awslocal lambda create-function \
  --function-name CalculateRunRateFunction \
  --runtime nodejs12.x \
  --handler calculateRunRateHandler.handler \
  --role arn:aws:iam::012345678901:role/DummyRole \
  --zip-file fileb://dist/calculateRunRate.zip

The output of these commands will look something like this,



{
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "CodeSha256": "iA5Rqi3x8PvmLWPZBvWXYrpOQC9tx6zi6TjpU9K8uKI=",
    "FunctionName": "CreateMatchesFunction",
    "LastModified": "2020-03-30T21:17:30.828+0000",
    "RevisionId": "70a91d5c-c42d-422e-9e93-fbc8d6ef638c",
    "CodeSize": 995,
    "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:CreateMatchesFunction",
    "Version": "$LATEST",
    "Role": "arn:aws:iam::012345678901:role/DummyRole",
    "Timeout": 3,
    "Handler": "createMatchesHandler.handler",
    "Runtime": "nodejs12.x",
    "Description": ""
}
{
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "CodeSha256": "GF2chlh/6N9mcPAyg6ysY537GsN+fff6zAuNeQUaz3Y=",
    "FunctionName": "CalculateRunRateFunction",
    "LastModified": "2020-03-30T21:17:31.469+0000",
    "RevisionId": "e1bfe0ab-bce5-4e21-b7c4-8a2bfa980a22",
    "CodeSize": 815,
    "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:CalculateRunRateFunction",
    "Version": "$LATEST",
    "Role": "arn:aws:iam::012345678901:role/DummyRole",
    "Timeout": 3,
    "Handler": "calculateRunRateHandler.handler",
    "Runtime": "nodejs12.x",
    "Description": ""
}

Ok, so now we have successfully deployed our lambda functions to localstack. Now we just need to create our state machine and execute the functions. For that, we can run the following commands,



aws stepfunctions \
  --endpoint http://localhost:8083 \
  create-state-machine \
  --definition file://state-machine.json \
  --name "NetRunRateCalculatorStateMachine" \
  --role-arn "arn:aws:iam::012345678901:role/DummyRole"

aws stepfunctions \
  --endpoint http://localhost:8083 \
  start-execution \
  --state-machine arn:aws:states:us-east-1:123456789012:stateMachine:NetRunRateCalculatorStateMachine \
  --name test
  --input "{\"teamA\":\"India\", \"teamB\": \"Australia\", \"numMatches\": 3, \"matchType\": \"t20\"}"

This will give you the following output,



{
    "creationDate": 1585603052.224,
    "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:NetRunRateCalculatorStateMachine"
}
{
    "startDate": 1585603052.766,
    "executionArn": "arn:aws:states:us-east-1:123456789012:execution:NetRunRateCalculatorStateMachine:test"
}

Now to check the status of the execution you can run the describe-execution utility on step functions, like this,



aws stepfunctions \
  --endpoint http://localhost:8083 \
  describe-execution \
  --execution-arn arn:aws:states:us-east-1:123456789012:execution:NetRunRateCalculatorStateMachine:test

If the machine already completed execution you will get the following output,



{
  "status": "SUCCEEDED",
  "startDate": 1585603052.766,
  "name": "test",
  "executionArn": "arn:aws:states:us-east-1:123456789012:execution:NetRunRateCalculatorStateMachine:test",
  "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:NetRunRateCalculatorStateMachine",
  "stopDate": 1585603055.931,
  "output": "{\"teamA\":-3.4393976210462283,\"teamB\":3.4393976210462283}
",
  "input": "{\"teamA\":\"India\", \"teamB\": \"Australia\", \"numMatches\": 3, \"matchType\": \"t20\"}"
}

That's it. You have successfully created a state machine and run it against the lambdas in localstack. You can find the full code here.
Success...

SAM Local with Localstack

Vikas Garg — Thu, 19 Mar 2020 16:00:08 +0000

As a developer, I look for making local dev experience as smooth as possible. In the same vein, one thing that bothers most of us working with aws cloud is to test it locally. Good thing is that we have localstack to our rescue. This post will focus on how you can have AWS SAM working with Localstack.

Tools
Project Setup
Setting up Localstack
Hooking it up

Tools

NodeJs 12.x
Docker
aws-cli
awscli-local

Project Setup

For this post, we will use sam init to pull in the hello-world project with nodejs runtime. More information on SAM. After following the prompts, you will get a sample hello world project setup running in node.

Setting up Localstack

In order to setup localstack in the project, we will first create a docker-compose.yml in the root of our project.



version: "3"
services:
  localstack:
    build: localstack
    container_name: localstack
    environment:
      - DATA_DIR=/tmp/localstack/data
      - SERVICES=s3 # this can be a comma separated list of services you want to start
      - TZ=America/New_York
    ports:
      - '4572:4572' # these ports will be different for different services.

Another thing that we will need to do is create a directory called localstack with following contents,



.dockerignore
Dockerfile
recorded_api_calls.json

Let's see what each of these files will look like, .dockerignore



.dockerignore
Dockerfile

Docekrfile



FROM localstack/localstack

COPY /. /tmp/localstack/data

We will get to the contents of the json file later. Once we have this, we are ready to start the localstack in a docker container. You can do this by running,
docker-compose up -d. This will pull the localstack image and build it and start it.

S3 bucket

Now we will need to create a s3 bucket in our localstack container. For this we can just run this: awslocal s3 mb s3://test-bucket. This will create the bucket. Now bash into the container, docker exec -it localstack /bin/bash and then go to vi /tmp/localstack/data/recorded_api_calls.json. Copy the contents of this file and paste them in the file recorded_api_calls.json in the localstack directory we created earlier. We need to do this, so that we don't need to create the bucket every time we start the container.

Hooking it up

Now if you want to setup your project to use any AWS components like S3, IAM, STS etc. you can do so and test them without going to AWS directly.

In the code, you will need aws-sdk to initialize the s3 client, you can do this by,



const s3 = AWS.S3({ endpoint: 'http://localhost:4572' });

Coming back to make sam local invoke work with localstack. For this, bare minimum you would need to do this:



sam local invoke --docker-network host

For more information on sam local

Full working code can be found here:

vikasgarghb / sam-local-with-localstack

sam-local-with-localstack

This project contains source code and supporting files for a serverless application that you can deploy with the SAM CLI. It includes the following files and folders.

hello-world - Code for the application's Lambda function.
events - Invocation events that you can use to invoke the function.
hello-world/tests - Unit tests for the application code.
template.yaml - A template that defines the application's AWS resources.

The application uses several AWS resources, including Lambda functions and an API Gateway API. These resources are defined in the template.yaml file in this project. You can update the template to add AWS resources through the same deployment process that updates your application code.

If you prefer to use an integrated development environment (IDE) to build and test your application, you can use the AWS Toolkit.
The AWS Toolkit is an open source plug-in for popular IDEs that uses the SAM CLI to build…

View on GitHub

Lambdas, SAM and Dotnet

Vikas Garg — Wed, 18 Mar 2020 13:02:23 +0000

Recently, had a chance encounter with lambdas and dotnet. AWS SAM provides a pretty good hello-world for dotnet application. So, I gave that a chance and below are some of my findings.

Dotnet SDK
Dotnet Tools
SAM Build
Dotnet Publish
Dotnet Lambda

Dotnet SDK

Currently, SAM works with dotnetcore2.0 and dotnetcore2.1. And the latest available version of dotnet SDK is 3.1. In order to work with the hello-world example of AWS SAM with dotnet, you would need at least sdk version 2.1 installed on your machine (I was able to make it work with sdk 3.1 too).

Dotnet Tools

You might also want to install aws dotnet tools. These are required by sam to create, build and deploy the lambda functions. If you don't have them installed, then you can do so by

dotnet tool install -g Amazon.Lambda.Tools

If you get an error that dotnet not found, then you might want to check the location /usr/local/share/dotnet/dotnet. This needs to have symlink to /usr/local/bin. In order to do that, just run
ln -s /usr/local/share/dotnet/dotnet /usr/local/bin

While running sam build, you might run into the following issue:

Could not execute because the specified command or file was not found.
Possible reasons for this include:
  * You misspelled a built-in dotnet command.
  * You intended to execute a .NET Core program, but dotnet-xyz does not exist.
  * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.

To resolve it, add the following $HOME/.dotnet/tools to your PATH variable. This should do it, export PATH=$PATH:$HOME/.dotnet/tools

SAM Build

Now when you run sam build, it will build the code and put the compiled files into .aws-sam/build directory and will also create the package to deploy to aws lambda via sam deploy. If all of your setup is correct, then you can either invoke this lambda locally via sam local invoke OR deploy it to AWS via sam deploy -g.

Dotnet Publish

If you don't want to run the sam build and have some more control over how you build your code, you can do that via dotnet publish. With this you can provide more options like runtime, configuration etc for the build. Also you can specify an alternate location for the output files. Only other thing you would need to do is that you will have to manually create the zip file to deploy to lambda. You can do that by using any compression tool (zip for unix, for eg.).

Dotnet Lambda

Another way you can do is via dotnet lambda. This feature is available via Dotnet Tools. Using this, you can combine the build and package steps together. This creates the deployable package which you can either deploy to lambda manually or via the tool itself (dotnet lambda deploy-function).

These were some of my findings and being an alien to dotnet universe, these things were very frustrating and tricky to resolve. So, thought of putting it all together in one place.

Sample code can be found here

vikasgarghb / dotnet-sam

dotnet-sam

This project contains source code and supporting files for a serverless application that you can deploy with the SAM CLI. It includes the following files and folders.

src - Code for the application's Lambda function.
events - Invocation events that you can use to invoke the function.
test - Unit tests for the application code.
template.yaml - A template that defines the application's AWS resources.

Deploy the sample application

The Serverless Application Model Command Line Interface (SAM CLI) is an extension of the AWS CLI that adds functionality for building and testing Lambda applications. It uses Docker to run your functions in an Amazon Linux…

View on GitHub

Polling with redux saga

Vikas Garg — Mon, 09 Mar 2020 19:30:45 +0000

I had been working on a solution to wait for a result to be available to update the results on a UI upon a user action. We are using react and redux. Looking at the other solutions available everywhere using redux-saga, they were all using an infinite loop and using specific redux-saga provided events to halt the polling. This made me think of making use of redux store and reducers to implement polling. So, this is my attempt to solve this problem.

What is polling?

Polling in general term means checking the status of something OR making repeated calls for a response. Eg. news ticker; twitter feeds etc.

Redux Saga

Redux Saga -- It uses an ES6 generator functions which are similar to async/await syntax.

Let's get to the business

For this, we are going to use a dummy service which have following APIs,

- POST /job -- Creates a job.
- GET /:jobId/status -- Returns the status of the job

Note: Not going into the details on the implementation of the APIs. Our focus will be on redux-saga implementation.

redux-saga

Let's see the implementation

export function* createJob(action) {
  const { payload } = action;

  yield post({
    url: '/job',
    onSuccess: ON_SUCCESS,
    onError: ON_ERROR,
    data: payload,
  });
}

export function* startPollingForStatus() {
  const job = yield select(state => getJob(state));
  yield put(getStatus(job.id));
}

export function* getStatus(action) {
  const { jobId } = action;

  yield get({
    url: `/${jobId}/status`,
    onSuccess: ON_SUCCESS_GET_STATUS,
    onError: ON_ERROR,
  });
}

export function* isJobComplete() {
  const job = yield select(state => getJob(state));
  if (job.status === 'complete') {
    return yield put(jobIsComplete(job.guid));
  }

  if (['created', 'processing'].includes(job.status)) {
    yield delay(4000);
    return yield put(getStatus(job.id));
  }

  yield put({
    type: ON_ERROR,
  });
}

export function* jobIsComplete() {
  return 'success';
}

export default function* root() {
  return yield [
    takeLatest(ON_CREATE_JOB, createJob),
    takeLatest(ON_GET_STATUS, getStatus),
    takeLatest(ON_SUCCESS, startPollingForStatus),
    takeLatest(ON_SUCCESS_GET_STATUS, isJobComplete),
    takeLatest(ON_ERROR, onError),
  ];
}

This might look a bit too much but let's break it down.

startPollingForStatus
As soon as the job is created and is returned successfully, we start the polling. As a result of the creation job, we have the returned job stored in redux store and then use the job id to fetch its status.

isJobComplete
Upon successful return of the status, we check for different statuses: created, complete and processing. If the status is not complete, we want to continue polling. So, we make use of delay function provided by redux-saga. This will delay for given time (in ms) and then resume the polling. If the status is complete, we will halt the polling and do something else to show the user that the job is complete.

Conclusion

This makes use of redux store and state and get us out of using an infinite loop to poll for updates. Let me know in comments if you see any issues with this approach.

Plan to add some samples for tests and github link for working solution. :)

UPDATE: Code can be found here

Forem: Vikas Garg

Using PingAccess and PingFederate for Authentication and Authorization

Contents

PingAccess and PingFederate

Services

Getting the Token

Call the APIs

Decode the Token

Conclusion

vikasgarghb / blog-ping-id

Application integration with PingAccess and PingFederate

Feature Flags with Spring Boot

Table Of Contents

Problem at hand

vikasgarghb / blog-feature-flags-spring

Feature Flag Annotation usage with Spring Boot.

Project Setup

Dependencies

Feature Loading

Enhancement

Remarks

Namespaced Stacks via SAM

Create Sam Configuration File

Update your template

Handle or use environment in the code

Deploy

RDS Proxy via SAM

Table Of Contents

Secret

IAM Role 1

RDS Proxy

Database Connection

Lambda

IAM Role 2

Alternative to IAM Role 2

ASL Expressions with Step Functions

Table of Contents

ASL

Input Processing

Operations

Subset of List

Nested Property

Nested Property for an item in the list

Logical Filtering

Existence Check

Streaming Files to S3 using Axios

TOC

PreRequisites

Wrong Solution 1

Wrong Solution 2

Correct Solution

Conclusion

Step Functions with Localstack

Table of Contents

Tools

Functions

State Machine

docker-compose

Run everything

SAM Local with Localstack

Table Of Contents

Tools

Project Setup

Setting up Localstack

S3 bucket

Hooking it up

vikasgarghb / sam-local-with-localstack

sam-local-with-localstack

Lambdas, SAM and Dotnet

Table of Contents

Dotnet SDK

Dotnet Tools

SAM Build

Dotnet Publish

Dotnet Lambda

vikasgarghb / dotnet-sam

dotnet-sam

Deploy the sample application

Polling with redux saga

What is polling?