Forem: Jeiman Jeya

🎯 🚀 The Ultimate Guide to Software Engineering, Zombie Apocalypse Style

Jeiman Jeya — Thu, 06 Apr 2023 05:56:42 +0000

This guide presents a comprehensive set of rules for software engineering inspired by the movie Zombieland. The rules are not simply a list of arbitrary guidelines, but rather a carefully crafted set of principles that are designed to promote efficiency and effectiveness in software development. By drawing inspiration from the movie, these principles are not only practical but also engaging and memorable.

The Rules in Zombieland

Rule #1: Cardio - Keep your code lean and agile to ensure quick and efficient processing.
Rule #2: Double Tap - Test your code thoroughly to ensure it's not just "undead" but truly dead before moving on.
Rule #3: Beware of Bathrooms - Be cautious when accessing sensitive parts of the codebase, such as login or payment processing modules.
Rule #4: Seat belts - Always use version control and back up your codebase regularly to safeguard against crashes and data loss.
Rule #7: Travel light - Avoid bloated software by minimising dependencies and optimising code for performance.
Rule #8: Kill Efficiently - Identify and fix bugs quickly and efficiently to prevent them from spreading and causing more harm.
Rule 9: A club does not have to be reloaded - Automate repetitive tasks to increase productivity and reduce errors.
Rule 10: Don't make any noise - Keep your code clean and organised to make it easier to understand and maintain.
Rule 15: Keep an exit free - Plan for scalability and flexibility in your code to allow for future changes and updates.
Rule #17: Don't be a Hero - Don't try to reinvent the wheel. Use proven software engineering patterns and best practices to avoid common pitfalls.
- Rule #17: Be a Hero - Occasionally, taking calculated risks and thinking outside the box can lead to innovative solutions and advancements.
Rule #18: Limber up - Stay up-to-date with the latest technologies and tools to remain competitive in the industry.
Rule #22: When in Doubt, Know Your Way Out - Have a plan in place for handling unexpected errors and failures.
Rule #23: Ziploc™ Bags - Protect sensitive data and prevent breaches by implementing robust security measures.
Rule #31: Check the Back Seat - Don't forget to review and test all aspects of your code, including less frequently used features.
Rule #32: Enjoy the little things - Take time to celebrate small successes and accomplishments throughout the software development process.
Rule #36: Sunscreen - Protect your eyes and health by taking breaks and practising good ergonomics while coding.
Rule #42: Keep your hands to yourself - Follow coding standards and respect the work of your colleagues to maintain a positive and efficient work environment.
Rule #52: Don't be afraid to ask for help - Seek advice and guidance from more experienced developers to learn and grow in your career.
Rule #53: Wet naps - Clean up messy code and fix technical debt regularly to maintain a healthy and sustainable codebase.

Conclusion

This guide presents a unique and engaging set of principles for software engineering inspired by the movie Zombieland. By following these rules, software developers can improve the efficiency, effectiveness, and quality of their work. The rules cover a wide range of topics, from testing and security to teamwork and innovation. Whether you're a seasoned developer or just starting out, the principles presented in this guide can help you succeed in the challenging and dynamic world of software engineering.

Resources

Zombieland Rules

Best Practices for Building Multi-Cloud Modules in Terraform

Jeiman Jeya — Wed, 05 Apr 2023 12:32:32 +0000

Organisations today are increasingly adopting cloud services from different providers to meet their business needs. However, managing multi-cloud infrastructure can be challenging, especially when it comes to infrastructure management, which is where Terraform comes in. With Terraform, you can utilise modules to build a comprehensive, multi-cloud configuration.

In this post, we will cover the best practices for creating Terraform module configuration that is robust and reusable for large-scale, multi-cloud architecture. With this knowledge, you can confidently build and deploy infrastructure with ease.

The Gold Standard - Best Practices for Building Cloud/Multi-Cloud Modules

Build modules to be extremely and easily reusable.
Create cloud-agnostic modules that support AWS and Azure for multi-cloud architecture.
Avoid hard-coded or semi-hard-coded values at the module level; instead, use input arguments.
Use conditional expressions and feature toggles to create comprehensive and flexible modules.
Group common resources into the same folders.

The Foundation - The Folder Structure

It all starts with the folder structure. How we structure it will determine how we can further automate our IaC workflows.

The folder structure can be broken down into 2 sections:

Modules: Allows you to create logical abstractions on top of a set of resources. In other words, it lets you group resources together and reuse this group later, possibly multiple times. Once you create these modules, they become “child modules”, which can then be referenced in configuration files.
Patterns: These are repeated configuration files that derive from defining child modules. They are also referred to as “root modules”.

The photo above depicts a Terraform module grouping resources together.

Philosophy on grouping similar and common resources into the same folders

It's best to store resources that share a common infrastructure resource category in the same folder. For example, consider all elements of networking, such as VPCs, subnets, security groups/firewalls, network security groups, load balancers, key vaults, and secrets. All of these either belong to the same underlying cloud category or they form the foundation of your cloud architecture. To keep things organized, it's recommended to group all of these resources in a folder labelled "foundation". As the name implies, they are the essential building blocks for any cloud architecture you set up for a company.

Use case example: Your company needs to set up the foundation for your cloud architecture and at the same time create a Kubernetes cluster with various node pools and settings that does a dependency reference of the foundation resources. This is followed by creating SQL databases, blob storages/S3 buckets, and more that will be used across the company. There are plans to introduce future projects that may not have the same folder structure as the current ones but will reside in a separate folder that uses the same Terraform modules.

IaC stages

These folders can be called "stages" because we need to create certain resources in a specific order. Therefore, stages have sequences, which are like orchestration steps. You need to execute items 1 to 10 in a particular order. Folder prefixes can be addressed as {sequence}_{stage}. They correspond to the following structure:

sequence - order of which to implement the Terraform resources - 01, 02, 03
stage - corresponds to the infrastructure category - foundation, common, kubernetes

Therefore, the folders can be organised in the following manner:

01_foundation - contains all networking resources, followed by load balancers, CDNs, key vaults, and secrets
02_kubernetes - contains all Kubernetes-related resources that does a dependency reference to networking resources from the 01_foundation folder
03_common - contains all the “common” resources that are usually created in any tech company - S3 buckets, blob storages, NoSQL databases, SQL databases, and more
04_custom_project_name - contains custom resources that need to be isolated for better management on special or custom projects

With stages, you can expand the folders to meet your engineering needs for provisioning new resources that are tied to a specific project. This allows teams to group resources together in the same folder, resulting in quicker execution of Terraform commands and faster results.

Visual representation of the folder structure for AWS Cloud

The visual representation below depicts the idea behind structuring your Terraform configuration into a "patterns" folder. As mentioned above, a "patterns" folder contains Terraform configurations that are repeated and injected into child modules for use. With this approach, you can create several of the same resources with different naming conventions and settings to meet your technical requirements.

terraform/
  patterns/
    iac/
     test/          
       01_foundation/
         aws-vpc.tf
         aws-subnets.tf
         aws-cdn.tf
         aws-secrets-manager.tf
       02_kubernetes/
         aws-eks.tf
         aws-kubernetes-cluster.tf
         .....
       03_common/
         aws-s3.tf
         aws-rds.tf
         aws-dynamodb.tf
       ..... 
   modules/
    aws/
     aws-vpc/
       main.tf
           output.tf
           variables.tf
         aws-s3/
           main.tf
           output.tf
       variables.tf
      ....

The folder structure mentioned above is just the beginning. We will further expand on this structure later in this post.

The Modularity - Building modules to be re-usable

To build a robust child module that can be reused multiple times without manipulation, start your design process from the root module. The role of a child module is to simply process input arguments and create the necessary resources. However, some teams have needed to create custom child modules for specific requirements, which can cause problems later on. Improving the functionality of a child module in this design pattern can affect existing resources created. This issue can be easily avoided if you appropriately create those data properties/objects in the root module.

Method 1 - Use conditional expressions and feature toggles from the beginning

Regardless of the outcome of your overall IaC configuration for your company, always keep the "feature toggle" concept in mind when building your modules from the beginning. This means creating resources with a toggle switch, which provides flexibility in deleting and re-creating said resources easily from a root module instead of a child module. Your child module can also have feature toggles, but they should use conditional expressions in combination with the root module.

For example, suppose you need to create two different types of S3 buckets: one with public access and the other with private access. You can use the following feature toggle logic in your root module, and a conditional expression in the child module.

Root module

# Root module

module "project-zeus-s3-bucket-public" {
  source = "./aws/aws-s3-bucket"
  bucket_name = "public-app-bucket"
  is_private = false
}

module "project-zeus-s3-bucket-private" {
  source = "./aws/aws-s3-bucket"
  bucket_name = "private-app-bucket"
  is_private = true
}

Child module

# Child module

resource "aws_s3_bucket" "bucket" {
  bucket = var.bucket_name

  tags = {
    Name = var.bucket_name
  }
}

resource "aws_s3_bucket_acl" "bucket_acl" {
  bucket = aws_s3_bucket.bucket.id
  acl    = var.is_private == true ? "private" : "public-read"
}

As you can see, we’re parsing in the is_private argument from the root module and the child module is performing a conditional expression check to see if the boolean value is set to true or false accordingly. With that condition, we will set the Access Control List for the bucket to either private or public accordingly.

Method 2 - Parsing all data from the root module into the child module

Imagine your child module is designed to only receive orders from the root module and simply create resources without manipulating the data. However, you can still achieve the same outcome as above by parsing all argument data types from the root module. The child module can then interpret the data types.

Root module

# Root module

module "s3-bucket-public" {
  source      = "./aws/aws-s3-bucket"
  bucket_name = "public-app-bucket"
  is_private  = false
  acl         = "public-read"
  policy      = file("policy.json")
  website     = {
    index_document = "index.html"
    error_document = "error.html"

    routing_rules = <<EOF
      [{
      "Condition": {
         "KeyPrefixEquals": "docs/"
      },
      "Redirect": {
          "ReplaceKeyPrefixWith": "documents/"
      }
      }]
    EOF
  }
}

Child module

# Child module
...
....

resource "aws_s3_bucket" "bucket" {
  bucket = var.bucket_name
  policy = var.policy

  website {
    index_document = var.website.index_document
    error_document = var.website.error_document
    routing_rules  = var.website.routing_rules
  }

  tags = {
    Name        = var.bucket_name
  }
}

resource "aws_s3_bucket_acl" "bucket_acl" {
  bucket = aws_s3_bucket.bucket.id
  acl    = var.acl
}

To summarise, any of these approaches will suffice to ensure that your child modules are reusable on a large scale.

The Jack of All Trades - Building an IaC stack to be Cloud-agnostic

By following the aforementioned steps and methods, your organisation can easily adopt a multi-cloud architecture. As previously mentioned, it's the structure of your folders that enables this approach. You still adhere to the same common resources folder concept as mentioned above and simply create new folders for each cloud provider you would like to adopt in your IaC stack.

Root module: visual representation

terraform/
  patterns/
    iac/
    test/
       01_foundation/
          aws/
        aws-vpc.tf
            aws-subnets.tf
            aws-cdn.tf
            aws-secrets-manager.tf
          azure/
        az-vnet.tf
            az-subnets.tf
            az-cdn.tf
            az-key-vault.tf
          gcp/
        gcp-vpc.tf
            gcp-subnets.tf
            gcp-cdn.tf
            gcp-key-management.tf

Child module: visual representation

terraform/
    modules/
    aws/
       aws-vpc/
          main.tf
              output.tf
          variables.tf
           aws-s3/
              main.tf
          output.tf
          variables.tf
    azure/
       azure-vnet/
          main.tf
              output.tf
          variables.tf
           azure-storage-account/
              main.tf
          output.tf
          variables.tf
    .....

As you can see, it's easy to adopt any cloud provider of your choice into your IaC stack if the folders are structured properly.

Folder breakdown

Patterns

iac - represents Terraform
test - represents the environment you’re provisioning the resources in. You may have entirely different resources for uat and prod environments. Therefore it’s best to keep them separate for feasibility.
01_stage - the common resource folders for your root modules to share
[provider-resource].tf - containing all of your root modules for that infrastructure category

The Automaton - Build a single multi-cloud, multi-environment pipeline

Based on the discussion above, we can now build a robust and modular pipeline to support running Terraform on any cloud provider and environment. In this example, we'll be using Azure DevOps. However, the same principles can be applied to any CI/CD toolset.

To cater to a multi-cloud and multi-environment architecture, it's best to build a modular and reusable pipeline. This involves creating a parent pipeline that invokes a template pipeline to execute Terraform commands using template parameters.

Parent pipeline

With a parent pipeline, you can define all the values and content that need to be parsed into the template or child pipeline as recognised input arguments.

# iac automation - multi-stage ci-cd pipeline

name: $(BuildDefinitionName)-$(Build.BuildId)-$(date:yyyyMMdd)$(rev:.r)

pool:
  vmImage: "ubuntu-latest"

parameters:
  - name: iacEnvironment
    displayName: IAC Environment (test, prod)
    type: string
    default: test
    values:
      - test
      - prod
  - name: cloudProvider
    displayName: Cloud Provider (aws, azure)
    type: string
    default: aws
    values:
      - aws
      - azure

variables:
  - template: ../../variables/generic-variables.yml
  - ${{ if eq(parameters.iacEnvironment, 'test') }}:
      - template: ../../variables/test-variables.yml
  - ${{ if eq(parameters.iacEnvironment, 'prod') }}:
      - template: ../../variables/prod-variables.yml
  - template: ../../variables/foundation-variables.yml

trigger:
  branches:
    include:
      - master
  paths:
    include:
      - "azure-devops/iac/pipelines/01-foundation-pipeline.yml"
      - "terraform/patterns/iac/test/aws/01_foundation/**"
      - "terraform/patterns/iac/test/azure/01_foundation/**"

pr: none

stages:
  - stage: planning_stage
    displayName: Planning
    jobs:
      - template: ../../templates/planning-template.yml
        parameters:
          terraformarguments: >-
            $(terraformarguments)
          environment: ${{ parameters.iacEnvironment }}
          sequence: $(sequence)
          stage: $(stage)
          cloud: ${{ parameters.cloudProvider }}

  - stage: deployment
    dependsOn: planning_stage
    condition: and(succeeded('planning_stage'))
    displayName: Deployment
    jobs:
      - template: ../../templates/deployment-template.yml
        parameters:
          terraformarguments: >-
            $(terraformarguments)
          environment: ${{ parameters.iacEnvironment }}
          sequence: $(sequence)
          stage: $(stage)
          cloud: ${{ parameters.cloudProvider }}

Parent pipeline breakdown

The goal is to create a single pipeline that can achieve a multi-cloud and multi-environment approach to running Terraform commands. However, there may be edge cases where you need to define separate pipelines for each environment. In the example above, we're using a single pipeline approach.

NOTE: This post will not go into too much detail about the components in the parent pipeline, as they differ from one CI/CD toolset to another. However, the general concept can be inferred from your understanding and experience of using CI/CD toolsets.

Template Parameters

We are utilising Azure DevOps' native process called Parameters. These are essentially template parameters that allow us to define values that will be used at runtime. For the example above, we’re specifying 2 parameters:

IaC environment
Cloud provider

With these two values, we can dynamically invoke the corresponding root modules in Terraform to create, update, or delete our resources.

Variable Groups

We have organised the necessary IaC stages and environment details into user-friendly and easily identifiable groups:

generic-variables.yml - contains all of the generic information, such as shared terraform secrets, and shared credentials
test-variables.yml - contains all of the environment-specific values, such as Terraform backend storage names (S3 bucket or Azure storage account names)
foundation-variable.yml - contains all of the IaC stage information such as Terraform arguments, stage folders to load (foundation, common, ecs, kubernetes), location, and more for that specific IaC stage

Path Triggers

As shown in the example above, we added AWS and Azure folder paths to ensure the pipeline triggers. However, since we are using template parameters, manual pipeline invocation is required to deploy to other non-default environments and cloud providers mentioned. Path triggers differ in each CI/CD toolset. Therefore, you could technically provide a wildcard to search for all cloud providers for the foundation IaC stage.

Pipeline Stages / Jobs

This section is where all of your template parameters and variable groups are injected to be utilised by the template pipeline.

Template pipeline or child pipeline

As the name suggests, templates allow you to define reusable content, logic, and parameters into smaller parts that are easier to understand.

Parameters and variables parsed from the parent pipeline can be processed to dynamically set the folders for your IaC stage and cloud provider. The following example demonstrates a step in a job for executing a Terraform plan. Depending on your CI/CD toolset and the marketplace plugins you use, you can write conditional logic to switch steps based on the cloud provider.

- task: TerraformCLI@0
  displayName: "Terraform Plan"
  name: tfPlanOutput
  inputs:
    command: "plan"
    workingDirectory: "$(System.DefaultWorkingDirectory)/terraform/patterns/iac/$(environment)/${{ parameters.cloud }}/${{ parameters.sequence }}_${{ parameters.stage }}"
    providerServiceAws: "$(aws_service_connection)"
    providerAwsRegion: "$(location)"
    commandOptions: '-out="$(System.DefaultWorkingDirectory)/terraform/patterns/iac/$(environment)/${{ parameters.cloud }}/${{ parameters.sequence }}_${{ parameters.stage }}/terraform.tfplan" -detailed-exitcode -var-file="$(System.DefaultWorkingDirectory)/terraform/patterns/iac/$(environment)/${{ parameters.sequence }}_${{ parameters.stage }}/values.tfvars" ${{ parameters.terraformarguments }}'
    allowTelemetryCollection: false
    publishPlanResults: "terraform_$(environment)_output_${{ parameters.sequence }}_${{ parameters.stage }}_plan"

The Conclusion

In today's world, organisations are increasingly adopting cloud services from various providers to meet their business needs. As a result, multi-cloud infrastructure is becoming more common, and managing it efficiently is becoming more challenging. Terraform is a popular tool that simplifies the process of infrastructure management by providing a declarative language for defining infrastructure as code (IaC).

In this context, building multi-cloud modules in Terraform requires a well-structured folder system, a focus on reusability, and a cloud-agnostic IaC stack. By following the steps outlined in this post, you can make your Terraform configuration more modular, easier to maintain, and more adaptable to changes in your cloud infrastructure.

Resources

What Are Terraform Modules and How Do They Work?

What is DevOps?

Jeiman Jeya — Tue, 04 Apr 2023 02:44:09 +0000

DevOps is more than just a software development methodology – it's a culture. The DevOps culture emphasises 3 main pillars: collaboration, communication, and shared responsibility between development and operations teams. In this post, we will discuss the importance of DevOps culture in software development and how developers and operations teams can benefit from it.

DevOps is not a Title

This conundrum needs to be addressed. If you’ve seen career titles such as DevOps Manager, and DevOps Lead, it means those individuals are managing the DevOps culture in that team. The title does not necessarily imply the term literally.

The Importance of DevOps Culture - The 3 Pillars

Collaboration and Communication

One of the key principles of DevOps culture is collaboration and communication. By breaking down silos between teams, DevOps promotes cross-functional collaboration, which leads to better outcomes. Collaboration between development and operations teams can help identify potential problems earlier in the development cycle, saving time and resources down the line.

Effective communication is also essential in DevOps culture. By sharing information and feedback, teams can work together to ensure that everyone is on the same page and working towards the same goals. This helps identify areas for improvement and implement changes more effectively.

Shared Responsibility

Another principle of DevOps culture is shared responsibility. In a DevOps environment, developers and operations teams share the responsibility of delivering software. This means that developers are responsible not only for writing code but also for ensuring that it runs smoothly in production. Similarly, operations teams are responsible not only for deploying and maintaining infrastructure but also for providing feedback to developers on how their code is performing in a production environment. In other words, “you build it, you ship it!”; developers can utilise the DevOps tools available to deploy their code to production.

DevOps teams can create a culture of accountability and ownership by sharing responsibility. When everyone is responsible for the success of the software, it creates a sense of ownership and pride in the work being done.

Continuous Improvement

Continuous improvement is a fundamental value of DevOps culture. DevOps teams continuously seek ways to enhance their processes, tools, and practices. By continuously evaluating and improving their work, DevOps teams can remain competitive in a rapidly changing digital landscape.

Continuous improvement also assists teams in learning from their mistakes. When something goes wrong, DevOps teams utilize the incident as an opportunity to learn and improve. By implementing changes based on what they have learned, teams can avoid making the same mistakes in the future. DevOps teams should always be looking for ways on improving their automation and processes.

Tools

DevOps teams can employ a range of tools and practices to achieve their goals, such as continuous integration and delivery (CI/CD), automated testing, and infrastructure as code (IaC). By automating tasks and using standardized processes, DevOps teams can reduce errors and improve the quality of their software. Some of the tools include:

GitHub Actions, Azure DevOps - CI/CD tool
Terraform, Pulumi, AWS CDK, Azure Resource Manager - IaC tool
SonarCloud, Codacy - Code analysis and security tool
Azure Test Plan - Automated testing tool

Conclusion

DevOps culture is more than just a set of practices; it is a mindset that emphasizes collaboration, communication, shared responsibility, and continuous improvement. By embracing these values, DevOps teams can create a culture of innovation and agility that can help them stay competitive in today's fast-paced digital landscape with the toolsets available.

Serverless Stack, a Terraform alternative for developers

Jeiman Jeya — Sat, 01 Apr 2023 05:41:09 +0000

Picture yourself being able to confidently deploy your application, whether it's full-stack or partially serverless, without having to rely on your DevOps or Platform team. Envision yourself seamlessly hosting your Infrastructure as Code (IaC) configurations in the same repository as your application code for maximum feasibility. Most importantly, consider the possibilities if you were able to eliminate the development feedback loop that necessitates the local mocking of services and redeployment for testing. There’s a solution for this.

Introducing Serverless Stack

Serverless Stack, also known as SST, is a powerful framework that enables you to effortlessly create serverless applications with backend stores on AWS. Since 2021, the SST framework has evolved into a comprehensive platform that empowers you to develop and deploy a cutting-edge full-stack application with robust backend features, including databases, GraphQL APIs, authentication, cron jobs, and various other AWS services such as ECS, S3, and CloudWatch. SST provides an array of features that efficiently addresses common developer challenges and concerns when building serverless applications, such as seamless integration with other backend tools and services in AWS.

What type of solutions can I build with it?

APIs
Web Apps
Mobile Apps (not literally, but tight coupling with serverless architecture)
GraphQL
Databases
Authentication
Asynchronous Tasks / Event Driven Systems
Monitoring
Lambda Layers
All other AWS services

SST is an excellent framework for building various solutions. However, for microservices that require loosely coupled services like account, identity, and payment services, it may not be the best choice. These solutions require a complex software design and cloud architecture, which are better suited for Kubernetes or other containerization solutions.

Overall, SST empowers you to confidently build a full-stack application comprising a front-end application, back-end application, database, and multiple integration systems or services. Thanks to the underlying architecture of SST, you can seamlessly incorporate other AWS services and third-party tools and libraries that are closely intertwined with your application. These can cover various areas such as authentication, monitoring, cron jobs, asynchronous operations, and more.

How does it all work and come together?

SST framework is built on top of the notable AWS Cloud Development Kit. SST follows monorepo architecture because it combines your application code along with the IaC. By default, SST uses TypeScript to define your architecture, however, you can still use JavaScript.

The project folder structure is as follows:



my-sst-app
├─ package.json
├─ sst.config.ts
├─ packages
│  ├─ core
│  │  └─ migrations
│  ├─ functions
│  ├─ graphql
│  └─ web
└─ stacks

The stacks/ folder contains all of your Infrastructure as Code. SST typically recommends grouping related resources together into stacks. For example, you can have a Database.ts file that contains the IaC to provision a RDS instance or a DynamoDB resource. Api.ts file could contain configuration to provision a API Gateway. You could then bind your database to the API in order for your functions that powers your API to have access to it. Your front-end application, Web.ts could host a static website powered by Vuejs on S3 and serve content through a CDN, such as CloudFront. In conclusion, it’s all about building up the stacks and binding them together using properties that are returned from the individual resources you provision.

packages/ contains all the components that power your backend systems, including systems like your GraphQL API, Lambda functions, front-end application, business logic, and any other programming you require. SST recommends maintaining a Domain Driven Design pattern within the packages/core folder to isolate your business logic from your API and Lambda functions. This ensures simple and maintainable code within your project. The folder contains all the implementation necessary for your application to function, which is then called by external-facing services, similar to an API. Within the packages/core/migrations folder, houses all of your SQL migrations. The packages/web folder houses your front-end application, whether it’s React, Vue, Next, Gatsby. packages/functions houses all of your code for your Lambda functions. They should be designed with simple functionality while being invoked by services code defined in the packages/core . Finally, packages/graphql houses outputs generated by the GraphQL. Ideally, you would not alter this folder as it’s shared between your frontend and backend, therefore should be committed into Git.

One great feature of SST is its use of Workspaces from npm. This provides support for managing multiple packages, or in this case, a monorepo, from within a single top-level, root package. Workspaces help manage dependencies for separate packages in your repository that have their own package.json file. This is especially useful if you want to keep several projects in a single monorepo that are related in some way.

SST includes its own set of npm scripts that handle various tasks for your application, such as starting it up, building it, deploying it, removing it, and testing it.



export default {
  config(_input) {
    return {
      name: "my-sst-app",
      region: "us-west-2",
    };
  },
  stacks(app) {
    app.stack(Database).stack(Api).stack(Web);
  },
} satisfies SSTConfig;

Finally, SST provides a configuration file that specifies your project configuration and the stacks that follow in your application. This configuration file is crucial in initializing your entire project, both on your local machine and when you deploy your application.

What features does it provide?

Live Lambda Development

SST offers a local development environment that enables you to debug and test your Lambda functions locally while being invoked remotely from AWS. This is a significant advantage of SST as it allows developers to test their changes locally without the need to mock the servers and redeploy to test them using conventional methods. Behind the scenes, SST employs AWS IoT over Websocket to communicate and forward requests between your local machine and the remote Lambda function (sort of like a proxy). This feature enables SST to execute a function locally using the same event, context, and credentials as the remote Lambda function. Changes are automatically detected, built, and reloaded live in just a few milliseconds. You can take full advantage of breakpoints to debug your code live with an IDE of your choice.

SST Console

SST Console is a web-based dashboard used to manage your SST applications. You can view real-time logs, invoke functions, replay invocations, view your deployed stacks, run migrations, make queries, view uploaded files in S3, query your GraphQL APIs, and more. For multitasking, you can use separate tabs or explorers to manage different parts of your application, providing a seamless monitoring experience.

Editor Integration

SST is designed to seamlessly integrate with your preferred IDEs. It comes with a range of built-in features including autocomplete, inline documentation, the ability to set breakpoints in your Lambda functions, and type checking (if you're building your application in TypeScript). Currently, SST provides support for Visual Studio Code, WebStorm, and IntelliJ IDEA.

Migration from AWS CDK and Serverless Framework

SST offers assistance in migrating your resources and applications from both AWS CDK and Serverless Framework with a comprehensive guide. The transfer process from AWS CDK should be relatively simple as SST was constructed on the AWS CDK. However, when dealing with the Serverless Framework, it is recommended to adopt an incremental update approach, which results in a hybrid Serverless Framework and an SST application.

How do I set up a project ?

It ultimately depends on the type of application you are looking to create. There are a few different approaches you can take:

Standalone application - this involves building your application from the ground up using the CDK API reference.
SST application - this approach involves using the SST API construct to build a variety of elements and/or resources such as frontend applications, backend applications, databases, APIs, and more.

To get started, follow these steps:

Have an AWS account ready to deploy your application.
Setup AWS CLI on your local machine and configure IAM user credentials for it. For now, you may provide full administrative access. You may change this later on to meet your project requirements based on the AWS resources you will be utilising in order for SST to deploy your application
Run npx create-next-app@latest to create a new Next.js application.
Then, initialise SST in your project root directory with npx create-sst@latest.
Start your local development environment by running npx sst dev to start SST and npm run dev to start Next.js.

Once you've completed these steps, you can start building your application.

How do I deploy it to my AWS environment?

CLI

The most convenient method for deployment is likely through the CLI. By running npx sst deploy --stage prod at the root level of your project, you can deploy your application to an environment with a prod identifier. SST utilizes CloudFormation stacks to carry out deployments, which is why you may notice your AWS account accruing CloudFormation stacks. To deploy your application to various environments, simply reference the -stage option in the CLI. This will allow you to design your application to be adaptable when it comes to switching between resources and environments.

CI/CD

To automate your deployments, push your code to a Git provider such as GitHub and then connect it to a CI/CD service to deploy your application. There are several CI/CD providers available, including GitHub Actions, Azure DevOps, Gitlab, Bitbucket Pipelines, Buildkite, and more. As with setting up an IAM user on your local machine, it is important to store the access credentials securely in your CI/CD tool. You can add additional steps in your pipeline that prompt the pipeline agent to log in securely to your AWS account, and SST will use the credentials to deploy your application.

How much customisation can I do with SST?

It depends on the application architecture and design patterns you employ. If your architecture is event-driven, then you can take full advantage of the SST framework. If your architecture is centered around building a mid-level API system using something like Express.js, you can still make use of the SST framework by installing Node package libraries in your project. This will help you leverage the capabilities of SST, while also designing a simple API system that uses REST design.

For example:

You can build an AWS API Gateway using SST
Build a Lambda function that utilises the Express.js REST API and bind the function to the API Gateway
You can extend from SST by using the AWS CDK constructs such as, S3, Cloudwatch Logs, Secrets Manager to store artifacts, send/stream your application logs, access sensitive information from the vault and bind them to your Lambda function

Summary

SST is a fantastic choice for developers seeking to build robust and scalable serverless applications that operate on AWS. SST offers a broad spectrum of features, such as a local development environment, and effortless integration with other backend tools and services in AWS. This provides developers with the flexibility and simplicity they require to create and deploy modern applications. Furthermore, SST is expanding its internal tooling to encompass additional constructs from AWS CDK, thereby providing an increasingly comprehensive yet user-friendly framework for building a modern, full-stack application.

Resources

How to set up an index alias that points to multiple indices in Elasticsearch

Jeiman Jeya — Wed, 11 Aug 2021 15:19:30 +0000

Index aliasing is a powerful feature in Elasticsearch that gives us the ability to control the indices. One could create as many indices as possible and use an alias to point to the right index or one can point the index alias to all indices, depending on your application needs.

Scenarios

Imagine if you had the following scenarios when working with Elasticsearch for your application needs:

Scenario 1: A snapshot of your index that needs to be restored to a new cluster, however, you forgot to restore it, and instead, you created a new index. As time passed, you realised that new data had amassed in the newly created index and you needed to combine the data from the snapshot with the new index.
Scenario 2: You have an index that requires certain schema changes or re-indexing. To be able to ensure your users are able to search from your instance with almost no downtime, set up a new index with new schema changes and re-index the data.

Introducing multi-index aliasing. You can create a single index alias that points to multiple indices. There's an endpoint from Elasticsearch that allows you to perform one or more alias actions in a single atomic operation.

Prerequisites

Verify if the alias you'd like to use has not been used yet by using the _alias/alias-name endpoint - HEAD _alias/my-alias
- If it returns a 404, it means the alias is available and can be used in setting up the multi-index aliasing
- If it returns a 200, it means it's being used and you need to choose another alias
Verify your indices are readily available within your cluster/instance.

Steps

Ensure that your old snapshot is ready (if you don't have your secondary index setup).
When you do restore the snapshot index, ensure that the index name is different than the newly created to avoid conflicts.
- Clone the index from the old snapshot/cluster if you have to
Run the following POST request to your designated cluster:



POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "new_index",
        "alias": "new_index_alias"
      }
    },
    {
      "add": {
        "index": "old_index",
        "alias": "new_index_alias"
      }
    }
  ]
}

But wait, what if I want to specify the new data is written to a specific index?

You can pass in the **is_write_index** parameter in the payload as follows:



POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "new_index",
        "alias": "new_index_alias",
        "is_write_index": true
      }
    },
    {
      "add": {
        "index": "old_index",
        "alias": "new_index_alias"
      }
    }
  ]
}

As you can see, it's fairly straightforward to set up an index alias that points to multiple indices within your Elasticsearch instance.

If you want to query data from the alias, simply perform the following:



GET /new_index_alias/_search

This would help you query across two indices new_index and old_index.

Benefits

Zero downtime: Bring up a new index with new changes, point alias to new index when it's ready. Keep the old index stale, in case it's needed for future reference or just as a backup.
Multiple Indices: We want to be able to query multiple indices? Then create an alias that points to multiple indices.

Resources

How To Automate App Releases using Fastlane and SemVer for Hybrid Applications

Jeiman Jeya — Tue, 10 Aug 2021 13:22:23 +0000

NOTE: This tool/implementation is useful for when you have both iOS and Android binary applications residing within the same repository (usually a React Native application) or hybrid app development tools which produce iOS and Android apps. However, it can be customisable to your needs with a little re-configuration.

The Story

Imagine if we could automate the entire app versioning process for our mobile applications by just typing in whether you want to bump it by:

Major
Minor
or Patch

Imagine if you don’t have to worry about the app versioning process of knowing what is already in production and what version we need to bump it to?

Lo and behold, the App Release Management (ARM) tool. 🎉

NOTE: Semantic versioning MAJOR, MINOR, and PATCH is the de facto standard you use to version your applications. Please refer to this link to find out more about SemVer standards.

Fancy word, but what does it actually do?

It pretty much does what is mentioned above. The process is as follows:

It will fetch the latest versions from TestFlight and Google Play Console respectively.
It will show you a summary of what the versions are.
If both Android and iOS match, the tool will select either one of them to use as the base of bumping the version.
The tool will prompt you whether you would like to bump it based on major, minor, or patch
Based on the input, the tool will run a SemVer operation in calculating the next iteration in the version and display the results for you.
If you’re happy with the result, you may press yes to proceed.
It will update the following files for you:
1. package.json
2. android/app/build.gradle
3. ios/ProjectName.xcodeproj/project.pbxproj
Operation completed!

It all looks fancy, but why this approach?

Well, imagine if you are a product owner or a new engineer who is new to the application code and is required to manage the App Releases. This tool will aid them in completing their task with ease, without worrying about configuring the wrong version code bump by accident.

Ok cool, how do I set it up?

Please refer to the documentation README file for detailed instructions on how to set up the tool.
Clone the devops-fun repository.
Once completed, navigate to development > app-release-management-tool folder.
Ensure that the .env file is present before proceeding any further. Refer to the README file for more information.
Run fastlane command.
Choose Option 1.
The rest are pretty self-explanatory thereafter.

What if I want to manually bump the version via a tool?

You may run the bump-version.sh and pass in a version code as an argument: ./development/app-release-management-tool/bump-version.sh 3.2.0

What does it look like?

Based on the image above, the following actions occur:

It will do a check against your iOS TestFlight build number and Google App Version and identify if the versions match (Please refer to the README.md file on how to set up a service connection to the respective stores in order to retrieve the app details programmatically).
If they do, it will display your current version in the App stores and will ask you for input, whether you want to bump the major, minor, or patch.
Based on your input, it will use the semver.sh script to perform the necessary version bump according to the Semantic Versioning standards.
If they don't match up, you'll be defaulted to using the bump-version.sh which requires manual input of your SemVer for your application - ie ./bump-version.sh 2.2.0

I hope this article has proven beneficial and useful to your release management process. If you have any inquiries, please reach out to me. Happy automating your release process! Do leave a comment or recommendations on improving the tool.

Resources

References

Part 2: Automating code quality scanning using Sonar Cloud and GitHub Actions

Jeiman Jeya — Mon, 09 Aug 2021 16:58:29 +0000

Part 1 of our article talks about the fundamentals of code quality with respect to Sonar Cloud. However, these are generic terms that can be applied to any code quality tool. In this article, we will explain how we can use Sonar Cloud to automate our code quality scanning with our GitHub repositories using GitHub Actions. The goal of this article is to ensure that you have a good understanding of how Sonar Cloud performs code analysis, how the integration features on GitHub (PR decorators, inline commenting, code quality overview using widgets) will benefit your engineering teams in the long run.

Some of the neat features that I found Sonar Cloud provides is:

Enhances your workflow with continuous code quality and code security
Supports all major programming languages you can think of
Provides a clear overview of your overall code health in your repository, pull requests and pipelines
Works with all famous Git providers (GitHub, Bitbucket, Azure DevOps and GitLab)
Works with all famous CI/CD tools (GitHub Actions, Bitbucket Pipelines, Azure Pipelines, GitLab CI/CD, CircleCI, TravisCI, etc)
Bug, Vulnerability, and Code Smell detection
Top-notch coding rules
In-ALM pull request feedback
Go/No Go Quality Gate checks
Automatic analysis (Currently limited to GitHub repositories)

Preliminaries

You will need the following:

A GitHub repository with a backend or frontend application
Access to GitHub Actions
A Sonar Cloud account
A Sonar Cloud project
Some basic knowledge on CI/CD

Setup Process

1. Sign up and import repositories

Ensure you have a GitHub repository readily available. For this, we will be choosing one of my NodeJS backend applications.
Signup for a Sonar Cloud account and log in using your GitHub account to have a seamless experience.
Once you have signed up, you will be redirected to this page:
From this page, you can import an organisation from GitHub.
Click on Import an organisation from Github.
You will be redirected to your GitHub account where it will prompt you to choose an organisation to install SonarCloud. In this article, we will be choosing my own personal GitHub repository.
From there, you will be redirected back to Sonar Cloud, where you will create your Sonar Cloud organisation. Please note that at this point, you have already installed and connected the Sonar Cloud App on your GitHub account.
From the image above, you can provide an organisation key that will be used later in GitHub Actions to send your code analysis metrics.
Next, you would need to choose a Pricing plan. There are 2 options: Paid and Free. For this article, we will be choosing the Free plan.
Click Create Organization.
You will now be able to import any Public repository. If you have Paid for Sonar Cloud, you may import your Private repositories.
We have selected our public repository, named nodejs-backend-starter. This repository contains a very basic Node.js backend application with Unit Testing and Code Coverage setup.
- NOTE: There's also an option to set up a monorepo. Based on the image above, you may see a small section at the bottom right of the image that mentions setting up a monorepo.
Once you have selected your repositories, click Set Up.
It will do an Automatic Analysis of your project providing an initial analysis of it.
Although Sonar Cloud offers automatic analysis for GitHub repositories only, we will be disabling this option as we want our CI pipeline to handle the analysis process for us. With the CI pipeline, you may replicate this entire process for any Git provider and any CI/CD tool available.
Navigate to Administration > Analysis Method > SonarCloud Automatic Analysis. Turn off this option.
Navigate back to your project root page. Your project page should look something like the image below:
As you can see, it states that they can't display any Quality Gate without a New Code definition. Based on the Part 1 article, we will be setting up the following:
1. A New Code Definition
2. A Quality Gate

2 Sonar Cloud Configuration

2.1 Set Up New Code definition

There are 2 options available at this moment. You can either set up a New Code definition on:

A project level
An organisation level

For this article, we will set it up on a project level.

On your project page, click on Set New Code definition. If that is not available, you may navigate to Administration > New Code.
If you have read my previous article, you may know that there are a number of options to choose from. In this case, we will be choosing the Number of days that equates to 30 days by default. The reason is we are not going to be maintaining a version for the project on Sonar Cloud for now.

2.2 Setup a Quality Gate

Navigate to your organisation page (https://sonarcloud.io/organizations/{username}/projects).
Select Quality Gates from the sub-navigation.
From the image above, there's already a default Quality Gate set up for you. We're going to be cloning this and creating our own. Click on Copy on the Sonar way Quality Gate*.*
Name your new cloned Quality Gate.
Since there are already New Code conditions, we are simply appending Overall Code conditions. For this article, we have added the following:
1. Coverage is less than 80%
2. Duplicated Lines is greater than 5%
3. Maintainability Rating is worse than A
4. Reliability Rating is worse than A
5. Security Rating is worse than A
Set your new Quality Gate as Default.
Now that we have our project settings in place, we need to set up a project properties file next.

2.3 Setup a project properties file

This properties file will enable the CI pipeline to perform the necessary configuration and filtering in order to perform the code analysis. Furthermore, this will ensure we have the right settings in place when we push our commit to a branch or create a pull request that triggers the code analysis.

In your code repository, create a sonar-project.properties file.



sonar.exclusions=**/*.bin
sonar.organization=jeimanjeya
sonar.projectKey=jeiman_nodejs-backend-starter
sonar.projectName=jeiman_nodejs-backend-starter
sonar.projectVersion=1.0
sonar.sourceEncoding=UTF-8
sonar.sources=src
sonar.exclusions=node_modules/**
sonar.test.inclusions=test/**
sonar.typescript.lcov.reportPaths=coverage/lcov.info

In this file, we're basically narrowing the focus of the scanner to configure the project source path, unit test inclusion path, coverage report path and exclude redundant folder paths (node_modules).

Commit this file to your:
Root directory: If it is a single application repository (microrepo) architecture (the case for us)
Monorepo directory: If you have a monorepo architecture

Now that we have set up our project configuration and settings, we can focus on configuring the CI pipeline on GitHub Actions.

3. GitHub Actions

3.1 Pipeline/Workflow Structure

You have several to choose from options:

Store all of your workflows in a single YAML file - ci.yml
Separate them into branch workflows:
1. ci_develop.yml - for PR triggers and branch triggers on develop branch
2. ci_master.yml - for PR triggers and branch triggers on main branch
Separate them into trigger workflows:
1. PR triggers - ci_pr.yml
2. Branch triggers - ci_branch.yml
Separate them into monorepo workflows:
1. spa-frontend.yml and ms-backend.yml - contains PR and branch triggers for both develop and main branch

Option 1 is difficult the maintain in the long run if you follow a monorepo architecture
Option 2 or 3 is the best way of maintaining your workflows on GitHub
Option 4 is useful for when you have a monorepo architecture

For this article, we are choosing Option 3 for simplification.

Your final GitHub workflow for branch analysis will look like this:



name: Sonar Cloud - Branch Analysis

# Controls when the action will run. Triggers the workflow on push
# events but only for the main and release-* branch
on:
  push:
    branches: 
      - main
      - release-*

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  sonarcloud:
    name: Build (Sonar Cloud)
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
      with:
        # Disabling shallow clone is recommended for improving relevancy of reporting
        fetch-depth: 0
    - uses: actions/setup-node@v2
      with:
        node-version: '15'

    - name: Node install dependencies
      run: npm install

    - name: Run unit tests
      run: npm run test

    - name: SonarCloud Scan
      uses: sonarsource/sonarcloud-github-action@master
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

We have included an Action from GitHub Actions and that is the Sonar Cloud Scanner.
In order for Sonar Cloud Scanner to authenticate and upload the analysis reports and metrics, you will need to store the SONAR_TOKEN secret in your GitHub repository.
Navigate to your project settings and retrieve the project token, Project > Administration > Analysis Method > Analyze with a GitHub Action. It will present you with a token.
Navigate back to your repository settings on GitHub, Repository Settings > Secrets > New repository secret. Name it as per the workflow secret name.
For the GITHUB_TOKEN, GitHub automatically appends the token for each workflow that runs on GitHub.
Commit your code into the main branch and watch the pipeline run.
The summary for the branch analysis pipeline has indicated that the analysis was successful.

Navigating back to Sonar Cloud, you will notice that your project branch analysis report has been updated.

We'll move on to running code analysis on a long-living branch next.

3.2 Running code analysis on a long-living branch

Simply check out a new branch from main and name it to release-{anyname}
Commit your code and the CI pipeline will pick it up based on the branch trigger logic in the workflow YAML file.
The SC Scanner will upload your analysis reports to your project on Sonar Cloud.
Don't be alarmed if it shows Not computed as Sonar Cloud requires a second analysis to be able to show your Quality Gate on that long-living branch. It mentions Next scan will generate a Quality Gate.
We pushed another commit to the repository to the long-living branch to show the Quality Gate.

Your final Quality Gate for branch analysis (on long-living branch) will produce the following result.

3.3 Running code analysis on pull requests (PR)

Clone the ci_branch.yml. Name it ci_pr.yml
The only section we are changing is the event trigger. Instead of a push event, we are using the pull_request event.



on:
  pull_request:
    branches:
      - main
      - release-*

Commit the code and raise a PR in your code repository.

Your PR page should produce pipeline status checks alongside SonarCloud Code Analysis on your Merge PR section. Furthermore, your PR will contain decorations provided by the SonarCloud Bot.

Your PR status checks details page should give you a better overview of what needs fixing.

If you navigate back to your Sonar Cloud project and choose your PR analysis report from the branch dropdown menu, you will see similar statistics that were displayed on GitHub.

Let's go ahead and fix our code to ensure we don't have any code smells and fine tune our Quality Gate conditions to match with a Passed state.

Now our PR has clean code (however overall code requires fixes), we can now merge our PR. The pipeline will run another branch analysis on the main branch, which is our default branch on Sonar Cloud.

Piecing it all together

We have accomplished having a Sonar Cloud project connected to a GitHub repository that has GitHub Actions workflows in place to automate our code analysis for branch and PR triggers. Once these analyses are completed on the pipeline, you can navigate back to your Sonar Cloud project and view branch and PR analysis on both your main branch and long-living branches.

One of the plus points in having integration between GitHub and Sonar Cloud is that engineering teams can benefit from having decorations forming up in their pull requests and repository. This will speed up the developer's productivity in fixing bugs, code smells, and vulnerabilities and avoid having further technical debts in their projects.

Tips

You can use any CI/CD tools to perform your code analysis. It does not need to be with GitHub Actions even though your repository resides on GitHub, however, for consistency you can keep them all under one umbrella.
To have a complete end-to-end experience for your code quality scans on your pull requests, ensure you connect and import repositories from one unique Git organisation. The aforementioned features mainly work for that reason. Try not to mix and match repositories from different organisations (even though it's possible to do so).
- 1 GitHub organisation = 1 Sonar Cloud subscription
- 1 Azure DevOps organisation = 1 Sonar Cloud subscription
Create long-living branches on Sonar Cloud if you're following Git flow technique of develop and master branch. You may read up on their community post on how to achieve this. This will ensure that you are performing branch analysis on them alongside your pull request analysis.
Fine-tune your Quality Gate to ensure you get the best experience from it that matches your engineering teams needs
Sonar Cloud supports monorepo projects!

I hope this 2-part series has been beneficial and helpful to your engineering needs. Sonar Cloud is a powerful code analysis tool. Please do explore their documentation to find out more or kindly reach out to me for any assistance.

Demo links to this article

References

Part 1: Concepts of Code Quality in Sonar Cloud

Jeiman Jeya — Mon, 09 Aug 2021 06:50:48 +0000

In engineering teams/tribes, we often find ourselves stuck in a dilemma of choosing a suitable and efficient code analysis tool. A tool that will provide us all of the essential features and metrics to analyse codebases using best design practices in mind. Teams would want to prevent code problems from being merged by detecting code smells, bugs, and vulnerabilities sooner. Furthermore, they would want to view fast and accurate feedback from their respective pull requests and code merges in their repositories.

The primary goals of software quality engineering are:

Process control and oversight
Implementing standards and metrics
Data collection and analysis
Test development
Identification of issues and solutions
Follow-up to ensure corrective actions

Overview

Over the years, I have worked with many code analysis tools to help developers in various teams such as Codacy, Code Climate, DeepScan, and Sonar Cloud. After spending a considerable amount of time experimenting and setting up Sonar Cloud projects, I realised it stands out from the crowd. It has a comprehensive analysis engine that offers features encompassing all of the aforementioned software quality engineering goals. Some of those features which fascinated me and my team was:

Pull Request decorators
- Inline commenting through report annotations
- Pull request widgets: Provides overall code quality health on your Pull requests
Repository widgets: Provides overall code quality health on your project
Scanning Old code vs New code
- Code Coverage
- Code Duplications
- Maintainability Ratings
Defining custom Quality Gate checks for different projects
Defining New Code definitions for different projects

Some of the aforementioned features (Quality Gate, Profile, New Code Definitions) are in fact settings that need to be configured in Sonar Cloud. You will need to give your attention to configuring these settings as you will not reap the benefits of fully utilising Sonar Cloud.

We are going to explore the Sonar Cloud ecosystem and how all of its core features come together to provide you a comprehensive code analysis experience.

Sonar Cloud Ecosystem

Projects

In Sonar Cloud, a single repository corresponds to a single project. It is how they maintain a unique set of code quality data and metrics for each repository you have.

Monorepo Support

Sonar Cloud does support monorepo projects. You can create multiple projects, each corresponding to a separate monorepo project which are all bounded to the same repository. This will allow you to:

Configure one Quality Gate per project
Receive multiple Quality Gate results
Read project-labeled messages from SonarCloud

Each monorepo project must have a unique project key in Sonar Cloud, which will be used to uniquely identify your projects using your CI tool.

The standard practice is to have the following naming convention: {organisationName-project-monorepoName}

Project 1: sampleorg-domain-frontend
Project 2: sampleorg-domain-backend

This is useful if your organisation maintains a big number of monorepo projects across various engineering tribes so the projects can be easily identified in Sonar Cloud. However, you can follow any naming convention you see fit that suits your needs.

New Code Definitions

Sonar Cloud follows the concept of Clean As You Code. The core idea is that you focus your attention and effort on new code. As you work on features and improvements, SonarCloud analyses your code on each new commit and alerts you to any code quality problems and vulnerabilities. This allows you to address the issues right away and ensure that all new code added to the project is always clean. You may read their documentation to find out more information.

Accompanying this are New Code Definitions. Setting up the right New Code definition for your project is vital to getting the most out of SonarCloud by determining which changes to your code are considered recent enough to merit your full focus and allow you to use the Clean as You Code methodology when addressing issues in your code. There are several options to consider when configuring your New Code Definitions:

Previous version: Issues in the code that have appeared since the most recent version increment of the project
Specific version: Issues that have occurred on a specific version of the project
Number of days: Issues that have appeared on your code since the specified number of days (A numerical number)
Specific date: Issues that have appeared on your code since the specified date

When you perform initial code analysis, it does not provide you a "New Code" analysis. Instead, it scans the whole project, providing you with an overall code quality health. As you go along and start analysing new commits, Sonar Cloud will present you with both an Overall and New code quality health analysis (depicted from the picture above).

You can set a New Code Definition either on a project level or an organisation level, with the latter providing you a way of automatically applying the definitions to new projects that are created.

Quality Gates

A quality gate is technically a metric of sorts that informs you whether your code meets a minimum level of quality required for the project. This consists of a set of conditions that are applied to the results of each analysis performed. If the results meet or exceed the quality gate conditions, it will show one of the following statuses accordingly: Passed or Failed. You can define conditions on New Code and Overall Code. Some examples of the conditions are:

Coverage is less than 80.0%
Duplicated Lines is greater than 3.0%
Maintainability Rating is worse than A
Reliability Rating is worse than A
Security Hotspots Reviewed is less than 100%

The quality gates are analysed and calculated on Main Branch (master by default), other Branches, and Pull Requests. You can create any number of Quality Gates for your projects and enable them per project, or create a default Quality Gate that will be applied to all projects that have been or will be created.

An example of a customised Quality Gate is shown below:

Quality Profiles

Quality profiles as you may have guessed it are programming language rules that are applied during code analysis. By default, the programming languages that are supported on Sonar Cloud have a built-in profile, called "Sonar way", using the standard best practices that are currently in the market. Although the "Sonar way" is best suited for most projects, there are cases where engineering tribes would want to customise their profiles that best suit their needs.

Branches

In Sonar Cloud, there are 2 types of branch analysis: Short-lived branches and Long-lived branches.

Short-Living Branches

As the name suggests, these branches are meant to be used to temporarily perform analysis on them, usually via pull requests. Short-lived branches are deleted automatically after 30 days with no analysis.

Long-Living Branches

These branches are useful for when tribes follow Agile methodology, utilising Git flow techniques to maintain a set of upstream branches (sprint, release). These branches will remain in your Sonar Cloud project history until it is deleted. Some companies maintain upstream branches for a very long time, hence this option is extremely useful to conduct code analysis on these branches alongside your main branch (master).

Defining long-living branches

Long-living branches are defined on a project level. Simply navigate to your Project Settings > Administration > Branches & Pull Requests. The long-living branches follow a regular expression pattern.

By default, the pattern is (branch|release)-.*. This means that when the name of the branch starts with branch- or release-, it will be considered a long-living branch.

Project Settings

All of the aforementioned concepts and terminologies are in fact project and organisation settings (Quality Gates, Quality Profiles, Long-living branches, New Code Definitions, Monorepo Support). Accompanied with are general settings such as Code Coverage Exclusions, Test File Inclusions and Exclusions, Duplication Inclusions, Source File Exclusions and Inclusions, and many more. You may refer to their documentation to set these up.

Summary

I hope this Part 1 article has provided you with insights on how Sonar Cloud operates and the software engineering quality it follows. Part 2 of this article will provide you a walkthrough on how to set up a Sonar Cloud project with a GitHub repository and perform code analysis using GitHub Actions.

Image Source: https://alexandrebrisebois.files.wordpress.com/2014/05/2011-09-18_code_reviews.png

Automating mobile application deployments using Fastlane and CI/CD tools

Jeiman Jeya — Mon, 03 May 2021 14:00:21 +0000

The Problem

Engineering teams these days find it troublesome to build, test and deploy their mobile application changes locally without having to maintain the tools required for it. There is a lot of maintenance involved as you need to keep track of what versions of these tools have been installed to avoid compatibility issues when you’re building the application bundles, especially hybrid apps built on React Native.

The Cure

fastlane, an automation tool that aids in handling all of the tedious tasks so you don't have to. It's by far, the easiest way to build and release your mobile apps.

fastlane can easily:

Distribute beta builds to your testers
Publish a new release to the app store in seconds
Reliably code sign your application - alleviates all of your headaches
Reliably maintain your provisioning profiles and application certificates in a Git repository

How it works

All actions in fastlane are written into lanes. Defining lanes are easy. Think of them as functions in any programming language of your choosing. You define all of your actions within that lane.

An example of a lane is as follows:

lane :beta do
  increment_build_number
  build_app
  upload_to_testflight
end

So when it comes to executing that lane, all you do is run fastlane beta in your terminal.

Installation & Setup

In this article, we will look at setting up a fastlane script to build, sign and deploy an iOS application to Testflight.

Pre-requisites

As with most projects, you need to perform the initial project setup to support building your application, such as:

Installing your project dependencies
Install XCode and Android Studio
Install Java SDK
Setup git repository for Android and iOS certificates - will be explained later in the article

fastlane folder structure

In ideal cases, you would have an Android application project and an iOS application project, both hosted in the same code repository as your project

projectFolder/
    app/
    scripts/
    ios/
        fastlane/
            ....
  android/
        fastlane/
            ....
    package.json
    .gitignore
  ...

In this case, you will want to initialise the fastlane folder within the respective android and ios project sub-directories.

Setup

Install fastlane
- via Homebrew (brew install fastlane)
- via RubyGems (sudo gem install fastlane)
Navigate your terminal to your respective android and ios project directory and run fastlane init
You should have a folder structure that is similar to the one below:

fastlane/
    Fastfile
    Appfile

The most interesting file is fastlane/Fastfile, which contains all the information that is needed to distribute your app.

Inside your Fastfile, you can start writing lanes:

lane :my_lane do
  # Whatever actions you like go in here.
end

You can start adding in actions into your lanes. fastlane actions can be found here.
Copy the following file structure to your Fastfile

default_platform(:ios)

def beta(arg1, arg2)
  # You may use Ruby functions to write custom actions for your app deployment
end

platform :ios do

    desc "Building the IPA file only"
      lane :build_ios_app_ipa do
        app_identifier = "com.appbundle.id"
        beta("AppSchemeName", app_identifier)
      end

end

default_platform(:ios) - Initialise your Fastfile file with a default platform
platform :ios do - Add all actions under a platform

From 5 to 6, it will inform fastlane that this particular Fastfile is purely for iOS operations. So instead of running the command fastlane <lane_name>, you will actually run fastlane ios <lane_name>. Therefore, anything parked under platform :ios do will be executed when lanes are invoked in your terminal.

If you are well-versed in Ruby, you may write your own Ruby functions to help you write custom actions that you require for further flexibility, especially when it comes to building several apps with different environments across your organisation.

fastlane will identify them as an action regardless due to the fact that fastlane is written in Ruby.

In this article, we will follow writing the actions using a Ruby function. This is so we can promote action re-usability across other lanes deployments.

Action Steps

Before we start writing our functionality in the lanes, we first list down our action steps:

Setup API Key for App Store Connect (app_store_connect_api_key) - This will allow fastlane to connect to your App Store to perform other actions that requires user authentication
Setup CI (setup_ci) - This will setup a temporary keychain to work on your CI pipeline of your choice
Create and sync provisioning profiles and certificates (match) - This will help us maintain our provisioning profiles and certificates across your teams
Update code signing settings (update_code_signing_settings) - This is to update the code signing identities to match your profile name and app bundle identifier
Increment your app build number (increment_build_number) - This will automate your application build number by retrieving the latest Testflight build number and incrementing the value
Build the app (build_app) - This will build the app for us and generate a binary (IPA) file
Upload your binary to Testflight (upload_to_testflight) - This will automate the process of uploading the binary file to Testflight and informing your testers accordingly

Steps

Step 1: Setup your App Store Connect API key

Visit the following page. It will provide you a step-by-step process in generating an API key
Once you have generated a key, take note of:
- Issuer ID
- Key ID
Download the generated API key - A .p8 file
Store it in a secure location in which you can easily access them. Avoid storing them in the same repository as anyone in your organisation will have access to the company's Apple account.
- In this article, we are storing them in another Git repository with limited read and write scopes to specific engineers within our organisation.

Step 2: Setup the fastlane script

Setup your App Store API Key to generate a hash used for JWT authorization

def setup_api_key()
  sh "if [ -d \"appstoreapi\" ]; then echo 'Folder exist, executing next step'; else git clone #{ENV['APPSTORE_API_GIT_URL']} appstoreapi; fi"
  app_store_connect_api_key(
    key_id: ENV['APPSTORE_KEY_ID'],
    issuer_id: ENV['APPSTORE_ISSUER_ID'],
    key_filepath: Dir.pwd + "/appstoreapi/AuthKey_xxx.p8",
  )
end

In the snippet above, I am cloning a Git repository which contains my App Store Connect API key, followed by utilising the app_store_connect_api_key action from fastlane. It takes in several parameters, however, there are 3 vital parameters:

key_id - The Key ID from which you took note when you generated the API key
issuer_id - The Issuer ID from which you took note when you generated the API key
key_filepath - The file path to your .p8 file

This step will generate a hash that will be used to authenticate the App Store using JWT.

I would highly recommend storing sensitive credentials or URLs and access them from an Environment Variable (.env) file in the same project folder - fastlane/.env

Once you have prepared the file, you can easily access any Environment Variable by simply passing in ENV['ENV_NAME'] into the Fastfile.

2. Define a Ruby function with 2 arguments

def beta(scheme, bundle_id)
end

Within this function, we specify the action steps we mentioned above

def beta(scheme, bundle_id)
    setup_api_key() # Import the setup_api_key function
    setup_ci() # Uses fastlane action to create a temporary keychain access on a CI pipeline
end

3. Utilise the match action from fastlane

match is a fastlane action that allows you to easily sync your certificates and provisioning profiles. It takes a new approach to iOS and macOS code signing, where you share one code signing identity across your engineering team to simplify the setup and prevent code signing issues. The foundation of match was built using the implementation of codesigning.guide concept.

You can store your code signing identities in:

Git repository
Google Cloud
Amazon S3 bucket

In this article, we have chosen to store it a Git repository. However, with the case of storing in a Git repository, you would need to provide a form of basic authorization in order for match to access and clone your repository.

Whichever Git provider you choose (GitHub, Bitbucket, Gitlab or Azure DevOps), you would need to setup a Personal Access Token (PAT), which fastlane will use to clone repository and sync your code signing identities.

In your Fastfile, you will need to encode your PAT with a Base64 encryption

authorization_token_str = ENV['GITHUB_TOKEN']
basic_authorization_token = Base64.strict_encode64(authorization_token_str)

The basic_authorization_token variable will be used in setting up the match implementation below.

match(
  git_url: "<git_url>",
  git_basic_authorization: basic_authorization_token,
  readonly: true,
  type: "appstore",
  app_identifier: [
    bundle_id
])

From the snippet above, this is a very simple implementation. Take note on the readonly parameter. This is crucial in creating and syncing your profiles and certs with your Apple Developer account. If you set readonly to false, you will allow match to automatically sync and create new profiles and certs, should it deem your existing certs and profiles expired or corrupted. Once it has provisioned the profiles, you can set readonly to true. This is to avoid egde cases where it might accidentally create new certs and profiles.

4. Update code signing identities

This step is mainly used to update the Xcode settings on a CI pipeline due to its default behaviour of selecting a default cert and profile from the Mac OS agent.

update_code_signing_settings(
  use_automatic_signing: false,
  path: "../ios/AppName.xcodeproj",
  code_sign_identity: "iPhone Distribution",
  profile_name: "match AppStore com.appbundle.id",
  bundle_identifier: "com.appbundle.id"
)

From the above snippet, you can retrieve your profile_name and bundle_identifier from your Apple Developer account or taking note of them from the match step once it has been executed.

5. Increment Application build number

We simply increment the number by accessing your latest testflight build number and incrementing the value with 1.

increment_build_number({
  build_number: latest_testflight_build_number + 1
})

puts "BUILD_NUMBER: #{lane_context[SharedValues::BUILD_NUMBER]}"

The puts command will print out the Build number in the console, which is shared across the lane context.

You may have other app build versioning models than the one mentioned in this article. Increment the build number however you see fit to your project needs.

6. Build the application

This step will involve building your application to generate a binary (APK) file.

build_app(
  workspace: "../ios/AppName.xcworkspace",
  export_xcargs: "-allowProvisioningUpdates",
  scheme: scheme,
  clean: true,
  silent: true,
  sdk: "iphoneos"
)
puts "IPA: #{lane_context[SharedValues::IPA_OUTPUT_PATH]}"

The build_app action is provided by fastlane.

The puts command will print out the file path location of the IPA file in which you can use to manually upload to Testflight.

7. Upload the binary file to Testflight

This step will involve uploading your binary file to your Testflight account.

app_identifier = "com.appbunde.id"
upload_to_testflight(app_identifier: app_identifier)

upload_to_testflight is an action provided by fastlane.

Putting them all together

The final lane should look something like this:

desc "Build and push a new build to TestFlight"
  lane :release_build do
    app_identifier = "com.appbundle.id"
    beta("AppSchemeName", app_identifier) # The custom function we wrote earlier
    upload_to_testflight(app_identifier: app_identifier)
  end

When you execute fastlane ios release_build in your terminal, it will operate based on the aforementioned steps above.

As you can see, we utilised a Ruby function to group all common operations under the same roof so is to ensure:

Code re-usability
Consistency
Clean code

Integration with CI/CD tools

With the fastlane scripts, you can easily integrate it with any CI/CD tools of your choosing:

GitHub Actions
Gitlab CI
Azure Devops
Bitbucket Pipelines

When setting up your pre-requisite steps in your pipeline YAML file, you can simply include a bash script that executes your fastlane script

cd ios/android folder

fastlane ios/android release_build

The command above will execute your fastlane script by following the aforementioned steps above.

Conclusion

fastlane is a powerful tool that helps streamline your build process across your organisation. It can be used along side your existing CI/CD pipelines or as a standalone pipeline script to be used on your local machines or custom pipeline tools such as Buildkite. It would require additional steps such as, cloning the repository and checking out branches, however, all readily available from fastlane as actions.

Spend some time reading through the documentation as it contains a lot of actions out of the box that will prove useful for your engineering teams (match, pilot, cert, sigh, appium, xctool, supply, and many more) . Furthermore, they provide an extensive list of plugins for third-party integrations such as Firebase, App Center, Yarn, Android Versioning, Dropbox, Slack Upload, S3, Bugsnag and many more. You can supercharge your fastlane scripts by coupling it with a CI/CD tool, such as GitHub Actions, Bitbucket Pipelines or Azure DevOps. The sky's the limit!