Forem: Watcharin(start)

🐣Why idempotence was important to DevOps

Watcharin(start) — Sat, 04 May 2024 19:52:50 +0000

🚀 I just need to improving myself in everyday and sharing my learned for everyone.

Hi reader 👋🏼, The adoption of DevOps practices has become increasingly prevalent within development team today. Many organizations have integrated the role of a DevOps engineer into their teams or have expanded the responsibilities of existing team members to encompass DevOps methodologies , akin to the Agile framework. In these scenarios, development team either recruit dedicated individuals to drive DevOps processes or distribute these responsibilities across the job description of all team members. This approach acknowledges that DevOps aims to enhance productivity by optimizing the workflows and responsibilities associated with each role. 👨🏻‍💻🧑‍💻👩🏻‍💻

📜 A crucial aspect of DevOps is the application of the Idempotence principle (alternatively referred to as idempotent or idempotency) in coding practices. This concept ensures that Infrastructure can be consistently managed and controlled through code. Fundamentally, it builds upon the previously established concept of Infrastructure as Code , which is widely implemented using tools such as Ansible, Terraform, Puppet, and other.😺

What is Idempotence

🐈 It’s a important principle in IaC or several programming, particularly in the context of operations or functional that modify state or data. An operation is considered idempotent if it can be applied multiple times without changing the result beyond the initial application.

In programming, idempotence is desirable for several reasons:

Reliability: Idempotent operations can be safely retried or repeat without causing unintended side effects or data corruption. This is especially crucial in distributed systems, where network failures or other issues may cause requests to be duplicated or retried.
Consistency: Idempotent operations ensure that the final state of the system is consistent and predictable, regardless of how many times the operation was performed.
Simplicity: Idempotent operations are easier to reason about and debug since their behavior is deterministic and predictable.

The example use case of Idempotence

Idempotence is particularly valuable in scenarios like:

RESTful APIs: In this context is about HTTP methods like GET, PUT, DELETE, and HEAD — are designed to be idempotent. For instance, multiple identical PUT requests should have the same effect as a single PUT request, ensuring data integrity. Furthermore, multiple request DELETE should have only first request was completed, hence other requests will returned the previous state.
Database operations: Operations like UPDATE or DELETE queries that target specific rows based on a unique identifier (e.g., a primary key) are often idempotent. Executing the same query multiple times will result in the same final state.
Caching: Caching invalidation or cache warming operations are typically idempotent. Repeated cache invalidation requests for the same key will effectively invalidate the cache entry once, ensuring consistency. Moreover, cache invalidation is the process of invalidating cache by removing data from system when those data is no longer valid or useful.
Infrastructure as Code: Tools like Ansible, Terraform, and Puppet often rely on idempotent operations to manage infrastructure resources. Running the same configuration for multiple times should result in the desired state without unintended modifications. Example, you executing the Terraform to provisioning instances more than one time with the same configuration. You could saw the desired state of your resources without modification.
Data processing: The operations as deduplication, normalization, or transformation of data can be designed to be idempotent, ensuring that repeated applications produce the same output.

It’s significant to note that idempotence is a property of an operation, nor necessarily of an entire system or application. In practice, developers usually strive to make critical operations idempotent to improvement reliability, consistency, and simplicity in the software operational. 🐈‍⬛

The example Golang code

The provided example code written in Golang illustrates the practical implementation of the idempotent concept within the context of a basic data management application.

A Person struct is defined to represent individual records, with the Email field serving as the unique identifier. Two simple functions are implemented:

For inserting new Person records into an in-memory database.
For retrieving all existing records.

Let’s writing a code

Beginning with define a Person struct

type Person struct {
    Email string
    Name  string
    Age   int
}

Define a new schema of in-memory database with init()

func init() {
    // Create the DB schema
    schema := &memdb.DBSchema{
        Tables: map[string]*memdb.TableSchema{
            "person": &memdb.TableSchema{
                Name: "person",
                Indexes: map[string]*memdb.IndexSchema{
                    "id": &memdb.IndexSchema{
                        Name:    "id",
                        Unique:  true,
                        Indexer: &memdb.StringFieldIndex{Field: "Email"},
                    },
                    "age": &memdb.IndexSchema{
                        Name:    "age",
                        Unique:  false,
                        Indexer: &memdb.IntFieldIndex{Field: "Age"},
                    },
                },
            },
        },
    }

    var err error

    // Create a new data base
    db, err = memdb.NewMemDB(schema)
    if err != nil {
        panic(err)
    }
}

Create an Insert function for built transaction into database.

func InsertDataWithID(member *Person) error {
    // Create a write transaction
    txn := db.Txn(true)

    // Insert the new person
    fmt.Printf("Inserting ID: %s\n", member.Email)
    if err := txn.Insert("person", member); err != nil {
        return err
    }

    // Commit the transaction
    txn.Commit()

    // Create read-only transaction
    txn = db.Txn(false)
    defer txn.Abort()
    return nil
}

func InsertMultipleData(people []*Person) error {

    for _, p := range people {
        if err := InsertDataWithID(p); err != nil {
            return err
        }
    }
    return nil
}

Create a retrieving all existing data from database and print out to console.

func ListAllValues() error {
    txn := db.Txn(false)
    // List all the people
    it, err := txn.Get("person", "id")
    if err != nil {
        return err
    }
    fmt.Println("All the people:")
    for obj := it.Next(); obj != nil; obj = it.Next() {
        p := obj.(*Person)
        fmt.Printf("[+]  %s | %s | %d\n", p.Name, p.Email, p.Age)
    }

    return nil
}

Define main() function to call both functions as insert and then retrieve them.

func main() {

    // Insert some people
    people := []*Person{
        &Person{"joe@aol.com", "Joe", 30},
        &Person{"lucy@aol.com", "Lucy", 35},
        &Person{"joe@aol.com", "Joey", 26},
        &Person{"tariq@aol.com", "Tariq", 21},
        &Person{"dorothy@aol.com", "Dorothy", 53},
        &Person{"joely@aol.com", "Joe", 35},
    }

    if err := InsertMultipleData(people); err != nil {
        panic(err)
    }

    if err := ListAllValues(); err != nil {
        panic(err)
    }
}

You can see fully code in one file below. you can see my test data that have a duplicated index as 1 and 3.

// main.go
package main

import (
    "fmt"

    "github.com/hashicorp/go-memdb"
)

type Person struct {
    Email string
    Name  string
    Age   int
}

var db *memdb.MemDB

func init() {
    // Create the DB schema
    schema := &memdb.DBSchema{
        Tables: map[string]*memdb.TableSchema{
            "person": &memdb.TableSchema{
                Name: "person",
                Indexes: map[string]*memdb.IndexSchema{
                    "id": &memdb.IndexSchema{
                        Name:    "id",
                        Unique:  true,
                        Indexer: &memdb.StringFieldIndex{Field: "Email"},
                    },
                    "age": &memdb.IndexSchema{
                        Name:    "age",
                        Unique:  false,
                        Indexer: &memdb.IntFieldIndex{Field: "Age"},
                    },
                },
            },
        },
    }

    var err error

    // Create a new data base
    db, err = memdb.NewMemDB(schema)
    if err != nil {
        panic(err)
    }
}

func InsertDataWithID(member *Person) error {
    // Create a write transaction
    txn := db.Txn(true)

    // Insert the new person
    fmt.Printf("Inserting ID: %s\n", member.Email)
    if err := txn.Insert("person", member); err != nil {
        return err
    }

    // Commit the transaction
    txn.Commit()

    // Create read-only transaction
    txn = db.Txn(false)
    defer txn.Abort()
    return nil
}

func InsertMultipleData(people []*Person) error {

    for _, p := range people {
        if err := InsertDataWithID(p); err != nil {
            return err
        }
    }
    return nil
}

func ListAllValues() error {
    txn := db.Txn(false)
    // List all the people
    it, err := txn.Get("person", "id")
    if err != nil {
        return err
    }
    fmt.Println("All the people:")
    for obj := it.Next(); obj != nil; obj = it.Next() {
        p := obj.(*Person)
        fmt.Printf("[+]  %s | %s | %d\n", p.Name, p.Email, p.Age)
    }

    return nil
}

func main() {

    // Insert some people
    people := []*Person{
        &Person{"joe@aol.com", "Joe", 30},
        &Person{"lucy@aol.com", "Lucy", 35},
        &Person{"joe@aol.com", "Joey", 26},
        &Person{"tariq@aol.com", "Tariq", 21},
        &Person{"dorothy@aol.com", "Dorothy", 53},
        &Person{"joely@aol.com", "Joe", 35},
    }

    if err := InsertMultipleData(people); err != nil {
        panic(err)
    }

    if err := ListAllValues(); err != nil {
        panic(err)
    }
}

When you executed the above code, you will see this result. you can see function was inserted the duplicated data and update that row with a new information, which you will see from last function. It isn’t applied idempotent concept.

Inserting ID: joe@aol.com
Inserting ID: lucy@aol.com
Inserting ID: joe@aol.com
Inserting ID: tariq@aol.com
Inserting ID: dorothy@aol.com
Inserting ID: joely@aol.com
All the people:
[+]  Dorothy | dorothy@aol.com | 53
[+]  Joey | joe@aol.com | 26
[+]  Joe | joely@aol.com | 35
[+]  Lucy | lucy@aol.com | 35
[+]  Tariq | tariq@aol.com | 21

Let’s see how can apply idempotence

For implement the idempotence, I should add code for query as verify existing data before insert or update data.

Firstly, I just create new function to retrieving existing data.

func GetDataByID(key string) *Person {
    // Create a read-only transaction
    txn := db.Txn(false)

    raw, err := txn.First("person", "id", key)
    if err != nil || raw == nil {
        fmt.Printf("[-] Not found %s", key)
        return &Person{}
    }

    return raw.(*Person)
}

Then, I will update the InsertMultipleData function to get data that will skipped if it found.

func InsertIdempotentData(people []*Person) error {

    for _, p := range people {

        existed := GetDataByID(p.Email)
        if existed.Email == p.Email {
            fmt.Printf("Skipping insertion for ID: %s (already exists)\n", p.Email)
            continue
        }

        // Insert the new person
        if err := InsertDataWithID(p); err != nil {
            return err
        }
    }
    return nil
}

If you replace the previous function call from InsertMultipleData to InsertIdempotentData , you could see result below. You can found 1 logging message that skipped insert operation.

[-] Not found joe@aol.comInserting ID: joe@aol.com
[-] Not found lucy@aol.comInserting ID: lucy@aol.com
Skipping insertion for ID: joe@aol.com (already exists)
[-] Not found tariq@aol.comInserting ID: tariq@aol.com
[-] Not found dorothy@aol.comInserting ID: dorothy@aol.com
[-] Not found joely@aol.comInserting ID: joely@aol.com
All the people:
[+]  Dorothy | dorothy@aol.com | 53
[+]  Joe | joe@aol.com | 30
[+]  Joe | joely@aol.com | 35
[+]  Lucy | lucy@aol.com | 35
[+]  Tariq | tariq@aol.com | 21

Conclusion

🐯 Idempotence is an important concept in programming, particularly when dealing with operations that modify state or data. An operation is considered idempotent if it can be applied multiple times without changing the result beyond the initial application. Idempotent operations are desirable because they enhance reliability, consistency, and simplicity in software designs. They ensure that the final state remains predictable and free from unintended side effects or data corruption, even if the operation is retried or repeated due to failures or other issues. 👻

Why we should provide standard for IaC on our team

Watcharin(start) — Mon, 25 Dec 2023 05:07:37 +0000

😺 Today, let's delve into a common challenge faced by non-developers who take on roles that mimic those of actual developers, such as DevOps Engineers, Cloud Engineers, Automated QA, and others. As we navigate through the intricate landscape of the modern tech world, we encounter numerous like application design patterns, frameworks, and architectural designs.

In this dynamic environment, the ability to craft applications and share them within your team or with others becomes a pivotal aspect. The cohesion achieved by adopting shared standards and frameworks ensures that your ideas, designs, and functionalities are comprehensible to everyone involved. This shared foundation not only facilitates mutual understanding but also provides a guiding framework and utility tools that enhance overall performance.

By adhering to a common working standard, you not only streamline communication within your team but also benefit from the efficiency and support offered by standardized frameworks. This collaborative approach not only fosters understanding but also contributes to a more efficient and high-performing work environment.

Understand several software design principle

📄 Understand about several design pattern and framework for programming languages.

🧑🏼‍💻 Software design principles are fundamental guidelines that software developers follow to achieve high-quality software design. The principles provide a framework for making design decisions that lead to more maintainable, scalable, and robust software.

Example are here:

DRY (Don't Repeat Yourself): This principle emphasizes the importance of reducing repetition in code. Every piece of knowledge or logic should be unique and single-represented in a system. This reduces the risk of inconsistencies and makes code more maintainable. Read more here.
KISS (Keep It Simple, Stupid): This principle suggests that systems work better if they are kept simple rather than made complex. Simplicity should be a key goal in design, and unnecessary complexity should be avoided. Read more here.

From the principle of designs, it provide us to Design patterns and Frameworks that we actually do on our application. It just a common standard for our team to collaborative in the same project that we work together.

The IaC programmings

📄 Definition of IaC, What is purpose to use programming to control infrastructure.

🔨 The Infrastructure as Code(also IaC) is a key concept in modern software deployment and operation, fundamentally transforming infrastructure managed in IT environment via readable documentation for developer and machine. We’re trying to reduce the manually setting up and configuring resources. We’re use code to automate the process, which can be more consistent and error-free than manual process.

Example tools of IaC.

Terraform: An open-source tool that uses a declarative configuration language for building, changing, and versioning infrastructure safely and efficiently.
Ansible: An open-source tool that uses YAML for defining infrastructure configurations, primarily for automation of application deployment and intra-service orchestration.

Relationship between IaC and several programming languages

📄 Said about same purpose and different on both

🤝 We must experience about programming and infrastructure before apply the IaC. The IaC can be implemented using standard programming language like Python, Javascript, Golang or specialized domain-specific language(DSLs). The DSLs are designed specifically for defining and managing infrastructure resources like Hashicorp’s Terraform and AWS Cloudformation.

📜 In the IaC, script or code are written to automatically set up and manage infrastructure. This code act as a blueprint for our resources. You can be executed to deploy, manage and destroy the infrastructure. We can integrates closely modern deployment practices as the same common programming like CI and CD.

How to adopt development practices to IaC

📄 Talk about current tools and a community experience to improve IaC writing.

⚒️ Adoption the software development principle in IaC is essential for maintaining the quality, efficiency, and reliability of our the source code to infrastructure automation. Here are several key programming principle and practices that can be effectively applied to IaC.

Use version control: Just like code, our infrastructure configuration should be stored in a version control system like Git. This practices allow tracking changes, collaborate with team and maintaining a history of our infrastructure state.
Write modularity: Write modular script, just as with traditional programming, break down your code into smaller, reusable modules can make them more manageable, and understandable.
DRY principle(Don’t Repeat Yourself): Avoid duplication in your code. You can use templates, modules, or function to reuse code as most as possible, which makes it easier to manage and reduces the risk of error.
Code review: We must implement code review process to IaC, just as you would for code. This principle encourages collaboration, improve code quality, and help in sharing knowledge among team member.
Use automate testing: Apply automate testing into your IaC. You can implement tools and frameworks to IaC script to ensure they perform as expected before deployment.
Documentation: Write the document of IaC script and configurations. Good documentation helps in understanding the purpose and function of your infrastructure, especially in complex environment.
Keep it idempotency: Ensure your script is idempotent, meaning running your script multiple times in the same environment result in the same state, without unintended side effects. This is crucial for reliability and predictability in automated environments.
Apply security practices: Most important to embedded the security practice on your IaC development process. This includes using secure access credentials, encrypting sensitive data, and should follow the principle of least privilege.
Often refactoring: Regularly refactor your IaC code for improvements and optimizations, similar to how you would refactor application code for better performance, readability, and maintainability.

By adopting these practices, team can ensure that their IaC approach is as robust, efficient, and maintainable as their software development process.

Example guideline to contribute IaC

📄 Show my standard to writing Terraform and Ansible

You can apply the previous practices into your IaC code and see what happen in your work. I can sharing some guideline to initial your IaC project here.

For the Hashicorp’s Terraform:

You might to create a new repository for versioning your script. You can create a folder structure in different purpose like
1. components to store your reusable and maintainable for script in consistent modules.
2. deployments to store your variable files that contain each environment configurations.
3. modules in optional to download and freeze external Terraform’s module for your environments.
You can integrates automation plugins for verify syntax, credentials leak, and missing fields. Use third party tools like Terrasec, TFlint, and other.
For documentation, you can use terraform-docs for automate provision code spec when you write a description in your scripts.
You can integrates your process with CI/CD pipeline for Jenkins, Github-action, and other platforms.
For integration testing your script, you can use Grantwork’s Terratest to verify your actually provisioning resources.

You can see my example terraform code for initialize from Github repo.

Conclusions

📄 Recap all topics and drop some question to audiences.

🧳 In our journey towards modernizing infrastructure management, we have embraced to use modern technology for provisioning, managing, and maintaining infrastructure. This is achieved through programming languages and domain-specific languages(DSLs). By writing code and script, we’re revolutionizing the way we handle infrastructure.

The purpose of this is to establish standard practices for our team in crafting code that enhances collaboration, readability, and maintainability. This adoption will not only streamline our development processes but also significantly improve the quality of code.

Lambda parameter from Secret Manager🤫

Watcharin(start) — Sat, 19 Aug 2023 13:54:25 +0000

Hey everyone, I'm back and ready to dive into some Serverless func with AWS Lambda. Today, I've set myself a cool goal: I want to figure out how to use a Lambda function to grab a secret credential from AWS Secret Manager, you know, the stuff that's locked up tight by AWS KMS service. I'm all about learning how Lambda can smoothly handle secrets from Secret Manager and whatever other secret stashes are out there.

Alright, check it out. This lab is hooking me up with some sweet Golang code. It's like magic – this code can summon that parameter from Secret Manager and stash it in Lambda’s memory, all sorted out by the code itself. Talk about slick, right? So, here's to cracking open the door to the secrets realm, and getting cozy with Lambda's skills and the hidden treasures they can unlock. Let's rock this journey! 🚀

Required Golang library

github.com/aws/aws-lambda-go/lambda
github.com/aws/aws-secretsmanager-caching-go/secretcache
errors
log
os

Coding

Create a new Golang project.

go mod init $YOUR_PROJECT_NAME
# Example
go mod init lamda-secret

Let’s start with first function to ask Lambda handle it.

func main() {
    // Parameter in Start(...) is func name to handled
    lambda.Start(ExportSecret)
}

Create a function to export your secret value

func ExportSecret() {
    // Define new variable for cache object
    sc, _ := secretcache.New()

    // Get a result from getter function and logged it
    word, _ := GetterSecret(sc)
    log.Printf("We have a secret word is %s", word)
}

Create a function that get secret from Secret Manager API into cache memory

func GetterSecret(sc *secretcache.Cache) (string, error) {
    // Request secret from SecretManager API with Secret ID from ENV
    secr, err := sc.GetSecretString(os.Getenv("SECRET_WORD_ID"))
    if err != nil {
        return "", errors.New("Can't get you secret")
    }
    return secr, nil
}

Build and upload them to S3 bucket

export BINARY_NAME="lsc"
export GOOS="linux"
export GOARCH="amd64"
export CGO_ENABLED="0"
export S3_BUCKET_NAME=$(aws s3api list-buckets --query 'Buckets[0].Name' --output text)

# Build binary file from code
go build -o $BINARY_NAME lamda-secret

# Compress this file into ZIP format
zip ../../lambda-$BINARY_NAME.zip $BINARY_NAME

# Upload to s3 with aws cli
aws s3api put-object --bucket $S3_BUCKET_NAME --key lambda-$BINARY_NAME.zip --body ./lambda-$BINARY_NAME.zip

Proof your result

Create AWS lambda function and upload your ZIP binary file. Example CLI to create

aws lambda create-function \
--function-name YourFunctionName \
--runtime go1.x \
--role your-iam-role-arn \
--handler your-bin-name \
--zip-file s3-url-with-zip

Manual trigger to Lambda function and you can see result on CloudWatch log group.

aws lambda invoke \
--function-name YourFunctionName

!! Before execute function you should attach IAM role to Lambda and that role will provide below permission.

{
    "Id": "AllowLambdaAccessSecret",
    "Statement": [
        {
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": "AllowLambdaAccessToSecretManager"
        },
        {
            "Action": [
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:Encrypt",
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:kms:ap-southeast-1:xxxxxx:key/xxxxx-xxx-xxxx-xxxx-xxxxxd96cc",
            "Sid": "AllowLambdaAccessToKMSKey"
        }
    ],
    "Version": "2012-10-17"
}

Or you can create and invoke function via web console

Reference

Conclusion

👾 Handling secrets within a Lambda function becomes a breeze with the assistance of the AWS SDK. This concept can be seamlessly extended to explore alternative techniques, such as accessing retained secrets, orchestrating secret updates, or even orchestrating the deletion of secrets through automated processes.

Excitingly, my forthcoming blog will delve into the practical implementation of the aforementioned functionalities, all achievable with a single click using the powerful tool that is Terraform. Stay tuned for the upcoming content! Catch you later! ✌️😋

Jmeter load test with real time dashboard(InfluxDB)

Watcharin(start) — Sun, 13 Aug 2023 12:47:19 +0000

🤖 Hey there! Just wanted to give you a quick update. Right now, I'm working on setting up a way to test how well our app performs. We're building a web app for our customers, so making sure it works smoothly is a big deal. We're using a user-friendly interface to create different performance test scenarios. The idea is to show how the app holds up under different conditions.

To do this, we're using JMeter to create scripts that simulate a bunch of people using the app at once. The only catch is that JMeter doesn't create fancy comparison reports for our customers. It mostly gives us raw data and documentation-style reports. So, I've come up with a plan. We're going to hook up InfluxDB to help us store performance data in a way that's easy to understand. Later on, we'll turn this data into cool visual dashboards.

Setting up

Develop a docker-compose into workspace

---
version: '3.8'

volumes:
  influx-data:

networks:
  dev:

services:
  influxdb:
    image: influxdb:2.7-alpine
    container_name: influxdb
    restart: unless-stopped
    ports:
      - 8086:8086
    networks:
      - dev
    volumes:
      - influx-data:/var/lib/influxdb2
      - ./influxdb2/conf:/etc/influxdb2
  demo-app:
    image: go-example:1.0 # build from my local app
    container_name: go-app
    ports:
      - 3000:3000
    networks:
      - dev

You can replace demo-app with your app

Provide docker compose with docker-compose up
Verify application is available with cURL
```
$ curl localhost:3000/healthz
OK%
```
Then open web browser to config InfluxDB http://localhost:8086
Create a user to manage InfluxDB
Save a token for access database via API and select configure later. Finish!!
Now, we’re already to collect data from Jmeter

Create a Jmeter script

You can develop your own scenarios like below

In the last step of your script, you add backend listener to InfluxDB.

Or you can add this XML to your JMX file.

...
                <BackendListener guiclass="BackendListenerGui" testclass="BackendListener" testname="Influxdb" enabled="true">
          <elementProp name="arguments" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" enabled="true">
            <collectionProp name="Arguments.arguments">
              <elementProp name="influxdbMetricsSender" elementType="Argument">
                <stringProp name="Argument.name">influxdbMetricsSender</stringProp>
                <stringProp name="Argument.value">org.apache.jmeter.visualizers.backend.influxdb.HttpMetricsSender</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="influxdbUrl" elementType="Argument">
                <stringProp name="Argument.name">influxdbUrl</stringProp>
                <stringProp name="Argument.value">http://localhost:8086/api/v2/write?org=start&amp;bucket=jmeter</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="application" elementType="Argument">
                <stringProp name="Argument.name">application</stringProp>
                <stringProp name="Argument.value">demo-go</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="measurement" elementType="Argument">
                <stringProp name="Argument.name">measurement</stringProp>
                <stringProp name="Argument.value">jmeter</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="summaryOnly" elementType="Argument">
                <stringProp name="Argument.name">summaryOnly</stringProp>
                <stringProp name="Argument.value">false</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="samplersRegex" elementType="Argument">
                <stringProp name="Argument.name">samplersRegex</stringProp>
                <stringProp name="Argument.value">.*</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="percentiles" elementType="Argument">
                <stringProp name="Argument.name">percentiles</stringProp>
                <stringProp name="Argument.value">90;95;99</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="testTitle" elementType="Argument">
                <stringProp name="Argument.name">testTitle</stringProp>
                <stringProp name="Argument.value">Test simple</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="eventTags" elementType="Argument">
                <stringProp name="Argument.name">eventTags</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="influxdbToken" elementType="Argument">
                <stringProp name="Argument.name">influxdbToken</stringProp>
                <stringProp name="Argument.value">NykJl8CX4paZEN3q7yNnIs9F_mLlnpITDWnWyfDSDv4yTyx6y0JOTMbExQljc_sq1qBVEYl5r4FFW3_NJd9SmQ==-7IV6tg8MsuwYU-5l4UyrtA==</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
            </collectionProp>
          </elementProp>
          <stringProp name="classname">org.apache.jmeter.visualizers.backend.influxdb.InfluxdbBackendListenerClient</stringProp>
        </BackendListener>
...

Import InfluxDB dashboard

InfluxDB provide a web service that you can import a dashboard template on it.

Go to InfluxDB web UI
Go to Dashboard >> Click Create Dashboard >> Add a Template
Parse a YAML file URL into block number 2(You can copy URL here)
Click Lookup Template >> Install template
After finish you can see a dashboard on your page

Execute load test

After you prepare your workspace, Let’s start your Jmeter script to test your application.

Then, you can see realtime result on your dashboard.

Conclusion

Enhance your load testing experience by seamlessly integrating your testing tools with a visualization platform, allowing you to create insightful dashboards that display real-time results of your testing scenarios. For this purpose, I've employed a widely recognized tool, JMeter, which conveniently offers a plugin for seamless integration with InfluxDB.

I'm optimistic that my upcoming workshop will prove invaluable in enhancing your testing journey. Looking forward to sharing knowledge and boosting your testing prowess. 🤖👾

Terraform AWS lambda integrated with EventBridge

Watcharin(start) — Tue, 01 Aug 2023 11:57:09 +0000

😎🎶 In the previous session, I shared how to develop a Lambda function that requests RDS to copy a snapshot from a source region to a target region. It was developed using Golang and AWS SDK to build data into the AWS API structure. Then, I used some programming logic to manage already replicated snapshots in the target region by copying only those that do not yet exist in the target region. You can read more here.

😜🤪 This session covers how to deploy the previous source code to real infrastructure on AWS using Terraform. I will demonstrate how to use Terraform to provide Infrastructure as Code and create a cloud service environment for serving the Lambda function from the requirements mentioned earlier.

AWS service is related

EventBridge; event bus and event rules.
IAM; policy and roles.
S3; bucket and objects.
Lambda; functions and policy
Cloudwatch; log group
KMS; multi-region key

Preparations

Before develop a Terraform code, you should install the following.

Terraform cli
AWS cli
Go bin

Get started develop

Define the Terraform information.

# versions.tf
terraform {
  required_version = ">=0.12"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.7.0"
    }
  }
}

Define AWS provider

# providers.tf
provider "aws" {
  # Configuration options
  region = local.region
}

To define local variables for use in this environment, create a main terraform file. Define variables that may be changed in other scenarios, and replace those values with resources from other files.

# main.tf
locals {
  # Use in providers.tf
  region = "ap-southeast-1"

  # For S3
  S3_prefix = "lambda-lake-"

  # For Lambda components
  lambda_func_name = "rds_copy_snap"
  lambda_s3_key    = "lambda-hello.zip"
  handler_bin_name = "hello"
  func_env_var = {
    OPT_SRC_REGION        = "ap-southeast-1"
    OPT_TARGET_REGION     = "ap-southeast-2"
    OPT_BEDUG             = "True"
    OPT_DB_NAME           = ""
    OPT_OPTION_GROUP_NAME = ""
    OPT_KMS_KEY_ID        = ""
  }

  # For EventBridge
  bus_name = "rds-bus" # Leave with `default` to use AWS service event bus.
  rds_events = {
    description = "Capture for RDS snapshot event"
    event_pattern = jsonencode({
      "source" : ["aws.rds", "demo.event"],
      "detail" : {
        "EventID" : ["RDS-EVENT-0042", "RDS-EVENT-0091"]
      }
    })
  }

  # Default tags in all resources created by this code.
  tags = {
    Owner   = "watcharin"
    Project = "local-dev"
  }
}

# Use to get my AWS session information
data "aws_caller_identity" "current" {}

First, I created an S3 resource in a separate file. This resource provides a new S3 bucket to store compressed binary files from the built Go code and upload them into Lambda. To generate the bucket name, I defined a prefix and added a random suffix.

# s3-buckets.tf
module "s3_lambda" {
  source = "terraform-aws-modules/s3-bucket/aws"

  bucket_prefix     = local.S3_prefix
  create_bucket     = true
  block_public_acls = true

  control_object_ownership = true
  object_ownership         = "ObjectWriter"

  tags = local.tags
}

After created S3 bucket, I built Go binary and push into its.

# Use taskfile help a pipeline step. Or you can read a manual step in Taskfile.yml.
task build
task upload:lambda

Let’s create Lambda resource. I will provision an IAM role for assume privilege from function.

# lambda.tf
data "aws_iam_policy_document" "lambda_assume" {
  statement {
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
    actions = [
      "sts:AssumeRole"
    ]
  }
}
resource "aws_iam_role" "lambda_iam_role" {
  name               = "iam_for_lambda"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume.json
}

Then, provide a policy document and attachment to Lambda role. This function will permission for access to AWS resource as Cloudwatch log group, KMS key, and RDS snapshot.

# lambda.tf
data "aws_iam_policy_document" "allow_logging" {
  policy_id = "AllowLambdaPushLog"
  statement {
    effect = "Allow"
    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]
    resources = ["arn:aws:logs:*:*:*"]
  }
}

data "aws_iam_policy_document" "allow_copy_rds_snapshot" {
  policy_id = "AllowLambdaCopyRdsSnapshot"
  statement {
    effect = "Allow"
    sid    = "AllowLambdaAccessToRdsSnapshot"
    actions = [
      "rds:CopyDBSnapshot",
      "rds:ModifyDBSnapshot",
      "rds:DescribeDBSnapshots",
      "rds:ModifyDBSnapshotAttribute"
    ]
    resources = ["*"]
  }

  statement {
    sid    = "AllowLambdaAccessToKMSKey"
    effect = "Allow"
    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*"
    ]
    resources = ["${aws_kms_key.rds_snap.arn}"]
  }
}

resource "aws_iam_policy" "lambda_logging" {
  name   = "lambda-logging-cloudwatch"
  policy = data.aws_iam_policy_document.allow_logging.json
}

resource "aws_iam_policy" "lambda_rds" {
  name   = "lambda-access-rds"
  policy = data.aws_iam_policy_document.allow_copy_rds_snapshot.json
}

resource "aws_iam_role_policy_attachment" "func_log_policy" {
  role       = aws_iam_role.lambda_iam_role.id
  policy_arn = aws_iam_policy.lambda_logging.arn
}

resource "aws_iam_role_policy_attachment" "func_rds" {
  role       = aws_iam_role.lambda_iam_role.id
  policy_arn = aws_iam_policy.lambda_rds.arn
}

I define an information about the Lambda that I want to created. In this function, I define configuration as source file from S3, memory size, and env var.

# lambda.tf
resource "aws_lambda_function" "rds_snap" {
  function_name = local.lambda_func_name
  description   = "Func to handle event from EventBridge to request RDS copy snapshot."
  role          = aws_iam_role.lambda_iam_role.arn

  handler   = local.handler_bin_name
  runtime   = "go1.x"
  s3_bucket = module.s3_lambda.s3_bucket_id
  s3_key    = local.lambda_s3_key

  package_type = "Zip"
  memory_size  = 128

  environment {
    variables = local.func_env_var
  }

  tags = local.tags
}

We still cannot access this function from EventBridge because I did not provide a Lambda policy that allows invoking this function. You can see the Terraform code below.

# lambda.tf
resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowExecutionFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.rds_snap.function_name
  principal     = "events.amazonaws.com"
  source_arn    = module.eventbridge.eventbridge_rule_arns.orders
}

I have created a Cloudwatch log group to store activity logs for Lambda. However, to reduce costs for this lab, I have set the retention period for the logs to one day. (Note: You can adjust this setting as necessary.)

# cloudwatch-log-group.tf
resource "aws_cloudwatch_log_group" "lambda" {
  name              = "/aws/lambda/${local.lambda_func_name}"
  retention_in_days = 1
  lifecycle {
    prevent_destroy = false
  }
}

Create a KMS key that will be used by a Lambda function to encrypt a snapshot from a source region and transfer it to a target region.

data "aws_iam_policy_document" "kms" {
  # Allow root user to full managed this key
  statement {
    effect = "Allow"
    actions = [
      "kms:*"
    ]
    resources = ["*"]
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
  }

  # Allow user for limited access key
  statement {
    effect = "Allow"
    actions = [
      "kms:CreateGrant",
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:DescribeKey",
    ]
    resources = ["*"]
    principals {
      type        = "AWS"
      identifiers = ["${data.aws_caller_identity.current.arn}"]
    }
  }
}

resource "aws_kms_key" "rds_snap" {
  description         = "CMK for AWS RDS backup"
  enable_key_rotation = true
  policy              = data.aws_iam_policy_document.kms.json
  multi_region        = true
}

resource "aws_kms_alias" "rds_snap" {
  target_key_id = aws_kms_key.rds_snap.id
  name          = format("alias/%s", lower("START_RDS_SNAP"))
}

Finally, I provide EventBridge rules to filter the interested events and send them to a Lambda function.

Note: For event bus, If you need to tracking AWS services event, you will define with default.

# eventbridge.tf
module "eventbridge" {
  source  = "terraform-aws-modules/eventbridge/aws"
  version = "2.3.0"

  bus_name = local.bus_name
  rules = {
    orders = local.rds_events
  }

  targets = {
    orders = [
      {
        name = "event-to-lambda"
        arn  = "${aws_lambda_function.rds_snap.arn}"
      }
    ]
  }

  tags = local.tags

  depends_on = [aws_lambda_function.rds_snap]
}

Once you finish developing the Terraform code, you can deploy it to your environment and wait for Terraform to create the necessary resources.

terraform init
terraform apply

How to prove it

You can simulate an event that you interesting.

Goto AWS web console.
Goto EventBridge >> Event buses

Click Send events and define event details like below. Then, the event source you will use demo.event that I define in the terraform and event pattern.

{
  "EventCategories": ["creation"],
  "SourceType": "SNAPSHOT",
  "SourceArn": "arn:aws:rds:us-east-1:123456789012:snapshot:rds:snapshot-replica-2018-10-06-12-24",
  "Date": "2018-10-06T12:26:13.882Z",
  "SourceIdentifier": "rds:snapshot-replica-2018-10-06-12-24",
  "Message": "Automated snapshot created",
  "EventID": "RDS-EVENT-0091"
}

After click send. you can wait a minute and see your resource on lambda monitor(metrics and log)

Conclusion

👏🏻😉 We now have the ability to efficiently manage and provide the Lambda function with Terraform while simultaneously preparing related resources. In this article, I'll present an innovative approach to design the Serverless architecture, enabling Lambda to achieve what it couldn't do independently. By creating a co-worker to collaborate with Lambda, we open up new possibilities and enhance its capabilities.

😋😜 My hope is that this article will serve as an inspiration for you to leverage Serverless services in your own scenarios. Embrace the power of Serverless and discover how it can revolutionize your projects!😘😘

AWS Lambda replicate RDS snapshot

Watcharin(start) — Fri, 28 Jul 2023 11:59:09 +0000

A few days ago, I encountered an issue with AWS RDS services, specifically with SQL server databases that could not be backed up using the AWS Backup service. Currently, there are limitations for requesting RDS to backup databases. However, it is still possible to create backup requests for our database using the RDS API.

Therefore, I want to remove the activity of engineers having to create backups manually through API or web console, and instead develop Serverless services to automate this process. This approach can also be extended to solve other issues in the future.

Let’s go from this guideline

Preparations

Before creating any Lambda function, we need to read a document that explains how to develop them. You can find it here. Then, you should have the following.

Golang version 1.16 or higher
Pre-commit that use to ensure you are pushing a verified source code to the SCM
Taskfile the optional, that use to reduce routine step in your development.

Overview solution is…

We focus on events generated by RDS when a snapshot is created. These events are triggered by AWS EventBridge, which provisions a function in AWS Lambda to execute the necessary actions.

First thing

You can provide a folder structure for develop a Lambda function in your workspace.

Create new Github repository and clone it into your local machine.
```
git clone <your_repo_url> 
```

Go to your folder and init a new Golang project. it will create a new go.mod file.

go mod init <your_repo_url>
#Example
go mod init github.com/StartloJ/lambda-rds-utils

(Optional) Create a pre-commit policy file to help you improve development quality.

#.pre-commit-config.yaml
repos:
- repo: https://github.com/dnephin/pre-commit-golang
  rev: master
  hooks:
  - id: go-fmt
  - id: go-vet
  - id: go-imports
  - id: go-mod-tidy

Create a first function

Now, we are ready to develop the Lambda function to handle the event. You can follow the steps below to create a real use case function that replicates the RDS snapshot across regions.

Remember the event structure from AWS EventBridge

You can see more event on official docs here.

{
    "version": "0",
    "id": "844e2571-85d4-695f-b930-0153b71dcb42",
    "detail-type": "RDS DB Snapshot Event",
    "source": "aws.rds",
    "account": "123456789012",
    "time": "2018-10-06T12:26:13Z",
    "region": "us-east-1",
    "resources": ["arn:aws:rds:us-east-1:123456789012:snapshot:rds:snapshot-replica-2018-10-06-12-24"],
    "detail": {
      "EventCategories": ["creation"],
      "SourceType": "SNAPSHOT",
      "SourceArn": "arn:aws:rds:us-east-1:123456789012:snapshot:rds:snapshot-replica-2018-10-06-12-24",
      "Date": "2018-10-06T12:26:13.882Z",
      "SourceIdentifier": "rds:snapshot-replica-2018-10-06-12-24",
      "Message": "Automated snapshot created",
      "EventID": "RDS-EVENT-0091"
    }
}

Define a configuration for functions

I will define parameters to reuse this function below.

OPT_SRC_REGION is a source region of RDS snapshots for copy.
OPT_TARGET_REGION is a target region that we need to store snapshots.
OPT_DB_NAME is a identity of RDS DB name that we need to copy snapshots.
OPT_OPTION_GROUP_NAME is a target option group to attachment to replicate snapshot in the target region.
OPT_KMS_KEY_ID is a KMS key that use to encrypt snapshot on target region.

Define main func

package main

import (
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/sirupsen/logrus"
)

func main() {
    // Make the hangler available for remote procedure call by Lambda
    logrus.Info("we starting handle lambda...")
    lambda.Start(HandleEvents)
}

Create a function handle by lambda

// Define func to receive a variable from `BridgeEvent`
// and it should return `error` if any mistake.
func HandleEvents(events BridgeEvent) error {
    return nil
}

Define a new AWS session

// define AWS session for source and target region to copy snapshot
func HandleEvents(event BridgeEvent) error {
    // start copy code
    des_sess := session.Must(session.NewSessionWithOptions(session.Options{
        Config: aws.Config{
            Region: aws.String(viper.GetString("target_region")),
        },
    }))
    src_sess := session.Must(session.NewSessionWithOptions(session.Options{
        Config: aws.Config{
            Region: aws.String(viper.GetString("src_region")),
        },
    }))
    // end copy code
    return nil
}

Create func to get all existing snapshot

// This func input session of RDS service. Then, it get all list of database
// snapshot that already in your resources in List of Snapshot name.
func ListAllSnapshot(svc *rds.RDS) ([]string, error) {

    db := viper.GetString("DB_NAME")

    // This line will return a list of snapshots included snapshot name.
    out, err := svc.DescribeDBSnapshots(&rds.DescribeDBSnapshotsInput{
        DBInstanceIdentifier: &db,
    })
    if err != nil {
        panic(err)
    }

    var snapShotName []string

    // this loop is extract a snapshot name only.
    for _, b := range out.DBSnapshots {
        logrus.Infof("We have %s", ConvertToString(*b.DBSnapshotArn))
        snapShotName = append(snapShotName, ConvertToString(*b.DBSnapshotArn))
    }

    return snapShotName, nil
}

Get a list of RDS snapshot name from source and target region

func HandleEvents(event BridgeEvent) error {
    ...
    // start copy code
    targetSvc := rds.New(des_sess)
    rep_dbSnapshots, err := ListAllSnapshot(targetSvc)
    if err != nil {
        return fmt.Errorf("we can't get any Snapshot name")
    }

    srcSvc := rds.New(src_sess)
    src_dbSnapshots, err := ListAllSnapshot(srcSvc)
    if err != nil {
        return fmt.Errorf("we can't get any Snapshot name")
    }
    // End copy code
    ...
}

Create a func to remove duplicate snapshot as source and target

// this func use to remove duplicate name source if found in target
// you shouldn't switch position input to this func.
// `t` is mean a target that use double check to `s`
// it will remove a value in `s` if found it in `t`
func GetUniqueSnapShots(t, s []string) ([]string, error) {
    //Create a map to keep track a unique strings
    uniqueMap := make(map[string]bool)

    //Iterate over `s` and add each string to map
    for _, str := range s {
        uniqueMap[str] = true
    }

    //Iterate over `t` and remove any string that are already in uniqueMap
    for _, str2 := range t {
        delete(uniqueMap, str2)
    }

    //Convert the unique string from Map to slice string
    result := make([]string, 0, len(uniqueMap))
    for str := range uniqueMap {
        result = append(result, str)
    }

    if len(result) == 0 {
        return nil, fmt.Errorf("not any Snapshot unique between source and target region")
    }

    return result, nil
}

Define a unique snapshot

func HandleEvents(event BridgeEvent) error {
    ...
    // start copy code
    dbSnapShots2Copy, err := GetUniqueSnapShots(rep_dbSnapshots, src_dbSnapshots)
    if err != nil {
        logrus.Warnf("it doesn't any task copy snapshot to %s", viper.GetString("target_region"))
        return nil
    }
    // End copy code
    ...
}

Create a func copy RDS snapshot across region

// Request to AWS RDS API for create event copy a specific snapshot
// into across region and encrypt it by KMS key multi-region
func CopySnapshotToTarget(svc *rds.RDS, snap string) (string, error) {
    targetSnapArn := strings.Split(snap, ":")
    targetSnapName := targetSnapArn[len(targetSnapArn)-1]

    // Copy the snapshot to the target region
    copySnapshotInput := &rds.CopyDBSnapshotInput{
        OptionGroupName:            aws.String(viper.GetString("option_group_name")),
        KmsKeyId:                   aws.String(viper.GetString("kms_key_id")),
        CopyTags:                   aws.Bool(true),
        SourceDBSnapshotIdentifier: aws.String(snap),
        TargetDBSnapshotIdentifier: aws.String(targetSnapName),
        SourceRegion:               aws.String(viper.GetString("src_region")),
    }

    _, err := svc.CopyDBSnapshot(copySnapshotInput)
    if err != nil {
        logrus.Errorf("Copy request %s is failed", snap)
        return "", err
    }

    logrus.Infof("Copy %s is created", snap)
    return fmt.Sprintf("Copy %s is created", snap), nil
}

Define loop to execute all snapshot value

func HandleEvents(event BridgeEvent) error {
    ...
    // start copy code
    for s := range dbSnapShots2Copy {
        logrus.Infof("trying to copy DBSnapshot to %s...", viper.GetString("target_region"))
        _, err := CopySnapshotToTarget(targetSvc, dbSnapShots2Copy[s])
        if err != nil {
            logrus.Error(err.Error())
        }
    }
    // End copy code
    ...
}

If all workflow is complete, it will normal end function from AWS Lambda controller.

Finally steps

If you finished to develop function, you will build and pack it with ZIP format.

export BINARY_NAME="lambda-rds-util" #you can change this name.
go build -o $BINARY_NAME main.go
zip lambda-$BINARY_NAME.zip $BINARY_NAME

You can upload your compress file(also ZIP file) to S3.

#you can push it with below command
export S3_NAME="<your_s3_bucket_name>"
aws s3api put-object --bucket $S3_NAME --key lambda-$BINARY_NAME.zip --body ./lambda-$BINARY_NAME.zip

In the web console of AWS Lambda, you can define a new resource and select source from S3.
In the web console, you can test function in AWS Lambda to ensure your code is work!.

Conclusion

You can automate certain activities on AWS or other cloud providers using the Serverless service, such as AWS Lambda. It can enhance your development experience and enable you to explore new possibilities. You can even extend this idea to develop a new workflow in your work.

I hope this blog has inspired you to consider using Serverless in your development journey.

Get starting DevOps at Opsta: Your Path to Success

Watcharin(start) — Sun, 11 Jun 2023 08:47:22 +0000

A good DevOps person is someone who acts as a connector between the development and operations teams, seamlessly integrating automation, collaboration, and continuous improvement to orchestrate and drive organizations to the pinnacle of innovation.

! In today’s rapidly evolving technologies landscape, The role of the DevOps professional is becoming increasingly crucial. Our organization strive for faster development cycles, efficient collaboration, reliable deployment, innovation researcher, and business partnership. We’re focus the demand for skilled DevOps practitioners continues to grow.

Opsta, a leading company in the industry, present a remarkable opportunity to embark on a fulfilling DevOps career. This guide aims to equip aspiring DevOps professional with the knowledge and skill necessary to excel in the VUCA(acronym, which stands for "Volatile," "Uncertain," "Complex," and "Ambiguous”) world of DevOps at Opsta.

Understanding the DevOps culture

Explore the core principles and methodology that underpin the DevOps culture at our company.
Learn how to promotes collaboration, communication, and shared responsibilities among team
Discover the significance of automation, continuous integration & continuous delivery, observability(metrics, log, trace), Infrastructure as Code, and Network & OS in DevOps approach.
To drive cross-functional teams for tools, you need to be adaptable, goal-focused, and able to innovate towards your target.

Essentials technical skills for the DevOps engineer

To design, manage, and troubleshoot networks and operating systems, you need a solid foundation of knowledge in these areas..
Dive into Infrastructure as Code(IaC) conceptual and implementation such as Terraform and Ansible for provisioning and managing infrastructure. Maybe, the others can do it same.
Strong knowledge in containerize technologies such as Docker, Podman, or Kubernetes for scalable, resilience, and portable.
Gain proficiency in public cloud platform like GCP, AWS, and Azure to leverage our services for building and deploying application.
Understanding into Observability(also monitoring, logging, tracing) platform such as Prometheus, Grafana, ELK stack, Istio, and others to ensure reliability, consistency, and availability on system.

Collaboration and communication in DevOps environment

Explore the importance of effective communication among team members and related teams.
Learn about agile methodologies, such as Scrum or Kanban, that are adopted to streamline development and delivery.
Discover significance of cross-function teams and encourage a culture of knowledge sharing and continuous learning.

Embracing automation in DevOps

Learn about the advantages of implementing a configuration management tool such as Ansible, Puppet, or others. These tools can be used to manage and configure both infrastructure and applications.
Explore the benefits of using a CI/CD pipeline with tools such as Jenkins, Gitlab-CI, Github Actions, ArgoCD, Tekton, and others to reduce human effort, team workload, and ensure compliance.
Learn the benefits of using cloud native services like AWS codepipeline, GCP cloudbuild, and Azure DevOps to automate for effective CI and CD applications.

Continuous learning and professional development

Continue moving and keeping abreast of DevOps along with technologies' innovations, trends, and expertise.
Discover resources that can help you improve your knowledge and skills, such as blogs, online and offline courses, and conferences.
Learn how to contribute to the community in order to assist the development of teams and other individuals through knowledge sharing, documentation, and training programs.

Conclusion

Technical know-how, teamwork, and a commitment to continual development were necessary for becoming an effective DevOps practitioner. You may start your journey toward becoming a successful DevOps professional by adopting the DevOps culture, developing key technical skills, and fostering good communication. Take embrace the chance to contribute meaningfully by promoting efficiency, creativity, and technical excellence within the DevOps environment.

I'll go into further detail about the technical skills you need to adopt the DevOps culture. Then, I'll talk about my experience working in a DevOps position and how it contributed to shifting a tech team toward DevOps culture. If you living in Thailand and have faith in technical professionals, you are welcome to join us in tackling challenging problems and empowering customers with technical standards.

Level up your development in Git

Watcharin(start) — Sat, 03 Jun 2023 12:41:07 +0000

As developers, we strive to improve ourselves every day and become better than we were before. I have found some tools and solutions that can help us. I would like to share a trick that I often use to simplify complex processes.

Today, we use GIT to version control in daily work. We have many styles in GIT workflow like feature branch, single branch, and other. The Git is concept and software that help us to saving and tracking file(code) history in standard idea. We use Git in different platforms such as GitHub, Gitlab, Bitbucket, and other.

We have process in step to review and ensure our code source before we commit code and push them to the Git server. How can we ensure our source code that correct syntax or available to execute in another place? Therefore, we make an Unit-test and a linter(also linting framework) that we can execute to verify our code in local. Maybe, We have a separate environment that make to test our source code, up to you!.

To help us to accelerate in development feature. I prefer you to use a open source project from community that called Pre-commit. We can use them to replace our manual step such as run unit-test and fix format. Then, how can we use it? I will show you in this blogs.

What is pre-commit

It is a framework for managing and maintaining multi-language pre-commit hooks.

Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every commit to automatically point out issues in code

Usage step by step

Install into local machine

Before you can run hooks, you need to have the pre-commit package manager installed.

Using pip for Linux

# for Python2
pip install pre-commit

# for Python3
pip3 install pre-commit

Using homebrew for MacOS

brew install pre-commit

Initial pre-commit policy in root directory source code

Add a pre-commit configuration. You must created file named .pre-commit-config.yaml. You can generate a very basic configuration using pre-commit sample-config .

repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.3.0
    hooks:
    -   id: check-yaml
    -   id: end-of-file-fixer
    -   id: trailing-whitespace
-   repo: https://github.com/psf/black
    rev: 22.10.0
    hooks:
    -   id: black

Install Git hook script

If you want to automate execute pre-commit in every committed. You should install Git hook before.

$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

now pre-commit will run automatically on git commit!

Run against all files

$ pre-commit run --all-files
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/psf/black.
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Check Yaml...............................................................Passed
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1

Files were modified by this hook. Additional output:

Fixing sample.py

black....................................................................Passed

oops! looks like I had some trailing whitespace. Now, you can fix your error first and re-add your code to commit stage in Git again.

Conclusion

We can automate our development processes with framework called pre-commit. If you need to use pre-commit in advance step, you can read more details in below reference link and kindly share your idea to improve our process with us.

Ref

Data ETL in DevOps processes

Watcharin(start) — Sat, 04 Jun 2022 10:06:34 +0000

🤗 Hi everyone, I have new blog's series to you. It have long time that I need to public this content. I would shared my perspective about relative of DevOps process and Data process. This post base on my experience.

In the normal Development and Operation process, we have a lot of data or message generation.
We can see the behavior of events from user(client) to the latest service in workflow. When a user accesses our platform or service, it is called a request.
The request is one of the message types that we can use to process or analytic. For all transactions on user workflow we have many many messages(also data or events)
to see what happens in your services. Further, we can see other messages on the infrastructure layer and Networking layer.

Oftenly, DevOps processes focus on the What happened in our platform, so they use logs messages created by application and observability tools.
In this session, I will present how DevOps uses previous data to observe what happened in your platform and what we should do if we have data.

What is ETL

ETL is called Extract, Transform and Load. It is data integration that refers to three distinct processes. We use to transfer data from multiple sources with any format to any data storage and already to visualize future trends or current information.

Next, we can work each function in following:

Extract is used to scrape data from any sources and any format like app log, flat file, metrics, or api response.

Transform is used to convert different data formats to the same format, cleaning data, filter useless data, add metadata in each message, and other processes.

Load is used to transfer data that is transformed to store in a data lake, data warehouse, or other storage.

In the data pipeline, we can adapt with

extract → transformation → load

or

extract → load → transformation.

You can design your data pipeline base-on your platform and tools.

ETL in basically

Now, you understand what ETL is, but when should we implement them? In begin to, answer this question, you need to design data flow from top layer to bottom or bottom layer to top.

Data design top layer to bottom

We will gather requirements from business about what we need to see in the end.

Example:

We need to see how many users have had bad experiences on our platform.
We need to see where the web page that users focus and oftenly use.
We need to see what the root cause of our service is broken.
We need to see what services we should expand resources, improve performance, and scale. ETC.

We were started with what we need to design data flow in the platform lifecycle. You can use goals from the engineer team, business team, or developer team to design and implement.

Data design bottom layer to top

We will survey the existing resources and data in our system. Next, we try to expand ideas from our data to what we can see in current data.

Example:

We have requested errors and many transactions in a few minutes, so that we can detect this event and analyze what happened and who is facing this problem.
We have frequency response time more than 3 sec in some features that should have response time less than 3 sec. We can use this data to analyze to see what service has a delay and what is the root cause.
We have an error log that shows the transaction failed. We can use this data and surround log to analyze what happened.

In my example, you can see patterns on what we have, then how can we use this data to analyze what happened currently. In advance, we can analyze information to get insight on your information.

Challenge in data integration

In application factor, we do not think about data in the software design phase. We expose useless information in our services or we do not have necessary data that can tell us what we want.
In person factor, who is using data not designed, who is designed do not use them. As explained, the developer team is designing architecture for their services with business requirements, but in the design phase, they have no other team to participate like the operation team or person who is using data from this service to analyze.
In observed tools, each team will have their stack for data integration and other teams can not use this data because they can not access this data or incompatibility between tools. For example, team-A uses AWS Redshift to store data and native AWS service to build their data pipeline, but team-B uses an on-premise stack like Hadoop and their developed tool to integrate Apache Hadoop. Another team can not access AWS Redshift because they have compliance or some reason. Then, we have 2 locations to store data that we can not merge and analyze.

Next session, I will talk about simple scenario to use ETL concept on DevOps world.

Simple PKI with CA issuer

Watcharin(start) — Sun, 22 May 2022 10:37:26 +0000

This workshop will simulate the scenario in how we can use PKI to make secure communication. All this session we work on terminal, you can following in step-by-step.

In the late session, we learned about how to use public/private keys in an easy case.

I explained the workshop to implement a solution. Then, We upgrade to an advanced workshop

to understand how to implement public/private CA intermediate servers.

If you are not read previous topic, you can go to:

Prerequisite

OpenSSL
cURL
Docker
tree

Create root CA certificate

In first thing, we will generate simple self-sign root CA certificate with following step:

Create SSL private key

   $ openssl genrsa -out rootCAkey.pem 4096
   Generating RSA private key, 4096 bit long modulus (2 primes)
   .........................................++++
   .........++++
   e is 65537 (0x010001)

We generate random keys with 4,096 character and you can use other lengths like 1024, 2048, 8192, or ETC. You can increase or decrease your key length to optimize cpu power to process them.

Create Certificate from private key

   $ openssl req -x509 -sha256 -new -nodes -key rootCAkey.pem \
      -days 14 -out rootCAcert.pem
   You are about to be asked to enter information that will be incorporated
   into your certificate request.
   What you are about to enter is what is called a Distinguished Name or a DN.
   There are quite a few fields but you can leave some blank
   For some fields there will be a default value,
   If you enter '.', the field will be left blank.
   -----
   Country Name (2 letter code) [AU]:TH
   State or Province Name (full name) [Some-State]:Bangkok
   Locality Name (eg, city) []:BangKhea
   Organization Name (eg, company) [Internet Widgits Pty Ltd]:Opsta Thailand
   Organizational Unit Name (eg, section) []:DevOps
   Common Name (e.g. server FQDN or YOUR name) []:OpstaRootCA
   Email Address []:watcharin@opsta.co.th

For the previous command, I request a new certificate file with some information. I use the digest algorithm SHA-256 and set it to expire before 14 days. It shows a prompt on your terminal for adding more information. You can see an example above.

Now, we can verify own root CA

   openssl x509 -text -in rootCAcert.pem

You can see your certificate information. Then go to the next step to create a chain certificate.

Create server certificate with issuer

Generate random private key with same step root CA

   # You can replace rsaServerKey.pem to your name
   # And can change key size from 2048 to other length
   $ openssl genrsa -out rsaServerKey.pem 2048
   Generating RSA private key, 2048 bit long modulus (2 primes)
   ......................................................................+++++
   ...............+++++
   e is 65537 (0x010001)

Create CNF file that provide meta data for CSR

   # req.cnf
   [req]
   req_extensions = v3_req
   distinguished_name = dn
   prompt = no

   [dn]
   CN = ssl-lab.example.local
   C = TH
   L = Bangkok
   O = Opsta Thailand
   OU = DevOps

   [v3_req]
   subjectAltName = @san_names

   [san_names]
   DNS.1 = ssl-lab.example.local
   DNS.2 = localhost
   DNS.3 = 127.0.0.1

Provide simple configuration to create CSR file. This file has SAN(Subject Alternative Name) with 3 names in [san_names]. For @san_names, I use for extend alternative name for server.

Generate CSR certificate

   $ openssl req -new -out req.csr -key rsaServerKey.pem -sha256 -config req.cnf
   $ cat req.csr
   -----BEGIN CERTIFICATE REQUEST-----
   MIIC9zCCAd8CAQAwaTEeMBwGA1UEAwwVc3NsLWxhYi5leGFtcGxlLmxvY2FsMQsw
   CQYDVQQGEwJUSDEQMA4GA1UEBwwHQmFuZ2tvazEXMBUGA1UECgwOT3BzdGEgVGhh
   aWxhbmQxDzANBgNVBAsMBkRldk9wczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
   …
   k9jkEKhPeICqJgHLbyiEcEc9xaPTMPb35cBQT8irnUq1+WbQamhWDIBmwzDHCSyf
   jCK672ACRleQCj8kk5l+dOB0wzWEaDcCoQNwIwqg2RpfDKc1lHARroBzm1P/4grg
   NaF5EYJJWhXUXGKq68meHpCTGzgC7M06rwBdOR+8l0GIUZ5K25MvNg3+ZA==
   -----END CERTIFICATE REQUEST-----

Verify CSR information, we focus on Requested Extension.

   $ openssl req -in req.csr -noout -text
   …
   Requested Extensions:
               X509v3 Subject Alternative Name: 
                   DNS:ssl-lab.example.local, DNS:localhost, DNS:127.0.0.1

Create trusted certificate with root CA with CSR file

   $ openssl x509 -req -sha256 -in req.csr -CA ../rootCAcert.pem \
     -extfile <(printf "subjectAltName=DNS:ssl-lab.example.local,DNS:localhost") \
     -CAkey ../rootCAkey.pem -CAcreateserial -out CertServer.pem -days 7
   …
   Signature ok
   subject=CN = ssl-lab.example.local, C = TH, L = Bangkok, O = Opsta Thailand, OU = DevOps
   Getting CA Private Key

We create trusted certificates with digest algorithm SHA-256 and create root CA serials that track how many certificates were issued with -CAcreateserial , so if not the first certificate you will use -CAserial instead. Then, this certificate will stay for 7 days only.

Verify certificate information that should have issuer

   $ openssl x509 -text -in CertServer.pem
   …
   Issuer: C = TH, ST = Bangkok, L = BangKhea, O = Opsta Thailand, OU = DevOps, CN = OpstaRootCA, emailAddress = watcharin@opsta.co.th

We will see Issuer information under Certificate >> Data.

Run demo web server

Now you can replace this certificate and private key with the previous session (also topic name little-step-to-use-pki-easiest) and test again. You will install your custom root CA to your browser or machine to see a difference between a single self-signed certificate and certificate with CA issuer.

Run demo web server

Create Dockerfile with simple configuration

FROM nginx:1.20-alpine

ADD default.conf /etc/nginx/conf.d/default.conf

COPY ssl /etc/nginx/ssl

RUN chown -R 0:0 /etc/nginx/ssl \
    && chown -R 0:0 /etc/nginx/conf.d/default.conf

Create Nginx site configuration in following code

server {
    listen       80;
    server_name  localhost ssl-lab.example.local;
    return 301 https://localhost:8443$request_uri;
}
server {
    listen              443 ssl;
    server_name         localhost ssl-lab.example.local;
    keepalive_timeout   70;
    ssl_certificate     /etc/nginx/ssl/CertServer.pem;
    ssl_certificate_key /etc/nginx/ssl/rsaServerKey.pem;
    ssl_protocols       TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers off;
    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

Start docker with default options

docker build -t nginx-ca:demo .
docker run -it --rm -p 8080:80 -p 8443:443 nginx-ca:demo

Use cUrl with verbose argument to verify

$ curl -vIL --cacert ../rootCA/rootCAcert.pem -XGET https://localhost:8443

*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: rootCA/rootCAcert.pem
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=ssl-lab.example.local; C=TH; L=Bangkok; O=Opsta Thailand; OU=DevOps
*  start date: Sep 16 06:36:23 2022 GMT
*  expire date: Oct 16 06:36:23 2022 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: C=TH; ST=Bangkok; O=Opsta Thailand; OU=DevOps; CN=OpstaRootCA; emailAddress=watcharin@opsta.co.th
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/7.81.0
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: nginx/1.20.2
Server: nginx/1.20.2
< Date: Mon, 19 Sep 2022 02:57:09 GMT
Date: Mon, 19 Sep 2022 02:57:09 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 612
Content-Length: 612
< Last-Modified: Tue, 16 Nov 2021 15:04:23 GMT
Last-Modified: Tue, 16 Nov 2021 15:04:23 GMT
< Connection: keep-alive
Connection: keep-alive
< ETag: "6193c877-264"
ETag: "6193c877-264"
< Accept-Ranges: bytes
Accept-Ranges: bytes

< 
* Excess found: excess = 612 url = / (zero-length body)
* Connection #0 to host localhost left intact

Little step to use PKI easiest

Watcharin(start) — Sun, 01 May 2022 12:13:28 +0000

From previous posts, I said about What is PKI. Then this post, I will explain How to use them in some case and next post I will give an advanced technique for PKI.

This lab will simulate the scenario in how we can use PKI to make secure communication. All this session we work on terminal, you can following in step-by-step.

Prerequisite

CLI terminal(On linux or something like that)
OpenSSL
cURL
Docker
tree

Use self-sign to secure communication

Scenario: We will provide simple web server and map SSL certificate on it. We create connection with cURL and debug it to see what's happen in handshake state. Then, we compare between insecure and secure.

Generate SSL with CN is ssl-lab.example.local.

    $ openssl req -newkey rsa:4096 -nodes -keyout ssl/ssl-lab.example.local.key \
        -x509 -sha256 -days 3 \
        -subj "/C=TH/ST=BKK/O=Opsta/OU=DevOps/CN=ssl-lab.example.local" \
        -addext "subjectAltName = DNS:ssl-lab.example.local, DNS:localhost, DNS:127.0.0.1" \
        -out ssl/ssl-lab.example.local.crt

    Generating a RSA private key
    ...................++++
    ..............................................................................++++
    writing new private key to 'ssl-lab.example.local.key'
    -----

Explain command.

For create certificated, we use OpenSSL library to create and manage them.

You can use following arguments to generate 2 files like private and public key:

req = request new resources

-newkey rsa:4096 = create new private key with length 4096 keys.

-nodes = export output key with plaintext.

-keyout /path/to/save/file.key = specific location to save private key.

-x509 = output with x509 structure.

-sha256 = choose a message digest algorithm.

-days 3 = expired date for this certificate in 3 day.

-subj "something" = quick add metadata for certificate. The arg must be formatted as /type0=value0/type1=value1/type2=..., characters may be escaped by \ (backslash), no spaces are skipped.

-addext "something" = add some extension metadata. The arg must be formatted as key=value

-out = specific location to save public key and digital certificate.

Check file should created.

    $ tree ssl

    ssl
    ├── ssl-lab.example.local.crt
    └── ssl-lab.example.local.key
    0 directories, 2 files

Verify certificate is matched with openssl cli.

$ openssl x509 -noout -modulus -in ssl/ssl-lab.example.local.crt | openssl md5
(stdin)= ac9173e222ab6b766e49da3069268dd0
# OUTPUT will difference in your terminal

$ openssl rsa -noout -modulus -in ssl/ssl-lab.example.local.key | openssl md5
(stdin)= ac9173e222ab6b766e49da3069268dd0

If your see same stdin in 2 output, you're right.

Now, you already to create web server in docker for learn how SSL work. First, you will check your docker daemon working and run a sample Nginx web server.

Create a Nginx site config in following.

<!-- file location in ./config/default.conf -->
server {
    listen       80;
    server_name  localhost ssl-lab.example.local;
    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}
server {
    listen              443 ssl;
    server_name         localhost ssl-lab.example.local;
    keepalive_timeout   70;
    ssl_certificate     /etc/nginx/ssl/ssl-lab.example.local.crt;
    ssl_certificate_key /etc/nginx/ssl/ssl-lab.example.local.key;
    ssl_protocols       TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers off;
    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

Create a Dockerfile in following.

FROM nginx:1.20-alpine

ADD config/default.conf /etc/nginx/conf.d/default.conf

COPY ssl /etc/nginx/ssl

RUN chown -R 0:0 /etc/nginx/ssl \
    && chown -R 0:0 /etc/nginx/conf.d/default.conf

Then, build customize http server in docker.

$ docker build -t http:lab .

Sending build context to Docker daemon  233.5kB
Step 1/4 : FROM nginx:1.20-alpine
 ---> 5c05ca045835
Step 2/4 : ADD config/default.conf /etc/nginx/conf.d/default.conf
 ---> Using cache
 ---> 1284fcf47f5b
Step 3/4 : COPY ssl /etc/nginx/ssl
 ---> 920c31460cbf
Step 4/4 : RUN chown -R 0:0 /etc/nginx/ssl     && chown -R 0:0 /etc/nginx/conf.d/default.conf
 ---> Running in f8bfa038944f
Removing intermediate container f8bfa038944f
 ---> 725ee42e058b
Successfully built 725ee42e058b
Successfully tagged http:lab

You can run http(s) server in terminal.

docker run -it --rm -p 8080:80 -p 8443:443 http:lab

Verify http server has running with docker ps or curl http://localhost:8080. It just return result on STDOUT.

$ docker ps

CONTAINER ID   IMAGE            COMMAND                  CREATED         STATUS         PORTS                                                                            NAMES
1235d24fe059   http:lab   "/docker-entrypoint.…"   7 minutes ago   Up 7 minutes   0.0.0.0:8080->80/tcp, :::8080->80/tcp, 0.0.0.0:8443->443/tcp, :::8443->443/tcp   relaxed_montalcini

You can use cURL to see what happen.

Get web page with insecure protocol(HTTP).

$ curl -v http://localhost:8080

*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.20.2
< Date: Mon, 18 Apr 2022 11:45:35 GMT
< Content-Type: text/html
< Content-Length: 612
< Last-Modified: Tue, 16 Nov 2021 15:04:23 GMT
< Connection: keep-alive
< ETag: "6193c877-264"
< Accept-Ranges: bytes
< 
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host localhost left intact

Use curl in default option and see error in https tunnel.

$ curl -v https://localhost:8443

*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

In the error response, We can translate in a part of message.

They're try to make a connection to endpoint.

Trying 127.0.0.1:8443...  
Connected to localhost (127.0.0.1) port 8443 (#0)

They're prepared information for connection.

ALPN, offering h2  
ALPN, offering http/1.1  
successfully set certificate verify locations:  
  CAfile: /etc/ssl/certs/ca-certificates.crt  
  CApath: /etc/ssl/certs

They're send first contact message in hello.

TLSv1.3 (OUT), TLS handshake, Client hello (1):  
TLSv1.3 (IN), TLS handshake, Server hello (2):

Server send digital certificate to client.

 TLSv1.2 (IN), TLS handshake, Certificate (11):

Client verify certificate integrity and found in self-sign issue.

TLSv1.2 (OUT), TLS alert, unknown CA (560):
SSL certificate problem: self signed certificate

Client close connection and report error message.

Closing connection 0  
curl: (60) SSL certificate problem: self signed certificate  
More details here: 'https://curl.se/docs/sslcerts.html'  

curl failed to verify the legitimacy of the server and therefore could not  
establish a secure connection to it. To learn more about this situation and  
how to fix it, please visit the web page mentioned above.

You can use crt certificate file in previous step to trust this communication.

You can see normal establish connection in debug log.

$ curl -v --cacert ssl/ssl-lab.example.local.crt https://localhost:8443

*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: ssl/ssl-lab.example.local.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=TH; ST=BKK; O=Opsta; OU=DevOps; CN=ssl-lab.example.local
*  start date: Apr 17 15:41:24 2022 GMT
*  expire date: Apr 20 15:41:24 2022 GMT
*  subjectAltName: host "localhost" matched cert\'s "localhost"
*  issuer: C=TH; ST=BKK; O=Opsta; OU=DevOps; CN=ssl-lab.example.local
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.20.2
< Date: Mon, 18 Apr 2022 12:03:50 GMT
< Content-Type: text/html
< Content-Length: 612
< Last-Modified: Tue, 16 Nov 2021 15:04:23 GMT
< Connection: keep-alive
< ETag: "6193c877-264"
< Accept-Ranges: bytes

Conclusion

You can see how to use PKI concept to make secure communication in this post. Then, you can adapt this step to your application to basic secure communication.

See ya in next posts to use CA for intermediate issuer.

What we should know in PKI

Watcharin(start) — Mon, 18 Apr 2022 12:36:31 +0000

PKI(Public Key Infrastructure)

In this topic, we're learn about What is PKI and How can we use it. You can imagine the importance of security and privacy in customer’s data. This topic should help you to understand.

What is PKI? Before answering this question, you should know: What is Symmetric and Asymmetric encryption.

You can understand a key difference between Symmetric and Asymmetric encryption. In a simple scenario, Asymmetric encryption will be stronger and more secure than Symmetric encryption because we do not use same secret key for encrypt/decrypt messages.

Information security has grown to be a hugely important factor in any communication, transaction, and ETC. A little vulnerability or weakness in communication that can be leveraged to devastating effects. This session will show you two encryption techniques that can be used to tighten communication security.

Little note

Algorithm

a process or set of rules to be followed in calculations or other problem-solving operations.

An encryption algorithm that means formula in mathematical procedure for shift a bits or convert in original data. An information is made in cipher text and requires use of the key to transform the data into its original form.

Cryptography

The study of secure communications techniques that allow only the sender and intended recipient of a message to view its contents. The term is derived from the Greek word kryptos, which means hidden. An encryption algorithm will take an original message(plain text) converted to cipher text. We can use the key to encrypt or decrypt messages.

Symmetric encryption

This is a simplest procedure used to convert messages between plain text and cipher text. We can use the same key in the encryption and decryption process. It’s easy to manage and exchange for sender and receiver to use it. You can allow your services to generate the secret key and exchange in their trusted zone to communicate with a secure tunnel. You mostly set timeout or expire in a secret key to avoid loopholes in capture pattern messages.

Example encryption algorithm such as Blowfish, AES, RC4, DES, RC5, and RC6.

Asymmetric encryption

There are public and private keys to encrypt/decrypt, which is a new method for isolating one secret key to encryption and one secret key to decryption. You will create two secret keys: public key and private key. It is most likely used to exchange on extranet(also internet) or untrust zone to secure communication. You can share a public key to another person for only encrypting some message and send it to a person that has a private key for decrypting the cipher into the original message.

Example encryption algorithm such as RSA, ECC, Diffie-Hellman, DSS, and DSA.

SSL/TLS communication(combination of Asymmetric and Symmetric)

For most likely cases, We can summarize steps for creating communication between client and server. Before any communication, They must initial rules and keys for security. You can follow these steps to understand how they work to create secure communication.

Client --> send hello package to target server. In message has client's information such as support cipher suits(algorithms), SSL/TLS versions, and other.
Server --> Check client's information that match a policies you configure. If not match they will reject request. For match policies, They send certificate with public key to client.
Client --> Verify server's certificate that trusted or not. The certificate information should match in your request like common name, expired date, and ETC.
Client --> Create session key(also symmetric secret key) for secure communication tunnel. Then encrypt payload with public key in certificate before send them to server.
Server --> Receive message with session key, so decrypt with private key. They send acknowledge message to client for accept this key in secure tunnel.
Both --> Send/receive messages with session key. If session key expire, they will repeat above step to establish communication.

More secure with CA(Certificate Authority)

In the previous topic, We can see some weaknesses in the initial process. If your contact server endpoint is not your target server, they will fake the server or something like that. Can you trust it?

For solved, You can use CA to help you verify certificates that you can trust. In step 3, you can get the issuer from the server's certificate and ask them to guarantee this certificate.

If you have doubt in CA issuer from the server certificate. You can trust the root CA in your host, so you will integrity check with the root CA for a guarantee certificate. We call this process is Chain of trust.

Next, I will show you how it work.