Forem: Yutaro Mori

OpenTofu forked Terraform is now available

Yutaro Mori — Sun, 17 Sep 2023 14:14:25 +0000

I write articles in English with the help of DeepL

Introduction
Target Audience
About Open Source
Impact of HashiCorp's license change
The Foundation's claims
- About monetization of Terraform
Forked OpenTofu
Hands-on
- Is it possible to migrate existing resources
Conclusion
Original
References

Introduction

In August 2023, HashiCorp, a developer of OSS such as Terraform, switched Terraform from an open source license to the Business Source License (BSL).

HashiCorp adopts Business Source License

This license change will not directly affect HashiCorp users of Terraform and other products. However, there is a movement against HashiCorp's license change, and a group has formed to pursue true open source, forked from Terraform. It is OpenTF Foundation.

According to the manifesto, the OpenTF Foundation had demanded that HashiCorp switches Terraform back to an open-source license in order for Terraform to remain truly open source, and stated that it would maintain OpenTF forked from Terraform, if HashiCorp would not.

And so, on September 6, 2023 OpenTF was forked. Moreover, OpenTF has been renamed OpenTofu and the OpenTF Foundation joined the Linux Foundation.

// Detect dark theme var iframe = document.getElementById('tweet-1699076153968095494-250'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1699076153968095494&theme=dark" }

What kind of OSS is OpenTofu? And why did they decide to fork from Terraform? Let's take a look at their manifesto.

Target Audience

People who want to know about OpenTofu
People who want to know about the Foundation's claims

About Open Source

The term "truly open source" appears several times in the above manifesto.

I wasn't sure what "truly open source" meant, so I checked the definition of open source. The Open Source Initiative (OSI)¹ has the following definition of open source.

Free Redistribution
Source Code
Derived Works
Integrity of The Author’s Source Code
No Discrimination Against Persons or Groups
No Discrimination Against Fields of Endeavor
Distribution of License
License Must Not Be Specific to a Product
License Must Not Restrict Other Software

Impact of HashiCorp's license change

HashiCorp's newly adopted license, the Business Source License(BSL)², allows the release of the source code, but restricts commercial use. Because it a restricted use license, the owner clearly states that the BSL is not an open source license.

Q: Is the BSL an Open Source license?³

A: The BSL does not meet the Open Source Definition (OSD) maintained by the Open Source Initiative (OSI). OSD does not allow limitations on specific kinds of such, such as production use. However, most of the OSD criteria are met. Most important, the source code is made available. The BSL allows for copying, modification, creation of derivative works, redistribution, and non-production use of the code. It allows for (and encourages) the licensor to define an Additional Use Grant (e.g., allowing for free use below a specified level, like in this example).

However, HashiCorp clearly states that it allows the commercial use, except for use such as incorporating Terraform into a competing company's product.

All non-production uses are permitted. All production uses are allowed other than hosting or embedding the software in an offering competitive with HashiCorp commercial products, hosted or self-managed.⁴

What does the term “embedded” mean under the HashiCorp BSL license? HashiCorp answered this question.

Under the HashiCorp BSL license, the term “embedded” means including the source code or object code, including executable binaries, from a HashiCorp product in a competitive product. “Embedded” also means packaging the competitive product in such a way that the HashiCorp product must be accessed or downloaded for the competitive product to operate.

Don't get me wrong, even if we are building a competing product for HashiCorp, there is nothing wrong with using their Vault for the purpose of securing a product that competes with Terraform, or deploying in Terraform to build a product that competes with the Vault.⁵

The Foundation's claims

According to the manifesto, the Foundation strongly criticizes that the BSL is a poison pill for Terraform.

Even if the license change does not affect end users or integration partners currently using Terraform, they claim that HashiCorp could change the license terms or its interpretation of them in the future so that they would not be able to freely use it.

The Foundation is concerned that Terraform's influence will diminish and the community will wane as developers and enterprises leave Terraform out of their choices when selecting a tool to manage their infrastructure.

So, to return to the claim at the beginning, the Foundation had demanded that HashiCorp rescinds the license change, and if they would not do so, they would maintain OpenTF forked from Terraform.

At the end of the manifesto is a Q&A summary. Here is the question I found interesting.

Didn't HashiCorp adopt BUSL to deter vendors who were using Terraform but not contributing back?

HashiCorp's license change was prompted by some vendors free-riding on OSS. HashiCorp also states in its blog that using OSS only for one's own commercial purposes without contributing to the open source community is against the spirit of open source.

On the other hand, the Foundation refutes HashiCorp's claims as follows.

This is inaccurate and misleading. First of all, many of the vendors affected by the change to BUSL have made considerable contributions to the Terraform community.

For example, contributions have included Terraform binaries or modules, learning contents, and third-party tools such as Terragrunt⁶ and tflint⁷.

Terraform are part of a large ecosystem. The same is true of Kubernetes or Linux or Go or other major infrastructure tools. They stated that Terraform has developed because it is open source and it would be a bad thing not to acknowledge the work of vendors.

Also, they claim that HashiCorp might not support Terraform OSS issues and PR because they will not be able to direct monetization to Terraform Cloud and Terraform Enterprise.

About monetization of Terraform

As a side note, "Could HashiCorp's license change be due to poor monetization of Terraform?" A certain article claim that this is the reason HashiCorp has changed its license.

Asked why HashiCorp opted for a change of license, Stadil, CEO of Scalr⁸ claimed in the article.

The reality here is that HashiCorp is not executing well on its Terraform cloud product. And because they're not executing well, other competitors like my company Scalr and a number of others have stepped in.

To be honest, I think Terraform Cloud has become a little more difficult to because of the change in billing plans. The former plan allowed up to 5 users per organization to use Terraform Cloud for free, but the new plan allows up to 500 resources for free. 500 resources will soon exceed the free limit, so I think many users would have switched to the paid plan.

Forked OpenTofu

OpenTofu forked on September 6, 2023 already has over 6,000 stars.

Git clone and run opentf on own local machine.

$ git clone git@github.com:opentffoundation/opentf.git
$ cd opentf

$ make
$ go install

$ opentf version
OpenTF v1.6.0-dev
on linux_amd64

Hands-on

As a tutorial, play hands-on to deploy Docker containers of nginx from OpenTF.

terraform {
  required_providers {
    docker = {
      source = "kreuzwerker/docker"
      version = "~> 3.0.1"
    }
  }
}

provider "docker" {}

resource "docker_image" "nginx" {
  name         = "nginx:latest"
  keep_locally = false
}

resource "docker_container" "nginx" {
  image = docker_image.nginx.image_id
  name  = "tutorial"
  ports {
    internal = 80
    external = 8000
  }
}

Run opentf init command on the directory where main.tf file is placed just as with Terraform and create the project.

$ opentf init
~~~~~~~~~~~~~~~~~~~~~~~
OpenTF has been successfully initialized!

You may now begin working with OpenTF. Try running "opentf plan" to see
any changes that are required for your infrastructure. All OpenTF commands
should now work.
~~~~~~~~~~~~~~~~~~~~~~~

After running opentf apply command and launching Docker containers and nginx was displayed in the browser.

$ opentf apply

OpenTF used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTF will perform the following actions:

  # docker_container.nginx will be created
  + resource "docker_container" "nginx" {
      + attach                                      = false
      + bridge                                      = (known after apply)
      + command                                     = (known after apply)
      + container_logs                              = (known after apply)
      + container_read_refresh_timeout_milliseconds = 15000
      + entrypoint                                  = (known after apply)
      + env                                         = (known after apply)
      + exit_code                                   = (known after apply)
      + hostname                                    = (known after apply)
      + id                                          = (known after apply)
      + image                                       = (known after apply)
      + init                                        = (known after apply)
      + ipc_mode                                    = (known after apply)
      + log_driver                                  = (known after apply)
      + logs                                        = false
      + must_run                                    = true
      + name                                        = "tutorial"
      + network_data                                = (known after apply)
      + read_only                                   = false
      + remove_volumes                              = true
      + restart                                     = "no"
      + rm                                          = false
      + runtime                                     = (known after apply)
      + security_opts                               = (known after apply)
      + shm_size                                    = (known after apply)
      + start                                       = true
      + stdin_open                                  = false
      + stop_signal                                 = (known after apply)
      + stop_timeout                                = (known after apply)
      + tty                                         = false
      + wait                                        = false
      + wait_timeout                                = 60

      + ports {
          + external = 8000
          + internal = 80
          + ip       = "0.0.0.0"
          + protocol = "tcp"
        }
    }

  # docker_image.nginx will be created
  + resource "docker_image" "nginx" {
      + id           = (known after apply)
      + image_id     = (known after apply)
      + keep_locally = false
      + name         = "nginx:latest"
      + repo_digest  = (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  OpenTF will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

docker_image.nginx: Creating...
docker_image.nginx: Still creating... [10s elapsed]
docker_image.nginx: Creation complete after 10s [id=sha256:f5a6b296b8a29b4e3d89ffa99e4a86309874ae400e82b3d3993f84e1e3bb0eb9nginx:latest]
docker_container.nginx: Creating...
docker_container.nginx: Creation complete after 2s [id=7440041e3dfebcd576edc3aacd39ce16601447b87bde79ac97f72d20996f78e6]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Is it possible to migrate existing resources

I have confirmed that I create a new resource with OpenTF. However, if already used Terraform for IaC, I would like to migrate it instead of creating a new one. I have confirmed that I can deploy updates and delete resources using opentf commands for resources created in Terraform.

#resource created by Terraform
$ terraform state list
module.module-network.aws_db_subnet_group.database[0]
module.module-network.aws_default_network_acl.this[0]
module.module-network.aws_default_route_table.default[0]
module.module-network.aws_default_security_group.this[0]
module.module-network.aws_internet_gateway.this[0]
module.module-network.aws_route.public_internet_gateway[0]
module.module-network.aws_route_table.private[0]
module.module-network.aws_route_table.private[1]
module.module-network.aws_route_table.private[2]
module.module-network.aws_route_table.public[0]
module.module-network.aws_route_table_association.database[0]
module.module-network.aws_route_table_association.database[1]
module.module-network.aws_route_table_association.database[2]
module.module-network.aws_route_table_association.private[0]
module.module-network.aws_route_table_association.private[1]
module.module-network.aws_route_table_association.private[2]
module.module-network.aws_route_table_association.public[0]
module.module-network.aws_route_table_association.public[1]
module.module-network.aws_route_table_association.public[2]
module.module-network.aws_subnet.database[0]
module.module-network.aws_subnet.database[1]
module.module-network.aws_subnet.database[2]
module.module-network.aws_subnet.private[0]
module.module-network.aws_subnet.private[1]
module.module-network.aws_subnet.private[2]
module.module-network.aws_subnet.public[0]
module.module-network.aws_subnet.public[1]
module.module-network.aws_subnet.public[2]
module.module-network.aws_vpc.this[0]

$ opentf state list
module.module-network.aws_db_subnet_group.database[0]
module.module-network.aws_default_network_acl.this[0]
module.module-network.aws_default_route_table.default[0]
module.module-network.aws_default_security_group.this[0]
module.module-network.aws_internet_gateway.this[0]
module.module-network.aws_route.public_internet_gateway[0]
module.module-network.aws_route_table.private[0]
module.module-network.aws_route_table.private[1]
module.module-network.aws_route_table.private[2]
module.module-network.aws_route_table.public[0]
module.module-network.aws_route_table_association.database[0]
module.module-network.aws_route_table_association.database[1]
module.module-network.aws_route_table_association.database[2]
module.module-network.aws_route_table_association.private[0]
module.module-network.aws_route_table_association.private[1]
module.module-network.aws_route_table_association.private[2]
module.module-network.aws_route_table_association.public[0]
module.module-network.aws_route_table_association.public[1]
module.module-network.aws_route_table_association.public[2]
module.module-network.aws_subnet.database[0]
module.module-network.aws_subnet.database[1]
module.module-network.aws_subnet.database[2]
module.module-network.aws_subnet.private[0]
module.module-network.aws_subnet.private[1]
module.module-network.aws_subnet.private[2]
module.module-network.aws_subnet.public[0]
module.module-network.aws_subnet.public[1]
module.module-network.aws_subnet.public[2]
module.module-network.aws_vpc.this[0]

#possible to delete resources by opentf command
$ opentf destroy
~~~~~~~~~~~~~~~~~~~~~~~

Plan: 0 to add, 0 to change, 29 to destroy.

Do you really want to destroy all resources?
  OpenTF will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes
~~~~~~~~~~~~~~~~~~~~~~~
Destroy complete! Resources: 29 destroyed.

However, OpenTofu is still under development for an alpha release and support is limited to test and development environments.⁹

Currently, OpenTofu supports local testing and development: you can build the code, run the tests, build opentf binaries, and so on. That means you can now start experimenting with OpenTofu and contributing back via Issues, PRs, and RFCs.

If you find any bugs or errors, contribute to the community by filing an issues or PR.

Conclusion

I tried OpenTofu, which is a forked of Terraform. I think there are pros and cons to what the Foundation is claiming. I watch how the de facto standard for IaC turns into a tool in the future.

Original

my original post

References

https://mariadb.com/ja/resources/blog/mariadb-bsl/

https://ja.wikipedia.org/wiki/%E3%82%AA%E3%83%BC%E3%83%97%E3%83%B3%E3%82%BD%E3%83%BC%E3%82%B9%E3%82%BD%E3%83%95%E3%83%88%E3%82%A6%E3%82%A7%E3%82%A2%E3%81%AE%E6%AD%B4%E5%8F%B2

https://zenn.dev/koduki/articles/45f65a5318f019

Not documented App Runner specification

Yutaro Mori — Sat, 31 Dec 2022 03:02:49 +0000

Introduction

I plan to monitoring system with Datadog at office. At first, I prepared EC2 server for running Datadog Agent to monitor media servers and DB. However, I think that it's a waste of cost to run server just starting Datadog Agent. So I decided to migrate Datadog Agent from EC2 to ECS.

About App Runner

https://aws.amazon.com/apprunner/

AWS App Runner is a fully managed container application service that lets you build, deploy, and run containerized web applications and API services without prior infrastructure or container experience.

I found it convenient that it is not need to set complex network configure and is able to implement fully managed container CI/CD infrastructure. But, I found something odd when I monitored the media server externally with Agent on App Runner.

Increase Response time

This is the graph of response time Datadog Agent obtained from our servers.

Response time increased rapidly immediately after migrating to App Runner. After a day had passed and it was not back to normal, I moved it to ECS on Fargate.

Curl inspection

I prepared curl container¹ which obtains response time and measured ECS on Fargate and App Runner to compare the performance of each container

FROM  curlimages/curl:latest

CMD  curl -s -o /dev/null -w '%{time_starttransfer}' <URL>

e.g.

$ docker build -t curl-yuta .
$ docker run --rm curl-yuta
0.252717

There are two screenshots that Agent from ECS on Fargate or App Runner obtains response time.

ECS on Fargate

App Runner

It found that the response time is slower when retrieved via App Runner, even in curl.

Mystery Container

Datadog automatically registers containers with agents in their dashboards.

ECS on Fargate

App Runner

On Fargate, when Datadog Agent was installed into the container, one unit was registered. However, on App Runner, two units was registered.

Mystery container, aws-fargate-request-proxy was running, but I could not find details about this container in the App Runner documentation.

In my guess, deployed container on App Runner doesn't connect directly external and it takes a long time since the proxy container exists between App Runner container and Internet.

Solution

I built monitoring system with Datadog Agent on Fargate instead of App Runner. And in combination with GitHub Actions, I built a CI/CD infrastructure so that datadog configuration files can be pushed to GitHub repository and automatically run up to container deployments.

Conclusion

I tried App Runner for the first time except tutorial. Unfortunately, I can not make use of it on business, however I feel that it is so convenient for App Runner to be deployed container easily. I would like to take advantage of this on another occasion.

Original

https://zenn.dev/yuta28/articles/app-runner-proxy

https://hub.docker.com/r/curlimages/curl ↩

What is IAM instance profile?

Yutaro Mori — Mon, 12 Sep 2022 06:33:42 +0000

Introduction

You attach IAM role with IAM policy to AWS resources which granted to operate the other AWS Resources.

However, EC2 is not attached to IAM role, but is attached to IAM instance profile.

In AWS CLI, There is the parameter about IamInstanceProfile, too.

 aws ec2 describe-instances --query "Reservations[].Instances[].IamInstanceProfile.Arn"
[
    "arn:aws:iam::XXXXXXXXXXXX:instance-profile/Yuta20210911"
]

So, what is the IAM instance profile?

About IAM instance profile

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

I found the following description in the AWS documentation.

Amazon EC2 uses an instance profile as a container for an IAM role. When you create an IAM role using the IAM console, the console creates an instance profile automatically and gives it the same name as the role to which it corresponds. If you use the Amazon EC2 console to launch an instance with an IAM role or to attach an IAM role to an instance, you choose the role based on a list of instance profile names.

Instance profile is the container stored the IAM role and described as attaching that container to EC2 in the document. It is the concept used only in EC2, and when you create IAM role, is created same name automatically. You don't have to be aware of the difference between IAM Instance profile and IAM role. At the important point, it is the case of Infrastructure as Code(IaC)

Points to note when developing IaC

When you attach IAM roles to Lambda or CodePipeline, write Arn of IAM roles in role parameter by CloudFormation or Terraform.

To create Lambda by IaC

# CloudFormation template file
AWSTemplateFormatVersion: '2010-09-09'
Resources:
  LambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      FunctionName: 'efs-s3'
      Role: !GetAtt IAMRole.Arn

  IAMRole:
    Type: 'AWS::IAM::Role'
    Properties:
      Path: '/'
      RoleName: 'efs-S3'
      AssumeRolePolicyDocument:
        Version:'2012-10-17'
        Statement:
          - Effect:Allow
            Principal:
            Service:
              - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'

# Terraform tf file
resource "aws_lambda_function" "LambdaFunction" {
    function_name = "efs-s3"
    role = "${aws_iam_role.IAMRole.arn}"
}

resource "aws_iam_role" "IAMRole" {
    path = "/"
    name = "efs-S3"
    assume_role_policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": "lambda.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
      }
    ]
    })
}

Passing on Arn info of IAM role created to the other AWS resources, the AWS resources are given to permissions to operate ones. On the other hands, what passing on to EC2 for IAM permission is not IAM role, but IAM instance profile. You have to create IAM instance profile expressly to create EC2 by IaC.

To create EC2 by IaC

# CloudFormation template file
AWSTemplateFormatVersion: '2010-09-09'
Resources:
    EC2Instance:
        Type: 'AWS::EC2::Instance'
        Properties:
            IamInstanceProfile: !Ref IAMRole
            Tags:
              - Key: "Name"
                Value: "Test"

    IAMRole:
        Type: 'AWS::IAM::Role'
        Properties:
            Path: '/'
            RoleName: 'Yuta20220815'
            AssumeRolePolicyDocument:
              Version:2012-10-17
              Statement:
                 - Effect:Allow
                   Principal:
                     Service:
                       - ec2.amazonaws.com
                     Action:
                       - sts:AssumeRole

    # Need to create an IAM instance profile
    IAMInstanceProfile:
        Type: 'AWS::IAM::InstanceProfile'
        Properties:
            Path: '/'
            InstanceProfileName: !Ref IAMRole
            Roles:
              - !Ref IAMRole

# Terraform tf file
resource "aws_instance" "EC2Instance" {
    tags = {
        Name = "Test"
    }
    iam_instance_profile = aws_iam_role.IAMRole.name
}

resource "aws_iam_role" "IAMRole" {
    path = "/"
    name = "Yuta20220815"
    assume_role_policy = jsonencode({
        Version = "2012-10-17"
        Statement = [
            {
                Effect = "Allow"
                Principal = {
                    Service = "ec2.amazonaws.com"
                }
                Action = "sts:AssumeRole"
            }
        ]
    })
}

# Need to create an IAM instance profile
resource "aws_iam_instance_profile" "IAMInstanceProfile" {
    path = "/"
    name = aws_iam_role.IAMRole.name
    role = aws_iam_role.IAMRole.name
}

If you deploy code that creates only EC2 and IAM role without creating IAM instance profile, you would get an error.

│ Error: creating EC2 Instance: InvalidParameterValue: Value (Yuta20220815) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name
│       status code: 400, request id: 7fe3f4c5-b023-40e9-8d31-69e799d5b18a
│
│   with aws_instance.EC2Instance,
│   on ec2.tf line 1, in resource "aws_instance" "EC2Instance":
│    1: resource "aws_instance" "EC2Instance" {

Therefore, when you create EC2 by IaC, you need to create IAM instance profile expressly.

Conclusion

I introduce IAM instance profile. When we pass on the permission of IAM role to EC2 from AWS console, we are not aware of IAM instance profile because we choose IAM role. However, there are prone to errors to create AWS resource by IaC and you should pay attention to them. I don't find the reason why EC2 are passed on IAM instance profile to not IAM role.

According to this article, IAM instance profile defines "who am I?". Just like an IAM user represents a person, an instance profile represents EC2 instances.

An instance profile, on the other hand, defines “who am I?” Just like an IAM user represents a person, an instance profile represents EC2 instances. The only permissions an EC2 instance profile has is the power to assume a role.

However, among the many AWS resources, I couldn't tell if EC2 is the only one that explicitly defines the target audience in the IAM instance profile.

Please comment me if you know the reasons.

Original

https://zenn.dev/yuta28/articles/ec2-iam-instance-profile

Customize Windows Terminal and Git operations

Yutaro Mori — Sat, 11 Jun 2022 07:29:24 +0000

Introduction

Windows Terminal is a powerful tool for managing your development environment. The settings as they are are sufficient, but you can customize your own settings for more convenient operation.

So, I will introduce my terminal settings.

Requirements

In advance, you need to install the following software:

PowerShell 7.x
Git for Windows
WSL2
Scoop

If you don't have these software, you can install them by following the link below.

Windows Terminal Settings

The default background color of Windows Terminal is black, making it difficult to tell which terminal is which when you install some WSL distribution or PowerShell. You change the color of the terminal and the font of the text.

I define my own color and font in the following way.

color scheme:
- One Half Dark
font style:
- Nerd Font

Nerd Font has a wide selection of development-oriented icons that are useful.

https://www.nerdfonts.com/#home

Installation Page

https://eng.fontke.com/font/24644369/

In my opinion, it is a little difficult to install the font in Windows by official page. You can install the one to drag and drop by Windows personal settings page.

Set color scheme via GUI or describe it in JSON file.

{
  "background": "#282C34",
  "black": "#282C34",
  "blue": "#61AFEF",
  "brightBlack": "#5A6374",
  "brightBlue": "#61AFEF",
  "brightCyan": "#56B6C2",
  "brightGreen": "#98C379",
  "brightPurple": "#C678DD",
  "brightRed": "#E06C75",
  "brightWhite": "#DCDFE4",
  "brightYellow": "#E5C07B",
  "cursorColor": "#FFFFFF",
  "cyan": "#56B6C2",
  "foreground": "#DCDFE4",
  "green": "#98C379",
  "name": "One Half Dark",
  "purple": "#C678DD",
  "red": "#E06C75",
  "selectionBackground": "#FFFFFF",
  "white": "#DCDFE4",
  "yellow": "#E5C07B"
}

Also, you generate it by this generator site.

https://windowsterminalthemes.dev/

Copy color scheme setting and paste it to the setting.json file, making the terminal so nice.

Git Customization

Next, I will customize the git operation. I think you make good use of the prompt display and tab completion functions to make the prompt easier to read when you use Git in the terminal. For example, you would describe .bashrc or .zshrc if you are using Mac.

There are several useful tools in PowerShell to help your Git Operation, and I will introduce them.

oh-my-posh

https://ohmyposh.dev/

Oh My Posh is a custom prompt engine for any shell(bash, zsh or PowerShell) that has the ability to adjust the prompt string with a function or variable.

scoop install oh-my-posh

jandedobbeleer(My favorite theme)

After installing it, you describe oh-my-posh command loading the oh-my-posh theme at terminal startup in $PROFILE.

# $PROFILE
oh-my-posh init pwsh --config ~/AppData/Local/Programs/oh-my-posh/themes/jandedobbeleer.omp.json | Invoke-Expression

There are various the oh-my-posh themes. You could find favorite theme.

https://ohmyposh.dev/docs/themes

posh-git

https://github.com/dahlbyk/posh-git#overview

posh-git is a PowerShell module that integrates Git and PowerShell by providing Git status summary information that can be displayed in the PowerShell prompt.

scoop install posh-git

After installing it, you add module to $PROFILE.

# $PROFILE
Import-Module posh-git

However, PowerShell's tab completion is a bit quirky by default, as it does not list search suggestions like zsh. Add the following PowerShell command to $PROFILE to allow multiple search suggestions to be displayed during tab completion.

Set-PSReadLineKeyHandler -Key Tab -Function MenuComplete

Scoop tab completion

In default, Scoop does not have tab completion. You install the module providing the tab completion and activate it.

https://github.com/Moeologist/scoop-completion

scoop install scoop-completion

# add $PROFILE
Import-Module "$($(Get-Item $(Get-Command scoop.ps1).Path).Directory.Parent.FullName)\modules\scoop-completion"

Display icons from terminal

https://github.com/devblackops/Terminal-Icons

This is a PowerShell module to show file and folder icons in the terminal. It is similar to Material Icon Theme, VS code plugin.

https://marketplace.visualstudio.com/items?itemName=PKief.material-icon-theme

Install and activate it in the same way.

scoop install terminal-icons

# add $PROFILE
Import-Module Terminal-Icons

Install GitHub CLI

https://cli.github.com/

After git pushing on the terminal, I usually think it is boring to open a browser and PR every time to merge into the GitHub repository. GitHub CLI provides a command line interface to GitHub. On the terminal, you can use it to push, pull and so on.

scoop install gh
gh --version
gh version 2.11.3 (2022-05-25)
https://github.com/cli/cli/releases/tag/v2.11.3

# need to authenticate from the terminal using it for the first time.
gh auth login
# pass ssh key or token to link terminal and GitHub together.

gh auth status
github.com
  ✓ Logged in to github.com as Yuhta28 (C:\Users\yuta_\AppData\Roaming\GitHub CLI\hosts.yml)
  ✓ Git operations for github.com configured to use ssh protocol.
  ✓ Token: *******************

You merge the PR by using the GitHub CLI, and main branch of local repository automatically is merged and pulled. Merged branches can be deleted, eliminating the accumulation of unneeded branches and reducing the operational load.

gh pr merge
? What merge method would you like to use? Create a merge commit
? Delete the branch locally and on GitHub? Yes
? What's next? Submit
✓ Merged pull request #139 (feature/add new blog windows git)
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), 648 bytes | 49.00 KiB/s, done.
From github.com:Yuhta28/zenn-blog
 * branch            main       -> FETCH_HEAD
   32be209..d7672f3  main       -> origin/main
Updating 32be209..d7672f3
Fast-forward
 .textlintrc                           |   3 +-
 articles/94744fc7a339cd.md            |   2 +-
 articles/eventbridge-slack.md         |   2 +-
 articles/first-article-by-cli-yuta.md |   2 +-
 articles/windows-git-dev.md           | 193 ++++++++++++++++++++++++++++++++++

✓ Deleted branch feature/add-new-blog-windows-git and switched to branch main

GitHub CLI also provides tab completion tools, and you activate it.

# add $PROFILE
Invoke-Expression -Command $(gh completion -s powershell | Out-String)

Conclusion

I share customization that makes Windows Terminal more convenient to use. I believe the change of color theme or implementation of functions such as tab completion and GitHub CLI will contribute to improve development efficiency.

If you have any other recommendations for Windows Terminal, please comment.

Original

https://zenn.dev/yuta28/articles/windows-git-dev

I manage my dev.to blog in GitHub repository

Yutaro Mori — Thu, 05 May 2022 02:31:08 +0000

Introduction

I blog in Japanese at Zenn, community of engineers to help one another out. Zenn makes us able to manage our blogs in GitHub repository and to write articles in my favorite editor. In addition, I merge articles into main branch they would automatically be published to my blog with CI/CD.

I wish that dev.to blog in the same way, and I realize that I manage my dev.to blog in GitHub repository and publish/update automatically it with reference to this article.

https://dev.to/maxime1992/manage-your-dev-to-blog-posts-from-a-git-repo-and-use-continuous-deployment-to-auto-publish-update-them-143j

I implement GitHub Actions CI/CD with DEV API. Based on above article, I introduce my own how to manage my dev.to blog.

How to build

1. Copy the template

Go to the repository of article authors. https://github.com/Yuhta28/dev-to-blog-template and copy the template.

2. Generate a DEV Community API key

Go to https://dev.to/settings/account and generate a DEV Community API key.

3. Set the API Key to GitHub Actions

Go to GitHub repository copied from the template and set the API Key to Actions secrets.

4. Build GitHub Actions workflow

In reference article, use prettier to format the markdown and the code snippets. I implement a text review using textlint and reviewdog in addition to that.

name: Build and Deploy

on:
  push:
    branches: [main]
  pull_request:

jobs:
  build:
    name: Build
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repo
        uses: actions/checkout@master

      - name: Install reviewdog
        uses: reviewdog/action-setup@v1
        with:
          reviewdog_version: latest

      - name: Install Dependencies
        uses: bahmutov/npm-install@v1

        #Make sure that cache is retrieved even if steps fail
      - name: cache-node-modules
        uses: pat-s/always-upload-cache@v3
        env:
          cache-name: cache-node-modules
        with:
          path: ~/.npm
          key: node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            node-

      - name: Install textlint
        run: 'npm install --save-dev textlint textlint-rule-common-misspellings textlint-rule-spellchecker'

      - name: Run textlint
        run: npx textlint -f checkstyle "blog-posts/**/*.md" >> .textlint.log

      - name: Run reviewdog
        if: failure()
        env:
          REVIEWDOG_GITHUB_API_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          cat .textlint.log
          cat .textlint.log | reviewdog -f=checkstyle -name="textlint" -reporter="github-pr-review"

      - name: Run Prettier
        run: yarn run prettier:check

      - name: Run Embedme
        run: yarn run embedme:check

  deploy:
    name: Deploy
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repo
        uses: actions/checkout@master

      - name: Install Dependencies
        uses: bahmutov/npm-install@v1

      - name: Deploy to dev.to
        run: DEV_TO_GIT_TOKEN=${{ secrets.DEV_TO_GIT_TOKEN }} yarn run dev-to-git

5. Link GitHub with dev.to

Define a property repository.url in package.json and set it to the GitHub repository URL.

  "name": "dev.to",
  "repository": {
    "type": "git",
    "url": "https://github.com/Yuhta28/dev-to-blog.git"
  }

6. Create new article

Create a template for a new article to use DEV API. go run create-post.go

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "os"
    "strings"
)

func main() {
    DEVAPIKEY := os.Getenv("DEVAPIKEY") //Set your dev.to API key in your environment variables
    client := &http.Client{}
    var data = strings.NewReader(`{"article":{"title":"Template","body_markdown":"Body","published":false,"tags":["tag1", "tag2"]}}`)
    req, err := http.NewRequest("POST", "https://dev.to/api/articles", data)
    if err != nil {
        log.Fatal(err)
    }
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("api-key", DEVAPIKEY)
    resp, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()
    bodyText, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%s\n", bodyText)
}

Unfortunately, the template cannot be connected to GitHub repository, so I need to get article ID and set it in dev-to-git.json to connect it to GitHub repository.

7. Set the article ID

Get the article ID to use DEV API, too. go run get-blog-id.go

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "net/http"
    "os"

    "github.com/itchyny/gojq"
)

func curl() interface{} {
    DEVAPIKEY := os.Getenv("DEVAPIKEY") //Set your dev.to API key in your environment variables
    client := &http.Client{}
    req, err := http.NewRequest("GET", "https://dev.to/api/articles/me/unpublished", nil)
    if err != nil {
        log.Fatal(err)
    }
    req.Header.Set("api-key", DEVAPIKEY)
    resp, err := client.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()
    var data interface{}
    err = json.NewDecoder(resp.Body).Decode(&data)
    if err != nil {
        log.Fatal(err)
    }
    return data
}

func main() {
    // Parse JSON
    query, err := gojq.Parse(".[].id")
    if err != nil {
        log.Fatalln(err)
    }
    input := curl()
    iter := query.Run(input) // or query.RunWithContext
    for {
        v, ok := iter.Next()
        if !ok {
            break
        }
        if err, ok := v.(error); ok {
            log.Fatalln(err)
        }
        fmt.Printf("%1.0f\n", v)
    }
}

8. Make the template for new article

Make the template for new article. go run make-template.go

package main

import (
    "fmt"
    "os"
)

func main() {

    var blog string

    print("Enter the name of the new article: ")
    fmt.Scan(&blog)

    // Create blog directory and article file
    if err := os.MkdirAll("blog-posts/"+blog, 0777); err != nil {
        fmt.Println(err)
    }
    file_article, err := os.Create("blog-posts/" + blog + "/" + blog + ".md")
    if err != nil {
        fmt.Println(err)
    }
    defer file_article.Close()

    // Add blog metadata in article file
    file_article.WriteString("---\ntitle: Title \npublished: false\ndescription: description\ntags: tag1, tag2\n---\n")

    // Create code directory
    if err := os.MkdirAll("blog-posts/"+blog+"/code", 0777); err != nil {
        fmt.Println(err)
    }
    file_code, err := os.Create("blog-posts/" + blog + "/code/.gitkeep")
    if err != nil {
        fmt.Println(err)
    }
    defer file_code.Close()

    // Create assets directory
    if err := os.MkdirAll("blog-posts/"+blog+"/assets", 0777); err != nil {
        fmt.Println(err)
    }
    file_assets, err := os.Create("blog-posts/" + blog + "/assets/.gitkeep")
    if err != nil {
        fmt.Println(err)
    }
    defer file_assets.Close()
}

9. Link article ID with the template

Connect the article ID with the template into dev-to-git.json.

[
  {
    "id": 773216,
    "relativePathToArticle": "./blog-posts/cw-oss-cloudwatch/i-use-cw-which-is-oss-to-tail-aws-cloudwatch-logs-2e9g.md"
  },
  {
    "id": 1056501,
    "relativePathToArticle": "./blog-posts/repography-make-readme-rich/repography-makes-github-repository-beautiful-3dn3.md"
  }
]

10. Deploy the article

It is completed to manage my dev.to blog in GitHub repository. After that, push branch and pull request to main branch, and run GitHub Actions CI/CD.

Conclusion

I have managed my dev.to blog in GitHub repository. It makes me easy version control and I can write articles using my favorite editor(Visual Studio code). Moreover, the linter tool helps me to reduce careless mistakes in writing. So, let's try this method when writing articles in dev.to.

Original

https://zenn.dev/yuta28/articles/dev-github-vscode

References

https://dev.to/maxime1992/manage-your-dev-to-blog-posts-from-a-git-repo-and-use-continuous-deployment-to-auto-publish-update-them-143j https://dev.to/beeman/automate-your-dev-posts-using-github-actions-4hp3

Repography makes GitHub repository beautiful

Yutaro Mori — Sat, 16 Apr 2022 02:29:27 +0000

Introduction

GitHub provides a GitHub Apps service that allows you to access various applications. We can gain some exciting Apps from GitHub marketplace. I introduce an exciting App called Repography that visualizes commits to GitHub repositories and enriches README.md.

What's Repography?

https://github.com/marketplace/repography

Repography is a GitHub App that provides visualized dashboard in markdown format for GitHub repositories. You embed it in README.md and the dashboard will be displayed in your repository to see past commits, issues, and ranking of top contributors. Access the site of Repography and can experience demo of Repography embedded OSS repository.

Click on the OSS installed Repography repository and Repography output the information for that OSS to the dashboard.

/ Recent activity

/ Structure

/ Top contributors

Repography offers several fee plans. These plans are no difference in functionally, only an increase in the private repository limit.

Choose your repository you wish to link to Repography and the dashboard display.

You can choose either markdown or reStructuredText from the format on the right. Copy code, embed it in README.md, and the dashboard will be displayed in your repository.

Poster

Repography provides another unique feature. It is that Repography output can be displayed in the office as a poster.

The fee is €60, but I don't know if they will ship overseas. Fortunately, You can download the PNG file.

Now you can have your posters printed print local print stores, even if you live far away. Also, you install the 4K desktop wallpaper (16:9). Stick it on your PC and make it a memory.

Conclusion

I introduced Repography. It visualizes the amount of your commitments on the dashboard, so if you introduce it to your daily learning motivation or when developing OSS with your team, it may help improve your motivation as you can visualize your hard work.

The poster would also be perfect for a stylish tech company office, so companies that are interested should consider it💡

Original

https://zenn.dev/yuta28/articles/repography-handson

Keep in mind when coding existing infrastructure

Yutaro Mori — Sun, 23 Jan 2022 09:20:04 +0000

Introduction

Many DevOps engineers would like to accomplish our existing cloud infrastructure using code.

However, it is difficult to manage existing infrastructure using code because there are few means and many constraints.

I share this post about getting into the trouble when I challenge the existing AWS infrastructure using code.

Which infrastructure as code tool

I survey what tools are most suitable for coding our existing AWS infrastructure.

I choose these three as candidates because they have a lot of documentation and articles and a low threshold for learning.

AWS CloudFormation
AWS CDK
Terraform

As a point of selection, I focus on whether or not these three things could be achieved.

Difficulty in coding existing resources
Readability of source code
Scalability for future operations¹

AWS CloudFormation

https://aws.amazon.com/cloudformation/

CloudFormation is a service that gives developers and businesses an easy way to create a collection of related AWS. Because of AWS service with a long history, there are many cases about IaC. However, when importing existing resources, I give up on coding existing resources because I have to prepare a template file suitable for the resource in advance.

AWS CDK

https://aws.amazon.com/cdk/

AWS CDK is an open-source software development framework to define your cloud application resources using familiar programming languages.

In Jan 2022, AWS CDK supports below programming languages.

JavaScript
TypeScript
Python
Java
C#
Go (in Developer Preview)

It is very helpful to define the infrastructure resources with my familiar programming languages. But, few people in our team can use programming languages supported by AWS CDK and AWS CDK does not have a function of import existing resources, so that I give up, too.

Terraform

https://www.terraform.io/

Finally, I decided to manage our infrastructure using code with Terraform.

Although Terraform has a sub-commandimport, the sub-command cannot import multiple resources such as EC2, VPC, and RDS at once, so that it became very hard when the number of target resources was large. So I use terraformer to import the existing resources. https://github.com/GoogleCloudPlatform/terraformer

Point of trouble

However, I still had some problems with the existing resources, so I introduce those parts.

Version differences between Terraform and terraformer

$ terraform --version
Terraform v1.1.3
on linux_amd64
$ cat terraform.tfstate | jq -r '. | { terraform_version: .terraform_version }'
{
  "terraform_version": "0.12.31"
}

Terraform converts infrastructure resources to IaC based on a file called .tfstate, which stores infrastructure resource configuration information. However, the .tfstate file generated by terraformer when importing existing resources has an old version of 0.12. If I run command terraform init with this file, I will get an error due to version conflicts and will not be able to run Terraform.

$ terraform init

Initializing the backend...
╷
│ Error: Invalid legacy provider address
│
│ This configuration or its associated state refers to the unqualified provider "aws".
│
│ You must complete the Terraform 0.13 upgrade process before upgrading to later versions.

To solve this, I upgrade to version 1 of terraformer's tfstate file to use tfenv,Terraform version manage https://github.com/tfutils/tfenv

0.12 -> 0.13

$ cd #<The directory where the tfstate file exists>
$ tfenv use 0.13.7
$ terraform init
$ terraform plan
$ terraform appy

0.13 -> 0.14

$ tfenv use 0.14.11
$ terraform init
$ terraform plan
$ terraform apply

0.14 -> 1.1

$ tfenv use 1.1.4
$ terraform init
$ terraform plan
$ terraform apply

AWS resources that cannot be imported

The terraformer can import not all AWS resources, and it can import only the resources listed in the below command.

$ terraformer import aws list
accessanalyzer
acm
alb
api_gateway
appsync
auto_scaling
batch
budgets
cloud9
cloudformation
cloudfront
cloudhsm
cloudtrail
cloudwatch
codebuild
codecommit
codedeploy
codepipeline
cognito
config
customer_gateway
datapipeline
devicefarm
docdb
dynamodb
ebs           #Pay attention to EBS
ec2_instance
ecr
ecrpublic
ecs
efs
eip
eks
elastic_beanstalk
elasticache
elb
emr
eni
es
firehose
glue
iam
igw
iot
kinesis
kms
lambda
logs
media_package
media_store
msk
nacl
nat
opsworks
organization
qldb
rds
resourcegroups
route53
route_table
s3
secretsmanager
securityhub
servicecatalog
ses
sfn
sg
sns
sqs
ssm
subnet
swf
transit_gateway
vpc
vpc_peering
vpn_connection
vpn_gateway
waf
waf_regional
workspaces
xray

Although not all AWS resources are supported, all major AWS resources are available, so I thought there would be no problem in this case. However, I could not import EBS even though it was in the list.

$ terraformer import aws -r ebs
2022/01/16 14:25:57 aws importing default region
2022/01/16 14:25:59 aws importing... ebs
2022/01/16 14:25:59 aws done importing ebs
2022/01/16 14:25:59 Number of resources for service ebs: 0 # no import
2022/01/16 14:25:59 aws Connecting....
2022/01/16 14:25:59 aws save ebs
2022/01/16 14:25:59 aws save tfstate for ebs

As a solution, I'm going to manage EBS to code manually tf file.

Conclusion

It turns out that existing resources using code is more difficult than I thought. My goals for this year is our existing resources using code and to spread code-managed infrastructure operations within the our team.

I'll do my best.

References

https://harness.io/blog/infrastructure-as-code/ https://aws.amazon.com/cloudformation/faqs/ https://www.terraform.io/language/upgrade-guides/1-0

Original

https://zenn.dev/yuta28/articles/iac-existing-infrastructure

In the future, I would like build on CI/CD with IaC tool, and it will be automatically deployed and the infrastructure resources will be updated when I git push it. ↩

GitHub Actions CI/CD without AWS Credential information

Yutaro Mori — Thu, 11 Nov 2021 14:48:18 +0000

Introduction

October 2021, GitHub announced that GitHub Actions supports OpenID Connect (OIDC) for secure deployments to cloud, which uses short-lived tokens that are automatically rotated for each deployment.

This feature helps me build CI/CD on GitHub Actions by AWS without AWS Credential such as IAM access key ID or secret access key.

I'll try to implement CI/CD for building EC2 instances by incorporating this feature into CI/CD using a combination of Terraform and GitHub Actions, and passing the IAM role to GitHub Actions.

GitHub Actions CI/CD with terraform

Previously, it is necessary to register IAM access key ID and secret access key in the environment variables in advance when deploying AWS services with terraform and building CI/CD with GitHub Actions or Circle CI.

- name: Configure AWS Credentials
  uses: aws-actions/configure-aws-credentials@v1
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} #access key ID
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} #secret access key

There are some problems.

Create IAM user to publish Access keys
Re-register Access keys in case of AWS multi account operation

To solve them, I add OpenID Connect provider. I build CloudFormation stack template to refer to this article and deploy IAM role & OIDC provider.

Deploying IAM role & OIDC provider

AWSTemplateFormatVersion: '2010-09-09'
Description: 'IAM Role for GHA'

Parameters:
  RepoName:
    Type: String
    Default: Yuhta28/terraform-githubaction-ci

Resources:
  Role:
    Type: AWS::IAM::Role
    Properties:
      RoleName: ExampleGithubRole
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action: sts:AssumeRoleWithWebIdentity
            Principal:
              Federated: !Ref GithubOidc
            Condition:
              StringLike:
                token.actions.githubusercontent.com:sub: !Sub repo:${RepoName}:*

  Policy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: test-gha
      Roles:
        - !Ref Role
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - 'ec2:*'
              - 'sts:GetCallerIdentity'
              - 's3:*'
            Resource: '*'

  GithubOidc:
    Type: AWS::IAM::OIDCProvider
    Properties:
      Url: https://token.actions.githubusercontent.com
      ClientIdList: [sigstore]
      ThumbprintList: [a031c46782e6e6c662c2c87c76da9aa62ccabd8e]

Outputs:
  Role:
    Value: !GetAtt Role.Arn

After launch stack, IAM Role and OIDC are displayed.

Create GHA Workflow

Create GitHub Actions workflow file. There is a template workflow file for terraform by HashiCorp and choose it.

name: 'Terraform'
on:
  push:
    branches:
      - main
  pull_request:

jobs:
  terraformCICD:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - run: sleep 5

      - uses: actions/checkout@v2

      - name: Configure AWS
        run: |
          export AWS_ROLE_ARN=arn:aws:iam::<AWS_AccountID>:role/ExampleGithubRole
          export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/awscreds
          export AWS_DEFAULT_REGION=ap-northeast-1

          echo AWS_WEB_IDENTITY_TOKEN_FILE=$AWS_WEB_IDENTITY_TOKEN_FILE >> $GITHUB_ENV
          echo AWS_ROLE_ARN=$AWS_ROLE_ARN >> $GITHUB_ENV
          echo AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION >> $GITHUB_ENV
          curl -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sigstore" | jq -r '.value' > $AWS_WEB_IDENTITY_TOKEN_FILE

      # Install the latest version of Terraform CLI and configure the Terraform CLI configuration file with a Terraform Cloud user API token
      - name: Setup Terraform
        uses: aws-actions/configure-aws-credentials@master
        with:
          role-to-assume: '${{ env.AWS_ROLE_ARN }}'
          web-identity-token-file: '${{ env.AWS_WEB_IDENTITY_TOKEN_FILE }}'
          aws-region: '${{ env.AWS_DEFAULT_REGION }}'
          role-duration-seconds: 900
          role-session-name: GitHubActionsTerraformCICD

      # Checks that all Terraform configuration files adhere to a canonical format
      - name: Terraform Format
        run: terraform fmt -check -diff

      # Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
      - name: Terraform Init
        run: terraform init

      - name: Terraform Validate
        run: terraform validate -no-color

      # Generates an execution plan for Terraform
      - name: Terraform Plan
        if: github.event_name == 'pull_request'
        run: terraform plan -no-color
        continue-on-error: true

        # On push to main, build or change infrastructure according to Terraform configuration files
        # Note: It is recommended to set up a required "strict" status check in your repository for "Terraform Cloud". See the documentation on "strict" required status checks for more information: https://help.github.com/en/github/administering-a-repository/types-of-required-status-checks
      - name: Terraform Apply
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        run: terraform apply -auto-approve

It is important for that step Configure AWS.

I set three environment variables on GitHub.

AWS_ROLE_ARN
AWS_WEB_IDENTITY_TOKEN_FILE
AWS_DEFAULT_REGION

AWS_ROLE_ARN

Specify Assume Role I just create.

AWS_WEB_IDENTITY_TOKEN_FILE

A path to Web ID token file.

AWS_DEFAULT_REGION

A default region. I select Tokyo region,ap-northeast-1.

With curl, two parameters, ACTIONS_ID_TOKEN_REQUEST_TOKEN and ACTIONS_ID_TOKEN_REQUEST_URL is passed to Web ID token file.

After that, web ID token file is passed to configure-aws-credentials, one of actions.

uses: aws-actions/configure-aws-credentials@master
with:
  role-to-assume: '${{ env.AWS_ROLE_ARN }}'
  web-identity-token-file: '${{ env.AWS_WEB_IDENTITY_TOKEN_FILE }}'
  aws-region: '${{ env.AWS_DEFAULT_REGION }}'
  role-duration-seconds: 900
  role-session-name: GitHubActionsTerraformCICD

Compared with workflow file described at the beginning of this section, IAM role and OIDC token are set instead of Access keys of environment variables. It is recommendation that role-session-name is set to examine the logs.

Build Terraform

Create one EC2 instance and store terraform.tfstate in S3.

Directory structure

|
├── main.tf
└── variables.tf

main.tf

terraform {
  backend "s3" {
    bucket = "specify bucket name"
    key    = "terraform.tfstate"
    region = "ap-northeast-1"

  }
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  region = "ap-northeast-1"
}

resource "aws_instance" "app_server_yuta" {
  ami                    = "ami-00cb500575fd9f9be"
  instance_type          = "t2.micro"
  vpc_security_group_ids = ["sg-XXXXXXXXXXXXXXXXX", "sg-XXXXXXXXXXXXXXXXX"]
  subnet_id              = "subnet-XXXXXXXXXXXXXXXXX"

  tags = {
    Name = var.instance_name
  }
}

variables.tf

variable "instance_name" {
  description = "Value of the Name tag for the EC2 instance"
  type        = string
  default     = "Yuta-ServerInstance"
}

After coding terraform file, create branch and PR. GitHub Action launches.

If terraform planstep is executed without any problems, merging will be possible. Once the main branch is merged, terraform applystep will run and the EC2 instance will be created.

Conclusion

I try to build GitHub Actions CI/CD without AWS Credential information. This feature is new and there are few reference materials. However, I think the feature is very useful and would like that everyone use it.

Original

https://zenn.dev/yuta28/articles/terraform-gha

Difference between EC2 RI/SP and RDS RI

Yutaro Mori — Wed, 01 Sep 2021 15:44:23 +0000

Introduction

Reserved Instances(RI) or Savings Plans(SP) of EC2 and Reserved Instances(RI) of RDS are for Cost Optimization, which is one of 5 pillar of the AWS Well-Architected Framework.

However, there are subtle differences in their specifications that made me confused. I summarize the difference of them and share post.

Feature

At first, I introduce these basics features.

About RI

Reserved Instances(RI) provide a significant discount compared to On-Demand pricing and provide a capacity reservation when used in a specific Availability Zone. If you would like to use some instances of specific instance type, you can reduce the cost to select RI.

And also, you pay for the entire Reserved Instance term with one upfront payment and get the best effective hourly price when compared to running the same instance on an On-Demand basis.

About SP

Savings Plans also provides a discount compared to On-Demand pricing. RI is applied to discount regarding specific instance type or availability zone. On the other hand, SP is a flexible pricing model in exchange for a specific usage commitment. The discount rate of SP is smaller than one of RI, but you need not to specify instance type and can make more flexible cost optimization plans.

Cautions for Purchase RI or SP

Both RI and SP are for cost reduction, but you pay attention to purchase. That is an order date. You can purchase EC2 RI to specify order date. Because of this future, you don't have to work to launch RI on holiday and you can purchase it in advance.

Moreover, in case of EC2 SP, you can specify order time. Because of this future, you can apply SP at midnight local time.

However, you cannot purchase RDS RI in advance.

Unfortunately RDS RI doesn't hove confirmation to order steps. You click submit button, and purchase is confirmed. I would like to purchase RDS RI in advance and made a mistake.

Let me show you below graph.

	EC2 RI	EC2 SP	RDS RI
specify order date	✓	✓
specify order time		✓

Digression

To tell you the truth, in case of EC2 RI, you can specify order time with AWS CLI.

purchase-reserved-instances-offering
--instance-count <value> \
--reserved-instances-offering-id <value> \
--purchase-time <value>

Conclusion

I share post about difference between EC2 RI/SP and RDS RI on order timing. There are subtle differences in their specifications that made me confused. Please be careful with these differences.

Original

https://zenn.dev/yuta28/articles/sp-attention https://zenn.dev/yuta28/articles/rds-attention

References

https://aws.amazon.com/ec2/pricing/reserved-instances/?nc1=h_ls https://aws.amazon.com/rds/reserved-instances/?nc1=h_ls https://aws.amazon.com/savingsplans/ https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/purchase-reserved-instances-offering.html

Be attention to replace a key pair for EC2

Yutaro Mori — Mon, 09 Aug 2021 02:34:15 +0000

Introduction

I will share points to note about replace a key pair for EC2.

About key pair

When you launch the instance, you select the key pair to connect with SSH.

EC2 instance cannot allow to be connected to with SSH by password authentication in initial settings.

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
PasswordAuthentication no

Once you select a key pair and launch EC2 instance, you can connect to it with public key authentication.

ssh -i ./NewWindows.pem ec2-user@18.182.24.156
Last login: Mon Aug  2 13:36:42 2021

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/

Replace a key pair

You create an Amazon Machine Image(AMI) based on running EC2 instance and you launch EC2 instance from that AMI. Then, select a key pair.

There is a case that you need to replace a key pair attached EC2 instance because of security, review of operations and so on.

In this case, you change a new key pair from existing key pair. Also, you can connect to EC2 instance with another secret key.

ssh -i .\tepkey.pem ec2-user@18.183.44.42
Last login: Mon Aug  2 13:45:37 2021

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/

At first, I expected that only new public key exists in the file, ~/.ssh/authorized_keys. But, it was wrong and an old public key existed there, too.

ssh-rsa ~~~~~~~~~~~~~~~~~~~XXXX~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ NewWindows
ssh-rsa ~~~~~~~~~~~~~~~~~~~XXXX~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ tepkey

Actually, I confirm that I can connect to EC2 instance with SSH by the former secret key.

ssh -i .\NewWindows.pem ec2-user@18.183.44.42
Last login: Mon Aug  2 15:07:23 2021

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/

According to the AWS documentation, you have to remove an old public key in ~/.ssh/authorized_keys manually when you replace a key pair.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#replacing-key-pair

Conclusion

I research replace a key pair for EC2 instance. I'm surprised at the specification and concerned about security🤔

If you disclosure an old secret key which is not used and not managed, your EC2 instances may be illegally accessed.

How does everyone do an operation? Let me hear your what you think.

Original

https://zenn.dev/yuta28/articles/ec2-keypair-replace

I use cw, which is OSS to tail AWS CloudWatch Logs

Yutaro Mori — Tue, 27 Jul 2021 18:01:11 +0000

Introduction

I will share cw, OSS to help to tail CloudWatch Logs.

Prerequisite knowledge

AWS CLI has a command which tails the logs collected CloudWatch Logs.

$ aws logs tail access_log
2021-07-22T07:45:55.422000+00:00 wordpress1 18.207.253.146 - - [22/Jul/2021:07:45:54 +0000] "GET /.env HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"
2021-07-22T07:45:59.986000+00:00 wordpress1 18.207.253.146 - - [22/Jul/2021:07:45:55 +0000] "POST / HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"
2021-07-22T07:49:31.689000+00:00 wordpress2 27.121.46.196 - - [22/Jul/2021:07:49:31 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
2021-07-22T07:49:31.689000+00:00 wordpress2 27.121.46.196 - - [22/Jul/2021:07:49:31 +0000] "GET /icons/apache_pb2.gif HTTP/1.1" 200 4234 "http://13.231.136.114/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"

If you would like to watch the logs in real time like tail -f, add the following option. aws logs tail --follow <Log Group>

$ aws logs tail --follow access_log
2021-07-22T07:45:55.422000+00:00 wordpress1 18.207.253.146 - - [22/Jul/2021:07:45:54 +0000] "GET /.env HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"
2021-07-22T07:45:59.986000+00:00 wordpress1 18.207.253.146 - - [22/Jul/2021:07:45:55 +0000] "POST / HTTP/1.1" 404 196 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"
2021-07-22T07:49:31.689000+00:00 wordpress2 27.121.46.196 - - [22/Jul/2021:07:49:31 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
2021-07-22T07:49:31.689000+00:00 wordpress2 27.121.46.196 - - [22/Jul/2021:07:49:31 +0000] "GET /icons/apache_pb2.gif HTTP/1.1" 200 4234 "http://13.231.136.114/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
2021-07-22T07:49:31.940000+00:00 wordpress2 27.121.46.196 - - [22/Jul/2021:07:49:31 +0000] "GET /icons/poweredby.png HTTP/1.1" 200 3412 "http://13.231.136.114/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
2021-07-22T07:49:33.195000+00:00 wordpress2 27.121.46.196 - - [22/Jul/2021:07:49:31 +0000] "GET /favicon.ico HTTP/1.1" 404 196 "http://13.231.136.114/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
2021-07-22T07:49:34.198000+00:00 wordpress2 27.121.46.196 - - [22/Jul/2021:07:49:33 +0000] "GET / HTTP/1.1" 403 4890 "https://www.google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
2021-07-22T07:49:34.198000+00:00 wordpress2 27.121.46.196 - - [22/Jul/2021:07:49:34 +0000] "GET /icons/poweredby.png HTTP/1.1" 200 3412 "http://13.231.136.114/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
2021-07-22T07:49:38.949000+00:00 wordpress2 27.121.46.196 - - [22/Jul/2021:07:49:34 +0000] "GET /icons/apache_pb2.gif HTTP/1.1" 200 4234 "http://13.231.136.114/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
2021-07-22T07:54:52.911000+00:00 wordpress1 27.121.46.196 - - [22/Jul/2021:07:54:52 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
2021-07-22T07:54:52.911000+00:00 wordpress1 27.121.46.196 - - [22/Jul/2021:07:54:52 +0000] "GET /icons/apache_pb2.gif HTTP/1.1" 200 4234 "http://54.168.233.33/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"

The output describes this.

Timestamp + time zone streams messages

It is useful that I can watch logs in real time to use tail -f, but I cannot specify each streams, and the access logs of multiple servers are mixed together.

I will share cw which solves this problem.

What's cw?

cw is the best way to tail AWS CloudWatch Logs from your terminal. https://github.com/lucagrulla/cw

cw is a native executable targeting your OS, and not needed external dependencies such as pip and npm. Compared to awslogs which is famous helpful tool for CloudWatch Logs¹, cw is written in golang and faster.

Moreover, it is very helpful that is possible to specify multiple groups. To use cw, I will try to execute tail -f for each streams, and watch logs.

Installation

You can install cw with brew command easily.

brew tap lucagrulla/tap
brew install cw

Usage

If you have installed the AWS CLI locally, you have probably used registered your credentials to your local machine with aws configure, but if you have used aws cli within an EC2 instance with an IAM role, you will need to set the default region information in the file ~/.aws/config.

[default]
region = ap-northeast-1

First, command cw -h.

$ cw -h
Usage: cw <command>

The best way to tail AWS Cloudwatch Logs from your terminal.

Flags:
  -h, --help               Show context-sensitive help.
      --endpoint=URL       The target AWS endpoint url. By default cw will use the default aws endpoints. NOTE: v4.0.0
                           dropped the flag short version.
      --profile=PROFILE    The target AWS profile. By default cw will use the default profile defined in the
                           .aws/credentials file. NOTE: v4.0.0 dropped the flag short version.
      --region=REGION      The target AWS region. By default cw will use the default region defined in the
                           .aws/credentials file. NOTE: v4.0.0 dropped the flag short version.
      --no-color           Disable coloured output.NOTE: v4.0.0 dropped the flag short version.
      --version            Print version information and quit

Commands:
  ls groups
    Show all groups.

  ls streams <group>
    Show all streams in a given log group.

  tail <groupName[:logStreamPrefix]> ...
    Tail log groups/streams.

Run "cw <command> --help" for more information on a command.

There is the introduction of basically how to use and sub command. Next, command cw ls groups and you see list of Log groups collected CloudWatch Logs.

$ cw ls groups
/aws/lambda/Yuta-rds-auto-stop
/ecs/first-run-task-definition
RDSOSMetrics
access_log

Command cw tail -f to watch CloudWatch Logs in real time.

$ cw tail access_log --follow --stream-name
wordpress2 - 27.121.46.196 - - [22/Jul/2021:09:19:33 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
wordpress2 - 27.121.46.196 - - [22/Jul/2021:09:20:24 +0000] "-" 408 - "-" "-"
wordpress2 - 27.121.46.196 - - [22/Jul/2021:09:20:29 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
wordpress2 - 27.121.46.196 - - [22/Jul/2021:09:20:29 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
wordpress1 - 27.121.46.196 - - [22/Jul/2021:09:20:48 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
wordpress1 - 27.121.46.196 - - [22/Jul/2021:09:20:49 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
wordpress1 - 27.121.46.196 - - [22/Jul/2021:09:20:49 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"

I can do that like tail -f. The main theme is that watching logs for each streams. You add option :stream after group.

$ cw tail access_log:wordpress1 --follow --stream-name
wordpress1 - 27.121.46.196 - - [22/Jul/2021:09:24:53 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
wordpress1 - 27.121.46.196 - - [22/Jul/2021:09:24:53 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
wordpress1 - 27.121.46.196 - - [22/Jul/2021:09:24:54 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
wordpress1 - 27.121.46.196 - - [22/Jul/2021:09:24:54 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
wordpress1 - 27.121.46.196 - - [22/Jul/2021:09:24:55 +0000] "GET / HTTP/1.1" 403 4890 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"

The stream, wordpress1 is only displayed.

Unresolved

I continue to output logs for a while, an error suddenly occurs and the program terminates. An error occurs a few minutes after command execution.

....

wordpress1 - 27.121.46.196 - - [22/Jul/2021:09:29:45 +0000] "GET /wordpress/wp-admin/plugin-install.php?tab=popular HTTP/1.1" 200 126481 "http://54.168.233.33/wordpress/wp-admin/plugin-install.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
wordpress1 - 27.121.46.196 - - [22/Jul/2021:09:29:47 +0000] "GET /wordpress/wp-admin/plugin-install.php?tab=recommended HTTP/1.1" 200 127412 "http://54.168.233.33/wordpress/wp-admin/plugin-install.php?tab=popular" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36"
operation error CloudWatch Logs: FilterLogEvents, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: 69b981bb-0bcf-4263-951e-73aabf9ab379, api error ThrottlingException: Rate exceeded

According to message, it is the API error that said The maximum number of attempts has been exceeded.

I don't know details, so I have a issue for developer. https://github.com/lucagrulla/cw/issues/214

Conclusion

The tool, cw is fast and useful. The error you saw made me bothered. I have a issue about that, but if someone has an idea, please submit a Pull Request.

Original

https://zenn.dev/yuta28/articles/cloudwatch-fast-tail

https://github.com/jorgebastida/awslogs ↩

reviewdog: this is not PullRequest build

Yutaro Mori — Tue, 23 Feb 2021 07:15:40 +0000

Feb 22 '21 Comments: 1 Answers: 1

Overview

I have a trouble with reviewdog I would like to be able run reviewdog on CircleCI, and when I did git push, it will check the documentation for text errors, and reviewdog will send out a pull-request if there is an error.

.config.yml


# ref: https://github.com/azu/textlint-reviewdog-example
version: 2.1

…

Open Full Question

Forem: Yutaro Mori

OpenTofu forked Terraform is now available

Table of Contents

Introduction

Target Audience

About Open Source

Impact of HashiCorp's license change

The Foundation's claims

About monetization of Terraform

Forked OpenTofu

Hands-on

Is it possible to migrate existing resources

Conclusion

Original

References

Not documented App Runner specification

Introduction

About App Runner

Increase Response time

Curl inspection

e.g.

ECS on Fargate

App Runner

Mystery Container

ECS on Fargate

App Runner

Solution

Conclusion

Original

What is IAM instance profile?

Introduction

About IAM instance profile

Points to note when developing IaC

To create Lambda by IaC

To create EC2 by IaC

Conclusion

Original

Customize Windows Terminal and Git operations

Introduction

Requirements

PowerShell 7.x

Git for Windows

WSL2

Scoop

Windows Terminal Settings

Installation Page

Git Customization

oh-my-posh

jandedobbeleer(My favorite theme)

posh-git

Scoop tab completion

Display icons from terminal

Install GitHub CLI

Conclusion

Original

I manage my dev.to blog in GitHub repository

Introduction

How to build

1. Copy the template

2. Generate a DEV Community API key

3. Set the API Key to GitHub Actions

4. Build GitHub Actions workflow

5. Link GitHub with dev.to

6. Create new article

7. Set the article ID

8. Make the template for new article

9. Link article ID with the template

10. Deploy the article

Conclusion

Original

References

Repography makes GitHub repository beautiful

Introduction

What's Repography?

/ Recent activity

/ Structure

/ Top contributors

Poster

Conclusion

Original