Forem: Tacio Nery

Create an AWS ECS Cluster Using Terraform

Tacio Nery — Fri, 09 Apr 2021 14:59:36 +0000

Hey everyone, I'd like to share my experience with Terraform and AWS. In this post I'll describe the resources I used to build a infrastructure on AWS and deploy a NodeJS application on it.

Resources

The application I needed to deploy is a monolithic NodeJS application, so, to deploy and make it scalable I decided to use containers with an autoscaling tool to scale the application based on CPU and Memory usage. To build this environment on AWS I used the services listed below:

VPC and Networking (Subnets, Internet Groups...)
Elastic Container Registry
Elastic Container Service
Application Load Balancer
Auto Scaling
Cloud Watch

Terraform Initial Configuration

The Terraform configuration I used was quite simple. The first step is create a Bucket on AWS S3 to store the Terraform State. It's not required but, it'll make our life easier if someone else needs to maintain this infrastructure. This is the main.tf file with this configuration.

# main.tf | Main Configuration

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "2.70.0"
    }
  }

  backend "s3" {
    bucket = "terraform-state-bucket"
    key    = "state/terraform_state.tfstate"
    region = "us-east-1"
  }
}

provider "aws" {
  region     = var.aws_region
  access_key = var.aws_access_key
  secret_key = var.aws_secret_key
}

The provider section is using some variables. We can define variables in a tfvars. I'll explain it later in this post.

UPDATE: With this initial configuration, just run terraform init.

VPC and Networking

Let's create a VPC and configure some Networking resources we're gonna use further. The sample code bellow will create a VPC.

# vpc.tf | VPC Configuration

resource "aws_vpc" "aws-vpc" {
  cidr_block           = "10.10.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true
  tags = {
    Name        = "${var.app_name}-vpc"
    Environment = var.app_environment
  }
}

For Networking, it is necessary to create Public and Private Subnets within the VPC, also a Internet Gateway and Route Tables for Public Subnets. The sample bellow will create these resources

# networking.tf | Network Configuration

resource "aws_internet_gateway" "aws-igw" {
  vpc_id = aws_vpc.aws-vpc.id
  tags = {
    Name        = "${var.app_name}-igw"
    Environment = var.app_environment
  }

}

resource "aws_subnet" "private" {
  vpc_id            = aws_vpc.aws-vpc.id
  count             = length(var.private_subnets)
  cidr_block        = element(var.private_subnets, count.index)
  availability_zone = element(var.availability_zones, count.index)

  tags = {
    Name        = "${var.app_name}-private-subnet-${count.index + 1}"
    Environment = var.app_environment
  }
}

resource "aws_subnet" "public" {
  vpc_id                  = aws_vpc.aws-vpc.id
  cidr_block              = element(var.public_subnets, count.index)
  availability_zone       = element(var.availability_zones, count.index)
  count                   = length(var.public_subnets)
  map_public_ip_on_launch = true

  tags = {
    Name        = "${var.app_name}-public-subnet-${count.index + 1}"
    Environment = var.app_environment
  }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.aws-vpc.id

  tags = {
    Name        = "${var.app_name}-routing-table-public"
    Environment = var.app_environment
  }
}

resource "aws_route" "public" {
  route_table_id         = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.aws-igw.id
}

resource "aws_route_table_association" "public" {
  count          = length(var.public_subnets)
  subnet_id      = element(aws_subnet.public.*.id, count.index)
  route_table_id = aws_route_table.public.id
}

Container Registry and ECS Cluster

Now, it's time to create the Container Registry and the ECS Cluster. First let's create the Container Registry with the code bellow:

# ecr.tf | Elastic Container Repository

resource "aws_ecr_repository" "aws-ecr" {
  name = "${var.app_name}-${var.app_environment}-ecr"
  tags = {
    Name        = "${var.app_name}-ecr"
    Environment = var.app_environment
  }
}

The ECR is a repository where we're gonna store the Docker Images of the application we want to deploy. It works like the Docker Hub, if you're familiar with Docker. You can build the Docker Image locally and push it to the ECR or use a CI/CD platform to do it.

Now we're going to create the ECS Cluster, Service and Task Definition.
A service is a configuration that enables us to run and maintain a number of tasks simultaneously in a cluster. The containers are defined by a Task Definition that are used to run tasks in a service.

Before we create the ECS Cluster, we need to create an IAM policy to enable the service to pull the image from ECR.

# iam.tf | IAM Role Policies

resource "aws_iam_role" "ecsTaskExecutionRole" {
  name               = "${var.app_name}-execution-task-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role_policy.json
  tags = {
    Name        = "${var.app_name}-iam-role"
    Environment = var.app_environment
  }
}

data "aws_iam_policy_document" "assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role_policy_attachment" "ecsTaskExecutionRole_policy" {
  role       = aws_iam_role.ecsTaskExecutionRole.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
}

Now let's create what we need for ECS. First we create the ECS Cluster:

resource "aws_ecs_cluster" "aws-ecs-cluster" {
  name = "${var.app_name}-${var.app_environment}-cluster"
  tags = {
    Name        = "${var.app_name}-ecs"
    Environment = var.app_environment
  }
}

I created a Log Group on CloudWatch to get the containers logs.

resource "aws_cloudwatch_log_group" "log-group" {
  name = "${var.app_name}-${var.app_environment}-logs"

  tags = {
    Application = var.app_name
    Environment = var.app_environment
  }
}

I created a Task Definition compatible with AWS FARGATE, I preferred to do so in order to have a better cost of this infrastructure.

data "template_file" "env_vars" {
  template = file("env_vars.json")
}

resource "aws_ecs_task_definition" "aws-ecs-task" {
  family = "${var.app_name}-task"

  container_definitions = <<DEFINITION
  [
    {
      "name": "${var.app_name}-${var.app_environment}-container",
      "image": "${aws_ecr_repository.aws-ecr.repository_url}:latest",
      "entryPoint": [],
      "environment": ${data.template_file.env_vars.rendered},
      "essential": true,
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "${aws_cloudwatch_log_group.log-group.id}",
          "awslogs-region": "${var.aws_region}",
          "awslogs-stream-prefix": "${var.app_name}-${var.app_environment}"
        }
      },
      "portMappings": [
        {
          "containerPort": 8080,
          "hostPort": 8080
        }
      ],
      "cpu": 256,
      "memory": 512,
      "networkMode": "awsvpc"
    }
  ]
  DEFINITION

  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  memory                   = "512"
  cpu                      = "256"
  execution_role_arn       = aws_iam_role.ecsTaskExecutionRole.arn
  task_role_arn            = aws_iam_role.ecsTaskExecutionRole.arn

  tags = {
    Name        = "${var.app_name}-ecs-td"
    Environment = var.app_environment
  }
}

data "aws_ecs_task_definition" "main" {
  task_definition = aws_ecs_task_definition.aws-ecs-task.family
}

An observation about the Task Definition is that I'm using the Terraform data function to set some environment variables that I defined in a JSON file (it needs an improvement to use AWS EKS or some other way to store secrets).

Ok, now let's create the ECS Service.

resource "aws_ecs_service" "aws-ecs-service" {
  name                 = "${var.app_name}-${var.app_environment}-ecs-service"
  cluster              = aws_ecs_cluster.aws-ecs-cluster.id
  task_definition      = "${aws_ecs_task_definition.aws-ecs-task.family}:${max(aws_ecs_task_definition.aws-ecs-task.revision, data.aws_ecs_task_definition.main.revision)}"
  launch_type          = "FARGATE"
  scheduling_strategy  = "REPLICA"
  desired_count        = 1
  force_new_deployment = true

  network_configuration {
    subnets          = aws_subnet.private.*.id
    assign_public_ip = false
    security_groups = [
      aws_security_group.service_security_group.id,
      aws_security_group.load_balancer_security_group.id
    ]
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.target_group.arn
    container_name   = "${var.app_name}-${var.app_environment}-container"
    container_port   = 8080
  }

  depends_on = [aws_lb_listener.listener]
}

I also defined a Security Group to avoid external connections to the containers.

resource "aws_security_group" "service_security_group" {
  vpc_id = aws_vpc.aws-vpc.id

  ingress {
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
    security_groups = [aws_security_group.load_balancer_security_group.id]
  }

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  tags = {
    Name        = "${var.app_name}-service-sg"
    Environment = var.app_environment
  }
}

Application Load Balancer

The next step is to setup a Load Balancer. As you could notice on the ECS configuration is that there's a reference to a load_balancer on it.

resource "aws_alb" "application_load_balancer" {
  name               = "${var.app_name}-${var.app_environment}-alb"
  internal           = false
  load_balancer_type = "application"
  subnets            = aws_subnet.public.*.id
  security_groups    = [aws_security_group.load_balancer_security_group.id]

  tags = {
    Name        = "${var.app_name}-alb"
    Environment = var.app_environment
  }
}

Now let's add a security group for the Load Balancer

resource "aws_security_group" "load_balancer_security_group" {
  vpc_id = aws_vpc.aws-vpc.id

  ingress {
    from_port        = 80
    to_port          = 80
    protocol         = "tcp"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }
  tags = {
    Name        = "${var.app_name}-sg"
    Environment = var.app_environment
  }
}

We also need to create a Load Balancer Target Group, it will relate the Load Balancer with the Containers.

resource "aws_lb_target_group" "target_group" {
  name        = "${var.app_name}-${var.app_environment}-tg"
  port        = 80
  protocol    = "HTTP"
  target_type = "ip"
  vpc_id      = aws_vpc.aws-vpc.id

  health_check {
    healthy_threshold   = "3"
    interval            = "300"
    protocol            = "HTTP"
    matcher             = "200"
    timeout             = "3"
    path                = "/v1/status"
    unhealthy_threshold = "2"
  }

  tags = {
    Name        = "${var.app_name}-lb-tg"
    Environment = var.app_environment
  }
}

One very important thing here is the attribute path within health_check. This is a route on the application that the Load Balancer will use to check the status of the application.

At last let's create a HTTP listener for out Load Balancer.

resource "aws_lb_listener" "listener" {
  load_balancer_arn = aws_alb.application_load_balancer.id
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.target_group.id
  }
}

Autoscaling

So, autoscaling is essential for the application I'm working on. To configure it on AWS I just needed to create an Autoscaling Target and two simple Autoscaling Policies. One to scale by CPU usage and another one for Memory usage.

# autoscaling.tf | Auto Scaling Group

resource "aws_appautoscaling_target" "ecs_target" {
  max_capacity       = 2
  min_capacity       = 1
  resource_id        = "service/${aws_ecs_cluster.aws-ecs-cluster.name}/${aws_ecs_service.aws-ecs-service.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
}

resource "aws_appautoscaling_policy" "ecs_policy_memory" {
  name               = "${var.app_name}-${var.app_environment}-memory-autoscaling"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.ecs_target.resource_id
  scalable_dimension = aws_appautoscaling_target.ecs_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.ecs_target.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageMemoryUtilization"
    }

    target_value = 80
  }
}

resource "aws_appautoscaling_policy" "ecs_policy_cpu" {
  name               = "${var.app_name}-${var.app_environment}-cpu-autoscaling"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.ecs_target.resource_id
  scalable_dimension = aws_appautoscaling_target.ecs_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.ecs_target.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }

    target_value = 80
  }
}

So, the application will scale up if the memory or the cpu usage reaches 80% of usage. When it comes bellow this value, the application will scale down.

Variables

I believe you noticed we used a lot of variables for the Terraform configuration files. To use variables I created a file called variables.tf. This file only have the variables definitions.

# variables.tf | Auth and Application variables

variable "aws_access_key" {
  type        = string
  description = "AWS Access Key"
}

variable "aws_secret_key" {
  type        = string
  description = "AWS Secret Key"
}

variable "aws_region" {
  type        = string
  description = "AWS Region"
}

variable "aws_cloudwatch_retention_in_days" {
  type        = number
  description = "AWS CloudWatch Logs Retention in Days"
  default     = 1
}

variable "app_name" {
  type        = string
  description = "Application Name"
}

variable "app_environment" {
  type        = string
  description = "Application Environment"
}

variable "cidr" {
  description = "The CIDR block for the VPC."
  default     = "10.0.0.0/16"
}

variable "public_subnets" {
  description = "List of public subnets"
}

variable "private_subnets" {
  description = "List of private subnets"
}

variable "availability_zones" {
  description = "List of availability zones"
}

The values for each variable are defined in a file called terraform.tfvars.

aws_region        = "us-east-1"
aws_access_key    = "your aws access key"
aws_secret_key    = "your aws secret key"

# these are zones and subnets examples
availability_zones = ["us-east-1a", "us-east-1b"]
public_subnets     = ["10.10.100.0/24", "10.10.101.0/24"]
private_subnets    = ["10.10.0.0/24", "10.10.1.0/24"]

# these are used for tags
app_name        = "node-js-app"
app_environment = "production"

This file is not committed in my repository. I created it locally and use S3 to manage access and control its versions. It needs some improvements as well that I'll do further.

UPDATE: Now, with all the configuration files properly written, run the command terraform plan to check what changes are going to be done and terraform apply to review and apply the changes.

You can also be asking about the Database. Well, in this project I created a Cluster on MongoCloud and put the credentials on the environment.

The full code can be found on my [Github].(https://github.com/thnery/terraform-aws-template)

Thanks

Thank you for reading this post. I hope it could be useful. If you have any feedback, please, let me know.

"E se trabalharmos como um enxame"?

Tacio Nery — Mon, 19 Oct 2020 20:23:27 +0000

This is a Brazilian Portuguese version of this post

O título desta postagem é uma pergunta que foi feita à minha equipe durante uma retrospectiva da Sprint.

Começamos a trabalhar como um Enxame recentemente e ... Está sendo realmente diferente e bem desafiador em relação a como trabalhavamos antes.

Eu gostaria de compartilhar algumas idéias sobre minha perspectiva desse "enxame" e o que nos levou a experimentá-lo.

Por que decidimos trabalhar como Enxame?

Como equipe, nos comprometemos a entregar os cards selecionados na Sprint Planning, mas nem todos esses cards são entregues (movidos para Done) até o fim da Sprint. Começamos a nos questionar por que isso estava acontecendo e a pergunta foi trazida a nós pelo Engenheiro Líder: "E se trabalharmos como um Enxame?"

Então, o que realmente é trabalhar como Enxame?

É o ato do time se reunir para resolver um problema ou fazer algo o mais rápido possível. A ideia é nos reunirmos e fazermos o que for preciso para finalizar as atividades atuais.

O enxame pode ser usado para focar em questões críticas, como o sistema inativo. Mas, também pode ser usado para entregar as atividades que a equipe se comprometeu no planejamento da Sprint.

Como aplicamos essa abordagem de "enxame"?

No início estávamos perdidos, de alguma forma. Qual a melhor forma de aplicar este método? Cabia a nós, o time, decidir como abordaríamos sobre os cartões da Sprint.

Após o Sprint Planning nos reunimos e iniciamos o cartão no topo do quadro (é importante que os cartões estejam devidamente priorizados) e definimos pequenas tarefas que poderiam ser feitas por cada membro. Decidimos também nos reunir no final do dia para verificar o andamento de cada tarefa. Se alguém terminar uma tarefa, a equipe será notificada e essa pessoa poderá ajudar outra pessoa. A equipe só iniciou um novo card após o término do atual.

Qual foi o impacto?

Trabalhando como um enxame, a equipe ficou muito mais integrada e colaborativa. Todos estão ajudando uns aos outros nas discussões, definições de escopo, decisões de implementação e codificação. Uma vez que todos estão envolvidos no desenvolvimento de um cartão desde o início, todos entendem o contexto do mesmo. Esta está sendo uma experiência muito boa para mim e para minha equipe.

Conclusão

Trabalhar em enxame é algo muito novo para mim, eu costumava trabalhar sozinho em minha mesa antes de entrar para minha equipe na Thoughtworks, onde tive minha primeira experiência com programação em pares, o que é ótimo. Minha equipe e eu aceitamos esse desafio e está sendo ótimo, principalmente porque todos se envolvem em todas as etapas do cartão.

Ainda estamos no início da aplicação deste método. Nosso quadro precisa ser aprimorado para abordar a criação de tarefas e talvez uma versão mais simples do quadro com apenas três colunas: Fazer, Fazer e Concluído. Também há métricas de como a equipe está se saindo com essa nova abordagem, mas acredito que vamos implementar isso em breve.

Então, e se sua equipe trabalhar como um enxame?

Referências

https://softwareengineering.stackexchange.com/questions/147664/what-is-swarming

https://www.infoq.com/news/2013/02/swarming-agile-teams-deliver/

"What if we work as a swarm?"

Tacio Nery — Mon, 19 Oct 2020 19:35:14 +0000

The title of this post is a question that was brought to my team during a sprint review.

We started working as a Swarm recently and... It's being some really different for what we were doing before and challenging.

I'd like to share some thoughts about my perspective of this "Swarming" and what lead us to try it.

Why did we decide to Swarm?

As a team, we commit to deliver the chosen cards on Sprint Planning but, not all of these cards were being delivered (moved to Done) at the and of the Sprint. We started to wonder why was that happening and the question was brought to us by the Lead Engineer: "What if we work as a Swarm?"

So, what really is Swarming?

It's the act of coming together to solve a problem or get something done as quick as possible. The idea is get together and do whatever it take to finish the current activities.

Swarming can used to focus on critical issues, such as the system down. But, it can also be used for delivering the activities the team committed on sprint planning.

How we applied this "Swarming" approach?

At the beginning we were lost, somehow. How is the best way to apply this method? It was up to us to decide how we'd approach over the sprint cards.

After the Sprint Planning we got together and started the card on the top of the board (it's important that the cards are prioritized) and defined small tasks that could be done by every member. We also decided to get together at the end of the day and check the status of each task. If someone finishes a task, the team would get notified and this person could help someone else. The team only started a new card after the current one was finished.

What was the impact?

Since we're working as a swarm the team got much more integrated and collaborative. Everyone is helping each other on discussions, scope definitions, implementation decisions and coding. As everyone is involved on the development of a card since the beginning, everyone gets context on it. This is being a very cool experience for me and my team.

Final thoughts

Swarming is something very new for me, I used to worked alone on my desk before join my team at Thoughtworks, where I had my first experience with pair programming, which is great. My team and I accepted this challenge and it's being great, specially because everyone gets involved with all the steps of the card.

We are still on the beginning of applying this method. Our board needs to be improved to address the creation of tasks and maybe, a simpler version of the board with just three columns: To Do, Doing and Done. There are also metrics of how the team is performing with this new approach, but I believe we're gonna implement this soon.

So, what if your team work as a swarm?

References

https://softwareengineering.stackexchange.com/questions/147664/what-is-swarming

https://www.infoq.com/news/2013/02/swarming-agile-teams-deliver/

How to remove Docker images?

Tacio Nery — Fri, 24 Jan 2020 17:25:14 +0000

Docker is a great tool to for software development and delivery. It enables you to separate applications from your infrastructure and deliver them softly and quickly.

How to remove Docker images?

Sometimes when we keep creating Docker images on our machine, or even on some cloud running Docker Container it'll soon start to run out of space. In this short post I'd like to share how I get rid of unnecessary Docker images.

The first step is to check the present images on your machine with this command:

$ docker images

The output should be like this:

REPOSITORY              TAG                 IMAGE ID    
mysql                   5.7                 b598110d0fff
mysql                   latest              ed1ffcb5eff3
<none>                  <none>              e81ad38baa90
redis                   latest              dcf9ec9265e0
ruby                    2.6.5-alpine        3304101ccbe9
postgres                latest              f9b3d2f9a593
mongo                   latest              58477a771fb4
rabbitmq                latest              86a895bb41d6
redis                   5.0.3-alpine        3d2a373f46ae

The output shows us the repository, it's the name at Docker Hub repository for the image. The TAG is the version of the image and the Image ID is the ID generated by Docker when we pull an image for our machine.

Before remove the desired image, it's important to know that the image is possibly associated with one or more containers. To check what containers are associated with images use the command below:

$ docker ps -a

The output will show us the ID of the container in our machine, the ID of an image we pulled and the entry command - this is the command to run our dockerized application.

CONTAINER ID        IMAGE                   COMMAND               
b8b33f28e089        ffb1b540cc8f            "/bin/sh -c 'apt-get…"
e2c8159a0ee6        ffb1b540cc8f            "/bin/sh -c 'apt-get…"
8d0910845ea5        ec6c8a057b83            "/bin/sh -c 'apt-get…"
c977dd983867        ec6c8a057b83            "/bin/sh -c 'apt-get…"
94f9fc8e6186        929b84501e14            "/bin/sh -c 'bundle …"
8d5de5c2c7a3        61f362c8bfb9            "/bin/sh -c 'bundle …"
3e3972e6aa86        15cc0f691e7b            "/bin/sh -c 'bundle …"
f3175c20bad5        ed8edff8b2e2            "/bin/sh -c 'bundle …"
f76d4f777146        ed8edff8b2e2            "/bin/sh -c 'bundle …"
4eed76085da4        postgres                "docker-entrypoint.s…"
505fc40d64c0        e81ad38baa90            "/bin/sh -c 'bundle …"
1804e73a047a        redis                   "docker-entrypoint.s…"
3241275444f3        postgres                "docker-entrypoint.s…"
154e700f0e65        postgres                "docker-entrypoint.s…"
28bfc42fdac6        mongo:latest            "docker-entrypoint.s…"
2e15f116cbf2        postgres                "docker-entrypoint.s…"
6ca6f9bd1f1b        rabbitmq:latest         "docker-entrypoint.s…"

To remove the container and the image we will use two commands:

$ docker rm <container_id>
$ docker rmi <image_id>

We can use the -f option for both commands to force the removal of a resource.
Here are the commands I use to remove all images and containers when I need to clean up my laptop.

$ docker ps -aq | xargs docker rm -f
$ docker images -aq | xargs docker rmi -f

Let's break it down. The docker ps is the command to show the running containers, the option -a will show all containers (even not running ones) and the -q option will output only the CONTAINER ID.

The xargs command is to build and execute a command from standard input. In our case we are passing to xargs the container id from the docker ps -aq command and the image id from the docker images -aq command. So, the xargs will build the exact command we need to remove the docker containers and images for each given ID we've got.

Removing Dangling Images

Dangling image means that a new build of the image was created, but wasn't given a name (untaged image). So, the old image becomes dangling. They'll be displayed as when you run docker images

If we need to remove only the dangling images, the command bellow works very fine. Add the option -a to also remove unused images.

$ docker images prune

So, hope this might be useful for you as it is for me.

Thanks for reading.

A short brief about Classification

Tacio Nery — Tue, 13 Aug 2019 14:06:40 +0000

What is Classification?

This is the first question I did when I heard the term Classification. The definition says it is, fundamentally, a model to predict labels. It brought me a new question. What are labels? Well, in a dataset for a classification models we will find features and labels, where a feature is a column used as an input data and the label is the value we want to predict.
So, when we know what value we want to predict with a Machine Learning model we have a Classification Problem.

Testing Classification Models

Let's get a NBA Log dataset. The goal is to predict if a player will last longer than 5 years in league. This data contains a target column, the TARGET_5Yrs column can be 0 (< 5 years) or 1 (>= 5 years). As we know our target (label), we can say for sure this is a Classification problem.

This dataset can be found here.

Requirements

Here are the libraries we will use in this example.

import os

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from pandas.plotting import scatter_matrix
from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC

# define the dataset path
DATASET_PATH = os.path.join("datasets")
SEED = 7

Loading the dataset

The first thing to do is load the dataset, let's use Pandas to do it and check how data is available.

# create a function to load the dataset
def load_nba_data(dataset_path=DATASET_PATH):
    csv_path = os.path.join(dataset_path, "nba_logreg.csv")
    return pd.read_csv(csv_path)

# load the dataset
nba_data = load_nba_data()
# replace NaN fields with 0
nba_data.fillna(0, inplace=True)
# show 10 first rows
nba_data.head(10)

	Name	GP	MIN	PTS	FGM	FGA	FG%	3P Made	3PA	3P%	...	FTA	FT%	OREB	DREB	REB	AST	STL	BLK	TOV	TARGET_5Yrs
0	Brandon Ingram	36	27.4	7.4	2.6	7.6	34.7	0.5	2.1	25.0	...	2.3	69.9	0.7	3.4	4.1	1.9	0.4	0.4	1.3	0.0
1	Andrew Harrison	35	26.9	7.2	2.0	6.7	29.6	0.7	2.8	23.5	...	3.4	76.5	0.5	2.0	2.4	3.7	1.1	0.5	1.6	0.0
2	JaKarr Sampson	74	15.3	5.2	2.0	4.7	42.2	0.4	1.7	24.4	...	1.3	67.0	0.5	1.7	2.2	1.0	0.5	0.3	1.0	0.0
3	Malik Sealy	58	11.6	5.7	2.3	5.5	42.6	0.1	0.5	22.6	...	1.3	68.9	1.0	0.9	1.9	0.8	0.6	0.1	1.0	1.0
4	Matt Geiger	48	11.5	4.5	1.6	3.0	52.4	0.0	0.1	0.0	...	1.9	67.4	1.0	1.5	2.5	0.3	0.3	0.4	0.8	1.0
5	Tony Bennett	75	11.4	3.7	1.5	3.5	42.3	0.3	1.1	32.5	...	0.5	73.2	0.2	0.7	0.8	1.8	0.4	0.0	0.7	0.0
6	Don MacLean	62	10.9	6.6	2.5	5.8	43.5	0.0	0.1	50.0	...	1.8	81.1	0.5	1.4	2.0	0.6	0.2	0.1	0.7	1.0
7	Tracy Murray	48	10.3	5.7	2.3	5.4	41.5	0.4	1.5	30.0	...	0.8	87.5	0.8	0.9	1.7	0.2	0.2	0.1	0.7	1.0
8	Duane Cooper	65	9.9	2.4	1.0	2.4	39.2	0.1	0.5	23.3	...	0.5	71.4	0.2	0.6	0.8	2.3	0.3	0.0	1.1	0.0
9	Dave Johnson	42	8.5	3.7	1.4	3.5	38.3	0.1	0.3	21.4	...	1.4	67.8	0.4	0.7	1.1	0.3	0.2	0.0	0.7	0.0

10 rows × 21 columns

Here we have a small sample of our dataset. Let's discard the Name and the TARGET_5Yrs columns, all the others are the features of every player, these will tell us if the player will last longer than 5 years in a league or not. The TARGET_5Yrs has the answer for every combination of features.

Let's check a quick description of our dataset with the info() function.

# first let's remove the uneeded Name column, 'cause it's not relevant for this experiment
nba_data = nba_data.drop('Name', 1)
nba_data.info()

# first let's remove the uneeded Name column, 'cause it's not relevant for this experiment
nba_data = nba_data.drop('Name', 1)
nba_data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1340 entries, 0 to 1339
Data columns (total 20 columns):
GP             1340 non-null int64
MIN            1340 non-null float64
PTS            1340 non-null float64
FGM            1340 non-null float64
FGA            1340 non-null float64
FG%            1340 non-null float64
3P Made        1340 non-null float64
3PA            1340 non-null float64
3P%            1340 non-null float64
FTM            1340 non-null float64
FTA            1340 non-null float64
FT%            1340 non-null float64
OREB           1340 non-null float64
DREB           1340 non-null float64
REB            1340 non-null float64
AST            1340 non-null float64
STL            1340 non-null float64
BLK            1340 non-null float64
TOV            1340 non-null float64
TARGET_5Yrs    1340 non-null float64
dtypes: float64(19), int64(1)
memory usage: 209.5 KB

We can also check some statistics information on our dataset with the describe function.

nba_data.describe()

	GP	MIN	PTS	FGM	FGA	FG%	3P Made	3PA	3P%	FTM	FTA	FT%	OREB	DREB	REB	AST	STL	BLK	TOV	TARGET_5Yrs
count	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000	1340.000000
mean	60.414179	17.624627	6.801493	2.629104	5.885299	44.169403	0.247612	0.779179	19.149627	1.297687	1.821940	70.300299	1.009403	2.025746	3.034478	1.550522	0.618507	0.368582	1.193582	0.620149
std	17.433992	8.307964	4.357545	1.683555	3.593488	6.137679	0.383688	1.061847	16.051861	0.987246	1.322984	10.578479	0.777119	1.360008	2.057774	1.471169	0.409759	0.429049	0.722541	0.485531
min	11.000000	3.100000	0.700000	0.300000	0.800000	23.800000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.200000	0.300000	0.000000	0.000000	0.000000	0.100000	0.000000
25%	47.000000	10.875000	3.700000	1.400000	3.300000	40.200000	0.000000	0.000000	0.000000	0.600000	0.900000	64.700000	0.400000	1.000000	1.500000	0.600000	0.300000	0.100000	0.700000	0.000000
50%	63.000000	16.100000	5.550000	2.100000	4.800000	44.100000	0.100000	0.300000	22.200000	1.000000	1.500000	71.250000	0.800000	1.700000	2.500000	1.100000	0.500000	0.200000	1.000000	1.000000
75%	77.000000	22.900000	8.800000	3.400000	7.500000	47.900000	0.400000	1.200000	32.500000	1.600000	2.300000	77.600000	1.400000	2.600000	4.000000	2.000000	0.800000	0.500000	1.500000	1.000000
max	82.000000	40.900000	28.200000	10.200000	19.800000	73.700000	2.300000	6.500000	100.000000	7.700000	10.200000	100.000000	5.300000	9.600000	13.900000	10.600000	2.500000	3.900000	4.400000	1.000000

Let's take a look at our target is distributed over the dataset.

nba_data.groupby('TARGET_5Yrs').size()

TARGET_5Yrs
0.0    509
1.0    831
dtype: int64

In resume, we have 1340 objects in our dataset where 509 will not last longer than 5 years in league and the others 831 will.

Machine Learning Models Evaluation

As we saw in the beginning of this post, this is a Classification Problem. We will create some models with different ML algorithms and check their accuracy.

Spliting Data

Let's split our dataset into two new datasets. We will use 80% of the dataset to train our classification models and 20% of it to perform the validation.

data = nba_data.values
# data = np.array(data)

# now let's separate the features columns from the target column
X = data[:, 0:19]
Y = data[:, 19]

# as said before,  we will use 20% of the dataset for validation
validation_size = 0.20

# split the data into traning and testing
X_train, X_test, Y_train, Y_test = model_selection.train_test_split(X, Y, test_size=validation_size, random_state=SEED)

Now that we have our training and testing set, we are going to create an array with the models we want to evaluate. We will use each model with the default settings.

models = []
models.append(('LR', LogisticRegression(solver='liblinear', multi_class='ovr')))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC(gamma='auto')))

To evaluate the models we will user K-Fold cross-validation and measure the accuracy for each model. This techinique randomly splits the training set into K distincts subsets (folds), then it trains and evaluates the model K times picking a different fold for every evaliation. The result will be an array with the K evaluation scores. For this example we will a Cross-Validation using the StratifiedKFold from SKLearn. We will use the Mean of the accuracies of each model to determinate which one has the best results.

scoring = 'accuracy'
models_results = []
for name, model in models:
    results = []
    skfolds = model_selection.StratifiedKFold(n_splits=10, random_state=SEED)
    for train_index, test_index in skfolds.split(X_train, Y_train):
        X_train_folds = X_train[train_index]
        Y_train_folds = (Y_train[train_index])
        X_test_folds = X_train[test_index]
        Y_test_folds = (Y_train[test_index])

        model.fit(X_train_folds, Y_train_folds)
        pred = model.predict(X_test_folds)
        correct = sum(pred == Y_test_folds)
        results.append(correct / len(pred))
    models_results.append((name, results))


names = []
scores = []
# the snippet bellow calculates the mean of the accuracies
for name, results in models_results:
    mean = np.array(results).mean()
    std = np.array(results).std()
    print("Model: %s, Accuracy Mean: %f (%f)" % (name, mean, std))
    names.append(name)
    scores.append(results)

Model: LR, Accuracy Mean: 0.705244 (0.026186)
Model: LDA, Accuracy Mean: 0.706205 (0.027503)
Model: KNN, Accuracy Mean: 0.674429 (0.026029)
Model: CART, Accuracy Mean: 0.634372 (0.047236)
Model: NB, Accuracy Mean: 0.632433 (0.040794)
Model: SVM, Accuracy Mean: 0.619384 (0.021099)

The results above show us that the Linear Discriminant Analysis has the best accuracy score among the models we tested. The boxplot below shows the accuracy scores spread accross each fold.

fig = plt.figure()
fig.suptitle('Models Comparison')
ax = fig.add_subplot(111)
plt.boxplot(scores)
ax.set_xticklabels(names)
plt.show()

Making Predictions

Now we will check the accuracy of the LDA model by making some predictions with the validation set we've prepared before. To do so, we will create an instance of the model and use the method predict.

model = LinearDiscriminantAnalysis()
model.fit(X_train, Y_train)
predictions = model.predict(X_test)
print("Accuracy: {}".format(accuracy_score(Y_test, predictions)))

Accuracy: 0.6902985074626866

We can also check the Confusion Matrix for this model

print(confusion_matrix(Y_test, predictions))

[[ 52  45]
 [ 38 133]]

Each row in a confusion matrix represents an actual target and each column represents a predicted target. The first row of this matrix contains the true negatives and the false positives. Which means that 52 samples were correctly classified and 45 were wrongly classified. The second row shows us the false negatives and the true positives, wich means that 38 samples were wrongly classified and 133 were classified correctly.

The confusion matrix provides a lot of information, but if you want to get a more concise metrics you can use the classification_report function of Scikit-Learn. It will provide the precision, recall and f1-score metrics.

print(classification_report(Y_test, predictions))

              precision    recall  f1-score   support

         0.0       0.58      0.54      0.56        97
         1.0       0.75      0.78      0.76       171

    accuracy                           0.69       268
   macro avg       0.66      0.66      0.66       268
weighted avg       0.69      0.69      0.69       268

The accuracy of the positive predictions is called precision. It's defined by the formula: TP/(TP + FP), where TP is the number of True Positives and FP is the number of False Positives. This metric is tipically used along the recall which is the true positive rate - the ratio of positive instances that are correctly detected by the model. It's equation is: TP / (TP + FN) where FN is the False Negatives.

Conclusion

This is a short brief about Classification with Python and Scikit-Learning. There is a lot more to cover, we can improve our models results by normalizing the data for example. There's also others metrics to cover. But the firts steps into Machine Learning world can be done with this tutorial. Hope you enjoy it!!

You can access the notebook for this example here.

References

Your First Machine Learning Project in Python Step-By-Step - https://machinelearningmastery.com/machine-learning-in-python-step-by-step/
Géron, A. (2017). Hands-on machine learning with Scikit-Learn and TensorFlow: Concepts, tools, and techniques to build intelligent systems.

Kubernetes Cluster Autoscaler on AWS

Tacio Nery — Fri, 26 Jul 2019 14:09:42 +0000

Running a production environment in any cloud (AWS, GCloud, MSAzure) could be very expensive if the resources are not used efficiently. Taking the scenario where a web platform is using 4 EC2 instances t2.large, this was the stack where I work before we change to Kubernetes - I will write about this some day. So taking this scenario, it's too much resource that is is not needed all the time.
With Kubernetes we created a more flexible environment, but we have the same amount of nodes running all the time. Cluster Autoscaler came to solve this. We could define a minimum amount of nodes and scale it when necessary. This is an approach of how we installed and configured Cluster Autoscaler for our environment. Our environment is basically this:

Amazon AWS EC2
Kubectl: Server Version 1.10.6
Kops: 1.10.0
Helm: 2.11.0

Before install, there are some steps to prepare the cluster. First, add some extra labels to nodes instance groups. Use the command below to open the editor.

$ kops edit ig nodes

Now Add the new labels in cloudLabels key.

spec:
  cloudLabels:
    k8s.io/cluster-autoscaler/my.cluster.com: ""
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/node-template/label: ""
    kubernetes.io/cluster/my.cluster.com: owned
  ...
  minSize: 5
  maxSize: 10

This configuration gives Kubernetes the ability to auto-discover instance groups based on cluster name. It's highly recommended, specially if there's multiple instance groups. Now add additional IAM policy rules for nodes, to do so, just edit the cluster config.

$ kops edit cluster

Now add the policies bellow:

...
kind: Cluster
spec:
  additionalPolicies:
    node: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "autoscaling:DescribeAutoScalingGroups",
            "autoscaling:DescribeAutoScalingInstances",
            "autoscaling:SetDesiredCapacity",
            "autoscaling:DescribeLaunchConfigurations",
            "autoscaling:DescribeTags",
            "autoscaling:TerminateInstanceInAutoScalingGroup"
          ],
          "Resource": ["*"]
        }
      ]
...

The command to review the updates is:

$ kops update cluster

To apply the updates just add the flag --yes

$ kops update cluster --yes

Ok, now that the cluster is updated it's time to install the Cluster Autoscaler. First install Helm, it's is a package manager for Kubernetes, it will make the job much easier. To find more about Helm, check: https://github.com/helm/helm.

The Cluster Autoscaler will be installed on Master node in kube-system namespace. Remember to choose the right AWS Region (the one where your cluster was created).

To install Cluster Autoscaler use this command, feel free to adjust the values for your environment:

helm install --name cluster-autoscaler \
 --namespace kube-system \
 --set image.tag=v1.2.0 \
 --set autoDiscovery.clusterName=my.cluster.com \
 --set extraArgs.balance-similar-node-groups=false \
 --set extraArgs.expander=random \
 --set rbac.create=true \
 --set rbac.pspEnabled=true \
 --set awsRegion=us-east-1 \
 --set nodeSelector."node-role\.kubernetes\.io/master"="" \
 --set tolerations[0].effect=NoSchedule \
 --set tolerations[0].key=node-role.kubernetes.io/master \
 --set cloudProvider=aws \
 stable/cluster-autoscaler

Obs.: To know what version of Cluster Autoscaler you should install, check the Releases section at https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler

One interesting value to check is expander . Cluster Autoscaler has, currently, 4 expander. They are designed to provide different strategies for selecting the node group to be scaled.

random: the default expander, should be used when there's no particular need for scaling node groups differently:
most-pods: select the node group with the most amount of pods scheduled to be scaled
least-waste: select the node group which waste the least amount of Memory/CPU resources
price: select the node group based on price

To get more information about other values for CA, check: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md
Use this command to see the Cluster Autoscaler logs:

$ kubectl -n kube-system get pods
NAME                                                         READY
<cluster-autoscaler-id>                                        1/1

$ kubectl -n kube-system logs -f <cluster-autoscaler-id>

To test if it's really working, create some NGINX Pods with a big amount of replicas:

$ kubectl run nginx --image=nginx --port=80 --replicas=200

Go to AWS EC2 and check the Autoscaling Groups, the Desired amount will increase as it's necessary. To scale down just remove the NGINX deploy:

$ kubectl delete deploy nginx

Summary

Cluster Autoscaler is the first step to have an efficient environment with Kubernetes. It will provide a very nice automated scaling of cloud resources. Hope it can be helpful, if you have any considerations, just let me know. Thanks for reading!!

Hi, I'm Tacio Nery

Tacio Nery — Wed, 22 Feb 2017 17:54:30 +0000

I have been coding for 10 years.

You can find me on Twitter as @thnery1

I live in Olinda.

I mostly program in these languages: Ruby, Python, ShellScript.

I am currently learning more about Data Science.

Nice to meet you.