Forem: bzinoun

[ 🇬🇧 ] Move from an IBM Cloud account to another

bzinoun — Mon, 16 Dec 2019 09:43:41 +0000

A few weeks ago, we had to move from an IBMCloud account to another , the cause was some enterprise constraints , you know the movie 🤷‍♂‍ .

As IBMCloud doesn’t provide tools out of the box for it, our migration was based on our daily CI/CD pipeline! Wow 🤦‍♂‍

Now, let's see what we need to move to the new Cloud:

➡️ 30 CloudFoundry
➡️ 10 PostgreSql
➡️ 2 MongoDB
➡️ 3 AppID (OPIDC)
➡️ 4 domain names

Okey 49 assets to move, seems good.

Now we have seen the most asset we have, the idea is to use CI/CD pipelines for most of our deployments.

So, what's about our application’s pipeline:

✅ Some applications are deployed with GitlabCI pipeline => should be OK
❌ Some applications are deployed with IBMCloud Devops pipeline => which is not cloud agnostic
❌ No Infrastructure as code for BDD, AppId ...etc. => Hmm manually provisioning ?
✅ Data migration? all our assets have an API for import/export => it's should be OK.

We have some bad and good thinks, moving all that stuff with a minimal down want be the best way , so without too much thinking

💡 Blue Green deployment is the best way
More about blue green deployment from Martin Fowler here

The Gif below describe in visual way how we proceed:

The process can be splitted to 6 steps describes as below:

Step 1 - unifying our deployment Pipeline by moving all our CI/CD pipeline to GitlabCI.

Step 2 - Move all non-production assets . As we are working in an agile mode we move the dev environment first then, when needed, staging , sandbox

Step 3 - Move the production assets, then data migration, then push the provisional production URL to the Business analyst / Product Owner for a Go/No GO

Step 4 - Once we have the GO -> Migrate the domain name from the legacy cloud to the New IBMCloud

Step 5 - Remove the death pipeline code , and remove the old assets from legacy Cloud

Step 6 Celebrate with the Team. 🎊 🎉🎊 🎉

So, without going too deep into details of every step , this how we do the work .

In the end, it's been not easy, and having a devops culture helps a lot and i can resume what we have learn :

1- we have to be cloud agnostic:
A toolchain must be a cloud agnostic. If we want a change our
deployment strategy, it must me smooth.

2 - Devops culture is a must
The pipeline source must be versioned, involved with the code source. The dev Team is the owner of the pipeline

3 - Start Easy
Dev environment first, as we work on agile , the other environments will be deployed in an iterative and incremental way

4 - If it’s hard manually, then script it .
Special attention to the Data migration for every type of asset. ( AppId , SQL , NoSql , Cloud object Storage …etc )

That's all folks, and how will you do in the same situation ?

[ 🇬🇧 ] -IBMCloud- K8S ready CI/CD toolchain

bzinoun — Sun, 09 Jun 2019 15:53:38 +0000

As i am an Ibmcloud (Bluemix)user, i share with you a ready Devops toolchain that i use for my Kubernetes Deployment .

In the interest of efficiency and continuous deployment, the implementation of a ToolChain of continuous deployment remains a must. The latter is divided into 3 Steps :

Step 1

Validate docker file
Build Docker image
Push docker images to Registry

Step2 : Non blocking Step

Vulnerability scanning for the docker image
Get the recommendations to fix them

Step3

validate deployment files
deploy changes to Kubernetes Cluster

🛠 Toolchain

Step 1 : Build

This step is responsible to verify docker file with a linter then build the image and push it to registry .
Bellow we describe the shell scripte that do the stuff :

Verify Docker file

#!/bin/bash
echo "REGISTRY_URL=${REGISTRY_URL}"
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}"
echo "IMAGE_NAME=${IMAGE_NAME}"
echo "ARCHIVE_DIR=${ARCHIVE_DIR}"
echo "DOCKER_ROOT=${DOCKER_ROOT}"
echo "DOCKER_FILE=${DOCKER_FILE}"

# View build properties
if [ -f build.properties ]; then 
  echo "build.properties:"
  cat build.properties
else 
  echo "build.properties : not found"
fi 

echo "=========================================================="
echo "Checking for Dockerfile at the repository root"
if [ -z "${DOCKER_ROOT}" ]; then DOCKER_ROOT=. ; fi
if [ -z "${DOCKER_FILE}" ]; then DOCKER_FILE=Dockerfile ; fi
if [ -f ${DOCKER_ROOT}/${DOCKER_FILE} ]; then 
echo -e "Dockerfile found at: ${DOCKER_FILE}"
else
    echo "Dockerfile not found at: ${DOCKER_FILE}"
    exit 1
fi
echo "Linting Dockerfile"
npm install -g dockerlint
dockerlint -f ${DOCKER_ROOT}/${DOCKER_FILE}

echo "=========================================================="
echo "Checking registry current plan and quota"
bx cr plan
bx cr quota
echo "If needed, discard older images using: bx cr image-rm"
echo "Checking registry namespace: ${REGISTRY_NAMESPACE}"
NS=$( bx cr namespaces | grep ${REGISTRY_NAMESPACE} ||: )
if [ -z "${NS}" ]; then
    echo "Registry namespace ${REGISTRY_NAMESPACE} not found, creating it."
    bx cr namespace-add ${REGISTRY_NAMESPACE}
    echo "Registry namespace ${REGISTRY_NAMESPACE} created."
else 
    echo "Registry namespace ${REGISTRY_NAMESPACE} found."
fi
echo -e "Existing images in registry"
bx cr images --restrict ${REGISTRY_NAMESPACE}

Build Docker Image

#!/bin/bash
echo "REGISTRY_URL=${REGISTRY_URL}"
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}"
echo "IMAGE_NAME=${IMAGE_NAME}"
echo "BUILD_NUMBER=${BUILD_NUMBER}"
echo "ARCHIVE_DIR=${ARCHIVE_DIR}"
echo "GIT_BRANCH=${GIT_BRANCH}"
echo "GIT_COMMIT=${GIT_COMMIT}"
echo "DOCKER_ROOT=${DOCKER_ROOT}"
echo "DOCKER_FILE=${DOCKER_FILE}"

# View build properties
if [ -f build.properties ]; then 
  echo "build.properties:"
  cat build.properties
else 
  echo "build.properties : not found"
fi 

# To review or change build options use:
echo -e "build --help"

 bx cr build --help

echo -e "Existing images in registry"
bx cr images

# Minting image tag using format: BUILD_NUMBER--BRANCH-COMMIT_ID-TIMESTAMP
# e.g. 3-master-50da6912-20181123114435
# (use build number as first segment to allow image tag as a patch release name according to semantic versioning)

TIMESTAMP=$( date -u "+%Y%m%d%H%M%S")
IMAGE_TAG=${TIMESTAMP}
if [ ! -z "${GIT_COMMIT}" ]; then
  GIT_COMMIT_SHORT=$( echo ${GIT_COMMIT} | head -c 8 ) 
  IMAGE_TAG=${GIT_COMMIT_SHORT}-${IMAGE_TAG}
fi
if [ ! -z "${GIT_BRANCH}" ]; then IMAGE_TAG=${GIT_BRANCH}-${IMAGE_TAG} ; fi
IMAGE_TAG=${BUILD_NUMBER}-${IMAGE_TAG}
echo "=========================================================="
echo -e "BUILDING CONTAINER IMAGE: ${IMAGE_NAME}:${IMAGE_TAG}"
if [ -z "${DOCKER_ROOT}" ]; then DOCKER_ROOT=. ; fi
if [ -z "${DOCKER_FILE}" ]; then DOCKER_FILE=${DOCKER_ROOT}/Dockerfile ; fi


#set -x
#bx cr build -t ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG} ${DOCKER_ROOT} -f ${DOCKER_FILE}
#set +x

bx cr build -t ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${TIMESTAMP} ${DOCKER_ROOT} -f ${DOCKER_FILE}


bx cr image-inspect ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${TIMESTAMP} 

# Set PIPELINE_IMAGE_URL for subsequent jobs in stage (e.g. Vulnerability Advisor)
export PIPELINE_IMAGE_URL="$REGISTRY_URL/$REGISTRY_NAMESPACE/$IMAGE_NAME:$IMAGE_TAG"

bx cr images --restrict ${REGISTRY_NAMESPACE}/${IMAGE_NAME}

echo "=========================================================="
echo "COPYING ARTIFACTS needed for deployment and testing (in particular build.properties)"

echo "Checking archive dir presence"
if [ -z "${ARCHIVE_DIR}" ]; then
  echo -e "Build archive directory contains entire working directory."
else
  echo -e "Copying working dir into build archive directory: ${ARCHIVE_DIR} "
  mkdir -p ${ARCHIVE_DIR}
  find . -mindepth 1 -maxdepth 1 -not -path "./$ARCHIVE_DIR" -exec cp -R '{}' "${ARCHIVE_DIR}/" ';'
fi

# If already defined build.properties from prior build job, append to it.
cp build.properties $ARCHIVE_DIR/ || :

# IMAGE information from build.properties is used in Helm Chart deployment to set the release name
echo "IMAGE_NAME=${IMAGE_NAME}" >> $ARCHIVE_DIR/build.properties
echo "IMAGE_TAG=${IMAGE_TAG}" >> $ARCHIVE_DIR/build.properties
# REGISTRY information from build.properties is used in Helm Chart deployment to generate cluster secret
echo "REGISTRY_URL=${REGISTRY_URL}" >> $ARCHIVE_DIR/build.properties
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}" >> $ARCHIVE_DIR/build.properties
echo "GIT_BRANCH=${GIT_BRANCH}" >> $ARCHIVE_DIR/build.properties
echo "TIMESTAMP=${TIMESTAMP}" >> $ARCHIVE_DIR/build.properties

echo "File 'build.properties' created for passing env variables to subsequent pipeline jobs:"
cat $ARCHIVE_DIR/build.properties

Step 2 : Validate

Security Check

#!/bin/bash
# uncomment to debug the script
#set -x

# View build properties
if [ -f build.properties ]; then 
  echo "build.properties:"
  cat build.properties
else 
  echo "build.properties : not found"
fi 

PIPELINE_IMAGE_URL=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${TIMESTAMP}

echo "PIPELINE_IMAGE_URL=${PIPELINE_IMAGE_URL}"
echo "REGISTRY_URL=${REGISTRY_URL}"
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}"
echo "IMAGE_NAME=${IMAGE_NAME}"
echo "IMAGE_TAG=${IMAGE_TAG}"
echo "TIMESTAMP=${TIMESTAMP}"

bx cr images --restrict ${REGISTRY_NAMESPACE}/${IMAGE_NAME}
echo -e "Checking vulnerabilities in image: ${PIPELINE_IMAGE_URL}"

for ITER in {1..30}
do
  set +e
  STATUS=$( bx cr va -e -o json ${PIPELINE_IMAGE_URL} | jq -r '.[0].status' )
  set -e
  # Possible status from Vulnerability Advisor: OK, UNSUPPORTED, INCOMPLETE, UNSCANNED, FAIL, WARN
  if [[ ${STATUS} != "INCOMPLETE" && ${STATUS} != "UNSCANNED" ]]; then
    break
  fi
  echo -e "${ITER} STATUS ${STATUS} : A vulnerability report was not found for the specified image."
  echo "Either the image doesn't exist or the scan hasn't completed yet. "
  echo "Waiting for scan to complete..."
  sleep 10
done
set +e
bx cr va -e ${PIPELINE_IMAGE_URL}
set -e
STATUS=$( bx cr va -e -o json ${PIPELINE_IMAGE_URL} | jq -r '.[0].status' )
[[ ${STATUS} == "OK" ]] || [[ ${STATUS} == "UNSUPPORTED" ]] || [[ ${STATUS} == "WARN" ]] || { echo "ERROR: The vulnerability scan was not successful, check the OUTPUT of the command and try again."; exit 1; }

Step 3 : Deploy

Verify K8S deployments file

#!/bin/bash
# uncomment to debug the script
#set -x
echo "IMAGE_NAME=${IMAGE_NAME}"
echo "IMAGE_TAG=${IMAGE_TAG}"
echo "REGISTRY_URL=${REGISTRY_URL}"
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}"
echo "DEPLOYMENT_FILE=${DEPLOYMENT_FILE}"

# Fix : after getting error with IMAGE_TAG variable
IMAGE_TAG=${TIMESTAMP}


# Input env variables from pipeline job
echo "PIPELINE_KUBERNETES_CLUSTER_NAME=${PIPELINE_KUBERNETES_CLUSTER_NAME}"
if [ -z "${CLUSTER_NAMESPACE}" ]; then CLUSTER_NAMESPACE=default ; fi
echo "CLUSTER_NAMESPACE=${CLUSTER_NAMESPACE}"

echo "=========================================================="
echo "DEPLOYING using manifest"
echo -e "Updating ${DEPLOYMENT_FILE} with image name: ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}"
if [ -z "${DEPLOYMENT_FILE}" ]; then DEPLOYMENT_FILE=deployment.yml ; fi
if [ -f ${DEPLOYMENT_FILE} ]; then
    sed -i "s~^\([[:blank:]]*\)image:.*$~\1image: ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}~" ${DEPLOYMENT_FILE}
    cat ${DEPLOYMENT_FILE}
else 
    echo -e "${red}Kubernetes deployment file '${DEPLOYMENT_FILE}' not found${no_color}"
    exit 1
fi    
set -x
kubectl apply --namespace ${CLUSTER_NAMESPACE} -f ${DEPLOYMENT_FILE} 
set +x

echo ""
echo "=========================================================="
IMAGE_REPOSITORY=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}
echo -e "CHECKING deployment status of ${IMAGE_REPOSITORY}:${IMAGE_TAG}"
echo ""
for ITERATION in {1..30}
do
  DATA=$( kubectl get pods --namespace ${CLUSTER_NAMESPACE} -o json )
  NOT_READY=$( echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | select(.ready==false) ' )
  if [[ -z "$NOT_READY" ]]; then
    echo -e "All pods are ready:"
    echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | select(.ready==true) '
    break # deployment succeeded
  fi
  REASON=$(echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | .state.waiting.reason')
  echo -e "${ITERATION} : Deployment still pending..."
  echo -e "NOT_READY:${NOT_READY}"
  echo -e "REASON: ${REASON}"
  if [[ ${REASON} == *ErrImagePull* ]] || [[ ${REASON} == *ImagePullBackOff* ]]; then
    echo "Detected ErrImagePull or ImagePullBackOff failure. "
    echo "Please check proper authenticating to from cluster to image registry (e.g. image pull secret)"
    break; # no need to wait longer, error is fatal
  elif [[ ${REASON} == *CrashLoopBackOff* ]]; then
    echo "Detected CrashLoopBackOff failure. "
    echo "Application is unable to start, check the application startup logs"
    break; # no need to wait longer, error is fatal
  fi
  sleep 5
done

APP_NAME=$(kubectl get pods --namespace ${CLUSTER_NAMESPACE} -o json | jq -r '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | .name' | head -n 1)
echo -e "APP: ${APP_NAME}"
echo "DEPLOYED PODS:"
kubectl describe pods --selector app=${APP_NAME} --namespace ${CLUSTER_NAMESPACE}
if [ ! -z "${APP_NAME}" ]; then
  APP_SERVICE=$(kubectl get services --namespace ${CLUSTER_NAMESPACE} -o json | jq -r ' .items[] | select (.spec.selector.app=="'"${APP_NAME}"'") | .metadata.name ')
  echo -e "SERVICE: ${APP_SERVICE}"
  echo "DEPLOYED SERVICES:"
  kubectl describe services ${APP_SERVICE} --namespace ${CLUSTER_NAMESPACE}
fi
#echo "Application Logs"
#kubectl logs --selector app=${APP_NAME} --namespace ${CLUSTER_NAMESPACE}  
echo ""
if [[ ! -z "$NOT_READY" ]]; then
  echo ""
  echo "=========================================================="
  echo "DEPLOYMENT FAILED"
  exit 1
fi

echo ""
echo "=========================================================="
echo "DEPLOYMENT SUCCEEDED"
if [ ! -z "${APP_SERVICE}" ]; then
  echo ""
  IP_ADDR=$(bx cs workers ${PIPELINE_KUBERNETES_CLUSTER_NAME} | grep normal | head -n 1 | awk '{ print $2 }')
  PORT=$( kubectl get services --namespace ${CLUSTER_NAMESPACE} | grep ${APP_SERVICE} | sed 's/.*:\([0-9]*\).*/\1/g' )
  echo ""
  echo -e "VIEW THE APPLICATION AT: http://${IP_ADDR}:${PORT}"
fi

Deploy to K8S

#!/bin/bash
# uncomment to debug the script
#set -x

echo "IMAGE_NAME=${IMAGE_NAME}"
echo "IMAGE_TAG=${IMAGE_TAG}"
echo "REGISTRY_URL=${REGISTRY_URL}"
echo "REGISTRY_NAMESPACE=${REGISTRY_NAMESPACE}"
echo "DEPLOYMENT_FILE=${DEPLOYMENT_FILE}"

IMAGE_TAG=${TIMESTAMP}

# Input env variables from pipeline job
echo "PIPELINE_KUBERNETES_CLUSTER_NAME=${PIPELINE_KUBERNETES_CLUSTER_NAME}"
if [ -z "${CLUSTER_NAMESPACE}" ]; then CLUSTER_NAMESPACE=default ; fi
echo "CLUSTER_NAMESPACE=${CLUSTER_NAMESPACE}"

echo "=========================================================="
echo "DEPLOYING using manifest"
echo -e "Updating ${DEPLOYMENT_FILE} with image name: ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}"
if [ -z "${DEPLOYMENT_FILE}" ]; then DEPLOYMENT_FILE=deployment.yml ; fi
if [ -f ${DEPLOYMENT_FILE} ]; then
    sed -i "s~^\([[:blank:]]*\)image:.*$~\1image: ${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}:${IMAGE_TAG}~" ${DEPLOYMENT_FILE}
    cat ${DEPLOYMENT_FILE}
else 
    echo -e "${red}Kubernetes deployment file '${DEPLOYMENT_FILE}' not found${no_color}"
    exit 1
fi    
set -x
kubectl apply --namespace ${CLUSTER_NAMESPACE} -f ${DEPLOYMENT_FILE} 
set +x

echo ""
echo "=========================================================="
IMAGE_REPOSITORY=${REGISTRY_URL}/${REGISTRY_NAMESPACE}/${IMAGE_NAME}
echo -e "CHECKING deployment status of ${IMAGE_REPOSITORY}:${IMAGE_TAG}"
echo ""
for ITERATION in {1..30}
do
  DATA=$( kubectl get pods --namespace ${CLUSTER_NAMESPACE} -o json )
  NOT_READY=$( echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | select(.ready==false) ' )
  if [[ -z "$NOT_READY" ]]; then
    echo -e "All pods are ready:"
    echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | select(.ready==true) '
    break # deployment succeeded
  fi
  REASON=$(echo $DATA | jq '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | .state.waiting.reason')
  echo -e "${ITERATION} : Deployment still pending..."
  echo -e "NOT_READY:${NOT_READY}"
  echo -e "REASON: ${REASON}"
  if [[ ${REASON} == *ErrImagePull* ]] || [[ ${REASON} == *ImagePullBackOff* ]]; then
    echo "Detected ErrImagePull or ImagePullBackOff failure. "
    echo "Please check proper authenticating to from cluster to image registry (e.g. image pull secret)"
    break; # no need to wait longer, error is fatal
  elif [[ ${REASON} == *CrashLoopBackOff* ]]; then
    echo "Detected CrashLoopBackOff failure. "
    echo "Application is unable to start, check the application startup logs"
    break; # no need to wait longer, error is fatal
  fi
  sleep 5
done

APP_NAME=$(kubectl get pods --namespace ${CLUSTER_NAMESPACE} -o json | jq -r '.items[].status | select(.containerStatuses!=null) | .containerStatuses[] | select(.image=="'"${IMAGE_REPOSITORY}:${IMAGE_TAG}"'") | .name' | head -n 1)
echo -e "APP: ${APP_NAME}"
echo "DEPLOYED PODS:"
kubectl describe pods --selector app=${APP_NAME} --namespace ${CLUSTER_NAMESPACE}
if [ ! -z "${APP_NAME}" ]; then
  APP_SERVICE=$(kubectl get services --namespace ${CLUSTER_NAMESPACE} -o json | jq -r ' .items[] | select (.spec.selector.app=="'"${APP_NAME}"'") | .metadata.name ')
  echo -e "SERVICE: ${APP_SERVICE}"
  echo "DEPLOYED SERVICES:"
  kubectl describe services ${APP_SERVICE} --namespace ${CLUSTER_NAMESPACE}
fi
#echo "Application Logs"
#kubectl logs --selector app=${APP_NAME} --namespace ${CLUSTER_NAMESPACE}  
echo ""
if [[ ! -z "$NOT_READY" ]]; then
  echo ""
  echo "=========================================================="
  echo "DEPLOYMENT FAILED"
  exit 1
fi

echo ""
echo "=========================================================="
echo "DEPLOYMENT SUCCEEDED"
if [ ! -z "${APP_SERVICE}" ]; then
  echo ""
  IP_ADDR=$(bx cs workers ${PIPELINE_KUBERNETES_CLUSTER_NAME} | grep normal | head -n 1 | awk '{ print $2 }')
  PORT=$( kubectl get services --namespace ${CLUSTER_NAMESPACE} | grep ${APP_SERVICE} | sed 's/.*:\([0-9]*\).*/\1/g' )
  echo ""
  echo -e "VIEW THE APPLICATION AT: http://${IP_ADDR}:${PORT}"
fi

Problems

⚠ Cluster kube always on "Pending"

After creating k8s cluster , the latter is still always in 'Pending' status

✅ Resolution : Update Kubernetes cluster to last proposed version

⚠ Can't publish docker image on registry

On the BUILD step of the Toolchain FAILED , because of the name of the image

The supplied image name was invalid.
Correct the image name, and try again.

✅ add a TIMESTAMP to docker image name . Replace the ${TAG_NAME} with ${TIMESTAMP}

That's all , thank you for reading and your feedback is highly appreciated

[ 🇬🇧 ] What Kubernetes is and what it is not

bzinoun — Wed, 05 Jun 2019 23:28:12 +0000

Two questions were very important to me when my interest was on Kubernetes :

What is Kubernetes ?
What Kubernetes is not ?

By answering to these two questions, i was sure to start from the right point and also know the responsibility of Kubernetes .

So let's make a pragmatic responses about that .

💡What is Kubernetes ?

Kubernetes is a platform that wrap a huge number of services and capabilities ; these latter is growing day after day .
From a top level we can say , the core functionalities of Kubernetes is it's ability to manage containers across linux based infrastructure .
But Kubernetes brings also a lot of feature such as :

Managing Secrets
Checking applications Health
Scaling application : Scale app and down
Auto Scaling
Instance replication
Discovering resource
Naming Resource
Monitoring resource
Managing and mounting Storage system
Providing security component : Authorization , Authentication

In one sentence the responsibility of K8S is

Platform for managing containerized workloads and services, that facilitates both declarative configuration and automation.

⛔️ What Kubenetes is not ?

The Common mistake is to say Kubernetes is a Paas !

Kubernetes is not a platform as a service (Paas) , in my opinion Rancher is a Paas for Kubernetes , It provide a high level configuration for kubernetes and provide some default configuration for Kubernetes to .

In fact Kubernetes doesn't dictate many aspects of our system and leaves them up to us to manage them .

In facts Kubernetes does :

not require a specific application type or a dependency framework
not require a specific programming language
not provide any database
not provide any message queues
allow us to choose our logging system
allow us to choose our monitoring & alerting system
..etc

That's all Folks

In the next post we will discuss the relation between K8S and containers . We will find response for questions such as :

What's benefits of containers
How we need to manage our containers
...etc

[ 🇬🇧 ] IBM AppId integration with Springboot 2

bzinoun — Tue, 28 May 2019 17:16:50 +0000

What we gonna do

through this article we will focus on how to integrate the OpenId Connect Server, AppID, proposed by IBM Cloud with a Java application under Springboot.

Prerequisites

An IBM Cloud Account: The free version is sufficient
Knowledge about OAuth2 flow
Knowledge knowledge of the Java ecosystem - Springboot.

IBM AppId in few lines

App ID is the OpenId Connect service from IbmCloud . It helps developers to easily add authentication to their web or mobile apps with few lines of code, and it helps them secure their cloud-native applications and services on IBM Cloud. AppID propose a connection trought Facebook Google etc ...

App ID also helps manage user-specific data that developers can use to build personalized app experiences.

/!\ We assume that you know how to create a basic Java Spring Boot App

Okay , let's show some code !

Configure AppId with Spring Boot

Add the dependencies to pom.xml

       <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-security</artifactId>
        </dependency>
        <!-- spring-security-oauth2-autoconfigure tells the framework to use OAuth2, instead of basic http security -->
        <dependency>
            <groupId>org.springframework.security.oauth.boot</groupId>
            <artifactId>spring-security-oauth2-autoconfigure</artifactId>
            <version>2.0.0.RELEASE</version>
        </dependency>

Create a ResourceSecurityConfiguration Java class to enable OAuth2, and Rest capabilities. Also, extend from ResourceServerConfigurerAdapter class, to later configure security access.

@Configuration
@RestController
@EnableResourceServer
public class ResourceSecurityConfiguration extends ResourceServerConfigurerAdapter {

}

Override security configuration by adding a configure method.

@Configuration
@RestController
@EnableResourceServer
public class ResourceSecurityConfiguration extends ResourceServerConfigurerAdapter {
    @Override
    public void configure(HttpSecurity http) throws Exception {
        http.authorizeRequests()
        .antMatchers("/api**", "/user", "/userInfo").authenticated()
        .and().logout().logoutSuccessUrl("/").permitAll()
        .and().csrf().disable();
    }
}

Add two endpoints to the ResourceSecurityConfiguration Java class that will be accessible when you perform GET Calls from the front end. The /user endpoint gives us the logged-in user object (principal) and the /userInfo endpoint returns a string with the details.

@Configuration
@RestController
@EnableResourceServer
public class ResourceSecurityConfiguration extends ResourceServerConfigurerAdapter {
    @Override
    public void configure(HttpSecurity http) throws Exception {
        http.authorizeRequests()
        .antMatchers("/api**", "/user", "/userInfo").authenticated()
        .and().logout().logoutSuccessUrl("/").permitAll()
        .and().csrf().disable();
    }
    // for open id
    @RequestMapping("/user")
    public Principal user(Principal principal) {
        //Principal holds the logged in user information.
        // Spring automatically populates this principal object after login.
        return principal;
    }

    // for open id
    @RequestMapping("/userInfo")
    public String userInfo(Principal principal){
        final OAuth2Authentication oAuth2Authentication = (OAuth2Authentication) principal;
        final Authentication authentication = oAuth2Authentication.getUserAuthentication();
        //Manually getting the details from the authentication, and returning them as String.
        return authentication.getDetails().toString();
    }

}

Add necessary AppId config values from information that you'll get from your service credentials / IBM Cloud account to application.properties to complete your configuration you can find these information in: Ibm cloud account -> services -> AppId -> Applications -> Add Application (if not done before) -> View Credentials

spring.main.allow-bean-definition-overriding=true

env.urlAppid= // OAuth server URL from IBM AppID account

# SECURITY OAUTH2 CLIENT (OAuth2ClientProperties)

security.oauth2.client.accessTokenUri=${env.urlAppid}/token
security.oauth2.client.userAuthorizationUri=${env.urlAppid}/authorization


# SECURITY OAUTH2 RESOURCES (ResourceServerProperties)
security.oauth2.resource.userInfoUri=${env.urlAppid}/userinfo


## Setting to change in case of changing app id dev instance
security.oauth2.client.client-id= // IBM AppId account client-id
security.oauth2.client.client-secret= // IBM AppId secret

Finally to Sign In and Signup you to make an HTTP Request to these endpoints:
Sign In: https://appid-oauth.eu-gb.bluemix.net/oauth/v3/{TenantId}/token

Request: {
username: '',
password: '',
grant_type: 'password' // don't change
}
RequestHeader:{
Authorization: 'Basic ClientId:Secret'
}

Response:{
access_token:'',
}

Signup: https://appid-management.eu-gb.bluemix.net/management/v4/{TenantId}/cloud_directory/sign_up

Request:{
name: '',
username: '',
password: '',
active: true, // don't change
emails: [
{
value: 'email@example.com',
primary: true
}
]
}

RequestHeader:{
   Authorization: 'Bearer {IAM_TOKEN}'
}

Response:{
HttpStatus: 200
}

That's all folks !

[ 🇬🇧 ] Gitlab CI to Build and Push containers to registry

bzinoun — Tue, 28 May 2019 16:31:26 +0000

We all know that Gitlab CI build uses docker image to do the job, But have you ever tried to build a docker image inside gitlab CI build ?

As we know gitlab CI start on docker container. So when we want to build a docker image inside gitlab CI build, it's docker in docker (DinD)

Without transition lets take a look at the .gitla-ci.yml file :

    image: docker:latest
    variables:
     DOCKER_DRIVER: overlay2
    stages:
      - build
      - push
    services:
      - docker:dind
    before_script:
      # docker login needs the password to be passed through stdin for security
      # we use $CI_JOB_TOKEN here which is a special token provided by GitLab
      - echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
      - docker version
      - docker info

    after_script:
      - docker logout registry.gitlab.com
    Build:
      stage: build
      script:
        - docker pull $CI_REGISTRY_IMAGE:latest || true
        - >
          docker build
          --pull
          --cache-from $CI_REGISTRY_IMAGE:latest
          --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
          .
        - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    Push_When_tag:
      stage: push
      only:
        # We want this job to be ran on tags only.
        - tags
      script:
        - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
        - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
        - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

Step 1 - Images & services

    image: docker:latest
    variables:
     DOCKER_DRIVER: overlay2
    stages:
      - build
      - push
    services:
      - docker:dind

yaml

We start by defining the docker image that will be used by GitlabCI build. In our case and as example we used the latest docker image .

image: docker:latest

⚠️ In production for example , I don't recommend using latestorstable versions. For many reasons ...
One of them is reproducibility, Another reason is we want our pipeline to work in 10 month or 10 years. If a new feature is needed , then an upgrade is planned .

    variables:
     DOCKER_DRIVER: overlay2

yaml
When using docker:dind , Docker uses the vfs storage driver which copies the filesystem on every run. This is a very disk-intensive operation which can be avoided if a different driver is used, for example overlay2

Step 2 - before and after Script

    before_script:
      # docker login needs the password to be passed through stdin for security
      # we use $CI_JOB_TOKEN here which is a special token provided by GitLab
      - echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
      - docker version
      - docker info

    after_script:
      - docker logout registry.gitlab.com

yaml
Nothing special on this step :

Connection to Gitlab Registry
Check docker daemon and config
Logout from docker registry

Step 3 - Build and Push

    Build:
      stage: build
      script:
        - docker pull $CI_REGISTRY_IMAGE:latest || true

        # notice the cache-from, which is going to use the image we just pulled locally
        # the built image is tagged locally with the commit SHA, and then pushed to 
        # the GitLab registry
        - >
          docker build
          --pull
          --cache-from $CI_REGISTRY_IMAGE:latest
          --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
          .
        - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

We pull the last pushed image on the registry ; the `|| true` assure that the pipeline will not fail if no image was found . 

After pulling the last image , this one will be used for the cache when building a new image using the `--cache-from` . 

Then we push to registry with the image flagged with `$CI_COMMIT_SHA` that contains the commit SHA .

yaml

Step 4 - Tag management

    Push_When_tag:
      stage: push
      only:
        - tags
      script:
        - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
        - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
        - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME

yaml
We want to keep our Git tags in sync with our Docker tags. That helps a lot when debugging and trying to reproduce specific version bugs .

If you have not automated this, you probably have found yourself in the situation of wondering “which git tag is this image again?”.

⚠️ This stage is triggered only when a tag is created .