Forem: Mert Simsek

You Aren’t Gonna Need It You Aint Gonna Need It (YAGNI) Principle

Mert Simsek — Mon, 28 Mar 2022 11:43:47 +0000

It tells that some features should NOT be developed with the thought that they will be needed in the future. The most common reason for developers to add features that they won't use at the moment is that they think that it will be easy to develop that feature at that time. The developer thinks: "This will be asked of me one day tomorrow. Then I don't need to change what I wrote." However, this feature is a supposing category as of the time of building and may never be requested from you. At this point, some may argue that this is planning and that such improvements can be made as a result of good analysis. However, real-life experience is far from this, and developers have to bear the cost of developing hypothetical features they predicted incorrectly. The analysis, programming, and testing time spent on this feature is wasted.

https://www.microbloggers.in/2020/08/yagni-principle-of-software-development.html

Let's say we add a concurrency system to our project. We added concurrency for all I/O processes. If it isn't a critical requirement at this point, YAGNI says "DO NOT". So why?

You won't need to do this and you'll have wasted your time, which will not only complicate your code even more. Imagine that we guessed right and we would really need that feature. Even in this case, this supposing has serious costs for us. First, we delay some of the things we need to do today while we are doing this feature that we think will be needed in the future. So this supposing brought us latency costs. Considering that only one-fourth of the features developed according to the research is really useful, delaying an already necessary feature on assumptions is an action that needs to be reconsidered.

In the case we guessed correctly, the delay cost is not the only cost. Transport cost is also one of the most important problems. We will have to move the code for this feature, which we will use in the future, in our project. With all changes, we need to change that part of the code as well. This code snippet, which we have never used at the moment, may cause errors. During debugging, we need to debug that part as well. In summary, we pay the cost of each line of code added to the project for the lines added for this feature. However, if we had added this feature to the project later, we would not have had to bear this cost during this period.

The issue of when the YAGNI principle applies is somewhat relative, but it can be said briefly as follows: As the size of the feature we assume, the importance of the YAGNI principal increases. If a few-hour job is to be done and this will cover a few days' developments in the future, it may not be a problem, but the longer this period, the higher your risks.

Solve Today's Problem, Not Tomorrow's

These features we've developed will require additional testing effort, documentation, and subsequent maintenance considerations. When we say let's make a tricky software with additional features, the possibility of developing a more complex and more complex software increases. Therefore, it is always beneficial to avoid developments that are not specified as needed. If we doubt some additionals, we can ask ourselves the following questions.

Is it 100% sure that the implementation is needed?
What are the investment costs?
How long does the implementation take?
What are the opportunity costs?

Often such questions cannot be answered suddenly. One of the problems with this is that statements about the future are mainly difficult and cannot be easily quantified. This is another reason why it makes sense to coordinate with colleagues and make a good decision together. In agile developments, for example, Sprint Planning 2 can be used.

To Sum Up

YAGNI is a principle that says not to include things we don't need in the system. It is quite similar to KISS from certain points. During the development phase, we sometimes act predictably(?) and write classes and methods that we think may be needed in the future. This is due to both the foresight that we may need in the future (?) and the fact that we want to see our development bigger.

Yagni
https://t2informatik.de

Go Plugin Development on Kong

Mert Simsek — Thu, 20 Feb 2020 19:29:56 +0000

Earlier this year, Kong has announced brilliant features. You check it out the following.

https://konghq.com/blog/kong-gateway-2-0-0-released/

I was very interested in one of them which is we will be able to develop Go plugins on Kong. I built a plugin to test and as soon as possible I'm writing this blog post eventually. I'm going to demonstrate a plugin that checks the consumer key from query string.

I used to run DB-less Kong on Docker and I prefer that either.

FROM kong/go-plugin-tool:2.0.4-alpine-latest AS builder

RUN mkdir -p /tmp/key-checker/

COPY . /tmp/key-checker/

RUN cd /tmp/key-checker/ && \
    go get github.com/Kong/go-pdk && \
    go mod init kong-go-plugin && \
    go get -d -v github.com/Kong/go-pluginserver && \
    go build github.com/Kong/go-pluginserver && \
    go build -buildmode plugin key-checker.go

FROM kong:2.0.4-alpine

RUN mkdir /tmp/go-plugins
COPY --from=builder  /tmp/key-checker/go-pluginserver /usr/local/bin/go-pluginserver
COPY --from=builder  /tmp/key-checker/key-checker.so /tmp/go-plugins
COPY config.yml /tmp/config.yml

USER root
RUN chmod -R 777 /tmp
RUN /usr/local/bin/go-pluginserver -version && \
    cd /tmp/go-plugins && \
    /usr/local/bin/go-pluginserver -dump-plugin-info key-checker
USER kong

In this way, we copy our required files. Let's go into details. Firstly, we create a builder image and I copy my key-checker.go file and I build it as a go plugin. After that, I get that file from the builder image and I put it in Kong image. That's straightforward.

Let's look over the Go file.

package main

import (
    "github.com/Kong/go-pdk"
)

// it represents to config parameters into the config.yml
type Config struct {
    Apikey string
}

func New() interface{} {
    return &Config{}
}

func (conf Config) Access(kong *pdk.PDK) {
    key, err := kong.Request.GetQueryArg("key")
    apiKey := conf.Apikey 

    if err != nil {
        kong.Log.Err(err.Error())
    }

    //it adjusts the header parameters in this way.
    x := make(map[string][]string)
    x["Content-Type"] = append(x["Content-Type"], "application/json")

    //If the key of the consumer is not equal to the claimed key, kong doesn't ensure the proxy
    if apiKey != key {
        kong.Response.Exit(403, "You have no correct consumer key.", x) 
    }
}

This file tells us that we check the key from the query string. If it is not equal to the key from the config file, we don't provide proxy service. In another case, it will ensure the proxy well and clearly the claimed URL from the config file. This is the config file. Also, I try to show changing header parameters and I write content type as a parameter for only showing.

# go.yml
_format_version: "1.1"
services:
- url: https://reqres.in/api/users?page=2
  routes:
  - paths:
    - "/"
  plugins:
  - name: key-checker
    config:
      apikey: mysecretconsumerkey

To build the Dockerfile I run this command.

docker build -t kong-demo .

Once it built, I run the following command. I point the related directories by using environment variables.

docker run -ti --rm --name kong-go-plugins \
  -e "KONG_DATABASE=off" \
  -e "KONG_GO_PLUGINS_DIR=/tmp/go-plugins" \
  -e "KONG_DECLARATIVE_CONFIG=/tmp/config.yml" \
  -e "KONG_PLUGINS=key-checker" \
  -e "KONG_PROXY_LISTEN=0.0.0.0:8000" \
  -p 8000:8000 \
  kong-demo

Right, we're able to check the service. As you can see, the key is not equal. Therefore the status code is 403 and proxy service is not working.

When I enter the correct consumer key, as expected Kong ensures the proxy service clearly and URL is requested.

Conclusion

This feature drastically will be useful for Go developers and you could build pretty cool plugins in this way. I also want to share some practical URLs like the following. I hope the blog post ignites the minds to build awesome Go plugins on Kong. Happy Konging!

https://docs.konghq.com/2.0.x/go

https://godoc.org/github.com/Kong/go-pdk

https://github.com/gszr/kong-2.0-demo

Symfony on Google Cloud Run

Mert Simsek — Mon, 11 Nov 2019 16:26:18 +0000

Nowadays, two of the biggest trends in application development have been the rise of serverless and containerization. This year, Google announced a new product called Cloud Run that is designed to bring the two together. At the same time, Google also announced Cloud Run for GKE which is specifically designed to run on Google’s version of Kubernetes. Likewise, I develop some of my applications and I refer them to survive on Cloud Run. I assume you have a Symfony 4 application as well. Let's start.

First of all, you need to read the container runtime contract page. There are some standards to follow the ways for Cloud Run.

https://cloud.google.com/run/docs/reference/container-contract

The crucial thing is that your application is supposed to listen 8080 port for now. It's the beta and in the future, this situation might be changed. Therefore, I've created an Nginx conf to listen to 8080.

server {
    listen       8080;
    server_name  localhost;
    root  /usr/share/nginx/html/public;

    location / {
        try_files $uri /index.php$is_args$args;
    }

    location ~ ^/index\.php(/|$) {
        fastcgi_pass unix:/run/php-fpm7.2/php-fpm.sock;
        #fastcgi_pass 127.0.0.1:9000;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;

        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        fastcgi_param DOCUMENT_ROOT $realpath_root;
        internal;
    }

    location ~ \.php$ {
        return 404;
    }

    error_log /var/log/nginx/project_error.log;
    access_log /var/log/nginx/project_access.log;
}

Also, I've created a Docker file to handle container issues.

FROM phpearth/php:7.2-nginx

COPY . /usr/share/nginx/html

COPY default.conf /etc/nginx/conf.d/default.conf

COPY php-fpm.conf /etc/php/7.2/php-fpm.d/www.conf

COPY php.ini /etc/php.ini

RUN apk add --no-cache composer

WORKDIR /usr/share/nginx/html

RUN mkdir -p /run/nginx && chown -R nginx:nginx /run/nginx && chown -R nginx:nginx /usr/share/nginx/html \
           && composer install -vvv \
           && mkdir -p /usr/share/nginx/html/var && chmod -R 777 /usr/share/nginx/html/var \
           && chown -R nginx:nginx /usr/share/nginx/html/var

I've created my own php.ini file.

[PHP]
engine = On
short_open_tag = On
precision = 14
output_buffering = Off
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 100
disable_functions = dl
disable_classes =
zend.enable_gc = On
expose_php = Off
max_execution_time = 30
max_input_time = 60
memory_limit = 512M
error_reporting = E_ALL & ~E_NOTICE
display_errors = On
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
error_log = error_l
variables_order = "EGPCS"
request_order = "GP"
register_argc_argv = On
auto_globals_jit = On
post_max_size = 64M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
include_path = ".:/opt/php72/lib/php"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 64M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = On
default_socket_timeout
cli_server.color
date.timezone = Asia/Riyadh
pdo_mysql.cache_size = 2000
pdo_mysql.default_soc
sendmail_path = /usr/sbin/sendmail -t
mail.add_x_header
sql.safe_mode =
odbc.allow_persistent = Off
odbc.check_persistent = Off
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M"
mysqli.max_persistent = -1
mysqli.allow_persistent = Off
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect =
mysqlnd.collect_statistics = Off
mysqlnd.collect_memory_statistics =
pgsql.allow_persistent = Off
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice
bcmath.scale
session.save_handler = files
session.save_path = "/tmp"
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 100
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=,fields"
zend.assertions
tidy.clean_output =
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit
ldap.max_links

And, we need to create a php-fpm.conf to figure out the PHP workers.

[www]

user = www-data
group = www-data

listen = /run/php-fpm7.2/php-fpm.sock

listen.owner = nginx
listen.group = nginx
listen.mode = 0660

pm = static
pm.start_servers = 1
pm.max_children = 5
pm.min_spare_servers = 1
pm.max_spare_servers = 5

You are going to have this structure in the end.

At this point, we need to create a Docker image into Google Cloud in the Container Registry service. Therefore, we run the following command.

gcloud builds submit --tag gcr.io/[Google-Cloud-Project]/[Image-Name]:[Version]

After creating the image, we will able to use that image like this.

gcloud beta run deploy [Service-Name] --image gcr.io/[Google-Cloud-Project]/[Image-Name]:[Version] --region europe-west1 --platform managed

By running this command, we push our earlier image to create a container on Cloud Run. When you look over the Google Cloud dashboard, you are going to reach all of the details about your service.

As you can recall, we have learned to automate this process the following post.

https://dev.to/_mertsimsek/continuously-build-test-and-deploy-with-symfony-on-google-cloud-5a6i

You also change like this the cloudbuild.yaml file to get autonomous for Cloud Run.

steps:
  # build the container image
  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'gcr.io/$PROJECT_ID/[Service-Name]', '.']
    # push the container image to Container Registry
  - name: 'gcr.io/cloud-builders/docker'
    args: ['push', 'gcr.io/$PROJECT_ID/[Service-Name]']
    # Deploy container image to Cloud Run
  - name: 'gcr.io/cloud-builders/gcloud'
    args: ['beta', 'run', 'deploy', '[Service-Name]', '--image', 'gcr.io/$PROJECT_ID/[Service-Name]', '--region', 'europe-west1', '--platform', 'managed']
images:
  - gcr.io/$PROJECT_ID/[Service-Name]

Conclusion

Google wants us to focus only on our codes. Each Cloud Run service has a unique endpoint and auto-scales deployed containers. Therefore, we don't need to suffer about the rest of the code. I hope, I succeed to change your ideas about to survive on Google Cloud platform.

Falcon API Framework on Docker - 2

Mert Simsek — Tue, 10 Sep 2019 13:46:34 +0000

In this article, we will need to build an API by using the Falcon API Framework in Python. We're going to cover some hints and methods to request. This article is part of a series:

1) Bootstrapping Falcon API Framework on Docker

2) Creating an API by using individual Falcon methods. (You're here.)

You had better read the previous article or you will be confused about that. Right, let's start.

My root directory is the following. First of all, I've created the requirements.txtfile.

attrs==19.1.0
falcon==2.0.0
falcon-autocrud==1.0.36
gunicorn==19.9.0
jsonschema==3.0.1
marshmallow==2.19.5
psycopg2==2.8.3
pyrsistent==0.15.4
python-dateutil==2.8.0
six==1.12.0
SQLAlchemy==1.3.6
webargs==5.4.0

It's ready to fetch the libraries by this command.

pip install -r requirements.txt

After arranging the libraries, we can create the Python files. My main file is app.py. In spite of this, you can define another name.

import falcon
from resources import athlete
from resources import plan
from resources import exercise
from services import database_service

from middlewares import (
    ContentEncodingMiddleware,
)

conn = database_service.connect()

api = falcon.API(middleware=[
    ContentEncodingMiddleware(),
])

# api = falcon.API()
athlete = athlete.Athlete(conn, database_service)
plan = plan.Plan(conn, database_service)
exercise = exercise.Exercise(conn, database_service)

api.add_route('/athletes/{id}', athlete)
api.add_route('/athletes', athlete, suffix='collection')
api.add_route('/plans/{id}', plan)
api.add_route('/plans', plan, suffix='collection')
api.add_route('/exercises/{id}', exercise)
api.add_route('/exercises', exercise, suffix='collection')

This file provides Falcon app to bootstrap. I added my routes and related files into this file. Now, as you see, there's resourcedirectory. This directory ships the my objecys. I've 3 objects to ensure as the resources. First one is plan.py.


import falcon
from webargs import fields
from webargs.falconparser import use_args


class Plan(object):
    post_request_args = {"name": fields.Str(required=True), "description": fields.Str(required=True),
                         "difficulty": fields.Int(required=True), "athlete_id": fields.Int(required=True)}

    def __init__(self, conn, database_service):
        self.conn, self.database_service, self.resource_name = conn, database_service, self.__class__.__name__

    def on_delete(self, req, resp, id):

        try:
            q = " ".join(
                ["DELETE", "FROM", self.resource_name.lower(), "WHERE", self.resource_name.lower() + "_id = %s"])
            q_resp = self.database_service.run_delete_query(self.conn, q, [id])
            if not q_resp['status']:
                output = {"status": True, "message": q_resp['message'],
                          "data": None}
            else:
                output = {"status": True, "message": self.resource_name + " was deleted successfully!", "data": None}

            resp.status = falcon.HTTP_200
            resp.body = output
        except Exception as error:
            output = {"status": False, "message": str(error), "data": None}
            resp.status = falcon.HTTP_500
            resp.body = output

    def on_get(self, req, resp, id):
        try:
            cur = self.conn.cursor()

            q = " ".join(
                ["SELECT", "*", "FROM", self.resource_name.lower(), "wHERE", self.resource_name.lower() + "_id = %s"])
            q_resp = self.database_service.run_get_query(cur, q, [id])
            if not q_resp['status']:
                output = {"status": True, "message": q_resp['message'],
                          "data": None}
            else:
                output = {"status": True, "message": None,
                          'data': self.database_service.set_columns(q_resp['data'], cur)}

            resp.status = falcon.HTTP_200
            resp.body = output
        except Exception as error:
            output = {"status": False, "message": str(error), "data": None}
            resp.status = falcon.HTTP_500
            resp.body = output

    def on_get_collection(self, req, resp):
        try:
            cur = self.conn.cursor()
            q = " ".join(
                ["SELECT * FROM", self.resource_name.lower()])
            q_resp = self.database_service.run_get_query(cur, q, [])
            if not q_resp['status']:
                output = {"status": True, "message": q_resp['message'],
                          "data": None}
            else:
                output = {"status": True, "message": None,
                          'data': self.database_service.set_columns(q_resp['data'], cur)}

            resp.status = falcon.HTTP_200
            resp.body = output
        except Exception as error:
            output = {"status": False, "message": str(error), "data": None}
            resp.status = falcon.HTTP_500
            resp.body = output

    def on_put(self, req, resp, id):
        try:

            cur = self.conn.cursor()
            # q = "SELECT name, description, difficulty FROM " + self.resource_name.lower() + " WHERE " + self.resource_name.lower() + "_id = %s;"
            get_q = " ".join(
                ["SELECT name,description,difficulty FROM", self.resource_name.lower(), "wHERE",
                 self.resource_name.lower() + "_id = %s"])
            get_resp = self.database_service.run_get_query(cur, get_q, [id])

            record = list(self.database_service.set_columns(get_resp['data'], cur))[0]

            request = req.media
            for index in record.keys():
                if index in request.keys():
                    record[index] = request[index]
            record['id'] = id

            update_q = " ".join(
                ["UPDATE", self.resource_name.lower(), "SET name=%s, description=%s, difficulty=%s WHERE",
                 self.resource_name.lower() + "_id=%s RETURNING ", self.resource_name.lower() + "_id;"])

            update_resp = self.database_service.run_upsert_query(self.conn, update_q, record.values())
            if not update_resp['status']:
                output = {"status": True, "message": update_resp['message'],
                          "data": None}
            else:
                response_data = {
                    "id": update_resp['data'],
                    "name": record['name'],
                    "description": record['description'],
                    "difficulty": record['difficulty']
                }

                output = {"status": True, "message": self.resource_name + " is updated successfully!",
                          "data": response_data}

            resp.status = falcon.HTTP_201
            resp.body = output

        except Exception as error:
            output = {"status": False, "message": str(error), "data": None}
            resp.status = falcon.HTTP_500
            resp.body = output

    @use_args(post_request_args)
    def on_post_collection(self, req, resp, args):
        try:
            # q = "INSERT INTO " + self.resource_name.lower() + " (name, description, difficulty, athlete_id) VALUES (%s,%s,%s,%s) RETURNING " + self.resource_name.lower() + "_id;"
            q = " ".join(
                ["INSERT INTO", self.resource_name.lower(),
                 "(name, description, difficulty, athlete_id) VALUES (%s,%s,%s,%s) RETURNING",
                 self.resource_name.lower() + "_id;"])

            params = {'name': args['name'], 'description': args['description'], 'difficulty': args['difficulty'],
                      'athlete_id': args['athlete_id']}

            q_resp = self.database_service.run_upsert_query(self.conn, q, params.values())

            if not q_resp['status']:
                output = {"status": True, "message": q_resp['message'],
                          "data": None}
            else:
                response_data = {
                    "id": q_resp['data'],
                    "name": args['name'],
                    "description": args['description'],
                    "difficulty": args['difficulty'],
                    "athlete_id": args['athlete_id']
                }

                output = {"status": True, "message": self.resource_name + " is added successfully!",
                          "data": response_data}

            resp.status = falcon.HTTP_201
            resp.body = output
        except Exception as error:
            output = {"status": False, "message": str(error), "data": None}
            resp.status = falcon.HTTP_500
            resp.body = output

I'm going to share only this file as a resource. Because other files are exact same according to code structure.

You'll see the prefix in front of the methods called on_and HTTP verbs come after this prefix. We understand, each method represents the HTTP verbs.

According to this, we'll have these endpoints;

1-) http://localhost:8000/plans (GET, POST, PUT, DELETE)

Perfection! It's straightforward to bootstrap. We can pass the configuration step. Therefore, I've created a conf directory. It ships two files. config.ini and gunicorn_conf.py. The ini file is like this.

[postgresqlDB]
host = postgresql
db = test_dev
user = mertingen
pass = mertingen

That's configuration file for web service.

import multiprocessing

bind = '0.0.0.0:8000'
workers = multiprocessing.cpu_count() * 2 + 1
timeout = 30
worker_connections = 1000

I've also servicesdirectory to handle some useful processes. That's the create_schema_service.py

import psycopg2
import database_service

conn = database_service.connect()


def create_tables():
    """ create tables in the PostgreSQL database"""
    commands = (
        """
        CREATE TABLE IF NOT EXISTS athlete (
            athlete_id SERIAL PRIMARY KEY,
            name VARCHAR(255) NOT NULL,
            email VARCHAR(255) NOT NULL,
            phone VARCHAR(255) NOT NULL,
            gender VARCHAR(255) NOT NULL,
            birthday DATE NOT NULL
        )
        """,
        """ CREATE TABLE IF NOT EXISTS plan (
                   plan_id SERIAL PRIMARY KEY,
                   athlete_id INTEGER NOT NULL,
                   name VARCHAR(255) NOT NULL,
                   description TEXT NOT NULL,
                   difficulty VARCHAR(255) NOT NULL,
                        FOREIGN KEY (athlete_id)
                            REFERENCES athlete (athlete_id)
                            ON UPDATE CASCADE ON DELETE CASCADE
               )
        """,
        """
                CREATE TABLE IF NOT EXISTS exercise (
                    exercise_id SERIAL PRIMARY KEY,
                    name VARCHAR(255) NOT NULL,
                    description TEXT NOT NULL
                )
                """
    )

    try:
        cur = conn.cursor()
        for c in commands:
            cur.execute(c)
            print("Table was created successfully!")
        cur.close()
        conn.commit()
    except (Exception, psycopg2.DatabaseError) as error:
        print(error)
    finally:
        if conn is not None:
            conn.close()


if __name__ == '__main__':
    create_tables()

And, database_service.py

import psycopg2 as pg
import configparser as cp
import os

dir_path = os.path.dirname(os.path.realpath(__file__))
c = cp.ConfigParser()

c.read(dir_path + '/../conf/config.ini')


def connect():
    try:
        connection = pg.connect(user=c['postgresqlDB']['user'],
                                password=c['postgresqlDB']['pass'],
                                host=c['postgresqlDB']['host'],
                                port="5432",
                                database=c['postgresqlDB']['db'])
        print("You are connected!")
        return connection
    except (Exception, pg.Error) as error:
        print("Error while connecting to PostgreSQL", error)

    # finally:
    #    if connection:
    #        connection.close()
    #        print("PostgreSQL connection is closed")


def set_columns(data, cur):
    items = []
    if data:
        for x in data:
            item = {}
            c = 0
            for col in cur.description:
                item.update({col[0]: x[c]})
                c = c + 1
            items.append(item)
        return items
    else:
        return []


def run_get_query(cur, query, params):
    try:
        if params:
            cur.execute(query, tuple(params))
        else:
            cur.execute(query)
        records = cur.fetchall()
        return {"status": True, "message": "", "data": records}
    except pg.InternalError as e:
        return {"status": False, "message": str(e), "data": None}


def run_upsert_query(conn, q, params):
    try:
        cur = conn.cursor()
        cur.execute(q, tuple(params))
        conn.commit()

        id = cur.fetchone()[0]
        return {"status": True, "message": "", "data": id}
    except pg.InternalError as e:
        conn.rollback()
        return {"status": False, "message": str(e), "data": None}


def run_delete_query(conn, q, params):
    try:
        cur = conn.cursor()
        cur.execute(q, tuple(params))
        conn.commit()
        return {"status": True, "message": "", "data": None}
    except pg.InternalError as e:
        conn.rollback()
        return {"status": False, "message": str(e), "data": None}

Lastly, I'd rather provide my resources as a JSON format. For this, I've created middleware in the root directory for Falcon.

import json
from datetime import date, datetime

class JSONEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, bytes):
            return obj.decode("utf-8")
        if isinstance(obj, (date, datetime)):
            return str(obj.isoformat())
        return super(JSONEncoder, self).default(obj)


class ContentEncodingMiddleware(object):
    def process_response(self, req, resp, _, req_succeeded):
        if not req_succeeded:
            return
        if req.client_accepts_json:
            resp.set_header('Content-Type', 'application/json')
            resp.body = json.dumps(resp.body, cls=JSONEncoder)

Let's create a plan by Postman.

That's it!

You'll able to find the codes and related repository on this link.

https://github.com/mertingen/python-falcon-framework-api

Falcon framework builds pretty-cool APIs. I hope, this article will be useful for you and see you in the next article. If you have any trouble, don't hesitate to ask.

Falcon API Framework on Docker

Mert Simsek — Thu, 01 Aug 2019 14:04:24 +0000

We're going to adjust the Falcon API Framework on Docker. In this way, we'll able to build our APIs. There are 2 kinds of frameworks, full-featured and bare frameworks. If you would rather develop a RESTFul API for your projects with Python, in general, you consider structure the following.

1) Flask with Flask-RESTFul

2) Django + REST Framework

On the other hand, there is a very good light-weight API framework available in Python called Falcon.

Accordingly, Falcon claims (I agree with them), other frameworks weigh you down with tons of dependencies and unnecessary abstractions. Falcon cuts to the chase with a clean design that embraces HTTP and the REST architectural style.

Friendly Reminder
In this article, we're going to see the adjustment and installation of the Falcon on Docker. My next post will demonstrate to build APIs with Falcon.

Well, let's start. Initially, we're supposed to modify the directory structure. It should be like this.

Firstly, I'm going to share docker-compose.yml file into the root directory.

version: '2'
services:
  backend:
    build: ./backend
    ports:
      - "9000:8000"
    volumes:
      - ./app/.:/app
    tty: true
    depends_on:
      - postgresql

  postgresql:
    image: postgres:alpine
    environment:
      POSTGRES_DB: test_dev
      POSTGRES_USER: mertingen
      POSTGRES_PASSWORD: mertingen
    ports:
      - '5432:5432'

As you can see, as a database solution, I refer to the Postgresql. Once you build this file, it will be created a database called test_dev and a user mertingen called. Also, our application exposes 9000 port to serve externally.

In the database directory, I've one file. Let's look over the Dockerfile.

FROM postgres:10.1-alpine

For now, I've been just pulling an image. Therefore, we'll able to connect the database and we'll refer those environment variables in the docker-compose.yml file. We're going to run this file then. In the backend directory, I've 2 files Dockerfile and start.sh. Let's look over them.

FROM python:3.6.2

COPY start.sh /scripts/start.sh
WORKDIR /app
ENTRYPOINT ["/scripts/start.sh"]

And, here is start.sh.

#!/bin/bash

pip install --upgrade pip
pip install -r /app/requirements.txt

gunicorn app:api -c /app/conf/gunicorn_conf.py --reload

You could find requirements.txt here.

attrs==19.1.0
falcon==2.0.0
falcon-autocrud==1.0.36
gunicorn==19.9.0
jsonschema==3.0.1
marshmallow==2.19.5
psycopg2==2.8.3
pyrsistent==0.15.4
python-dateutil==2.8.0
six==1.12.0
SQLAlchemy==1.3.6
webargs==5.4.0

start.sh file installs dependencies and creates a server to run the application. To serve the application, I create 2 configuration files, config.ini and gunicorn_conf.py in the conf directory. This is config.ini file.

[postgresqlDB]
host = postgresql
db = test_dev
user = mertingen
pass = mertingen

To connect the database, we'll use this file. The parameters will be environment variables in my next article. This file helps to connect to the database.

import psycopg2
import configparser
import os

dir_path = os.path.dirname(os.path.realpath(__file__))
config = configparser.ConfigParser()

config.read(dir_path + '/../conf/config.ini')


def connect():
    try:
        connection = psycopg2.connect(user=config['postgresqlDB']['user'],
                                      password=config['postgresqlDB']['pass'],
                                      host=config['postgresqlDB']['host'],
                                      port="5432",
                                      database=config['postgresqlDB']['db'])
        print("You are connected!")
        return connection
    except (Exception, psycopg2.Error) as error:
        print("Error while connecting to PostgreSQL", error)
    # finally:
    #    if connection:
    #        connection.close()
    #        print("PostgreSQL connection is closed")

Now, I share the configuration of the gunicorn server.

import multiprocessing

bind = '0.0.0.0:8000'
workers = multiprocessing.cpu_count() * 2 + 1
timeout = 30
worker_connections = 1000

We barely need to create a Falcon application. For this, let's create app.py and put this code.

import falcon
from resources import user
from services import database

#from middlewares import (
#    ContentEncodingMiddleware,
#)

conn = database.connect()

#api = falcon.API(middleware=[
#    ContentEncodingMiddleware(),
#])

api = falcon.API()
user = user.User(conn)
api.add_route('/users/{id}', user)
api.add_route('/users', user, suffix='collection')

Simply, to see a response from this API, I create a user resource file in the resources directory. It's like this.

import falcon

    def __init__(self, conn):
        self.conn = conn

    @staticmethod
    def set_columns(data, cursor):
        items = []
        for x in data:
            item = {}
            c = 0
            for col in cursor.description:
                item.update({col[0]: x[c]})
                c = c + 1
            items.append(item)
        return items

    def on_get_collection(self, req, resp):
        try:
            cursor = self.conn.cursor()

            query = """SELECT * FROM Users"""
            cursor.execute(query)
            records = cursor.fetchall()
            cursor.close()

            output = {"status": True, "message": None, 'data': self.set_columns(records, cursor)}

            resp.status = falcon.HTTP_200
            resp.body = output
        except Exception as error:
            output = {"status": False, "message": str(error), "data": None}
            resp.status = falcon.HTTP_500
            resp.media = output

In fact, I don't mention codes, as I said we're going to build an API in the next article. I've inserted some fake user data and the application returns the data.

From now on, we just need to run Docker Compose services with this command in the root directory.

docker-compose up --build

As a result, you will get an output like this.

After this point, as long as you request the http://localhost:9000/users URL, you will able to get this result. As I said, I played with fake data.

To Sum Up

Thanks all of you for reading till this point. Docker helps to build your development workspace and we should exploit its features in this way. Likewise, the Falcon framework builds pretty-cool APIs. I hope, this article will be useful for you and see you in the next article. If you have any trouble, don't hesitate to ask.

Naming and Creating API Endpoints

Mert Simsek — Sat, 20 Jul 2019 14:14:50 +0000

In this article, we're going to cover some standards and recommendations to create API endpoints. I hope, it will be very helpful for you and it will be helping to change your perspective about creating API endpoints, of course in a positive way.

First of all, we need to think about everything your API should handle somethings. Initially, that would be CRUD (create, read, update, delete) operations for your resources. If we had a customer endpoint, it would be like the following obviously.

Customer

Create
Read
Update
Delete
List
Image

The customer will also need to have an image uploaded via the API. In this instance, we will allow one image for a customer and a new image overrides the previous.

API Endpoint Approach

Let's talk about some theories. There're are no right answers at this point but some approaches have some cons than others. As we are developers, we have found to be the most beautiful and point the pros and cons of each.

GET Entities

GET /customers - List of customers, in some logical simply, for that specific list.
GET /customers/x - Just entity X. X might be an ID, hash, slug, username as long as it's unique to one resource.
GET /customers/x,y,z - The consumer demands multiple resources.

Friendly Reminder About Auto-Increment
In the above examples, X might be an auto-incrementing ID as many developers will admit. The significant thing is that anyone with access to your API will know exactly how many entities you have. This situation can be information for people.Someone could also write a script which hits /customers/1, then /customers/2 and /customers/3, scraping all entities in this way.

You could use a unique identifier. A universally unique identifier seems like a logical thing to do. You can find some libraries about it on the internet.

DELETE Entities

If you would rather delete entities? It's the easiest.

DELETE /customers/x - Delete a single customer.
DELETE /customers/x,y,z - Delete a bunch of customers.
DELETE /customers/x/image - Delete the image of customer.

POST vs PUT

What about creating and updating? I know lots of people who try to pair the POST and PUT to specific CRUD action is uncertain. In general speaking, PUT is used if you know the entire URL beforehand. For example, create could be a PUT if you're creating one image for a customer.

PUT /customers/1/image HTTP/1.1
Host: mertblog.net
Content-Type: image/jpeg

It would be a proper instance of when to use a PUT because. You already know the entire URL and you are able to do it time and time again. If we have multiple images for a customer, POSTing could be a good solution.

POST /customers/1/images

It's a tricky difference, but don't try and tie an HTTP method to one CRUD action only.

Plural, Singular or Both?

You can decide to make all endpoints singular, but there is an issue with this. Confusing. I would rather create a plural for everything as it is the most obvious:

/customers - If I run a GET on this, I will get a collection of customers.
/customers/7 - Obviously, I'm just interested in customer 45.
/customers/7,14 - customers 7, 14 and I got it, thank you.

Another solid reason for using the plural harmonious is that it accepts for harmonious named subentities:

/customers
/customers/7
/customers/7
/customers/7/venues
/customers/7/venues/2
/venues/2

Harmonious is key.

Verb or Noun?

Most of the time, APIs consist of a series of endpoints that described actions:

POST /sendUserMessage
id=5&message=hello

As you can imagine, this is not how things are done with REST. Furthermore, developers figure out the following approach to be more RESTful because it uses a subentity:

POST /users/5/send-message
{"message":"Hello"}

But unfortunately, this is still using a verb in the URL. A verb is an action - a doing term and our API only needs one verb - the HTTP Method. All other verbs need to stay out of the URL.

A noun is a place or a thing. Entities are things and a URL becomes the address on the Internet where a thing lives.

This instance would be drastically more RESTful.

POST /users/5/messages
{"message":"Hello"}

Tremendous. We have been creating a new message that belongs to a user. The best part about keeping it nice and RESTful like this is that other HTTP actions can be made to the identical URL:

GET /users/mert/messages
PATCH /users/mert/message/xdMRqwerty
DELETE /users/mert/message/xdMRqwerty

This is all much easier to document and much easier to understand for both humans and software which is RESTful.

Organizing Endpoints

Controllers

We are supposed to list customers, venues, users or categories? Straightforward. One controller for each type of entity:

CategoriesController
EventsController
UsersController
VenuesController

All of those should be an entity and each entity needs a controller. Subentities might sometimes just be a method. For instance, settings are a subentity of users. They can go in UsersController. But this is optional and flexible.

Routes

At this point, you should want to avoid having multiple endpoints doing painfully similar things with copy and paste logic. Your consumers will be mad that different endpoints provide the same entities but in a slightly different format.

Methods

When you have listed all of the routes you will demand your application, go and make suitable controller methods. Make them all empty and have one of them return "Hi there!" and check the output. You just wrote an API.

Conclusion

When you have a relatively extensive list the first step is to make a basic list of actions. This is planning an entity class. I tried to bring in that thinking all verbs should stay out of the URL. Another idea is using the plural. That provides the harmonious all of the entities. I hope, I could change your perspective about building APIs.

Continuously Build, Test, and Deploy with Symfony on Google Cloud

Mert Simsek — Tue, 16 Apr 2019 21:24:17 +0000

Hi buddies, I'm going to talk about Symfony build, test and deploy on Google Cloud. With this way, we will able to get full automation each process. We'll have less tolerance regarding any human mistake. In order to provide this, we're supposed to install gcloud. You could follow this document, it's piece of cake. After that, you should just run all of these commands.

gcloud init
gcloud components install app-engine-php
gcloud auth login
gcloud config set project [GC-PROJECT-ID]

From now on, our gcloud sdk is authenticated to our Google Cloud Project. You can also follow Google's documents, if you have any trouble.

Well, let's create a Symfony project with Composer. Run these commands to run a Symfony application.

composer create-project symfony/skeleton mertblog -vvv && cd mertblog
composer require dotenv annotations template log google/cloud -vvv
composer require web-server-bundle maker tests --dev -vvv
bin/console make:controller

The content of the controller is as following. I changed only route.

<?php

namespace App\Controller;

use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\Routing\Annotation\Route;

class DefaultController extends AbstractController
{
    /**
     * @Route("/", name="default")
     */
    public function index()
    {
        return $this->render('default/index.html.twig', [
            'controller_name' => 'DefaultController',
        ]);
    }
}

That's it! We've all libraries and files which we will need further. In order to deploy into Google Cloud App Engine, I create the app.yaml in root directory of the project.

runtime: php
env: flex

env_variables:
  APP_ENV: 'prod'
  DOCUMENT_ROOT: '/app/public'

Shortly after, I run this command to deploy.

gcloud app deploy

This command automatically reads this file and takes the actions. It may take a little long time. Whereas, everything is fine and going well, don't worry about it.
Once it's finished, run this, you're going to see our Default Controller response on App Engine production.

gcloud app browse

Perfect! From now on, I am able to create a GitHub repository to automation and push Symfony application into this repository. As you see, my repository is ready.

Right now, we have to integrate with GitHub and Google Cloud Build. For this, you can go to this page.

https://github.com/marketplace/google-cloud-build

You'll encounter this and click the Set up a plan button and go ahead. It will demand to a repository to run. I've selected my repository which I've created a moment ago.

With this way, Google Cloud Build and GitHub will integrated run smoothly. Now, I create a Dockerfile and Docker ignore file to create a container in root directory of the project. I'll use this container for testing my application.

FROM gcr.io/google-appengine/php

Here is .dockerfile.

.gcloudignore
.git
.gitignore
vendor/
var/
Dockerfile
.dockerignore

No more, we don't get any manual process. To automate my processes, I create a cloudbuild.yaml file in root directory of the project. There are 3 sections in this file. First of all, Google Cloud Build is building our application and testing according to our configuration. If these steps are succeed, it deploys to App Engine.

steps:
  # Build
  - name: 'gcr.io/cloud-builders/docker'
    args: [ 'build', '--build-arg', 'COMPOSER_FLAGS=--dev', '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA', '.' ]
  - name: 'gcr.io/cloud-builders/docker'
    args: [ 'push', 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA' ]

  # Test
  - name: 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA'
    args: [ 'bin/phpunit' ]
    dir: /app
    env:
      - 'APP_ENV=test'

  # Deploy
  - name: 'gcr.io/cloud-builders/gcloud'
    args: ['beta', 'app', 'deploy', '--quiet', '--image-url', 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA', '--version', '$BRANCH_NAME', '--no-promote']

But without deploying, we need to do somethings. For example, let's create a test class.

bin/console make:functional-test IndexControllerTest

Here you are.

<?php

namespace App\Tests;

use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;

class IndexControllerTest extends WebTestCase
{
    public function testSomething()
    {
        $client = static::createClient();
        $crawler = $client->request('GET', '/');

        $this->assertSame(200, $client->getResponse()->getStatusCode());
        $this->assertContains('Hello Mertblog', $crawler->filter('h1')->text());
    }
}

Nice! We check out the HTTP status code and the title of the page. We cannot pass this test because, our title is not Hello Mertblog. I won't change, because I would like to see failed build. Well, we have to grant permissions to Google Cloud Build for App Engine. As you see, I've set these permissions to Google Cloud Build.

In addition to this, you had better, not forget to add new files which we've just added on Git. I've pushed to GitHub my new files and it will be automatically builded on Google Cloud. If you go to Google Cloud Build page, you'll see it is failed because of our test. The HTTP response code is 200 but the title page is not as I demanded. It couldn't pass. By the way, deploying didn't run. Because, in order to deploy, other steps should succeed.

We can look over the details.

You can also check on GitHub.

Let's fix somethings such as our title on the page. I changed the title as Hello Mertblog and I push my new codes GitHub.

From now on, every step is perfect. In order they succeeded regarding cases.

Now, As long as you go to App Engine you see 2 versions to run. And master version is our main version which is deploying from Github by Google Cloud Build. We have to change traffic, If we need to see changing on our repository. You can stop old version and click Split Traffic button on this page. And give 100% traffic. Like this.

Let's check our homepage.As you see, my last codes were deployed obviously, the title was changed.

In addition to this, I am going to check on GitHub. I could see that there is no any errors or failures stuffs. Well done!

To Sum Up

If we gather, there is a big benefit and opportunity to use Google Cloud Build, Google Cloud App Engine and Symfony together. All processes are able to be automatic with this way and we can be away from mistakes. As long as if you have any trouble or problem, don't hesitate to stay in touch.

Handling Cookie From Guzzle FileCookieJar

Mert Simsek — Tue, 09 Apr 2019 11:24:59 +0000

Today, in this article, I would like to mention about cookie concept of the Guzzle. I've looked over on internet about this issue but I couldn't even find clear solution on the documentation. Well, my scenario is like this, I would like to authenticate a system by using username and password. In addition to this, I want Guzzle to store cookie data into a json file. With this way, I refer this file to request for further queries.

First of all, I would like to show my auth method to request.



    /**
     * @param Client $client
     * @return array
     * @throws GuzzleException
     */
    public function auth(Client $client): array
    {
        try {
            $apiUrl = getenv('API_URL');
            $username = getenv('USERNAME');
            $password = getenv('PASSWORD');

            $postData = array('email' => $username, 'password' => $password, 'keep_logged_in' => true);

            $response = $client->request('POST', $apiUrl . '/rest/authenticate', [
                'json' => $postData,
                'cookies' => new FileCookieJar('/Users/mertsimsek/Development/app/assets/cookies.json')
            ]);
            $results = \GuzzleHttp\json_decode($response->getBody()->getContents(), true);

            return ($results['success']) ? array('status' => true, 'message' => 'Authentication has set successfully.') : array('success' => false, 'message' => 'Authentication wasn\'t able to set.');
        } catch (RequestException $e) {
            return
                array(
                    'status' => false,
                    'message' => $e->getMessage()
                );
        }
    }

By FileCookieJar I am able to store cookie data into a file as JSON format. Thus, I will read this file to further requests. I've created a new method to read this file.



    /**
     * @return CookieJar
     */
    public function getCookieJarFromFile(): CookieJar
    {
        $cookieData = json_decode(file_get_contents('/Users/mertsimsek/Development/app/assets/cookies.json'));
        foreach ($cookieData as $cookie) {
            //If there are multiple cookie data, you could filter according to your case
            $cookie = json_decode(json_encode($cookie), TRUE);
            $setCookie = new SetCookie($cookie);
        }

        $jar = new CookieJar();
        $jar->setCookie($setCookie);

        return $jar;
    }

Well, from now on, I've a method to get cookie data from that file. Now, we could create a method to request new queries.As long as I request authpath, I am going to get this result and the json file will be created by Guzzle.

Right, we're able to request with this file to get data from APIs clearly. As you see, I get the cookie data from that file and I request by using Guzzle library.



    /**
     * @param Client $client
     * @return array
     * @throws GuzzleException
     */
    public function requestToApi(Client $client): array
    {
        $apiUrl = getenv('API_URL');
        $postData = array();

        try {
            $jar = $this->getCookieJarFromFile();

            $response = $client->request('GET', $apiUrl . '/rest/getUsers', [
                'json' => $postData,
                'cookies' => $jar
            ]);
            $results = \GuzzleHttp\json_decode($response->getBody()->getContents(), true);

            return ($results['success']) ? array('status' => true, 'message' => 'Authentication has set successfully.') : array('success' => false, 'message' => 'Authentication wasn\'t able to set.');
        } catch (RequestException $e) {
            return
                array(
                    'status' => false,
                    'message' => $e->getMessage()
                );
        }
    }

Hopefully, this article truly helps you to use FileCookieJar. If you have a trouble, don't hesitate to stay in touch.

Using Xdebug with Docker

Mert Simsek — Sun, 07 Apr 2019 22:08:27 +0000

I am going to show using Xdebug with Docker. Because, as long as you prefer using Xdebug on development workspace with Docker, you have to create remote connection of the Xdebug. First of all, you must have a Xdebug client. I would rather use PhpStorm for this. With this way, I am going to mention some hints and ways to use smoothly. Let's start!

I would like to start a Symfony project which is running on Docker. I will tell this independent from Symfony. Through Docker Compose helps me about it. My structure is like as the following.

Well, I share my Docker assets. Here you are. This is php.ini file.

[PHP]
engine = On
short_open_tag = On
precision = 14
output_buffering = Off
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 100
disable_functions = dl
disable_classes =
zend.enable_gc = On
expose_php = Off
max_execution_time = 30
max_input_time = 60
memory_limit = 256M
error_reporting = E_ALL & ~E_NOTICE
display_errors = On
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
error_log = error_l
variables_order = "EGPCS"
request_order = "GP"
register_argc_argv = On
auto_globals_jit = On
post_max_size = 64M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
include_path = ".:/opt/php72/lib/php"
doc_root =
user_dir =
enable_dl = Off
file_uploads = On
upload_max_filesize = 64M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = On
default_socket_timeout 
cli_server.color 
date.timezone = Asia/Riyadh
pdo_mysql.cache_size = 2000
pdo_mysql.default_soc
sendmail_path = /usr/sbin/sendmail -t 
mail.add_x_header 
sql.safe_mode =
odbc.allow_persistent = Off
odbc.check_persistent = Off
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M"
mysqli.max_persistent = -1
mysqli.allow_persistent = Off
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect =
mysqlnd.collect_statistics = Off
mysqlnd.collect_memory_statistics =
pgsql.allow_persistent = Off
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice
bcmath.scale
session.save_handler = files
session.save_path = "/tmp"
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 100
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=,fields"
zend.assertions 
tidy.clean_output =
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit
ldap.max_links

Second one is nginx.conf.

server {
    listen 80;
    server_name web;
    root /var/www/app/public;

    location / {
        # try to serve file directly, fallback to index.php
        try_files $uri /index.php$is_args$args;
    }

    location ~ ^/index\.php(/|$) {
        #fastcgi_pass unix:/var/run/php7.2-fpm.sock;
        fastcgi_pass php:9000;
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        fastcgi_param DOCUMENT_ROOT $realpath_root;
        fastcgi_buffer_size 128k;
        fastcgi_buffers 4 256k;
        fastcgi_busy_buffers_size 256k;
        internal;
    }

    location ~ \.php$ {
        return 404;
    }

    error_log /var/log/nginx/project_error.log;
    access_log /var/log/nginx/project_access.log;
}

And third one is the Dockerfile. The important points are on here.

FROM php:7.2-fpm-alpine

RUN apk update \
    && apk add  --no-cache git mysql-client curl libmcrypt libmcrypt-dev openssh-client icu-dev \
    libxml2-dev freetype-dev libpng-dev libjpeg-turbo-dev g++ make autoconf \
    && docker-php-source extract \
    && pecl install xdebug redis \
    && docker-php-ext-enable xdebug redis \
    && docker-php-source delete \
    && docker-php-ext-install pdo_mysql soap intl zip \
    && echo "xdebug.remote_enable=on" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
    && echo "xdebug.remote_autostart=off" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
    && echo "xdebug.remote_port=9001" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
    && echo "xdebug.remote_handler=dbgp" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
    && echo "xdebug.remote_connect_back=0" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
    && echo "xdebug.idekey=mertblog.net" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
    && echo "xdebug.remote_host=docker.for.mac.localhost" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini \
    && curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer \
    && rm -rf /tmp/*

CMD ["php-fpm", "-F"]

WORKDIR /var/www/app

EXPOSE 9000

If you haven't got Mac OS, you're supposed to change xdebug.remote_host as your machine IP. In addition to this, as you see, my idekey is mertblog.net and my remote_port is 9001. You are able to change it. Finally, let me show docker-compose.yml file.

version: '3'
services:
    php:
        build:
            context: ./php-fpm/.
        volumes:
            - ./app:/var/www/app
            - ./php-fpm/php.ini:/usr/local/etc/php/php.ini
        depends_on:
            - mysql
    web:
        image: nginx:latest
        ports:
            - "8888:80"
        volumes:
            - ./app:/var/www/app
            - ./nginx/app.conf:/etc/nginx/conf.d/default.conf
        depends_on:
            - php
    mysql:
        image: mysql:5.7
        environment:
            MYSQL_ROOT_PASSWORD: symf0ny
        ports:
            - "3333:3306"

I installed the Symfony application which I've refered this document. I started my web application by using this command.

docker-compose up --build

From now on, I could configure the Xdebug client. For this, follow this path from the interface. Phpstorm > Preferences > PHP > Debug. You are going to see Xdebug Debug Port. You should change as 9001 port. After that, follow this path Phpstorm > Preferences > PHP > Servers. You are going to see this screen.

I've set the name docker-server. My host is localhost and my web server post is 8888. Click the Use path mappings. You have to define the Absolute path on the server as /var/www/app. It links to our project directory on the local machine. Well, after this, you close this screen and click the Edit Configurations section on the left side.

On this section, we are going to create a configuration under to PHP Remote Debug. I've set the name as docker. My server is docker-server which I've created just before. And Ide Key is same thing in the Dockerfile.

I clicked the phone icon to start listening PHP Debug Connections.

Right, I can start my Xdebug session by getting this url. http://localhost:8888/home?XDEBUG_SESSION_START=mertblog.net

I've put red point on the 19th line. From now on, I am able to debug my variables on PHPStorm interface obviously. It truly helps us and with this way we could notice mistakes easily.

Automating Git Events with Github Webhook

Mert Simsek — Thu, 21 Mar 2019 16:42:58 +0000

Webhooks allow you to build or set up integrations which subscribe to certain actions on GitHub.com. When one of those events is triggered, Github sends a HTTP POST payload to the webhook's configured URL. Webhooks can be used to update an external issue tracker, trigger CI builds, update a backup mirror, or even deploy to your production server. You're only limited by your imagination. If you would like to configure URL or events, you are supposed to run this url the following.

https://github.com/[owner]/[repository]/settings/hooks.

You'll see this page on that and click the Add webhook button to configure our new webhook.

After the typing your password, from now on you are able to configure first webhook. It's truly so straightforward.

At this point, we're supposed to type a payload url. It should be the file which welcomes the payload from Github. And secret sections helps to authorization. After a webhook creation, Github sends a request with this secret. If the secret is same with our response from that file, Github figures out that. To allow that, I've created a PHP file github_webhook.php like this.

<?php

echo 'thisismertblogsecret';

If everything is smooth, you will have been seen Webhook list on your Github setting page. From now on, all push events will have been sent your payload url. To handle the request, I've changed the file.

#echo 'thisismertblogsecret';

if ($_POST && $_POST['payload']){
    $payload = json_decode($_POST['payload'], true);
    if (isset($payload['ref']) && $payload['ref'] == 'refs/heads/master'){
        exec("/usr/bin/git pull origin master >> ./webhook.log 2>&1 && echo $(date) >> ./webhook.log");
    }
}

Well, we've simple a file. It handles the requests from Github events. If the request to push, it runs the pull command. And it sends the output a file webhook.log named. Let's make a test.I am going to send push command and it will be running pull command.

As you see, the output was typed into our log file. In addition, pull process was succeed.

Common Errors

1-) error: cannot open .git/FETCH_HEAD: Permission denied

Just run this commands the following.

chown root:nginx .git/FETCH_HEAD

chmod 775 .git/FETCH_HEAD

2-) Permission denied (publickey). fatal: Could not read from remote repository.

This is easy as well. Imagine that, github_webhook.php is run by Nginx user. Not root user. So, every user has a key in Ssh connections. You are supposed to create a key for Nginx user.

sudo -u nginx ssh-keygen -t rsa

You could see the keys with this path, /var/lib/nginx/.ssh/.

Right, then you have to add this key into deploy keys of the repository, https://github.com/[owner]/[repository]/settings/keys. From now on, you will able to get rid of that.

Well, my talking is over about that. In some cases, Github Webhook is truly helpful for developers. If you have any questions or trouble don't hesitate to type.

Serializer Component with Symfony

Mert Simsek — Sun, 24 Feb 2019 23:31:49 +0000

This post leads to use Serialization in our projects. But What's the serialization and who needs that? We will answer these needful questions. They are help us to understand that obviously. Well, let's start.

What's the serialization?

For developers; It's transforming an object to a formatted string. We would have textual, JSON, XML. On the other hand, binary formats. Generally, we develop REST Apis commonly. Whereas, different programming languages and different machines should stay in touch each other by our REST Apis endpoints. At this point, serialization truly works for us.

Who needs serialization?

As a developer, we are able to refer this way. Initially, we send or receive data when we develop a REST Api. It should be strict format to manage all of communication. The most common use case is queue systems. We figure out queue systems with serialization. Other thing might be testing. We are able to detect some changes. Plus, If we want to store data in database instead of having columns with fields. We can have only one field for all data. If I need to give some instances, they might be like Redis, Elasticsearch.

How it works?

Here is the architecture of how it works.

-- Normalizers --
First of all, there is an object format. Normalizers are obviously classes that to convert objects to an array. Array structure is easily understand and manipulate.

-- Encoders --
These classes are convert from array to the claimed format. Or, claimed formatted to array.

Well, from now on we are able to start writing some codes. I am going to give some tips about that. For this, I installed the component.

composer require symfony/serializer -vvv

After that, I added my configuration lines in framework.yamlfile.

framework:
    serializer:
        enabled: true
        enable_annotations: true

I've adjusted the configuration. Now, I wrote my first code.

<?php

namespace App\Controller;

use App\Entity\User;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Component\Serializer\SerializerInterface;

class UserController extends AbstractController
{
    /**
     * @Route("/user", name="user")
     * @param SerializerInterface $serializer
     * @return JsonResponse
     */
    public function index(SerializerInterface $serializer)
    {

        $user = new User();
        $user->setFullname('Mert Simsek')
            ->setCity('İstanbul')
            ->setEmail('mertsmsk0@gmail.com')
            ->setPhone('+90-000-000-00-00')
            ->setCreatedAt(new \DateTime('2019-02-25 00:00:00'));

        $serializedUser = $serializer->serialize($user,'json');

        return JsonResponse::fromJsonString($serializedUser);

    }
}

I implement the serializer interface and I convert the claimed format. The response is going to be like this.

At the moment, I would like to highlight some features. First of all is groups. Picture that, you've all of these columns; fullname, phone, city etc. But you don't want to provide all of them. At this point, "groups" helps us. How? Like this. You write "groups" line each propery of the entity as below.

<?php

namespace App\Entity;

use Doctrine\ORM\Mapping as ORM;
use Symfony\Component\Serializer\Annotation\Groups;

/**
 * @ORM\Entity(repositoryClass="App\Repository\UserRepository")
 */
class User
{
    /**
     * @ORM\Id()
     * @ORM\GeneratedValue()
     * @ORM\Column(type="integer")
     * @Groups({"phone","web"})
     */
    private $id;

    /**
     * @ORM\Column(type="string", length=255)
     * @Groups({"phone","web"})
     */
    private $fullname;

    /**
     * @ORM\Column(type="string", length=255)
     * @Groups({"phone","web"})
     */
    private $city;

    /**
     * @ORM\Column(type="string", length=255)
     * @Groups({"phone"})
     */
    private $phone;

    /**
     * @ORM\Column(type="string", length=255)
     * @Groups({"phone","web"})
     */
    private $email;

    /**
     * @ORM\Column(type="datetime")
     * @Groups({"phone","web"})
     */
    private $createdAt;

    // at this point, there are getters and setters...

}

Now, we can serialize with this "groups" defining. As you see, If I serialize with "web" group, the phone property is not going to appear us. Because, the phone property is appearing for only "phone group". Let's see.

$user = new User();
$user->setFullname('Mert Simsek')
    ->setCity('İstanbul')
    ->setEmail('mertsmsk0@gmail.com')
    ->setPhone('+90-000-000-00-00')
    ->setCreatedAt(new \DateTime('2019-02-25 00:00:00'));

$serializedUser = $serializer->serialize($user,'json', ['groups' => ['web']]);

return JsonResponse::fromJsonString($serializedUser);

How my response appears? Here you are.

As you see, anymore we cannot see the phone property. Now let's check it out the other formats. For instance, xml? I changed the format from json to xml.

$user = new User();
$user->setFullname('Mert Simsek')
    ->setCity('İstanbul')
    ->setEmail('mertsmsk0@gmail.com')
    ->setPhone('+90-000-000-00-00')
    ->setCreatedAt(new \DateTime('2019-02-25 00:00:00'));

$serializedUser = $serializer->serialize($user,'xml', ['groups' => ['web']]);

return JsonResponse::fromJsonString($serializedUser);

Immediately, we make sure the result.

Or csv... For this thime, of course, I converted to xml to csv.

I would like to mention about other one feature. It's Name Convertor. If we need to change from camelCase to snake_case, we just add this line into configuration.

framework:
    serializer:
        enabled: true
        enable_annotations: true
        name_converter: serializer.name_converter.camel_case_to_snake_case

From now on, the created at property is going to be snake_case formatted.

To Sum Up

I would like to thank all of you for reading till this point. Serializer Component has plenty of feature. Specially, when you develop the REST Apis, it truly helps you.

An Introduction to the Kubernetes

Mert Simsek — Tue, 25 Dec 2018 22:51:46 +0000

In this article, my goal is to get purpose of Kubernetes, to understand philosophy. If i need to tell a little bit, The all ways lead the dependencies. Imagine that, you've a software project. All classes are depend on each other. At this point, Dependency Injection might be solution well. Let's look from out of the box a little bit. Let's get a Php project. We need to have other libraries, packages which have written by other programmers. Yes, you know the answer. Composer is the best solution for us. Before the saying, "perfect!" in fact, we still have a dozen dependencies. The services of operation systems, they are able to be Nginx, Mysql, Rabbitmq, Redis etc. It's not finish. Directly, we're up to the operation system. In addition, network? The solution is for like these things, Docker. But, again, there are a few confusing problems to manage plenty of containers. For this, fortunately, we've met container orchestration like Kubernetes.

I will give a brief introduction to the Kubernetes and demonstrate which problems that can be solution.

Basically, Kubernetes is container orchestration. Container orchestration is ensures to manage our containers. We are able to run containers on a single computer to develop or test applications. Yet, when we run production, our priorities vary.

Is it durable for errors?
Scalability.
Using the machine resources as little as possible.
Are accessible from the external.
Can we update/rollback without any downtime.

Container orchestrators are the systems which help us for all of these. There are many container orchestrators available.

Docker Swarm
It is a container orchestrator provided by Docker, Inc. It is part of Docker Engine.
Kubernetes
Kubernetes was started by Google, but now, it is a part of the Cloud Native Computing Foundation project.
Mesos Marathon
Marathon is one of the frameworks to run containers at scale on Apache Mesos.
Amazon ECS
Amazon EC2 Container Service (ECS) is a hosted service provided by AWS to run Docker containers at scale on its infrastructrue.
Hashicorp Nomad
Nomad is the container orchestrator provided by HashiCorp.

From now on, let's talk about Kubernetes a litte bit. According to the Kubernetes website. By the way, Kubernetes is also referred to as k8s, as there are 8 characters between k and s.

"Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications."

Kubernetes offers a very rich set of features for container orchestration. Some of its fully supported features. Kubernetes has written with Go programming language.

Automatic binpacking
Kubernetes automatically schedules the containers based on resource usage and constraints, without sacrificing the availability.
Self-healing
Kubernetes automatically replaces and reschedules the containers from failed nodes. It also kills and restarts the containers which do not respond to health checks, based on existing rules/policy.
Horizontal scaling
Kubernetes can automatically scale applications based on resource usage like CPU and memory. In some cases, it also supports dynamic scaling based on customer metrics.
Service discovery and Load balancing
Kubernetes groups sets of containers and refers to them via a Domain Name System (DNS). This DNS is also called a Kubernetes service. Kubernetes can discover these services automatically, and load-balance requests between containers of a given service.
Secrets and configuration management
Kubernetes can manage secrets and configuration details for an application without re-building the respective images. With secrets, we can share confidential information to our application without exposing it to the stack configuration, like on GitHub.

Before we dive into Kubernetes, we need to stay intouch with Cloud Native Computing Foundation.

The Cloud Native Computing Foundation (CNCF) is one of the projects hosted by The Linux Foundation. CNCF aim to accelerate the adoption of containers, microservices, and cloud-native applications.CNCF provides resources to each of the projects, but, at the same time, each project continues to operate independently under its pre-existing governance structure and with its existing maintainers.

We will see the Kubernetes architecture, the different components of the master and worker nodes, the cluster state management with etcd and the network setup requirements. We will also talk about the network, which is used by Kubernetes.

Master Node

The master node is responsible for managing the Kubernetes cluster, and it is the entry point for all administrative tasks. We can communicate to the master node via the CLI, the GUI (Dashboard), or via APIs.

Worker Node

A worker node is a machine (VM, physical server, etc.) which runs the applications using Pods and is controlled by the master node. Pods are scheduled on the worker nodes, which have the necessary tools to run and connect them. A Pod is the scheduling unit in Kubernetes.

To Conclusion

We will dive deeper into this in future chapters. Kubernetes might be the best solution for you, you are supposed to measure your target basically. Further, we'll discuss Master Node and Worker Node. Thus, after that, we are going to encounter topics which confusing.