Forem: Ahmed Ashraf

JAVA SpringBoot request filters

Ahmed Ashraf — Sun, 16 May 2021 19:50:14 +0000

Intro

Filter provides a convenient mechanism for inspecting and filtering HTTP requests entering your application. For example, you want to inject a custom header to a request/response based on some conditions, or you want to run some checks before accessing the controller and serve the request.

Bad

Sometimes we see people implement those kinds of checks and assertions in the controller. yes, it works but it's not the best place to put it there.

Good

Use Filters which it runs before accessing the controllers and serve your requests.

Filters flow

Hands On

How to write?

In order to create a request filter. you just need to create a class that implements the Filter interface.

package com.ahmedash95.example.filters;

import javax.servlet.*;

@Component
public class RoomsCreateFilter implements Filter 
{
  @Override
  public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
     // your code goes here.

    // then you need to pass it in the chain. 
    chain.doFilter(request, response);
  }
}

If you are not familiar with the code above. here are some clarifications:

@Component is an annotation that allows us to register the class as a bean in the ApplicationContext
implements Filter interface tells Spring how this class should be used when register via @Component
import javax.servlet.*; imports Filter, ServletRequest, ServletResponse, FilterChain, and ServletException.
chain.doFilter(request, response); at the end is a must. as it tells spring how to continue handling the request. without it the response will be empty as the chain got broken.

Return response on errors

If you validate something and want to stop processing the request and just return the response. you can simply modify the Response object and return it from your Filter class.

@Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
    boolean valid = false; // check for something
    if (!valid) {
        ((HttpServletResponse) response).setStatus(422);
        response.getOutputStream().write("Validation error".getBytes());
        return;
    }


    chain.doFilter(request, response);
}

Examples

Let's write some real-world examples and some different scenarios to make sure you get the full picture.

1.Ensure API key is provided and valid

Let's say you are working on API that provides something. and you want to ensure that all requests to your application will have the API key and the key is valid. of course, you don't want to do the key validation to be in every controller and method.

Let's start by creating our filter first

package com.example.demo;

import java.io.IOException;
import javax.servlet.*;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.springframework.stereotype.Component;

@Component
public class APIKeyValidatorFilter implements Filter {
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
        HttpServletRequest req = (HttpServletRequest) request;
        String key = req.getHeader("X-API-KEY");
        if(key == null) {
            ((HttpServletResponse) response).setStatus(401);
            response.getOutputStream().write("API Key is missing!".getBytes());
            return;
        }

        if(!KeyValidator.valid(key)) {
            ((HttpServletResponse) response).setStatus(403);
            response.getOutputStream().write("API Key is invalid".getBytes());
            return;
        }

        chain.doFilter(request, response);
    }
}

As you see. we basically do some checks. if the key is not provided we show the key missing error message. if is provided but not valid we show The key is an invalid message.

2.Ratelimit some endpoints

Let's say you want to protect the POST /comment endpoint. so your users must not be able to submit more than 2 comments in one minute. Again it can be done in the controller but It's not the best place to do it.

Let's create the Filter class

package com.example.demo;

import javax.servlet.Filter;
import java.io.IOException;
import javax.servlet.*;
import javax.servlet.http.HttpServletResponse;

public class PostCommentRateLimit implements Filter
{
    RateLimiter rateLimiter;

    public PostCommentRateLimit(RateLimiter rateLimiter) {
        this.rateLimiter = rateLimiter;
    }

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
        int userId = 1; // Get User
        boolean valid = rateLimiter.validate(String.format("ratelimit.user.comments:%d", userId), 2);
        if(!valid) {
            ((HttpServletResponse) response).setStatus(429);
            response.getOutputStream().write("Too Many Requests".getBytes());
            return;
        }


        chain.doFilter(request, response);
    }
}

Few things to explain here:

We did not use @Component here. because we want to apply the filter on POST /comment endpoint only. so we will register it ourself next.
We have RateLimiter in constructor. and you don't need to worry about it. use whatever library fits your needs.

As mentioned we did not use @Component because it would apply the filter for all requests. instead we will create another class to register our filters the way we want.

package com.example.demo;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.web.servlet.FilterRegistrationBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class RequestsFilterRegister {

    @Autowired
    RateLimiter rateLimiter;

    @Bean
    public FilterRegistrationBean<PostCommentRateLimit> registerPostCommentsRateLimiter(){
        FilterRegistrationBean<PostCommentRateLimit> registrationBean  = new FilterRegistrationBean<>();

        registrationBean.setFilter(new PostCommentRateLimit(rateLimiter));
        registrationBean.addUrlPatterns("/comment");

        return registrationBean;
    }
}

We created RequestsFilterRegister as @Configuration class
The only method there registerPostCommentsRateLimiter to register the filter as we want.
we used addUrlPatterns to apply the filter only on /comment endpoint.

Now we have one small problem. the filter is applied to any method on /comment either GET or POST. and to fix that we just need to modify the PostCommentRateLimit@doFilter to skip if the method is not post

HttpServletRequest req = (HttpServletRequest) request;
if (!req.getMethod().equals("POST")) {
    chain.doFilter(request, response);
    return;
}

Now the full class is

package com.example.demo;

import javax.servlet.Filter;
import java.io.IOException;
import javax.servlet.*;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class PostCommentRateLimit implements Filter
{
    RateLimiter rateLimiter;

    public PostCommentRateLimit(RateLimiter rateLimiter) {
        this.rateLimiter = rateLimiter;
    }

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
        HttpServletRequest req = (HttpServletRequest) request;
        if (!req.getMethod().equals("POST")) {
            chain.doFilter(request, response);
            return;
        }

        int userId = 1; // Get User
        boolean valid = rateLimiter.validate(String.format("ratelimit.user.comments:%d", userId), 2);
        if(!valid) {
            ((HttpServletResponse) response).setStatus(429);
            response.getOutputStream().write("Too Many Requests".getBytes());
            return;
        }


        chain.doFilter(request, response);
    }
}

Conclution

Now you know What is Request Filters, how to write them, and you applied some real-world examples. also you saw how to customize the routes and methods to apply filters on.

The full source code https://github.com/ahmedash95/java-spring-request-filters

For more, you can check baeldung.com/spring-boot-add-filter

Note

I'm still learning Java and spring. so if you see any mistakes or a better way in writing Filters Your comments and suggestions will be greatly appreciated.

Design Patterns in PHP: Facade (with Laravel)

Ahmed Ashraf — Tue, 03 Mar 2020 15:23:19 +0000

Design patterns are very important for every developer. it solves very common problems in every project you build.

Facade pattern definition:

Facade helps you to hide any complex implementation of a class to get an instance of it

Wikipedia:

a facade is an object that serves as a front-facing interface masking more complex underlying or structural code

The Problem:

Let's say we have a CMS and with every new post, we want to tweet about it.

First, we will use dg/twitter-php library to send tweets. from the README it says the following code should post a tweet

use DG\Twitter\Twitter;

$twitter = new Twitter($consumerKey, $consumerSecret, $accessToken, $accessTokenSecret);

$twitter->send('I am fine today.');

Notice here every time we need to get a twitter instance we need to pass some data considered secret. and we don't want to pass them every time we need to send a tweet.

also, we don't need a Factory (Factory Pattern) as we only want to send tweets and using a factory here would be an extra layer of complexity that we don't need.

so the Facade pattern can help us hide the complexity of creating an instance by creating this Facade class

class TwitterFacade 
{
  public static function get()
  {
    return new Twitter($consumerKey, $consumerSecret, $accessToken, $accessTokenSecret);
  }
}

as we are using laravel. let's put the secrets in config/services.php

'twitter' => [
    'api_key' => env('TWITTER_API_KEY'),
    'api_secret' => env('TWITTER_API_SECRET'),
    'api_token' => env('TWITTER_Access_TOKEN'),
    'api_token_secret' => env('TWITTER_Access_TOKEN_SECRET'),
],

and then let's update our facade class to use the config like following

class TwitterFacade 
{
  public static function get()
  {
      $config = config('services.twitter');
      return new Twitter($config['api_key'], $config['api_secret'], $config['api_token'], $config['api_token_secret']);
  }
}

and now whenever we want to send tweets we would just write one single line

TwitterFacade::get()->send('Test tweet from a facade class');

Implement Facade with Laravel

Laravel is supporting the Facade pattern and its part of its core system. if you open config/app.php and scroll to the aliases you would see a bunch of Facades

'aliases' => [

    'App' => Illuminate\Support\Facades\App::class,
    'Arr' => Illuminate\Support\Arr::class,
    'Artisan' => Illuminate\Support\Facades\Artisan::class,
    'Auth' => Illuminate\Support\Facades\Auth::class,
    'Blade' => Illuminate\Support\Facades\Blade::class,
    'Broadcast' => Illuminate\Support\Facades\Broadcast::class,
    'Bus' => Illuminate\Support\Facades\Bus::class,
    'Cache' => Illuminate\Support\Facades\Cache::class,
    'Config' => Illuminate\Support\Facades\Config::class,
    'Cookie' => Illuminate\Support\Facades\Cookie::class,
    'Crypt' => Illuminate\Support\Facades\Crypt::class,
    'DB' => Illuminate\Support\Facades\DB::class,
    'Eloquent' => Illuminate\Database\Eloquent\Model::class,
    'Event' => Illuminate\Support\Facades\Event::class,
    'File' => Illuminate\Support\Facades\File::class,
    'Gate' => Illuminate\Support\Facades\Gate::class,
    'Hash' => Illuminate\Support\Facades\Hash::class,
    'Lang' => Illuminate\Support\Facades\Lang::class,
    'Log' => Illuminate\Support\Facades\Log::class,
    'Mail' => Illuminate\Support\Facades\Mail::class,
    'Notification' => Illuminate\Support\Facades\Notification::class,
    'Password' => Illuminate\Support\Facades\Password::class,
    'Queue' => Illuminate\Support\Facades\Queue::class,
    'Redirect' => Illuminate\Support\Facades\Redirect::class,
    'Redis' => Illuminate\Support\Facades\Redis::class,
    'Request' => Illuminate\Support\Facades\Request::class,
    'Response' => Illuminate\Support\Facades\Response::class,
    'Route' => Illuminate\Support\Facades\Route::class,
    'Schema' => Illuminate\Support\Facades\Schema::class,
    'Session' => Illuminate\Support\Facades\Session::class,
    'Storage' => Illuminate\Support\Facades\Storage::class,
    'Str' => Illuminate\Support\Str::class,
    'URL' => Illuminate\Support\Facades\URL::class,
    'Validator' => Illuminate\Support\Facades\Validator::class,
    'View' => Illuminate\Support\Facades\View::class,

],

Now let's create a new class in App/Facades/TwitterFacade.php and put the following content

namespace App\Facades;

use Illuminate\Support\Facades\Facade;

class TwitterFacade extends Facade 
{
    protected static function getFacadeAccessor()
    {
        return 'twitter-poster';
    }
}

notice that we are extending Facade from Illuminate namespace and then we implemented getFacadeAccessor. we choose 'twitter-poster' as the name of the Twitter object that we want to use.

Now let's jump to the AppServiceProvider to bind the 'twitter-poster' to Twitter. in AppServiceProvider at Register method add the following

use DG\Twitter\Twitter;

public function register()
{
    $this->app->bind('twitter-poster',function(){
        $config = config('services.twitter');
        return new Twitter($config['api_key'], $config['api_secret'], $config['api_token'], $config['api_token_secret']);
    });
}

Now you can see the connection. first, we registered 'twitter-poster' and bind it to an instance of a Twitter class.

The final usage of our facade will be like following

Route::get('/tweet',function(){

    \App\Facades\TwitterFacade::send('Hello from ahmedash.com: Facade design pattern article');

});

Conclusion

We learned what is Facade and what problem does it solve and how to use it with Laravel

Understand Laravel Event::fake() with (Model Events)

Ahmed Ashraf — Fri, 21 Feb 2020 15:53:13 +0000

If you are a laravel developer who cares about writing well-tested software then you might have seen this problem before or at least you are going to face it soon,

In my case, It was a task that allows our users to create widgets for their websites, each widget has a configuration with the total number of ads and the total number of recommended articles.

So, I have a Widget model and WidgetConfiguration model. every time I want to create a new widget I would create a configuration row for it. I used Laravel model events to do so.

class Widget extends Model
{
    public static function boot()
    {
        parent::boot();

        static::created(function($model){
            dispatch(new CreateWidgetConfiguration($model));
       });
     }
}

// WidgetsController

public function store(WidgetStoreRequest $request){   
    Widget::create($request->all());
    dispatch(new NewWidgetCreated($model));
}

Every time I create a widget it will fire CreateWidgetConfiguration event and there is another event called NewWidgetCreated event that will send an email to account managers.

When It comes to writing tests, and you want to make sure every time you create a new widget it must create a configuration for it but you don’t want to fire a NewWidgetCreated event because you don’t want to send an email or notification every time we run tests.

So let’s write the unit tests first

public function test_create_widget_with_configuration()
{
    Event::fake();
    $user = factory(User::class)->create();

    $response = $this->actingAs($user)
    ->json("POST","/api/widgets",[
        'name' => 'My First Widget',
        'domain' => 'https://my-first-widget.com',
    ]);
    $this->assertDatabaseHas('widgets_configuration',[
        'widget_id' => 1
    ]);
}

the previous test will always fail because Event::fake() won’t fire the CreateWidgetConfiguration event.

Let’s take a look at the content of the fake method

public static function fake($eventsToFake = [])
{
    static::swap($fake = new EventFake(static::getFacadeRoot(), $eventsToFake));

    Model::setEventDispatcher($fake);
}

It says that it will fake events and replace the event dispatcher of the model to be the EventFake class.

In order to solve this issue and only fake the global Event class without model dispatcher, we have to set Model dispatcher manually after calling the event fake. so the final result will be something like

$initialEvent = Event::getFacadeRoot();
Event::fake();
Model::setEventDispatcher($initialEvent);

and the final test case method is

public function test_create_widget_with_configuration()
{
    $initialEvent = Event::getFacadeRoot();
    Event::fake();
    Model::setEventDispatcher($initialEvent);
    $user = factory(User::class)->create();

    $response = $this->actingAs($user)
    ->json("POST","/api/widgets",[
        'name' => 'My First Widget',
        'domain' => 'https://my-first-widget.com',
    ]);
$this->assertDatabaseHas('widgets_configuration',[
        'widget_id' => 1
    ]);
}

This is it for now. I hope you at least have got the idea of how event fake works. The main purpose of the article was to let you know what it is and how to deal with it.

If you find any kind of misinformation here, please feel free to let me know and I will update it.

Design Patterns in PHP: Decorator (with Laravel)

Ahmed Ashraf — Thu, 13 Feb 2020 19:02:09 +0000

Design patterns are very important for every developer. it solves very common problems in every project you build.

Decorator pattern definition:

It helps you to add extra behavior on an object without affecting the other objects from the same class, will see what does that means.

Wikipedia:

the decorator pattern is a design pattern that allows behavior to be added to an individual object, dynamically, without affecting the behavior of other objects from the same class

The Problem

Let's say we have a Post model

class Post extends Model
{
    public function scopePublished($query) {
        return $query->where('published_at', '<=', 'NOW()');
    }
}

and in our PostsController we have the index method like below

class PostsController extends Controller
{
    public function index() {
        $posts = Post::published()->get();
        return $posts;
    }
}

in order to cache posts and avoid querying the database every time we need to list posts we can do the following

class PostsController extends Controller
{
    public function index() {
        $minutes = 1440; # 1 day
        $posts = Cache::remember('posts', $minutes, function () {
            return Post::published()->get();
        });
        return $posts;
    }
}

Now we are caching posts for 1 day. but looking at the code. the controller knows too much. it knows how many days we cache, it initiates the cache object by himself.

also, imagine you are implementing the same for Tags, Categories, Archives for your HomePageController. it would be too much to read and maintain.

Repository pattern

for most of the cases, the Repository pattern is connected to the decorator pattern and you will see how.

First, let's separate how we get the posts using the Repository pattern, create app/Repositories/Posts/PostsRepositoryInterface.php with the following content

namespace App\Repositories\Posts;

interface PostsRepositoryInterface 
{

    public function get();

    public function find(int $id);

}

create PostsRepository in the same path with the following content

namespace App\Repositories\Posts;

use App\Post;

class PostsRepository implements PostsRepositoryInterface
{
    protected $model;

    public function __construct(Post $model) {
        $this->model = $model;
    }

    public function get() {
        return $this->model->published()->get();
    }

    public function find(int $id) {
        return $this->model->published()->find($id);
    }

}

getting back to the PostsController and apply the changes to be like

namespace App\Http\Controllers;

use App\Repositories\Posts\PostsRepositoryInterface;
use Illuminate\Http\Request;

class PostsController extends Controller
{
    public function index(PostsRepositoryInterface $postsRepo) {
        return $postsRepo->get();
    }
}

The controller became healthy and knows enough details to get the job done.

here we depend on Laravel's IOC to inject the concrete object of the Posts interface to get our posts

All we just need to do is to tell Laravel's IOC which class to create when using the interface.

In your app/Providers/AppServiceProvider.php add the bind method

namespace App\Providers;

use App\Repositories\Posts\PostsRepositoryInterface;
use App\Repositories\Posts\PostsRepository;

use Illuminate\Support\ServiceProvider;

class AppServiceProvider extends ServiceProvider
{
    public function register()
    {
        $this->app->bind(PostsRepositoryInterface::class,PostsRepository::class);
    }
}

Now anytime we inject PostsRepositoryInterface laravel will create an instance of PostsRepository and give it back.

Implement caching through Decorator

We said at the beginning that, Decorator pattern allows behavior to be added to an individual object without affecting other objects from the same class.

Caching here is the behavior and the object/class is PostsRepository

Let's create PostsCacheRepository in the same path app/Repositories/Posts/PostsCacheRepository.php with the following content


namespace App\Repositories\Posts;

use App\Post;
use Illuminate\Cache\CacheManager;

class PostsCacheRepository implements PostsRepositoryInterface
{
    protected $repo;

    protected $cache;

    const TTL = 1440; # 1 day

    public function __construct(CacheManager $cache, PostsRepository $repo) {
        $this->repo = $repo;
        $this->cache = $cache;
    }

    public function get() {
        return $this->cache->remember('posts', self::TTL, function () {
            return $this->repo->get();
        });
    }

    public function find(int $id) {
        return $this->cache->remember('posts.'.$id, self::TTL, function () {
            return $this->repo->find($id);
        });
    }
}

In this class, we accept the Caching object and PostsRepository object then we use the class (Decorator) to add caching behavior to the PostsReposiory instance.

we can use the same example to send HTTP requests to some service and then we fall back to the model in case of failure. I believe you got the benefit from the pattern and how it makes it easy to add behaviors.

The last thing is to modify AppServiceProvider interface binding to create PostsCacheRepository instance instead of PostsRepository

namespace App\Providers;

use App\Repositories\Posts\PostsRepositoryInterface;
use App\Repositories\Posts\PostsCacheRepository;

use Illuminate\Support\ServiceProvider;

class AppServiceProvider extends ServiceProvider
{
    public function register()
    {
        $this->app->bind(PostsRepositoryInterface::class,PostsCacheRepository::class);
    }
}

Check your files again now you will find it very easy to read and maintain. also, it's testable and if at some point you decided to remove the caching layer. you will just change the binding in the AppServiceProvider and that's it. nothing extra needs to change.

Conclusions

We learned how to cache models with the Decorator pattern
We showed how the Repository pattern is connected to Decorator Pattern
How DependecyInjection and Laravel IOC make our life easy
How powerfull are laravel components

Hope you enjoyed reading the article. and it showed you the powerfull of design patterns and how it makes your project easy to maintain and manage .

Simple steps to Dockerize your Laravel app

Ahmed Ashraf — Wed, 18 Dec 2019 16:15:35 +0000

Hello Friends, I want to share something that I used to do few months ago for my laravel projects.

The reason I decided to drop Valet is that I'm using a shared Macbook pro that has the company dev environment and the projects I'm working on every day after work.

Valet was not quite a good option as it somehow conflicts with the tools I use for my company dev setup. so I decided to dockerize everything to make it easy for me to work and even share projects with other friends if we work on it together.

here I share how I set up every project I start.feel free to share your thoughts, ideas, comments and feedback on it. also, I know it's not that fast to get it for every project. but as a template, it does a good job for the work.

Prerequisite

The setup I use can be for new projects or existing projects. tools used is docker and other project dependencies like MySQL, Redis, NPM for frontend development.

I'll start everything with an empty directory and you can jump between steps. also, I assume you already have docker and docker-compose on your machine and you can read docker-compose.yaml and Dockerfiles

Preparing the files

I'll start with 4 images

Nginx: as a web server
PHP: contains php-fpm, php-cli, composer. I even install npm there to easy run compile laravel assets instead of installing another image
MySQL: the database we need for our app
Redis: for caching, queues

    .
    ├── docker
    │   ├── nginx
    │   │   └── conf.d
    │   │       └── default.conf
    │   └── php
    │       └── Dockerfile
    └── docker-compose.yml

    4 directories, 3 files

We have docker-compose.yml and docker directory that has the configuration files. and for php we have a custom image

The docker-compose

We will define some volumes to copy important configurations and to keep data in sync between our machine and the containers so even with a rebuild or restart containers the data still safe and not destroyed.

Check the comments in the file to know more

    version: '2'
    services:
      webserver:
        image: nginx:alpine
        volumes:
          - ./code:/var/www
                ## copy nginx configuration for our application ##
          - ./docker/nginx/conf.d/:/etc/nginx/conf.d/
        ports:
                ## run the webserver on port 8080 ##
          - "8080:80"
      app:
            ## read php image from our custom docker image ##
        build:
          context: .
          dockerfile: ./docker/php/Dockerfile
        volumes:
          ## copy project files to /var/www ##
          - ./code:/var/www
        working_dir: /var/www

      db:
        image: mysql:5.7
        ## expose the mysql port to our machine so we can access it from any mysql-client like TablePlus ##
        ports:
          - "3388:3306"
       ## keep mysql data on localhost so we don't lose them ##
        volumes:
          - ./docker-volumes-data/db:/var/lib/database
        environment:
          MYSQL_DATABASE: laravel_db
          MYSQL_ROOT_PASSWORD: root
      redis:
        image: redis
        volumes:
          ## keep redis data on localhost so we don't lose them ##
          - ./docker-volumes-data/redis:/data

Now let's jump into each one and check it's content

Nginx: webserver

    webserver:
        image: nginx:alpine
        volumes:
          - ./code:/var/www
                ## copy nginx configuration for our application ##
          - ./docker/nginx/conf.d/:/etc/nginx/conf.d/

here we copy the project files and also copy all conf.d files. which has one file called default.conf and the file has the virtual host for our project

    server {
        client_max_body_size 30M;
        listen 80;
        index index.php index.html;
        error_log  /var/log/nginx/error.log;
        access_log /var/log/nginx/access.log;
        root /var/www/public;
        location ~ \.php$ {
            try_files $uri =404;
            fastcgi_split_path_info ^(.+\.php)(/.+)$;
            fastcgi_pass app:9000;
            fastcgi_index index.php;
            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $fastcgi_path_info;
        }
        location / {
            try_files $uri $uri/ /index.php?$query_string;
            gzip_static on;
        }
    }

PHP

    app:
        ## read php image from our custom docker image ##
        build:
          context: .
          dockerfile: ./docker/php/Dockerfile
        volumes:
          ## copy project files to /var/www ##
          - ./code:/var/www
        working_dir: /var/www

for php we have a custom dockerfile for it so we can install the extensions we need and customize the setup for our need. the file fits any project I work on so it might fit your need to

    # dockerfile
    FROM php:7.2-fpm

    # Copy composer.lock and composer.json
    COPY code/composer.lock* code/composer.json* /var/www/

    # Set working directory
    WORKDIR /var/www

    # Install dependencies
    RUN apt-get update && apt-get install -y \
        build-essential \
        libpng-dev \
        libjpeg62-turbo-dev \
        libfreetype6-dev \
        locales \
        zip \
        jpegoptim optipng pngquant gifsicle \
        unzip \
        curl

    # Clear cache
    RUN apt-get clean && rm -rf /var/lib/apt/lists/*

    # Install extensions
    RUN docker-php-ext-install pdo pdo_mysql mbstring zip exif pcntl
    RUN docker-php-ext-configure gd --with-gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ --with-png-dir=/usr/include/
    RUN docker-php-ext-install gd
    RUN pecl install -o -f redis \
        &&  rm -rf /tmp/pear \
        &&  docker-php-ext-enable redis

    # Install composer
    RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

    # NPM for frontend builds
    RUN apt install nodejs npm -y

    # Add user for laravel application
    RUN groupadd -g 1000 www
    RUN useradd -u 1000 -ms /bin/bash -g www www

    # Copy existing application directory contents
    COPY ./code /var/www

    # Copy existing application directory permissions
    COPY --chown=www:www . /var/www

    # Change current user to www
    USER www

    # Expose port 9000 and start php-fpm server
    EXPOSE 9000
    CMD ["php-fpm"]

here I use php7.2 and then I add some important extensions with composer and NodeJS and NPM to compile the assets from the same image. of course, you can move that part to another NodeJS image but I prefer to keep it there as it works perfectly for me.

MySQL

mysql has a very easy setup

    db:
        image: mysql:5.7
        ports:
          - "3388:3306"
        volumes:
          - ./docker-volumes-data/db:/var/lib/database
        environment:
          MYSQL_DATABASE: laravel_db
          MYSQL_ROOT_PASSWORD: root

I expose the port to 3388 so I can connect to it using any mysql-client like TablePlus or SequelPro

Also, I use volumes to sync the data between the host and the container so I don't lose the data if it restarts or stops

Redis

    redis:
        image: redis
        volumes:
          - ./docker-volumes-data/redis:/data

I use volumes to sync the data too but as you see I don't expose any ports as I use the terminal to check Redis.

laravel queue:listen command is enough for me to monitor the jobs during development

Run docker build and up

Now it's time to start our docker setup and then initialize our project

    $ docker-compose up --build

I run this command every time to build the PHP docker image and run the containers. it will take some time to install everything and the end result should be like:

    Successfully built b98842b1e665
    Successfully tagged docker-demo_app:latest
    Creating docker-demo_db_1        ... done
    Creating docker-demo_app_1       ... done
    Creating docker-demo_webserver_1 ... done
    Creating docker-demo_redis_1     ... done

Now we have every thing ready let's move to the next step

Initialize the project

If you are running a new laravel project then we will run the composer from the app docker container to install the new project. first, we need to get the container name. you can get it from the output of the previous command or just run

    $ docker-compose ps

    # output #

             Name                        Command               State                 Ports
    ----------------------------------------------------------------------------------------------------
    docker-demo_app_1         docker-php-entrypoint php-fpm    Up      0.0.0.0:2222->22/tcp, 9000/tcp
    docker-demo_db_1          docker-entrypoint.sh mysqld      Up      0.0.0.0:3388->3306/tcp, 33060/tcp
    docker-demo_redis_1       docker-entrypoint.sh redis ...   Up      6379/tcp
    docker-demo_webserver_1   nginx -g daemon off;             Up      0.0.0.0:8080->80/tcp

Then let's run

    $ docker exec -it docker-demo_app_1 composer create-project laravel/laravel code

it'll install laravel project inside /code directory

Basic laravel stuff

we need to run few commands to get the app ready

Key generate

    $ docker exec -it docker-demo_app_1 php artisan key:generate

    # output #
    Application key set successfully.

Modify .env file

we just need to change the database and redis to make them work for our app

    # database config
    DB_CONNECTION=mysql
    DB_HOST=db
    DB_PORT=3306
    DB_DATABASE=laravel_db
    DB_USERNAME=root
    DB_PASSWORD=root

    # redis config
    REDIS_HOST=redis
    REDIS_PASSWORD=null
    REDIS_PORT=6379

Let's run the migration files to make sure the database works well

    $ docker exec -it docker-demo_app_1 php artisan migrate

    # output #
    Migration table created successfully.
    Migrating: 2014_10_12_000000_create_users_table
    Migrated:  2014_10_12_000000_create_users_table (0.05 seconds)
    Migrating: 2014_10_12_100000_create_password_resets_table
    Migrated:  2014_10_12_100000_create_password_resets_table (0.02 seconds)
    Migrating: 2019_08_19_000000_create_failed_jobs_table
    Migrated:  2019_08_19_000000_create_failed_jobs_table (0.01 seconds)

Run tests

    $ docker exec -it docker-demo_app_1 ./vendor/bin/phpunit
    PHPUnit 8.5.0 by Sebastian Bergmann and contributors.

    ..                                                                  2 / 2 (100%)

    Time: 2.1 seconds, Memory: 16.00 MB

    OK (2 tests, 2 assertions)

Run on browser

just visit http://localhost:8080/ to see the app running

Source code

here are the full files on Github where you can clone and use them

https://github.com/ahmedash95/laravel-docker-template

Conclusions

I hope the article helped you to easily create a docker setup for your app. even make it work with the existing apps shouldn't be a problem at all.

I still see it's not smooth enough. but I hope to find a better way to make it simpler. you can help by sharing ideas or any experience you had with a similar case.

I know there is laradock but I feel it's too big to use it and just have 4 files make it easy to get it running.

Understand Golang channels and how to monitor with Grafana (Part 2/2)

Ahmed Ashraf — Mon, 09 Dec 2019 19:56:48 +0000

In the first part, we talked about what are go channels and how they work and how to implement them in a scalable way.

In this part, I'll show you how to monitor them and get some useful metrics like processing time per worker. How many jobs running per minute. And some other stuff

Stack

let's take a small brief of the tools we will use to do the job.

Prometheus: is an open-source time-series metric system. also, it supports alerts but it's not our topic for today.
Grafana: is an open-source analytics & monitor solution that supports many data sources and we will use it with Prometheus to display the metric we fire from our Go app.

Prometheus

first, we need to define the metrics in our app and register them in Prometheus package. but let's define the types of the metric and what is the purpose of each one.

Counter: a number that can only increase or reset to zero. like how many emails you sent. it can't be at some time 5 then became 3. it can only increase.
Gauge: represents a single numerical value that can arbitrarily go up and down. like memory usage or concurrent users and for our case it's running workers and running jobs. because these numbers go up and down.
Histogram: in just an observer. based on the samples it gives you the average. like response time, response size, request duration.
Summary: It's like a histogram but also it counts the total observations and a sum of all observed values.

I'm not sure I understand summary well. but anyway we don't need it in this article

// prometheus.go

package queue

var (
    JobsProcessed  *prometheus.CounterVec
    RunningJobs    *prometheus.GaugeVec
    ProcessingTime *prometheus.HistogramVec
    RunningWorkers *prometheus.GaugeVec
)

var collectorContainer []prometheus.Collector

//InitPrometheus ... initalize prometheus
func InitPrometheus() {
    prometheus.MustRegister(collectorContainer...)
}

//PushRegister ... Push collectores to prometheus before inializing
func PushRegister(c ...prometheus.Collector) {
    collectorContainer = append(collectorContainer, c...)
}

func InitMetrics() {
    JobsProcessed = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Namespace: "worker",
            Subsystem: "jobs",
            Name:      "processed_total",
            Help:      "Total number of jobs processed by the workers",
        },
        []string{"worker_id", "type"},
    )

    RunningJobs = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Namespace: "worker",
            Subsystem: "jobs",
            Name:      "running",
            Help:      "Number of jobs inflight",
        },
        []string{"type"},
    )

    RunningWorkers = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Namespace: "worker",
            Subsystem: "workers",
            Name:      "running",
            Help:      "Number of workers inflight",
        },
        []string{"type"},
    )

    ProcessingTime = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Namespace: "worker",
            Subsystem: "jobs",
            Name:      "process_time_seconds",
            Help:      "Amount of time spent processing jobs",
        },
        []string{"worker_id", "type"},
    )

    metrics.PushRegister(ProcessingTime, RunningJobs, JobsProcessed, RunningWorkers)
}

after defining the metrics we need to run our endpoint that Prometheus will call to get the metrics data

r.Handle("GET", "/metrics", gin.WrapH(promhttp.Handler()))

Now if we tried to make some calls

JobsProcessed.WithLabelValues("Worker-1", "ahmedash.com").Inc()
JobsProcessed.WithLabelValues("Worker-1", "ahmedash.com").Inc()

JobsProcessed.WithLabelValues("Worker-2", "ahmedash.com").Inc()
JobsProcessed.WithLabelValues("Worker-2", "ahmedash.com").Inc()
JobsProcessed.WithLabelValues("Worker-2", "ahmedash.com").Inc()

the result when you hit your Prometheus endpoint http://localhost/metrics

and if you opened Prometheus dashboard you will find the chart like below

this is the way of how you can work with Prometheus and send the data to it and check it's dashboard for simple visualization.

but as you can see the dashboard of Prometheus is not that fancy, and we need to create a nice dashboard so it's easy for anyone to understand what is going on with the system. and Grafana can tackle this job very well.

Push the metric from Go to Prometheus

From the first part, we will modify the code to send the data we need for the metrics

Follow arrows ⬅️

func (d *Dispatcher) Run() {
    for i := 0; i < d.maxWorkers; i++ {

        // increase the number of running workers
        RunningWorkers.WithLabelValues("Emails").Inc() ⬅️⬅️⬅️⬅️⬅️⬅️⬅️⬅️

        worker := NewWorker(d.WorkerPool)
        worker.Start()
        d.Workers = append(d.Workers, worker)
    }

    go d.dispatch()
}

With every job. we need to increase the jobs in queue metric


func (d *Dispatcher) dispatch() {
    for {
        select {
        case job := <-JobQueue:

            // Increase running jobs Gauge
            RunningJobs.WithLabelValues("Emails").Inc() ️⬅️⬅️⬅️⬅️⬅️⬅️⬅️

            go func(job Queuable) {
                jobChannel := <-d.WorkerPool
                jobChannel <- job
            }(job)
        }
    }
}

First metric: stores the number of jobs processed per worker
Second metric: decreases the number of current jobs running from the previous code snippet
Third metric: stores the processing time for each job

func (w Worker) Start() {
    go func() {
        for {
            w.WorkerPool <- w.JobChannel

            select {
            case job := <-w.JobChannel:
                startTime := time.Now()

                // track the total number of jobs processed by the worker
                JobsProcessed.WithLabelValues(w.Name, "Emails").Inc() ️⬅️⬅️⬅️⬅️⬅️⬅️⬅️
                if err := job.Handle(); err != nil {
                    log.Fatal("Error in job: %s", err.Error())
                }
                // Decrease the number of running jobs once we finish
                RunningJobs.WithLabelValues("Emails").Dec() ️⬅️⬅️⬅️⬅️⬅️⬅️⬅️

                // ⬇️ Register the proccesing time in the Histogram ⬇️
                ProcessingTime.WithLabelValues(w.Name, "Emails").Observe(time.Now().Sub(startTime).Seconds()) ️⬅️⬅️⬅️⬅️⬅️⬅️⬅️
            }
        }
    }()
}

Grafana

Now we have everything ready and our app is working as expected. the last step we need is just to have a beautiful dashboard with Grafana.

from Grafana dashboard we need to add Prometheus as a data source

then from HOME at the top bar create New dashboard

Then we need to add a Singlestat for counters & gauge metrics

Now:

Let's choose Prometheus from data-sources
then search for the metric name
have the full right name of the metric

Then from Options tab choose if it's current or total. in our case it's current

Finally from General tab rename the component

Also, you can control the size of each component in the panel

Let's make another one for the running workers with worker_workers_running metric name

then let's make a graph panel to show how many jobs per minute

Now you can make some requests to the app and see the data in realtime. just make sure to enable the auto-refresh.

Now we have the first part of our dashboard

The second part is just about adding a graph to show the processing duration for each worker

for this metric, the value would be worker_jobs_process_time_seconds_sum / worker_jobs_process_time_seconds_count

and then make sure we display the values in milliseconds

and the final result is

Now we have a dashboard shows us the running workers, how many jobs in queues. jobs processing duration. also, easily we can see if there are any spikes with our jobs or the system works as we expect from him to be.

Conclusion

In this article including the two parts. we learned how to

create channels
dispatcher/worker pattern
how to create&kill workers
push metrics and display them in a very nice dashboard

Things we didn't cover

We don't deal with failed jobs. so if an error happens to any job we don't have a way to re-try or even store them for later
No data persistence. so if the app restarts we will lose all jobs because they are in memory.
We didn't show how to make this work with a third-party messaging/queuing system like Redis or RabbitMQ.

Source code

This is a source code contains all the code we went through. should be easy to run as it's dockerized.

https://github.com/ahmedash95/go-channels-demo

References

These 2 articles helped me a lot to understand how I can build powerful apps that support async jobs and monitor them. only my article combines knowledge from.

Understand Golang channels and how to monitor with Grafana (Part 1/2)

Ahmed Ashraf — Sun, 08 Dec 2019 17:28:09 +0000

Just before we start I'd like to tell you this article contains the info I was able to understand about go channels and how you can use it in a good way. so if you find any mistake please leave a comment with it :).

What is Go Channels?

Golang docs say:

Channels are the pipes that connect concurrent goroutines. You can send values into channels from one goroutine and receive those values into another goroutine.

so it's like a simple pipe. you send to it from one side and receive from it from its other side. and with goroutines, you can make it async as I will show you the email queue next.

How to use Channels in Go

package main

import (
    "fmt"
)


func main() {
    // Create the channel
    messages := make(chan string)

    // run a go routine to send the message
    go func() { messages <- "ping" }()

    // when you receive the message store it in msg variable
    msg := <-messages

    // print received message
    fmt.Println(msg)
}

Line 1: we created the channel using make
Line 2: we send the message through a goroutine
Line 3: we receive the messages and store it in msg variable
Line 4: we print the message

so at line 2, we used a goroutine, without it, the code won't work as there are no listeners and we call that a deadlock

messages := make(chan string)

messages <- "ping"

msg := <-messages

fmt.Println(msg)

Output

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [chan send]:
main.main()
    /tmp/sandbox676902258/prog.go:11 +0x60

that happens because it's not clear for Go to understand where it should store the message if there are no listeners to send the message directly to them. so a simple solution is to use Buffers

Chanel Buffer

go allows us to set the buffer to a channel by using make(chan string,1), the second parameter means that we tell go to create the channel with a buffer of 1 string

messages := make(chan string,1)

messages <- "ping"

msg := <-messages

fmt.Println(msg)

Now if we try to send 2 messages before the initializing the listener it will throw a deadlock, because for the second message there is no place to store it, also we don't have a listener to send it directly

messages <- "ping"
messages <- "pong"

msg := <-messages

fmt.Println(msg)

Best design to use channels for background workers

so how we deal with channels? do we send all messages with goroutines? what about dropping all channels and just open a goroutine with every job?

all these questions we are going to answer them in the next points

a simple queue worker diagram can be like

simply we need to create a Dispatcher, a dispatcher opens X number of workers based on the requirements and these workers will receive the messages and handles them as we will see.
by Workers I mean channel listeners

Create a Dispatcher

//JobQueue ... a buffered channel that we can send work requests on.
var JobQueue chan Queuable

//Queuable ... interface of Queuable Job
type Queuable interface {
    Handle() error
}

//Dispatcher ... worker dispatcher
type Dispatcher struct {
    maxWorkers int
    WorkerPool chan chan Queuable
    Workers    []Worker
}

The Dispatcher has 3 main properties

Max worker: how many workers this dispatcher owns
Worker Pool: we register all workers in that pool so the dispatcher can pull one of them every time to send the message to it
Workers: contains all workers to talk to anyone of them like close a specific

Now let's create the dispatcher creator

//NewDispatcher ... creates new queue dispatcher
func NewDispatcher(maxWorkers int) *Dispatcher {
    // make job bucket
    if JobQueue == nil {
        JobQueue = make(chan Queuable, 10)
    }
    pool := make(chan chan Queuable, maxWorkers)
    return &Dispatcher{WorkerPool: pool, maxWorkers: maxWorkers}
}

The dispatcher now needs Run method to start his job

//Run ... starts work of dispatcher and creates the workers
func (d *Dispatcher) Run() {
    // starting n number of workers
    for i := 0; i < d.maxWorkers; i++ {
        worker := NewWorker(d.WorkerPool)
        worker.Start()
        // register in dispatcher's workers
        d.Workers = append(d.Workers, worker)
    }

    go d.dispatch()
}

the end of Run method starts dispatch of the Dispatcher

func (d *Dispatcher) dispatch() {
    for {
        select {
        case job := <-JobQueue:
            // a job request has been received
            go func(job Queuable) {
                // try to obtain a worker job channel that is available.
                // this will block until a worker is idle
                jobChannel := <-d.WorkerPool

                // dispatch the job to the worker job channel
                jobChannel <- job
            }(job)
        }
    }
}

dispatch runs an infinite loop, with every new message(job) it will pull one of the workers and send the message to that worker to handle it.

Create Worker

//Worker … simple worker that handles queueable tasks
type Worker struct {
    Name       string
    WorkerPool chan chan Queuable
    JobChannel chan Queuable
    quit       chan bool
}

Name: worker name so we can see it in logs
WorkerPool: the pool of the dispatcher to register itself to it
JobChannel: The messages(jobs) the worker receives from the dispatcher
quit: to close the worker. it helps for autoscale.

So how worker runs after the dispatcher sends the messages to it

//Start ... initiate worker to start listening for upcoming queueable jobs
func (w Worker) Start() {
    go func() {
        for {
            // register the current worker into the worker queue.
            w.WorkerPool <- w.JobChannel

            select {
            case job := <-w.JobChannel:
                // we have received a work request.
                if err := job.Handle(); err != nil {
                    fmt.Printf("Error in job: %s\n", err.Error())
                }
            }
        }
    }()
}

so Start method starts an infinite loop, the first thing the worker registers itself to the WorkerPool of the dispatcher. then it listens for any message within the select scope. once it receives the message it will call handle() method to process this channel.

a simple scenario

Dispatcher started with 2 workers
The first worker registers itself and waiting for any messages
The second worker registers itself and waiting for any messages
The dispatcher receives a message and pulls the first worker from the pool as it was the first registering itself.
Now the pool has the second worker only. because the first one has been pulled already to serve the first message.
a new message came to the dispatcher. dispatcher pulls from the pool the second worker to server the message.
now let's say the second worker finished handling and the first one still handling the job for any reason.
the second worker registers itself again to the pool.
the first worker finished handling and register itself again to the pool.
Now the pool has 2 workers and will do the same at step 4.

Implement the logic with an email service

Let's say in our application we want to send welcome emails to new users.


type Email struct {
    To      string `json:"to"`
    From    string `json:"from"`
    Subject string `json:"subject"`
    Content string `json:"content"`
}

func (e Email) Handle() error {
    r := rand.Intn(200)
    time.Sleep(time.Duration(r) * time.Millisecond)
    return nil
}

let's assume the handle method will call a third-party like SendGrid to send the email.

//EmailService ... email service
type EmailService struct {
    Queue chan queue.Queuable
}

//NewEmailService ... returns email service to send emails :D
func NewEmailService(q chan queue.Queuable) *EmailService {
    service := &EmailService{
        Queue: q,
    }

    return service
}

func (s EmailService) Send(e Email) {
    s.Queue <- e
}

The EmailService now has a queueable channel and Send method to send the email

Now our main.go file can be something like

var QueueDispatcher *Dispatcher

func main() {
    QueueDispatcher = NewDispatcher(4)
    QueueDispatcher.Run()

    mailService = emails.NewEmailService(JobQueue)

    r := gin.Default()
    r.GET("/email", sendEmailHandler)
    return r
}

func sendEmailHandler(c *gin.Context) {
    emailTo := c.Query("to")
    emailFrom := c.Query("from")
    emailSubject := c.Query("subject")
    emailContent := c.Query("content")

    email := emails.Email{
        To:      emailTo,
        From:    emailFrom,
        Subject: emailSubject,
        Content: emailContent,
    }

    mailService.Send(email)

    c.String(200, "Email will be sent soon :)")
}

In the next chapter (during next 2 days), we will use Grafana & Prometheus to monitor workers and processing time and get some beautiful metrics for the application.

Twitter @ShareAsPic app with Go & Chromedp

Ahmed Ashraf — Wed, 06 Nov 2019 18:13:48 +0000

This blog is a translation of its Arabic version on my website.

As a Facebook user when I see any tech tweet on twitter I share it with my FB community. so I thought having a twitter account that takes a screenshot from a tweet can be a nice idea.

Stack

Go-lang: you can use any lang as chromedp supporting several languages but I prefer Go to get a strong knowledge.
Go-lang Twitter API: to receive tweets through StreamAPI and post tweets through Twitter API
Chromedp: Chrome Debugging Protocol. it allows you to use the chrome browser and interact with it to simulate real user behavior with code. also it supports headless mode -no GUI- so it can run on CI/CD pipelines to do e2e tests or any automation you need like scrapping websites and other cool stuff.
Redis: an InMemory database we will use to store processed tweets so we don't respond to the same tweet more than once

Source Code

https://github.com/ahmedash95/shareaspic

Scenario would be like:

create a twitter account @shareAsPic
people can mention him in a reply to any tweet with a message like @shareAsPic share this please
use twitter stream API to listen for all these tweets
use Chromedp to open that tweet in the browser and take a screenshot in - a -headless- mode reply to the mention with the screenshot

Application flow

1 - Users always post tweets.

2 - These tweets land on Twitter's servers which it has a lot of systems. each system has it's own mission with that tweet (notifications, timeline, streaming API, etc..)

3 - We will use the StreamAPI to listen only to replies contain @ShareAsPic our twitter app name to process it

the below example shows how to prepare the stream params to listen only for our app's mentions

params := &twitter.StreamFilterParams{
  Track:         []string{"@ShareAsPic"},
  StallWarnings: twitter.Bool(true),
}

4 - At this step our app starts to receive tweets data twitter sending to us through StreamAPI

// Full struct https://github.com/dghubble/go-twitter/blob/master/twitter/statuses.go#L13
type Tweet struct {
    CreatedAt            string                 `json:"created_at"`
    CurrentUserRetweet   *TweetIdentifier       `json:"current_user_retweet"`
    ID                   int64                  `json:"id"`
    IDStr                string                 `json:"id_str"`
    InReplyToScreenName  string                 `json:"in_reply_to_screen_name"`
    InReplyToStatusID    int64                  `json:"in_reply_to_status_id"`
    InReplyToStatusIDStr string                 `json:"in_reply_to_status_id_str"`
    InReplyToUserID      int64                  `json:"in_reply_to_user_id"`
    InReplyToUserIDStr   string                 `json:"in_reply_to_user_id_str"`
    Text                 string                 `json:"text"`
    User                 *User                  `json:"user"`
}

from the above Struct we send InReplyToScreenName and InReplyToStatusID to chromedp to open tweet URL on twitter and take the screenshot

5 - here we realize the power of headless browsers as chromedp simulates real user behavior on a real browser and makes interactions.
for us, it opens https://twitter.com/{UserName}/status/{TweetID} and using one of the chromedp functions takes a screenshot of that tweet using element selector that contains tweet body

func TweetScreenShot(username string, tweetId string) (string, error) {

    chromedpContext, cancelCtxt := chromedp.NewContext(context.Background()) // create new tab
    defer cancelCtxt()

    // capture screenShot of an element
    fname := fmt.Sprintf("%s-%s.png", username, tweetId)
    url := fmt.Sprintf("https://twitter.com/%s/status/%s", username, tweetId)

    var buf []byte
    if err := chromedp.Run(chromedpContext, elementScreenshot(url, `document.querySelector("#permalink-overlay-dialog > div.PermalinkOverlay-content > div > div > div.permalink.light-inline-actions.stream-uncapped.original-permalink-page > div.permalink-inner.permalink-tweet-container > div")`, &buf)); err != nil {
        return "", err
    }
    fmt.Printf("write pic to path %s\n", fmt.Sprintf("%s/%s", PIC_STORAGE_PATH, fname))
    if err := ioutil.WriteFile(fmt.Sprintf("%s/%s", PIC_STORAGE_PATH, fname), buf, 0755); err != nil {
        return "", err
    }
    return fname, nil
}

// elementScreenshot takes a screenshot of a specific element.
func elementScreenshot(urlstr, sel string, res *[]byte) chromedp.Tasks {
    return chromedp.Tasks{
        chromedp.Navigate(urlstr),
        chromedp.WaitVisible(sel, chromedp.ByJSPath),
        chromedp.Sleep(time.Second * 3),
        chromedp.Screenshot(sel, res, chromedp.NodeVisible, chromedp.ByJSPath),
    }
}

6 - then we reply to the user with the screenshot attached

filename, err := TweetScreenShot(tweet.InReplyToScreenName, tweet.InReplyToStatusIDStr)
if err != nil {
    logAndPrint(fmt.Sprintf("Faild to take a screenshot of the tweet, %s", err.Error()))
    return
}

filePath := fmt.Sprintf("%s%s", PIC_STORAGE_PATH, filename)

logAndPrint("upload photo")
mediaId, _ := TwitterUploadClient.Upload(filePath)
logAndPrint(fmt.Sprintf("photo has been uploaded: %d", mediaId))

statusUpdate := &twitter.StatusUpdateParams{
    Status:             "",
    InReplyToStatusID:  tweet.ID,
    PossiblySensitive:  nil,
    Lat:                nil,
    Long:               nil,
    PlaceID:            "",
    DisplayCoordinates: nil,
    TrimUser:           nil,
    MediaIds:           []int64{mediaId},
    TweetMode:          "",
}

_, _, err2 := client.Statuses.Update(fmt.Sprintf("Hello @%s , Here u are", tweet.User.ScreenName), statusUpdate)
if err2 != nil {
    logAndPrint(fmt.Sprintf("Faild to reply pic tweet, %s", err2.Error()))
}

logAndPrint(fmt.Sprintf("replied to: %s\n", tweet.IDStr))

and the result of the whole flow is like

Conclusions

The app itself has no goal but it was just a simple idea to implement and see how can we automate browser interactions with Chromedp. it adds huge value for E2E tests especially for SPA. or scrapping websites content.

If you have any comments or ideas to share about chromedp. I'm more than interested to discuss and learn.