Forem: Islam Elgohary

Modules in DDD

Islam Elgohary — Sun, 20 Feb 2022 16:20:36 +0000

The 2nd part of chapter 5 in Domain-Driven Design by Eric Evans talks about modules (a.k.a. packages) and how to correctly divide the objects into modules to serve the purpose of the domain model and support the communication between the developers.

Low Coupling and High Cohesion

The first rule of creating good modules, is that modules should have low coupling between each others and each module should be highly cohesive in itself.

Low Coupling

Low coupling means that each module should be independent of other modules in the system. When that's achieved; it gets easier to understand the model since you're dealing with a small part of the system at a time without the need of thinking about the whole system (objects outside the module).

High Cohesion

High cohesion between the objects of the systems which have related responsibilities means that the components of the module are all related thus making it easier to understand what the module does as a self-contained subsystem as each module should focus on one idea/responsibility.

Choosing Modules

The meaning of the objects in the domain should drive the choice of the module and keep in mind that putting some classes together in a module tells the other developers that they should think of these classes together.

If the model is telling the story of the system then the modules are the chapters and they should be named in ways that convey their meanings. The names should be part of the Ubiquitous language (the common language between the technical team and the domain experts)

So modules should:

Tell the story of the system
Contain cohesive set of concepts
Have low coupling with other modules
Have meaningful names that become part of the Ubiquitous language

You might need to choose between technical cohesion and conceptual cohesion. In that case, choose conceptual cohesion because the developers can handle the results of low technical cohesion if they correctly understand the story of the model.

Agile Modules

Modules will need to be refactored as the system grows, however, since modules are large; refactoring them is disruptive and harder than refactoring model objects. That leads to modules reflecting a much older version of the model.

Initially, modules are highly coupled because the model changes as the project goes on. Lack of refactoring keeps that inertia going. That's why modules should be easy to refactor and easy to communicate to other developers what modules do.

Some frameworks divide entities into modules based on their technical responsibilities (example: separating entity's business logic from data access) but this makes modules harder to understand and therefore increases the cost of development and refactoring. The justification given for doing that is that the modules might be deployed on different machines but most of the time that doesn't happen and doing that is costly because:

The code no longer reveals the model
The developers no longer understand the model or reason about it as meaningful pieces.

Minimize the technical partitioning restrictions and apply only the essential ones. The most important purpose of the module is to separate the domain layer from other code.

Modeling Paradigms

This part in the book talks about object oriented languages and how they are the most fitting for the domain model. That's because at the time of writing this book, the usage of object oriented languages was on the rise and that's why it was considered to be the best fit.

However, currently many paradigms are being adopted and the book describes how to mix paradigms as well. That's why instead of talking about object-oriented paradigm I will try to generalize the rules of mixing the paradigms because I think it's more relevant for the current trends in software engineering.

Note that domain-driven design highly relies on OOP and most of the techniques are specific for OOP but I will try to generalize the concepts when possible. But that is my personal opinion and it's important to know that as per the book, Object-oriented languages are the best fit for applying Domain-Driven Design.

Rules for Mixing Paradigms

Don't fight the dominating paradigm: first try to model the domain object in the paradigm you're already using. You can often rethink the model and look at it from another perspective so try to do that before using a new paradigm.
Lean on the ubiquitous language: If you're using a new paradigm; stick to the ubiquitous language. That way the model can still be reasoned about.
Don't be limited by the modeling tools: if you're using a modeling tool like UML don't distort the model into what's easier to represent with the tool. Use tools that fit the paradigm.
Be skeptical: Is the paradigm you're introducing really worth its weight or can the object be represented with the current paradigm? Sometimes objects are not obvious to model but they can often be modeled to different paradigms.

Conclusion

The concept of modules is not new but it is important to have a process for creating them. This part is all about the rules of creating those modules.

I researched if DDD should mainly use object-oriented languages and I didn't find that to be the case and that's why I tried to not focus on it. It's important to know that the book is almost 20 years old and some parts are not very relevant. Also, I believe DDD is more about communicating the model than writing code.

Domain-Driven Design: Entities, Value Objects and Services (chapter 5.1)

Islam Elgohary — Wed, 09 Feb 2022 17:58:14 +0000

This chapter focuses on connecting model and implementation. The chapter is large and it has a lot of important information that's why I decided to split it.

Patterns of Model Elements

Entities: Something with an identity relevant to the domain. It has a state and is continuous through the system even if the state changes. (example: a transaction in a banking system)
Value Objects: Describes an attribute or a value of something else. It has no identity or state and is often created for an operation then destroyed (example: address of the customer in a banking system)
Services: Actions or operations that are done on client's requests. They belong to no specific object. ( example: sending an email of a transaction in a banking system)

Entities

Entities are objects that are in the core of the domain. The system needs to have the state of those objects and they need to be identified.
The implementation of entities should focus on the life cycle of the entity more than the attributes. The entity itself is important regardless of its attributes.
Across a system, entities might have different implementations in the sub-systems. However, In all implementations the objects must refer to the same entity.
Entities have an operation that gives a different value for each instance of the entity. This operation is used for identifying the instance.
The domain model defines what it means for 2 objects to be equal.
A quick way to identify an entity is to answer the question "Does the user need to know if X is the same entity as before?" or in other words "Does the user care about the identity of it?"
An example to that is having a customer object in a banking system. The customer itself is important to the User and the system must allow for the identification of different customers. The customer might fail to pay a loan or they might own a number of accounts. The identity of the customer is important to know which customer owns which accounts or which customer failed to pay.
Modeling Entities

1. Don't focus on the attributes, strip the object down to the most fundamental characteristics. The attributes that define the object and that are used to match the object with other objects that belong to the entity.

2. Add behavior that is essential to the identification of the entity and attributes that are used  by that behavior.

3. Remove other behaviors and attributes to other objects and associate them with the entity.

4. An example to that is a customer object. a customer would have a unique id, a name and an address. According to the domain of banking, the customer is not identified by the address but it is identified by its id and name. Thus, the address attributes should be moved to another "Address" object.

![1.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1644422864110/STcVVhgTJ.png)

Designing the Identity Operations

1. Defining attributes must be unique. For example, the national ID of a customer.
2. If there is no unique attributes, then attach a unique ID to the object. Most databases do that automatically.
3. Identify what makes 2 objects the same entity.
4. Identity operations can require human input if the ID does not correspond to something in the business domain.
5. When an entity needs matching on two different systems whose IDs are different, a third system is used for attaching for example the national ID or the Phone number

Value Objects

Value objects have no identity but their attributes describe something.
If you only care about the attributes of the object but not the identity, then it is a value object
An example to that is a marker pen. If you're coloring, you don't care which pen you use but you care about its color. So all instances of the pen that have the same color are the same and it doesn't matter which pen you're using. The pen is defined by its attributes but the pen itself has no identity.
Is an address a value object or an entity?
This is an important question. The short answer to that is "It depends". It depends on the domain. If you're running a delivery service, then knowing if 2 people who made an order live in the same address is not important so it is a Value Object. However, if the system plans the delivery route then it is important to know that they have the same address so they are delivered together then it is an Entity.

The same object can be an Entity or a Value Object depending on the domain.
Designing Value Objects
1. Which instance of the object we have doesn't matter as long as they have the same attributes.
2. Caring only about the attributes and not the identities allows for more optimization. If the value object is an electrical outlet in a home designing system, then we can either create multiple copy of the outlet or create one and refer to it a hundred times. When should we copy and when should we share?
Copying will send only a copy of the object but it means that you use more space to store redundant copies. However, sharing sends a reference to the object but it also sends a message for every change to the object so the main object changes. In short, Copying uses more space but less network overhead while sharing uses less space but more network overhead unless the object is immutable.
1. Associations should not be bidirectional. If bidirectional associations are needed then consider changing the object to an entity. That's because Value Objects have no identity so having a bidirectional relationship makes no sense.

Services

Services are operations that belong to no specific object but they are important to the domain.
Not anything that has no home in an Entity or a Value Object is a Service. You must first try to place the operation in an Entity or a Value Object and not give up and make it a Service.
Services have no state and no meaning beyond their operation.
The parameters and results should be domain objects.
To keep the design clear, those operations are declared as services instead of creating an object that represents nothing in the domain.
Good Service Characteristics

1. The operation is part of the domain but it is not a natural part of an Entity or a Value Object.
2. The interface is defined in terms of other domain elements.
3. The operation is stateless. A service can use global system info or change them but the service itself should have no state.

Service and the Isolated Domain Layer

This is a pattern focused on distinguishing domain layer services from services in other layers. Usually, a Service is something in the infrastructure layer like a mailing service and they lack any domain meaning. Services that have domain meaning are in the domain layer. But differentiating between services in the application layer and domain layer can be confusing. For example, If we have an operation that exports the transactions as excel then it should be on the application layer because the domain has no concept of file formats. However, an operation that transfers money from one account to the other should be in the domain layer because that's where the business rules reside and the action is a core part of the domain. Here is a more detailed example from the book (I made some changes to it).

A request to make a transfer from one account to another might be processed like this:

1. Application Layer: 
    - Get input request 
    - Ask domain layer service to do the transfer and wait for response 
    - Send notification through mailing service in infrastructure layer

2. Domain Layer:
    - Check if transfer is possible according to business rules 
    - Make changes in account balances to transfer fund 
    - Return result to application layer service

3. Infrastructure Layer:
    - Send notification email

Granularity

Making services medium-grained allows for more reusability because the service has a lot of functionalities. Having a fine-grained service can cause the application layer to have a lot of domain details so it can coordinate those services which blurs the line between application and domain layers. It is better to have medium-grained services to avoid that.

Conclusion

This is the first half of the chapter. It introduces Entities, Value Objects and Services as components of the implemented model. It explains how to differentiate between them and how to best use them to reflect the domain while achieving the system's goals.

Layered architecture and Smart UI: Domain-Driven Design by Eric Evans (Part II, chapter 4)

Islam Elgohary — Wed, 02 Feb 2022 09:45:43 +0000

In part I, the book clarifies the motivation behind using Domain-Driven Design and it laid the foundation of its goals and how to achieve them. It also defined what Model-Driven Design is. In chapter 4, The book explores two architecture frameworks and how they fit into Model-Driven Design.

Part II

Chapter 4: Isolating the Domain

It is important to understand that most of the code is not about the domain but about the technology of the software itself. We need to isolate the domain objects from the rest of the code so we do not lose sight of the domain.

For example, when you set the destination in a ride hailing app, you go through different parts of the system:

The frontend widget
The query that gets the possible destinations from the database
User input validation
Associate selected destination with the trip
Commit changes to the database

Only a little portion of that is part of the business domain.

Often UI, database and other code is written directly in the domain objects because it is faster to get things done. This makes the project harder to understand and harder to scale. To avoid that, systems should be implemented with separation of concerns and that's what Layered Architecture aims at.

Layered Architecture

Layered architecture divides the program into layers where each layer is responsible of an aspect of computer science. The layers are:

UI or Presentation layer
Application layer which coordinates tasks between domain objects and it has no business rules
Model layer or domain layer which is the most important layer for Model-Driven Design
Infrastructure layer which provide technical capabilities to support the above layers

An example to that can consist of:

Layer	Component
Presentation	Frontend web interface
Application	Backend for Frontend (bff)
Domain	Specialized Microservices handling different entities each
Infrastructure	Interfaces that utilize infrastructure (ex. Rabbit MQ, Database)

Layered Architecture Specifications

Partition system to layers according to the responsibilities
Each layer should depend only on the layers below it. This is achieved by using the public interfaces of the layers below.
Each layer is loosely coupled to the layer above it. To achieve that, layers should communicate with the layers above them through callbacks or observers.
Infrastructure layer should know nothing about the domain and should be used as services.
The Domain-Driven Design approach requires only the domain-layer to be separated, other layers can be mixed and the system can still follow Domain-Driven Design.

Advantages of Using Layered Architecture

Design is a lot cleaner
Less expensive to maintain since layers are loosely coupled
Easier to scale because each layer can evolve separately

Disadvantages of Using Layered Architecture

Has a learning curve that inexperienced developers might struggle through
Takes longer time to design and implement

That's why another architecture is introduced as an "anti-pattern" to Layered Architecture and that is Smart UI.

Smart UI

If the project is not very ambitious or is not expected to scale, or if the developing team is inexperienced then Smart UI is probably the way to go.

Smart UI specifications

Business logic is implemented in the UI
Application is divided to small functions and each function is implemented as a separate UI.
Relational databases are used as a shared data repository
Use the most automated UI building tools available. (I'm not sure what the book means by that exactly but I think it's about tools that let you drag and drop components to form the UI and then wire your code to it.)
Use fourth generation languages (4GL)

Advantages of Using Smart UI

Fast and easy to develop
Can be changed quickly since it's divided to small functions and small UIs
Easy to expand the system since the UIs are decoupled
Using relational databases provides integrity at data levels
4GL tools work well
Maintenance team can quickly redo parts of the code that they don't understand since changes should be localized to each separate UI

Disadvantages of Using Smart UI

Integrating applications is difficult since the tools and languages are not flexible
Business problem is not abstracted which means behavior cannot be reused
Lack of abstraction limits refactoring options
No flexibility to add richer behavior
If need for scalability arises later, system cannot migrate from this approach except by replacing the entire app.

It's important not to use sophisticated design approaches unless the development team is committed to those patterns all the way or if the project does not need such approaches. It's better to use 4GL and automated tools instead of a sophisticated, flexible language like Java. Do not overengineer your project.

Conclusion

This chapter introduces to approaches to design a system based on how complex the system is and how skilled the developers are. It explained when to apply Domain-Driven Design and when not to showing the pros and cons of each of the 2 patterns introduced.

As always, it's important to remember that designing a system is a process of constantly making tradeoffs. You get fast development but it's harder to scale. You have flexibility but it takes longer to develop.

Domain-Driven Design by Eric Evans (Part I)

Islam Elgohary — Thu, 27 Jan 2022 14:15:54 +0000

When I first started working as a software engineer I gradually started spending a lot more time designing than actually implementing. More often then not, designing the system to correctly fix the problem of the business is harder than implementation. This is why I started to focus more on design and as I'm currently reading Domain-Driven Design: Tackling Complexity in the Heart of Software I decided to write my notes about it as articles here; to better understand it and maybe help someone get the gist of it without having to read it.

In this article I will write my notes on Part I (chapters 1,2 and 3).

What is Domain-Driven Design?

Domain-Driven Design is a software design approach that focuses on modelling the software to match the business domain.

The book uses the term "model" to express the organized and selectively abstracted knowledge of the domain experts. Meaning that it's not about a specific diagram, document or "model" in the context of programming. But basically whatever shows what the system should do and how to do it is referred to as the model.

The domain is used in the book to refer to the business domain so if your software is for booking taxis, the domain would be the business of ride hailing.

The Utility of a Model

In the preface of part I, the factors defining the utility of a model are:

Model and Implementation shape each other
Model language is used by all teams (technical and business)
The model is the shared understanding of the domain (shared by the developers and the domain experts)

Chapter 1: Crunching Knowledge

The first chapter explains the process of creating a good model. The process should involve meetings between the developers and the domain experts to create the model and refine it to achieve the business goal.

The process is iterative, meaning that it's not a single meeting but multiple meetings that all incrementally create an effective model.

Effective Modeling

These are how modelling should be to produce an effective model

Bind the model to the implementation.
Cultivate a common language based on the model.
Develop a knowledge-rich model that shows the objects' behavior and how that solves the problem.
Distill the model by removing unnecessary details.
Brainstorming and experimenting.

Chapter 2: Communication and the Use of Language

The model's language is the set of terms and relationships that are used in the model.
As explained in chapter 1, the model's language should be understood by both domain experts and developers. Meaning, it should be tailored to the domain and be precise enough to be used for technical development.

As the model keeps changing to better fit its purpose, the language evolves as well by input from both the domain experts and the developers. A good language should be used by all teams and if that's not the case; then the model should be enriched to provide a good common language.

In the meetings, involved parties should play with words and phrases to find the easiest way to express the domain and model it. It's ok to abstract technical details for the domain experts but the core model should be known to them.

Diagrams

It's important to know that diagrams do not need to have the full context of the model. The goal of the diagram is to explain the model and they should be as simple as possible. You should use diagram and documents to explain the model but they don't have to be a replica of the model. In my opinion, a good model should be simple enough that it's understood from the code itself.

Chapter 3: Building Model and Implementation

This chapter stresses on the importance of binding the model to the implementation. A model can be very descriptive of the domain while being unrepresentable in code. If the implementation is not considered while building the model:

The model can have objects and relations that cannot be stored in a database.
Model will detail irrelevant subjects and ignore important subjects.
Some discoveries about the domain might not be apparent without considering the implementation details.

Design Model vs Analysis Model

There are 2 types of models:

Analysis Model: Captures the fundamentals of the domain concepts in comprehensible, expressive way. (Focuses on the domain)
Design Model: Specify a set of components that can be constructed using programming tools and correctly solves the problem. (Focuses on the implementation)

The book introduces another approach Model-Driven Design.

Model-Driven Design

Model-Driven Design integrates both Analysis and Design models and it serves both purposes.

To build the model:

Design the model to reflect the domain in every literal way.
Revisit the Model and modify it with implementation in mind.

If you have multiple systems then they can have different models, however, within a single sub-system you should have only one model. Development of that system then becomes an iterative process of refining the model, the design and the code as a single activity. If the code changes then the model must be changed and vice versa.

Modeling Paradigms and Tool Support

Object-oriented languages (Java) are powerful for implementing the model since they are already based on a modeling paradigm.

Logical languages (Prolog) are also a good fit for model-driven design because they declare a set of logical rules for the design to follow.

Purely procedural languages (C) are not a good fit because it has no modeling paradigm. Procedural languages rely on telling the computer what to do step by step and that's it.

Letting the Bones Show

The underlying model of the system should be the same model shown to users. Otherwise, confusion will arise and the language will not be descriptive enough. Having different models can also cause unexpected behaviors to the users. An example to that is how Microsoft Internet Explorer modeled the bookmarks.

For the user, "Favorites" was a list of URLs but to the actual implementation a "Favorite" is a file that contains a URL. This meant that URL names had to follow windows file naming restrictions. So if a user saved a URL and named it "AOT: 3" they will get an error stating that file names cannot contain special characters like ":" which is very confusing.

Hands-On Modelers

Separating the responsibility of the modeling and coding between different team members violates binding the model to the implementation. It eventually renders the model useless therefore the developers should be able to express the model and contribute to it and anyone who touches the model should participate in the implementation.

Conclusion

Part I is basically a motivation for why Domain-Driven Design is important. It explains the goals of the approach and shows what it hopes to achieve.

My biggest takeaway of Part I is that both domain experts and developers need to be involved in the modeling process and that having a common language that is understood by everyone involved is crucial.

I also couldn't help but notice how well this approach goes with the Lean approach introduced by Eric Ries in his book The Lean Startup. Specifically the focus on experimenting and iterations.

There are modeling examples in the book that better explain the approach's goals by showing the do and don't of modeling so you might want to have a look at that.

Design Patterns: Singleton Pattern

Islam Elgohary — Sat, 09 Oct 2021 13:42:12 +0000

Continuing my Intro to Design Patterns series; today I will explain the singleton design pattern.

You know how you can have one global variable in your application? Singleton pattern is the same but for classes.

The Singleton design pattern is a creational pattern and it's a very simple one actually. Basically, if you need only one instance of a class in your application then that's when you use a singleton pattern. The singleton pattern is a class that can have only one instance in the application.

Problem

I will use a simple example here of a counter. Let's assume that you have a chat room and you want to keep count of how many people are currently online without using a database. For the sake of simplicity let's assume that we don't need that information to be persistent so we are ok with the counters resetting when the application restarts. In this case we can just use a class with a variable that keeps track of users count. the class can look like this:

class UserCount {
  private counter int;

  constructor UserCount(){
    this.counter = 0;
  }

  function Increment() {
    this.counter++;
  }

  function Decrement() {
    this.counter--;
  }
}

class Auth {
  counterInstance UserCount;

  constructor Auth() {
    this.counterInstance = new UserCount();
  }

  function Login(request){
    // Check username and password then log the user in
    this.counterInstance.Increment()
  }

  function Logout(request){
    // Logout the user
    this.counterInstance.Decrement()
  }
}

With that implementation, we have a class that has one variable that is initialized to be zero. it can then be incremented and decremented.
Class Auth uses this class and whenever a user logs in, the number is incremented, whenever they logout it's decremented.

This works fine for our case since users are counted only when they login and we only have one class that uses the UserCounter class. But now we allow users to add a guest to the chat where guests can chat without being logged in.

We now have a new class GuestChat that will need to use the UserCounter class. If we use the same implementation of UserCounter each class will have its own independent counter because it will have its own instance of the UserCounter class.

class GuestChat {
  counterInstance UserCount;

  constructor GuestChat() {
    this.counterInstance = new UserCount();
  }

  function AddGuest() {
    // Create a new guest
    this.counterInstance.Increment();
  }

  function RemoveGuest() {
    // Remove a guest
    this.counterInstance.Decrement();
  }
}

Solution

Here comes the singleton pattern. Instead of having multiple instance of UserCounter, we can implement the class in a away that creates only one instance and whenever an instance exists, all classes will use it. The implementation would look like this:

class UserCount {
  private static instance UserCount;
  private counter int;

  static function GetInstance() {
    if (this.UserCount == null) {
      this.instance = new this.UserCount();
      return this.instance;
    } else {
      return this.instance;
    }
  }

  private constructor UserCount(){
    this.counter = 0;
  }

  function Increment() {
    this.counter++;
  }

  function Decrement() {
    this.counter--;
  }
}

Here are the changes that we applied to UserCount class:

Create a new private variable instance and make it static. static variables and static functions are associated with the class itself and not with the object. So whenever the class is called it will reference the same static variable/function.
Make the constructor private. This way other classes cannot create new instances of the class and it's completely up to the class itself to decide when to create a new instance.
Create a new function GetInstance which returns the instance if it has already been created or create a new instance (using the private constructor) if it does not exist.

Now classes Auth and GuestChat will be changed to the following:

class Auth {
  counterInstance UserCount;

  constructor Auth() {
    this.counterInstance = UserCount.GetInstance();
  }

  function Login(request){
    // Check username and password then log the user in
    this.counterInstance.Increment()
  }

  function Logout(request){
    // Logout the user
    this.counterInstance.Decrement()
  }
}


class GuestChat {
  counterInstance UserCount;

  constructor GuestChat() {
    this.counterInstance = UserCount.GetInstance();
  }

  function AddGuest() {
    // Create a new guest
    this.counterInstance.Increment();
  }

  function RemoveGuest() {
    // Remove a guest
    this.counterInstance.Decrement();
  }
}

In the constructor, instead of creating a new instance of UserCount, both classes call GetInstance() instead. This means that only one class will create a new instance and then that instance will be shared with both classes.

Conclusion

The singleton design pattern uses access modifiers to limit creation of instances of the class allowing for having a global instance of the class throughout the whole application where other classes can only get the instance but they cannot directly create new instances at will. The example I used is very simple but this is usually used for resources like database connections because you usually need to create the connection once and then use it everywhere instead of creating a new connection each time you want to connect to the database.

Photo by Markus Spiske on Unsplash

Design Patterns: Strategy Pattern

Islam Elgohary — Fri, 01 Oct 2021 14:49:51 +0000

In a previous blog post Intro to Design Patterns I used strategy pattern as an example for behavioral patterns. In this post, I will give a deeper explanation to strategy pattern and give examples to it.

You know how you can go to the grocery store through many different routes? Or how you can play music on both your phone and your laptop despite them being different devices? This is where strategy pattern shines.

Strategy pattern is a behavioral design pattern that is very helpful for when you have similar but different objects where each object has a different way to do the same task or when something can be achieved in more than one way and you need to accommodate for all these ways in your application. Let's explain with an example.

Problem

Let's assume that you have an application that calculates the area of a square. You can have a class for the square that looks like this:

class Square {
  side double;

  function area () {
    return this.side * this.side;
  }
}

and your main function would be:

function main() {
  squareInstance = new Square();
  squareInstance.side = 5;
  print(squareInstance.area);
}

Now a new requirement comes up where you also need to handle triangles. One way to do that is to add a "type" field to the class and calculate the area according to the type which would look like this:

class Shape {
  type string;
  side double;
  base double;
  height double;

  function area () {
    switch this.type {
      case "triangle": return 0.5 * this.base * this.height;
      case "square" : return this.side * this.side;
    }
  }
}

this would work but what if we add a circle to the class? as you can probably notice, this is a confusing solution and it will get more confusing as we add more types. This solution would introduce a lot of problems like:

We needed to add variables that are specific to a single shape and are irrelevant to other shapes. The triangle has a base and a height while the square has a side.
Initializing an instance of the class is error prone. A client can simply create a shape of type "triangle" and give it base and side instead of a height.
Handling object behavior requires switch statements or if statements. This will cause problems as the project gets larger because whenever you add a new type, you will find yourself adding a new if/case statement to every function that handles the object and before you know it, you will have a massive piece of code that is basically conditional statements.

Solution

The strategy pattern aims to allow for similar objects to have different behaviors without having to know the context. It also keeps the code clean and organized and allows for adding new behaviors seamlessly. Here is how it works:

Move the function that has different behaviors to an interface.
Split the different "types" to different classes.
each of the new classes should implement the interface.

The code will look like this:

interface AreaStrategy {
  function area(){};
}

class Square implements AreaStrategy {
  side double;
  function area() {
    return this.side * this.side;
  }
}

class Triangle implements AreaStrategy {
  base double;
  height double;
  function area() {
    return 0.5 * this.base * this.height;
  }
}

Now calculating the area can be done without having the context. You can have a function that takes a shape and calculate the area directly without the function having to know what the shape is. Since all classes implement AreaStrategy, shape can simply be of type AreaStrategy. The function would look like this:

 function calculateArea(shape AreaStrategy){
  return shape.area()
}

function main() {
  squareInstance = new Square();
  squareInstance.side = 5;
  print(calculateArea(squareInstance))
  // prints "25" i.e. 5 * 5

  triangleInstance = new Triangle();
  triangleInstance.base = 10;
  triangleInstance.height = 20;
  print(calculateArea(triangleInstance))
  // prints "100" i.e. 0.5 * 10 * 20
}

By using abstraction, a function needs to only have the object to work properly without having to know the object's type. It doesn't even have to know that there are types. As long as the object implements the interface it will work without adding anything to the function.

Now if we have a new shape, all we need to do is to create a new class and make it implement the AreaStrategy interface and it will work without having to make modifications to other parts of the code.

Conclusion

By using abstraction, strategy pattern makes it a lot easier and cleaner to have different behaviors/algorithms for the same task. Simply implement an interface and you're good to go. No context needed. It simply allows you to focus on the task rather than the method it's done.

Photo by Alvaro Reyes on Unsplash

Intro to Design Patterns

Islam Elgohary — Sat, 11 Sep 2021 11:28:06 +0000

If you have studied computer science or software engineering then you have probably came across design patterns. If you haven't, chances are you have faced problems that could have easily been avoided with a design pattern. In this article, I will explain what are design patterns and how it can save you a lot of effort and make your code cleaner.

Motivation

As your project gets larger, you can be tempted to just create code that works, however, this will inevitably introduce problems that are harder to solve in the future. Design patterns allow you to avoid those problems before they arise by writing your code in a way that handles those problem for you.

Prerequisite

To understand design patterns you have to know Object-oriented programming and be familiar with:

Encapsulation
Abstraction
Inheritance
Polymorphism

What are design patterns?

Over the years, numerous software engineers have faced recurring problems, accumulated solutions to these problems by trial and error and created design patterns to avoid those problems in the future.

You can think of design patterns as blueprints written by other developers to help you avoid problems that they have already faced.

They are basically time tested solutions to software design problems.

Design patterns categories

Every design pattern falls into one of 3 categories.

Creational
Structural
Behavioral

Creational Patterns

They provide mechanisms to create objects in a way that makes code reusable and clean. An example to that is Singleton pattern

Singleton pattern lets you create one instance of the class and allows for global access to such class. A common case where you need that is for accessing shared resources like the database or a shared state for example. You probably don't want your application to create a new database connection every time a function wants to access the database.

Structural Patterns

They provide mechanisms to assemble large objects and classes from smaller ones providing more flexibility. An example to that is Facade pattern

A facade is a class that provides a simple interface to complex systems. This is helpful in case you are using a library that provides a lot of functionality. Instead of exposing the whole library you can create a facade with only the functionality you need. It's also helpful if a client wants to integrate with you. Instead of giving them access to your whole system, you can give them a set of functionalities that they can use as a facade.

Behavioral Patterns

They provide mechanisms of running algorithms and communications between objects. An example to that is Strategy pattern.

Strategy pattern utilizes polymorphism to allow for using different behaviors for a specific task. If you have a class that does a specific task in different ways according to some input then the strategy pattern is useful. For example, if you have a class that calculates the area of a shape. Instead of having one class with a function Area() and changing the function behavior by checking what the shape is, you can create an interface that has the function Area() and create a class for each shape that implements that interface. Now other functions will not need to know what shape they're using they will simply call the function and each class will run its specific Area function.

Conclusion

That was a quick intro to design patterns just to give an idea about what they are. I gave a simple example to each category just to clarify what the category is about. I didn't dive into the examples too much here but I will probably create a separate post for each pattern explaining it more.

Photo by Glen Carrie on Unsplash

Live Data Processing Using Kafka and Ksql

Islam Elgohary — Mon, 05 Apr 2021 12:37:00 +0000

What is Kafka?

The Apache Kafka website defines Kafka as

an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

so in a nutshell, apache kafka is a platform designed for streaming data as events.

Kafka uses pub/sub messaging model. In that model, messages are published to topics and consumers can subscribe to these topics. All the consumers will receive the message unlike queues where the message is received by only one of the consumers.

Architecture of Kafka:

Kafka has 4 main components:

1- Producer:
The producer is the publisher of the event. It can be on any platform and can be written in any language. The producer appends the event to the topic. Multiple producers can publish events to the same topic.

2- Consumer:
The consumer subscribes to the topic and multiple consumers can subscribe to the same topic and all subscribed consumers will receive the message.

3- Broker:
Host for the topic. The broker maintains the replication and partitioning of the topics (more on that later)

4- Topic:
The topic can be considered as a folder that stores the events. It's basically the place where the events are stored. Events are stored in topics as logs meaning that they are append only and are immutable. This allows for replaying the events on a specific topic.

Topics can also be divided to partitions where events are partitioned according to their keys. each consumer subscribes to a specific partition which allows for scaling the topics and distributing work among multiple instances of the same consumer. Each topic has a leader partition and 0 or more follower partitions. That structure is maintained by the broker.

Kafka also allows for replicating topics which makes it fault tolerant. The broker is responsible for managing the replication accross the cluster.

Now that we have an idea about Kafka, let's dive into Ksql and how it leverages kafka for live data processing.

What is Ksql?

Ksql is defined as

The event streaming database purpose-built for stream processing applications

Ksql is basically a database that's built on kafka streams. It allows developers to run queries that are very similar to sql queries but the queries can run realtime using the stream. In other words, ksql runs queries on the event log and the queries are run on every new instance of the data.

Ksql has two important concepts:

1- Streams: A stream is a sequence of data where events are appended to the stream in order. Once an event is in the stream it cannot be deleted nor updated.

2- Tables: A table is a representation of the current state based on the data in the stream. It can be changed realtime according to the data in the stream but it is mutable and the data can be updated or deleted.

let's take as an example a financial transaction where Alice has $500 in her bank account and Bob has $200 in his bank account

The initial table and streams would look like this:

The arrow connecting the stream to the table represents that the table reads from that stream.
The stream currently has no new events and the table shows the current balance for both Alice and Bob.
Alice make a transfer to Bob's account and the event is published as "Alice sends $100 to Bob" the stream and table would then look like this:

As soon as the stream receives the event, the table will update its values accordingly

Now in the table, Alice has $400 and Bob has $300. The event of Alice transferring that money is still on the stream and cannot be deleted from the stream. However, if for any reason we want to change the balance of Bob for example, we can do that directly to the table without changing the stream.

Using Ksql:

Ksql is very similar to SQL in syntax. For example creating a stream from a topic can be done using

CREATE STREAM transactions (
    from_id BIGINT,
    to_id BIGINT,
    value DOUBLE
  ) WITH (
    KAFKA_TOPIC = 'transactions-topic',
    VALUE_FORMAT = 'JSON'
  );

We can also create tables from streams:

CREATE TABLE users (
     id BIGINT PRIMARY KEY
   ) WITH (
     KAFKA_TOPIC = 'users-topic', 
     VALUE_FORMAT = 'JSON'
   );

Furthermore, we can create streams from streams to filter the streams:

CREATE STREAM filtered AS
   SELECT 
     a, 
     few,
     columns 
   FROM source_stream;

We can also join 2 streams or join a stream with a table to add more information to the streams:

CREATE STREAM enriched AS
   SELECT
      cs.*,
      u.name,
      u.classification,
      u.level
   FROM clickstream cs
      JOIN users u ON u.id = cs.userId;

We can keep creating streams and tables from each others to further refine the results and aggregate the data by using the above syntax CREATE TABLE AS and CREATE STREAM AS.

However, dealing with a stream is different than dealing with a table since streams can be infinite and we can't possibly run aggregations on infinite data; that's why we have Windowing

Windowing:

Since we're dealing with streams in KSQL we need to have a time window to run some queries on streams. Every record on the stream has a timestamp which can be used for windowing the queries.

A stream would look like this:

A window over the stream would look like this:

Windowed queries run on the records inside the window and the result will change as the window shifts where the table will have a new row for each window.

There are different types of windows in Ksql which are:

1- Tumbling: The window has a fixed size and the windows don't overlap. It takes size as a paremeter.

2- Hopping: The window is fixed in size but windows can overlap. It takes size and advance as parameters where advance is the value the window moves with. for example, you can have a window of size 40 seconds that advances by 20 seconds.

3- Session: The window has a fixed size. It starts from the first entry of a particular key it doesn't shift until a new message with the same key happens within a specified period of time or inactivity gap. the inactivity gap is a parameter.

Conclusion:

Kafka is a powerful event-streaming tool and using KSQL leverages kafka by allowing for running queries directly on the stream and getting live statistics and analytics about the system with simple SQL-like queries.

Resources:

1- https://kafka.apache.org

2- KSQL: Streaming SQL Engine for Apache Kafka

3- Developer to Architect: How Kafka Differs From Standard Messaging

4- https://aws.amazon.com/msk/what-is-kafka/

5- Kafka Architecture & Internal by Narayan Kumar

6- Kafka consumer group

7- KSQL: Streaming SQL for Apache Kafka

8- Introducing KSQL: Streaming SQL for Apache Kafka

9- https://docs.ksqldb.io

On messaging and embedded messaging

Islam Elgohary — Sat, 20 Feb 2021 15:01:36 +0000

Nowadays almost every large software is built using microservices architecture. As I always say, this introduces a whole bunch of problems within the software. One of these problems is communication between services.

The problems that you would probably face in this aspect are:

Synchronization: synchronizing all the services so the system would eventually be in a consistent state
Homogeneity: More often than not, services will be using different protocols or different technologies

In order to solve these problems you can use REST or RPC, however, both protocols are synchronous which means that a service is bound to block on the other service to finish handling the request before it's able to free resources for new requests. This is where messaging comes in play. In this article I will explain briefly messaging brokers and transition to embedded messaging, however, I will not get into much details about messaging protocols or patterns.

Messaging is an asynchronous way of communicating events or tasks between different services. To illustrate how messaging works, let's take a simple example.

assume that you're working on a system for a bank. the system has 2 services:

Transactions service that logs the transactions
Account service that handles the customers accounts

assume that a customer makes a deposit. In that case, we need to add the deposited amount to the customer's account in the account service and log the transaction in the transactions service.

Using a messaging broker to do that would look like this:

so the request goes to the account service first which updates the customer's account with the new deposited amount. Then the account service would create an event with the deposit action and send it to the messaging broker. Then the transaction service would receive that message and log the transaction.

The account service's role in the process ends as soon as it sends the message to the broker which means that it no longer blocks on the transactions service.

However, we can take this one step further by using embedded messaging.

Embedded messaging uses a messaging broker that is inside the service itself. So instead of connecting to the broker externally the transactions service would have the broker embedded in it.

As you can see, there is no longer a connection between the transactions service and the broker, which means that we don't need to worry about them being disconnected. This also adds the possibility of different classes within the service to communicate using the broker and it allows for parallelizing some functionality by having different classes listening to the same event or message.

For example, assume that the transactions service logs the transaction in 2 different databases and sends a notification to a different service. Since we don't need to worry about losing connection to the broker; we can have 3 different classes within the transactions service listen to the same message so the 3 functionalities would start at the same time as soon as the message is received.

We can even take a step further and use embedded messaging for other functionalities replacing other protocols altogether. For example, if the transactions service needs to contact the users service to get the user's name, we can have a design that looks like this:

first the transactions service receives the message as the previous example

The transactions service then sends a new message to the broker inside the users service requesting the user's name. The message would have a unique identifier along with the payload to relate the response to the current task. Notice that the transactions service would connect to the users service's broker, send the message, then disconnect. We no longer need connections to an external intermediate service.

The users service can then send the name to the response queue in the transactions service's broker along with the unique identifier that was initially sent in the previous step (again connect, send, disconnect). Then the transactions service can carry on logging the transaction with the user's name.

Since we no longer have an intermediate service, we eliminated a single point of failure and we alleviated bottle necks associated with that intermediate service.

Resources:

The circuit breaker pattern

Islam Elgohary — Sun, 13 Sep 2020 01:16:02 +0000

In a system that applies microservices architecture, microservices often need to collaborate to get the job done. But How does the system act if a microservice is down? What would happen if a service is unavailable or not responsive enough to handle requests? How can this be handled gracefully with minimal effect on the user experience? In this article, I will explain the circuit breaker pattern which provides a solution to this problem.

First of all we must accept that, if a service is unavailble or unresponsive there is nothing we can do until the service is up again and the cause of the problem is solved. Which means that our only options are to fail fast and be fault tolerant. To understand why the circuit breaker is important, let's see how the error might be handled without a circuit breaker.

Assume that a user sends a request to buy some stock and somewhere on the path of that transaction the transactions microservice needs information from the users microservice but the users microservice is down or the network is slow. Assume that the timeout value is 10 seconds then the user will only know about the error after 10 seconds and each user trying to make a transaction will also have to wait for 10 seconds before each error. That will also happen for each retry by the user and obviously, this is not acceptable specially in a realtime system where time is critical and users expect things to happen instantaniously.

That's where a circuit breaker comes in.

What is a circuit breaker?

The circuit breaker is a proxy that a microservice receives requests through. It checks two things regarding that microservice:

1- availability: Is the microservice online?
2- responsiveness: Can the microservice handle requests?

by monitoring these states, the circuit breaker may or may not allow requests to reach the microservice. Which allows the system to fail fast and hence be more fault tolerant.

How a circuit breaker works:

Initially, the circuit breaker is closed (it allows requests) and it keeps monitoring the microservice at all times until the microservice is either unavailable or unresponsive. In that case, the circuit breaker opens and it automatically rejects all subsequent requests without sending them to the microservice or waiting for the timeout period. Eventually when the microservice is available and responsive again, the circuit closes and allows requests to reach the microservice normally. The most important part of that process is the monitoring.

How does the circuit breaker monitor a microservice?

One way to monitor the microservice can be an endpoint that simply replys with OK.The circuit breaker can keep pinging the endpoint, If it replies then the service is up, if it doesn't then it's down. However, this approach would be limited since it only checks for availability but not responsiveness. In order to work, a service might need to use a database and if the database is unavailable then the service is not responsive and should not receive requests even though it is online.

To solve that, another approach is a synthetic transaction. This is done by actually calling a real endpoint in the service with fake data and depending on whether the request is processed or not the circuit closes or opens. But this means that all services in all environments should be aware that this is a fake request and should not be handled by the system.

That's why another approach is more commonly used which uses real-user monitoring.

Real-user monitoring:

This approach monitors the requests done by the users in realtime, monitors the response time of those requests and analyizes the trends in them. If the response time keeps increasing above the mean time then the service is assumed to be overloaded and the circuit opens. The circuit breaker then moves to a half-open state in which it only allows a small number of requests and monitors them. When the response time is back to normal, the circuit is then closed again and all requests are handled normally.

This can also be threshold-based where a threshold is set for failed requests and the circuit breaker waits for requests to actually fail a number of times that is equal to that threshold before going to the half-open state.

In both cases, this approach will need to use cache or storage in order to keep track of the requests and the overhead of this approach depends on the effeciency of the storage used.

Applying circuit breaker pattern:

Now that we know what a circuit breaker is, We have two options to implement it:

1- Actually implement it from scratch.
2- Using one that's provided by a service that you already use.

As for the first one, it's relatively easy to implement and there are many examples online so I will not get into that and I will focus on the second option.

If you're using an API Gateway, chances are it already has a circuit breaker that you can configure. For example: Tyk and KrakenD both have their own implementations of a circuit breaker that you can simply use. There are also other opensource services that have a circuit breaker ready for configuration like Elasticsearch. Many orchestration and integration frameworks like Apache Camel also provide circuit breakers.

Conclusion:

Failing fast is important for most systems and is crucial for realtime systems. But eventhough the circuit breaker is useful, it does affect the performance since it adds a new layer that a request needs to go through. However, in software engineering there are no silver bullets. Using microservices introduces new problems and those problems are solved by approaches like the circuit breaker that in turn affects the performance. Things can also get more complex in distributed systems.

In short, there are always tradeoffs and it's a matter of what suits your usecase and what you can afford to sacrify in order to improve other aspects in your system.

How Heroku's 12-factor app aligns with modern industry trends

Islam Elgohary — Sat, 11 Jul 2020 17:03:12 +0000

The 12-factor app is a methodology developed by Heroku and was first presented in 2011 by Adam Wiggins. It consists of 12 rules or guidelines that dictate how an app should be built and deployed to improve 2 aspects:

1- App Scalability
2- App Maintainability

Originally, The methodology was created and was a must follow for apps to be deployed using Heroku. However, it has been adopted by many other companies either by applying it to their own products or creating new tools that make it easier to apply the methodology on other products. If we watch the current technology trends and commonly used methods in building and deploying apps, we can easily see how they align with those of the 12-factor app. In this article, I will list the 12-factor app rules and how they align with modern trends.

I. Code Base:

One code base tracked in revision controls, many deploys.

What this means is that you should not use a mono repo that has multiple apps, but rather have a separate repo for each app. It also dictates that each app can have multiple deployments (ex: staging, production,...) but they must be from the same code base even if they have different commits.

I'm half-hearted about this rule as it has 2 directions:

Each repo should have only one app.
ach app should have only one repo even if it has multiple deployments.

As for the first part, I don't think it's strictly better than having a mono repo and currently, many companies have mono repos including Heroku itself according to Heroku's Software Architect Joe Kutner on Episode 409 of the "Software Engineering Radio" Podcast (The episode explains a lot about 12-factor app).

However, I think it's currently a given to apply the 2nd part. It's actually the intuitive action to deploy an app from only one repo even if you have multiple deployments of the app from different commits. Otherwise, there will be a lot of inconsistencies

II. Dependencies:

Explicitly declare and isolate dependencies

An App should never rely on the existence of system-wide tools or packages, but it should be explicitly stating and using it's own dependencies regardless of the packages that exist on the system.

This can be divided into:

Dependency Declaration
Dependency Isolation

The first part is about having the app declaring exactly the dependencies and tools that it uses. An example to declaring packages is npm which declare dependencies in package.json as for declaring tools like curl this is satisfied by using Docker by downloading all the tools you need in the image before deploying.

The second part is about isolating the app dependencies from system-wide packages so the app uses only the its dependencies and doesn't accidentially use a system dependency. Again, Docker satisfies this condition since it separates the app's environment completely from the machine's environment.

III. Config:

Store config in the environment

Config include database URLs, third-party services credentials and per deployment values. These values are mostly different according to each environment

Config should be decoupled from the app itself so changing the database for example would not require changing anything in the code rather it would be a change in the environment itself.

To know if your app satisfies this condition, ask yourself, can you make the app opensource without changing the code?

This is definitely best practice and often github would warn you if you have credentials in your repo. However, it's not uncommon to make the mistake of having config in the app itself but this rule definitely still applies to modern apps for security and scalability reasons.

IV. Backing services:

Treat backing services as attached resources

Backing services are any services the app needs to run normally like databases, caches and messaging services. What this rule states is that these services should be treated as attachables and should be easily changed without changing the code. This is like a followup to the previous rule because having the backing services configs in the environment will satisfy this rule automatically.

V. Build, release, run:

Strictly separate build and run stages

Each deployment should have 3 stages:

Build: Generate an executable from the code
Release: The build is combined with the configs
Run: The app (build + config) is run in execution environment

This rule dictates that the 3 stages should always be separated and should go only in on direction. So changing the code in runtime is not possible and any change has to be done before the build stage.

The code of a release shouldn't be changed either and each release has a unique ID so any change must create a new release with a new ID.

Finally, the run stage deploys the release.

Considering most CI/CD tools, they abide by this rule. When a new commit is merged, the CI/CD tool creates a new build from that commit and gives it an ID. Afterwards, that build can be used to create a new release for the specific deployment environment. For example, you can select to deploy build v1.0.0 to testing and according to your pipeline, this might create a docker image that has the build and the config of the testing environment. Later, this build is run on your server through kubernetes for example.

I would say this is a very important rule specially now that CI & CD are part of most projects and separating the pipeline stages is crucial to be able to easily revert to a specific build/release when needed.

VI.Processes

Execute the app as one or more stateless processes

This is a very important rule and in my opinion, this is the foundation of current deployment methodologies.

This rule states that an app should have zero or more processes (instances) and they should be stateless.

Having multiple instances means that the app can handle more requests and scale horizontally instead of vertically which is something that most modern products do. In addition, having the processes stateless, means that they can easily be disposed of and replaced by new instances (as stated in rule IX). A stateless instance doesn't expect any thing to stay in memory or disk but rather uses stateful backing services like databases. Those instances should also never use sticky sessions so a user can have their requests handled by multiple instances and should keep going normally in case an instance is destryed for any reason.

Obviously, this is how Kubernetes works, Kubernetes allows you to choose the number of instances of the app to run and it manages stopping, restarting and replacing the instances automatically expecting them to be stateless.

VII. Port Binding:

Export services via port binding

This rule states that the webserver should be part of the app.

As a young developer, this rule is very confusing to me and I didn't get it at first. After some research, I learned that previously, the app didn't have to be itself the web server. Rather the app could be deployed somewhere and its web server somewhere else where the web server is responsible for routing to the app. I don't really think this is relevant today since the standard now is implementing the web server into the app itself. However, maybe that was influenced by this rule and that's actually why this technology is no longer used.

VIII. Concurrency:

Scale out via the process model

Another very important rule relating to the use of microservices.

This rule states that a program should be divided into multiple processes where each process has it's own functionallity. The 12-factor website explicitly states that this is different from in-app concurrency like running multiple threads inside the app. But the concurrency model should follow the "unix process model for running service daemons" where the processes are explicitly created and visible to the developer as processes are first class citizens in the sense that each process should be managed, controlled and configured by the developer (or at least the developer is aware of them) and not internally by the app.

Obviously, this is the fundamental concept of microservices architecture which has not been formally given the name "microservice" until later in 2012.

IX. Disposability:

Maximize robustness with fast startup and graceful shutdown

Another crucial rule that aligns with modern deployment methods.

This rule state that:

Processes should be disposable. (which we established in rule VI)
A process should always be ready to gracefully shutdown without breaking the app.

This means that each process should be ready for any unexpecting shutdown wether its a SIGTERM or a hardware failure. For example, if a process receives a SIGTERM it should have a mechanism to delegate it's current job to another process like putting it back to the queue so when another process is instantiated to replace it, the job can be finished.

Again, Kubernetes runs expecting processes to do that and this is why processes can be restarted/replaced at any time and why changing the release version of an app is now as easy as running one command (or even use a GUI) to select the release you want and Kubernetes can handle the rest.

X. Dev/Prod Parity:

Keep development, staging, and production as similar as possible

This one is self explainatory. You should always strive to have the development, staging and production environment as similar as possible. This is mainly to make the app ready for CD and to achieve that, you need to minimize 3 gaps between the environments:

Time Gap: The time between development and deployment to production. This is fixed by minimizing the time between development and deployment.
Personnesl Gap: Development and Deployment are handled by 2 different persons (Developer and DevOPs). This is fixed by involving the developer in the deployment process.
Tools Gap: Developer might use lightweight tools like (SQLite) while production uses more sophisticated tools like (MySQL). This is fixed by...well, using the same tools for development as in production.

This has become a lot easier now with CD tools that allow developers to handle deployment as soon as their changes are done. Using the same tools in all environments is easier now with packaging systems like homebrew and of course with docker where you can use any service you want without going through setting it up.

XI. Logs:

Treat logs as event streams

This rule prohibits writing or managing log files, rather it encourages treating logs as events in the sense that they are actually time-ordered events. So the app should just output these events as a stream through stdout and they should be handled outside the app.

Of course this is very relevant today given that processes are disposable and distributed which makes sence to aggreagate logs from different processes in one place.

Services that help with that provide that include Logstash and loggly.

XII. Admin Processes:

Run admin/management tasks as one-off processes

An example to these tasks are DB migrations. The rule states that these tasks must:

Run in the same release environment as the deployment
Be part of the app code even if they are a one-time script

This focuses on the consistency of the app where you shouldn't just ssh to the server and run commands there. Rather, you should create scripts that are shipped with the code to make those changes.

If you have ever used an ORM to manage migrations, then you will notice that it follows these rules as the migration scripts are stored in the project and can be run in multiple environments with the same consistency.

Conclusion:

The 12-factor App ,despite being almost a decade old, is mostly relevant and it has obviously changed the way of implementing and deploying apps and possibly influenced the tools that might be considered as the standard in the current software industry.

Some Resources:

Official 12-factor app website
SE-Radio Episode 409
AWS Blog: Applying the Twelve-Factor App Methodology to Serverless Applications

How Javascript handles Async code

Islam Elgohary — Fri, 08 May 2020 00:56:55 +0000

When I first started learning Javascript, the concept of async code was very hard to grasp for some reason and I had to read a lot and watch a lot of videos explaining how async code works on Javascript. What made things harder, was that Javascript is single-threaded so I couldn't understand how a single threaded language had the ability to run async code. In this article, I will share my knowledge about async code and how Javascript handles it.

Async programming has been around since early stages of computing And it has been heavily used in the past few years and it's currently in almost all websites and applications as it's now considered the default programming paradigm to obtain high performance and low latency. Before explaining how Javascript handles async programming, let me briefly explain why async programming is important and what exactly it is.

What is async programming?

Async programming refers to code that does not stop the main process of the program (non blocking code). Some parts of the code do not need to stop the main website/application process to execute. For example, Imagine a blog where users can do CRUD operations on blog posts where the blog posts are saved in MYSQL database. Imagine that the blog doesn't use async programming. Here is a possible scenario:

1- User A is creating a post
2- Now the program is busy writing the post to the database
3- At the same time, user B wants to read another post
4- Since the program is currently writing to the database, user B will have to wait for user A's new post to be written in the database before being able to read another blog post.

Now let's see the same scenario if the database operations are async:

1- User A is creating a post
2- The database write operation does not stop the main thread but instead, it's scheduled to run some other time in the future and the program can continue running normally.
3- At the same time, user B sends a request to read a different post.
4- Since the program is not busy, it can handle user B's request immediately.

Note that async is not the same as concurrent or multi-threading. As we will see Javascript runs async code but it is single-threaded.

How async code is modeled in Javascript?

Javascript uses Promises, promises are given that name because they promise to finish execution at some point in the future, however, they do not guarantee when they will finish running. In Javascript, async code is mostly I/O and network requests but most async libraries have corresponding sync APIs and normal code can also be written as promises or you can convert other async libraries APIs to be Promises with libraries like bluebird.

A promise is usually given 2 functions, one called in case of success of the promise and another called if the promise fails. Those functions are called callbacks, however, callbacks have problems like scope changes that made it hard to use (check Callback Hell for more). That's why Async/Await syntax was eventually introduced. You can learn more about promises Here.

An example of async code is the famous setTimeout function that takes one callback function and run it after a specified period of time.

How does Javascript handles async code?

As mentioned above, Javascript is single-thread. This means that the program cannot run the async code on multiple threads which means that it cannot run multiple processes at the same time. If that's the case then how does Javascript handle async code with a single thread?

The thing is that the async API used in Javascript doesn't run in the Javascript runtime but they are actually APIs exposed by the system running the Javascript program (web APIs for frontend developers or C++ for backend developers). So the async code is actually deferred to the system running the Javascript runtime. But How does the runtime know that the async code finished running? Here comes the Event Loop and the Callback Queue.

When an async function is done execution, the callback function is pushed to the Callback Queue. Once the Javascript runtime stack is empty, the Event Loop gets the function from the Callback Queue and adds it to the runtime stack and thus the function is executed by the Javascript runtime.

This is how Javascript has async code while being single-threaded. I hope this post was useful and I recommend watching this video for live examples as this helped me a lot when I first started. You can also read This and This to learn more about the Event Loop.