Forem: Ez Pz Developement

Notes about Chapter 02 of Web Scalability For Startup Engineers

Ez Pz Developement — Fri, 09 Aug 2024 21:54:16 +0000

Based on my understanding of this chapter, the author is expressing the idea that sometimes the need to scale forces us to break good design principles. However, it’s important to first understand these principles.

Some Good Software Design Principles:

This is a list of the good design priciples that a software engineer should understand:

Simplicity:

Keep it simple, but not too simple. How simple should we make things? We need to consider for whom we are designing and the delivery deadline.

Simplicity is not how fast or quick can we implement a solution , but how easy for another software engineer to use our solution, and being able to understand the system as it goes larger and more complex.

To implement this a software engineer need experience with different tools and languages, he also need to revisit his old solutions review them and try to fix them, finding a mentor or working with people who value this principles will make us progress faster.

Below is the steps to make our solutions more simple:

Hide complexity and build abstraction: we should be able to achieve the local simplicity, so if when take it is easy to understand how it works , zoom int out to modules and then to an entire app must be the same case. Complexity is about how many dependencies a single component like (class, module, …) have on the other components. We need to worry about how our components interact instead of how they fulfil their duties once we start seeing a bigger picture, in larger systems we can add services where each one is responsible for a specific functionalities and showing a higher level of abstraction.
Avoid Over Engineering: Do not spend time building over complicated and imaginary designs that no one uses, we should care about simplicity and most common scenarios,we should think about tradeoffs and if we will really need this.
Try TDD : allow us to reduce the amount of useless functionality, and act as a doc for our code by showing the expected results and behavior and the code function. Tdd allows us to do mental shift where we start thinking about how this component is going to be used by other components first rather than implementing the internal logic
Learn from models of simplicity: check other good software and learn from there design.

Loose Coupling:

We should keep coupling as low as possible between components, coupling is how much two components depend on each other , the less coupled the components the less they know about and depend on each other, a no coupling mean that two components does not know about each other.

Keep the low coupling is important for out ability to scale.

The higher the coupling the higher the number of changes/bugs that we might add to other components that depend on the internal implementation of specific component.
We will be able to hire more engineers as they don’t have to know the full details to work on specific parts of the system.

Promoting Loose Coupling: we should carefully manage dependencies between modules classes and applications.

Applications are the highest level functions, you might use an application for accounting, asset management or file storage.

Each application will contain a set of modules (for example: pdf rendering, credit card processing, portal document) that another team member can work on independently, if we can not do this the application might have some tight coupling problems .

Each module consist of classes , a class is the smallest unit of abstraction, we should keep our functions private or protected as much as possible, the less knowledge other classes know about our class the less they are aware how the class does the job, private functions can be refactored easily as they are called only in the class , in case of other protected and public functions we should search the code before refactoring them.

We should share only the minimum information and functionality that satisfies, sharing too much in early increase coupling.

How to avoid unnecessary coupling:

Hide as much as we can and avoid using getters and setter without needing them, as these getters and setter was introduced to provide a good ide support.

When client need to call methods of class in a certain order for the work to be done.

Do not allow circular dependencies between layers, classes and modules, in diagrams the relations between our components should look like a directed graph more.

Models of lose coupling:

A good example of loose coupling is the unix program where the commands like grep, sort, awk can be combined to perform a more complex tasks.

Simple logging Facade for java (SLIF4J), it act as layer to hide the complexity of the logging from the users

Read books that discuss this subject.

DRY:

Things that we should avoid to ensure that we are applying this principle

Following inefficient process: we should always try to get feedbacks, apply continuous improvements, incremental change, and repeat, we should not have the mentality of “we always did it this way” or “ this is how we do it”
Lack of automation: waste time deploying manually, configuring servers, writing documentation , and testing, this tasks can be simple at the start but it is going to be hard and time consuming when software get more complex.
Not invented here, reinventing the wheel: building things that already exists which waste our time.
Copy past programming: having a code that does the same things in a part of the system so we just copy that code and use it , we will face problems like bugs occurring in multiple parts , we can add a rule that we never copy past code.
I wont need it again so let’s just hack it quickly: we might work on a code or project quickly thinking that we will never need to go back to it again , but a problem occur and we have to go back to work on it , we will find a messy and inefficient , not tested , and unmaintainable code waiting for us, we should practice refactoring , inheritance, composition, and design patterns

A white paper by Nasa show that 10 to 25 percent large systems code is a result of copy past programming, In a higher level of abstraction we can create a common service that can be used by other services across the system.

If the usage of a library or a component is easy everyone will use it , if not they will not and we might end up with a duplication or hacks.

ِConding To Contract

Coding to contract or to interface You discuss things that are allowed to the client to see and expose only what the client need .

Contract: is a set of rules that the provider of functionality agrees to fullfile , and the client will depend on , but without knowing how this functionality is implemented, as long as we keep it intact, client and providers can be modified independently.

When designing the code we should create explicit contract , we should depend on the contract whenever is possible instead of implementation details.

We should think of the contract as a legal documentt, in legal document we should be more details oriented , because if our contract does not cover all what we want (in case of software if our contract expose too much details) we need to renegotiate every change with our client.

When we start building systems we should first define what features our client need and then expose the minimum details to achieve what he want.

Http is a good example of coding to contract because it gives the possibility for different application to communicate based on a specific interface(decoupling), things like web browsers, cache sever(Varnish), web server (nginx, apache) can communicate between each other and depend on the same contract( an example of this can be found in the figure 2.5).

Draw Diagrams

Diagrams are worth a thousand words, even when we don’t have so much time we should take time to design the architecture.

If it is difficult for us to draw diagrams we can follow this approach:

Draw diagrams of what we have already built , once we get comfortable with diagrams
And then we start drawing diagrams while coding and working on certain features
And then we start trying to do an up front design (design first and then code last)

We might want to design a circuit breaker component , is a design pattern that will prevent our system from falling by first checking if certain system is available before doing an action

So we can do the following to design it:

1- Create a draft of the interface (Listing 2.1)

2- draft the client code, can be a unit test or just some client code that does not have to compile.(2.2)

3- a draft of sequence diagram

4- a draft of class diagram

With this approach we can see the design from different angles , and avoid doing an unrealistic design.

We have three very important diagrams: use case diagram, class diagram, and module diagrams

Use case diagrams:

They show the users of the system and the operations they perform, they show the business requirement. Can also show interaction with other systems like apis or task scheduler.

We should keep it simple to we can maintain readability and maintainability.

Class diagram:

Are the best to show coupling between classes by simply watching how many dependencies a node include, and show the module structures and the relations between its classes , interfaces.

Interfaces should always depend on interfaces, never on concrete classes.

Classes on the other hand should depend on interfaces as much as possible.

Module diagram:

A module diagram is a zoom out of class diagram, show the interaction between modules. Can be a package or any logical part responsible for a certain functionality.

Module diagram focus on certain functionality that are relevant to the functionality that we want to document; when a system grow larger it is better to create a few separate diagrams to keep simple; easy to remember, and easy to recreate.

Single Responsibility

Reduce complexity, make it simple…

Some guidelines to promote the single responsibility:

Keep class below two to four screens of code

Ensure that our class does not depend on more than 5 classes or interfaces.

Ensure that class has specific goal and purpose.

Summarize the responsibility of class and put it on the top of class , if you find it hard to summarize this mean that we are breaking the rule.

An example of this is if we are adding an email verification feature to our software, we could add the verification to the code that create the user but this will make the code more complex and we will not be able to re use it in another code again, separating the logic of validation in a separate class will solve this.

A good way to learn more about this is to explore more about design patterns( strategy, iterator, proxy, and adapter), and learn more about Domain driven design.

Open Closed Principle:

Everytime we write a code with the intent to extend it and not modify it later we are using this principle, classes should be open for extension and closed for modification.

With a main reason of increasing flexibility and make future changes cheaper.

An example of this is when we have to implement a sorting algorithm , with a feature to sort employees, and we implement the solution inside a class called SortingEmployees with a sort method, this will cause problems if we want to do the same thing for Cities, we will be left with two dirty solutions , we either extend the SortingEmployees , but sorting Cities does not have to know about SortingEmployees , or to copy past code from sorting employees class and past it in SortingCities.

A solution here is to break the problem into smaller ones , by creating a sorter and Comparator interfaces , and then the new class SortEmployee will implement the Comparator interface and Comparator will have an instance of Sorter inside of it.

MVC framework is a good example of this and specially spring framework , if framework is well designed you don’t have to update the framework code to implement features but you just extend a component and create another one based on it . in spring most of classes does not even have to now about the existing of spring mvc framework.

Dependency injection

Reduce coupling and promote the open closed principle.

Reference the objects that the class depends on, it does not allow the class to know about the referenced object implementation details, or how they are assembled.

Dependency injection switch from the principle of letting class inherit other classes ( a pull approach) , to adding the objects directly into the class (a push approach) to decouple the class from dependencies and make it easier to test.

To understand more an example of a reader and cd can be used, fig 2.14 and 2.13 show examples of this

Dependency injection make the class responsibility less and make it dumber, make it simpler.

Without the need to know the contract of the injected object the class can focus in it’s own responsibility

Inversion of control

DI is included here , this is a larger principle that can be applied every where in all the levels of abstractions.

IOC is removing some responsibilities from the class to make it simple , and less coupled to other parts of the system

You don’t have to know who will use or create your objects, how or when.

It is used in a lot of frameworks , IOC look at requests and figure out which classes should be initiated and which services and components they depend on. (requests contain data like the url, headers and cookies that the ioc framework will use)

Can also be called as the “we call you, you don’t call use principle” the classes does not have who is using them, when their instances are created, or how their dependencies are put together, the classes become like a plugin.

Using frameworks will reduce the local complexity of our app.

The factors of an IOC framework ( FIG 2.16)

You can create plugins for your framework
Each plugin is independent and can be removed or added at anytime
Framework can auto detect these plugins , or there is a way to configure which plugins should be used
Your framework define the interfaces for each plugin and should not be coupled to a plugins themself.

IOC framework , is like a having fishes in tank , you decide how many fish we want there, and we decide when to feed them , so the fish is the plugin and you are the ioc framework.

Design for scale

A difficult thing to master, we should be careful to make a balance with designing to scale and overengineering.

Most of the startup fails and never need to scale (90%) , the other 9% will not need to really go for a horizontal scalability, only the 1%

Similar to coupling and complexity principles discussed above , scalability problems can categorized into:

Adding more clones: adding indistinguishable components
Functional partitioning: Dividing the system into smaller subsystems based on functionality
Data partitioning: keeping a subset of the data in each machine

Adding more clones While building a system the easiest way to scale it is to design it to be able to add more clone; a clone is the exact same of a current server or component where if we send a random request to any clone server we should get the same result. Figure 2.17 and 2.18.

We need to pay attention to where we keep state and sync it between the servers.

Scaling with clones work best for stateless servers or services (services that does not have any local state)

The problem with this scaling technique is in syncing data between stateful services.

Functional partitioning: The main idea is to look at the parts that work on the same functionality and create a separate subsystem of them.

In term of infrastructure it is the separation of our data center into multiple server types for example: Message queue server, cache server, Webserver, load balancer…

It is dividing the system into independent services, it help us allow the coding to contract principle , it is often used in web services layer , and one of the basics of service oriented architecture , this strategy also allow us to analyze and get the needs of each services independently and scale them separately.

It is also common to break the app into a database service layer and web service layer.

It is common in large companies to separate the app into a smaller independent services , where each team can work on a service separately analyze it and try to scale it.

Drawbacks of this approach is that it require more management and effort to start with, We can not keep rewriting our system and divide it endlessly , it might not solve our scalability problem there might be some other problems like architecture or optimization issues.

Data Partitioning Using a manifestation of share-nothing principle, where each service has it’s own subset of data, he have a complete control of his internal state without the need to sync state with others, no need for locking( locking is when we have multiple server trying to access the same resources and modify the data we need to handle concurrency to avoid data corruption).

Design for self healing

And because our systems might fail at any moment we should design them with high availability and self healing in mind.

We want to make our system always available for our users , even when experiencing partial failure or during maintenance (a system is considered available as long as it perform it’s functions as expected from client prospective).

There is no specific measurement of availability, but it can measured as the numbers of nines , if a system is available 99% of time it means which means that he is going to be out for 365 days * 0.01 = 3.65, and if we do 99.999% which make it available only 5 min a year.

The larger the system get the higher the chance of failure because we will be communicating with more services/data stores and components when the system get bigger, failure need to be considered as a norm during design not as special condition.

NEtflix use a system called chaos monkey , the system cause the failure of some component , so the team can test the availability of there Software.

Crash only: means that whenever a system failed and then start operating again he must be able to detect failure and fix the broken data

To ensure high availability of a system we should remove the single points of failure, and to ensure graceful failure, which means that our system should switch to a backup/clone component without impacting the user or cause loss/corruption of data.

We can draw a diagram of all our system components and ask ourself if we shutdown one service what could happen, we can then discuss the possibility of adding redundancy and see if it is going to be cheap or not, we also prepare a disaster recovery plan.

If we achieved the a high level of availability and handling graceful failure , we can start thinking about designing a self healing system, who can fix his issues without the need of human interactions, this is hard and expensive to build.

An example of a self healing system is Cassandra when a node failed the server stop requests that are coming to this node (this is the only stage where users might experience some failure or downtime), once the node is detected as failed the client continue reading data from other nodes in the cluster that provide redundancy of the failed node, when the failed node is back to work the system automatically provide it with the missing data

Mean time to recovery, measure how fast we can detect, repair and recover from a failure, the higher the availability of our system and it can be measured with this equation mean time to failure / (mean time to failure + mean time to recovery).

Summary:

The cleanest solution is not always the best solution for the business if it costs more time , money , and need more management, we need to think and make the best possible solutions for the business, we need to make tradeoffs in term of scalability, flexibility, high availability, costs, and time to market.

Don’t hesitate to challenge the rules, but it’s essential to understand the tools, basics, and principles of our craft first. This way, we can make informed decisions and balance tradeoffs effectively

Notes about The Chapter 01 of Web Scalability For Startup Engineers

Ez Pz Developement — Fri, 14 Jun 2024 22:46:44 +0000

Notes about The Chapter 01 of Web Scalability For Startup Engineers (Image by Herbert Austfrom Pixabay)

Last week, I wanted to learn more about system design and scalability. So, I started by watching YouTube videos and reading LinkedIn posts to get a general idea. I also looked into book reviews, and it seems like “Designing Data-Intensive Applications” is a popular choice among many people.

I wanted to begin with something simpler since “Designing Data-Intensive Applications” covers a lot of detailed topics. After talking to a friend and hearing recommendations from others in our field, I found out about “Web Scalability for Startup Engineers.” This week, I started with the first chapter, which gave an overview of the topics the book will cover.

Notes:

What is Scalability:

the ability of our system to handle more data, request, users and transactions , we must be able to scale up and down in a cheap and quick way.

Scalability Dimensions:

Handling more data : storing more content, with popularity if data analytics and big data this play an important role.
Handling higher concurrency levels: how many users (open connections, active threads, messages being processed at the same time) can our system server at the same time, how can we solve the problems of servers having few processing units, ensure parallel execution of code to ensure data consistency.
Handling higher interaction rates : how many times clients exchange communication with the server , must be able to handle responses quicker, faster read and writes and higher concurrency.

Scalability vs Performance:

Scalability is related to performance; Scalability determines the capacity to handle more users, Performance refers to how swiftly the system handles requests under load, such as the speed at which it can respond to 100 user requests every 5 seconds.

Scalability is also implied when it comes to team members, the more people a team the harder the communication.

DNS:

Is usually hosted in a different server , the customer connect to it get the dns server get the ip address of the domain and then he start requesting content from our server.

VPS:

Is virtual machine for rent; hosted with other virtual machines in a one physicall machine, this aproach is not good

Single Server configuration:

this option is better but we might switch if:

Our user base grow, and it will take more cpu, i/o to serve our users
Database grow we added alot of data , queries will take more time to execute so we need more cpu and I/O power.
Added new features which are going to make users interact more with the system, and this will need more resources.

How Can We Scale Vertically:

1- Adding more I/O capacity by adding more hard drives in RAID arrays:

RAID arrays: is a set of hard or solide state drives linked together to form a one logical storage, to protect data in case of failures, here is some popular RAID config used:

RAID 0( None drive can fail) : the data is split evenly between two drives
RAID 1( 1 drive can fail) : mirroring, at least two drive have the exact copy of data , if a disk fail others will be able to continue working
RAID 5( 1 Drive failure): Stripping with parity, requires the usage of 3 drives, splitting the data across multiple derives , but also has parity distributed across the drives
RAID 6( 2 drives failure) : Stripping with double parity, similar to RAID 5, but the parity is written in two additional drives
RAID 10 ( up to one drive in each array ): combine RAID 1 and RAID 0, mirror all data in secondary drives, and use stripping acrros each set of drives to speed up data transfers

Parity: is a calculated value stored in a drive from other drives in the array , used to reconstruct the data in a case of failure of one of the drives.

2- Improve I/O access time by switching to solide state drives (SSD):

SSD is fasster than hdd , but for databases it is not the case because alot of databases like mysql are optimized for sequentiel read , and databases like cassandra goes further and use mostly sequentiel read.

Sequentiel disk operations is like reading a disk page by page, while radoù disk I/O is like picking a random page in a book each time, ssd is so much faster than hdd when it comes to random , and also sequentiel because of the lack of physical head.

3- Reducing I/O by increasing RAM:

More caching space, more memory for the app to work in , better for DB because they cach frequently accessed data in ram.

4- improve network throughput:

by upgrading network interfaces, upgrade network adapters , upgrade providers.

5- switching to a more powerful server:

a server with more virtual cores, server with 12 or 24 threads( virtual cores), processes do not have to share cpu, cpu will perform less context switches.

Vertical scalability is a simple approach because we dont have to rearchitect anything we just need to upgrade our hardware and this come with cost.

OS may prevent use from scalling vertically also, in some databases increasing CPUs won’t do any improvements because of increasing lock contention.

Locks: are used to sync access between threads to a specific resources like memory or files, lock contention happen when a one lock is used for a big resources that has a lot of operations, to solve this fine grained locks must be introduced which create a much more specific locks for each task in the resource and allow threads to access the resource more effitiantly. Therefore adding more cpu when lock contention happen does not have any significant impact.

We should Design app with high concurrency in mind, so when we add more cores it is not going to be a waste.

Isolation Of Services:

Is the separation of services (db, ftp, dns, cache, web server ) into multiple phsysicall servers, when we do this we have no room to grow.

Deviding a system based on functionality to scale is called functional partitioning, for example dividing the admin service and client service in multiple physicall servers.

CDN: content delivery network , a hosted server that cares of the global distribution of static content like js, css, images ,videos, it works as an http proxy, if a client need to download a static content the cdn check if he has the content if not he request it from the server and cach it , and other clients will be server from cdn without even contacting the server.

Horizontal Scalability

It has to be considered before the application is built,systems that are true horizontally scalable does not need a powerful machine, they usually run on multiple cheap machines, and you can add as much servers as you want, you avoid the high price of buying top tier hardware , and vertical scalability ceiling problem( no more powerful hardwares).

Scaling with multiple data centers in case of having a global audience is important as it provide protection against rare outage events , and client in other countries can get a response faster.

Scaling horizontally with webservers and caches, is easier than scaling persistence stores and databases.

The usage of a round robin dns is the choice if we use multiple web server, this will distribute traffic between servers, what round robin dns does is that he just get a domain name and allow use to map the domain name to multiple ip addresses, when a client send a request the round robin dns map the client request to a server , a two clients might connect to different servers without even realising

A data center infrastructure:

Frontline: first part that user devices interactwith, does not cotaine any business logic, help us, can exist in or outside of our datacenter,
First client send a request , geoDNS to resolve the domain name and send the closet load balancer ip address ; then it get distributed to frontend cach server or directly to frontend app server
CDN, Load balancers, reverse proxies , can be used and hosted by third parties,

Load balancer: is a hardware or software that allow the addition and removal of servers dynamically, it also help in distributing traffic to multiple servers.

Reverse proxy: is an intermediate between client request and the actual server , can be used as a load balancer , or for web acceleration where compress inbound and outbound data and also cache content and also do the ssl encryption which boost main server performance (he have proxy taking care of side tasks for him), can act as a server who preserve the anonymity and encapsulate how the real internal structor look like because clients can access our data center or internal structor from a one record locator or url.

Edge cache server: is http cach server located near customers, can cache an entire http request , it can cach an entire page, or partially cache it and delegate some other parts to the server, it can also decide that the page is not cachable and delegate the entire page to the server.

A single data center can scale using edge cach and cdn, and It is not necessary to use alot of components and technologies to scale , instead we should use what is necessary

The application architecture should not involve around technologies (programming languages, databases) , it should focus on the domain model to create a mental picture of the problem that we are trying to solve.

The Frontend Must be kept as dumb as possible, and allow it to use message queues , and the cache backend , caching the html page along with the database query is more efficient than just caching the database query, Web services is critical part of the application as it contains the most important parts of our business logic.

Servers might have job processing servers or job running on schedule, with the goal of handling notifications, order fullfilement or some other high latency tasks.

SOA: service oriented architecture focused on solving business needs , where each service has very clear contract and use the same communication protocols.

SOA has some alternatives; layered architecture, hexagonal architecture, event-driven architecture.

Multi layer architecture is a way to represent functionality in a form of different layers, components in lower layer expose functionality to upper layer , lower layers can never depend on the functionality of a top layer.

In a layered architecture, having richer features in a specific layer usually leads to a more stable API. Conversely, simpler features may result in a less stable API, as changes to lower layers’ APIs can be costly due to dependencies from many other APIs.

Hexagonal architecture assume that the business logic of the app is the center of the app , there is a contract between the business logic and non business logic component , but no layers, the main reason of this is that we can replace any of the non business logic component at any time without effecting our core app.

Event-Driven Architecture (EDA) shifts the focus from responding to requests to handling actions. It works by creating event handlers that wait for an action to occur and then react to it.

In all the architectures dividing the app into smaller units that function independently will show a performance benefits, we can think of these web services as a an autonomous applications where each one become a separate app, each app hide it’s implementation details and present a high level api.

Message queue, app cache , main datastore, … , we should think of them as plugins that we can replace at any time with another technology.

Isolating third party services is a good for us as we dont know if they are scalable, or if we have full control on them , so isolating them and make it possible to replace them later is beneficial.

Create A Vim Plugin For Your Next Programming Language, Indentation and Autocomplete

Ez Pz Developement — Fri, 14 Jun 2024 22:45:59 +0000

In the previous post, i discussed how to structure our vim plugin, and how we can add the beautiful syntax highlight feature.

In this short post, i will explain how to add a simple auto-complete in addition to indentation, if you want to see a full example of a working vim extension for a new programming language, please check this GitHub repository

Indentation

So first we need to create a new file in ftplugin directory, if you want to know more about autoload and other vim extension folders make sure to check the first part of this blog post here

The name of the new file is iop.vim, we will show a short example of how we handled indentation in our extension, you can check the full file here

:inoremap { {<CR><tab><CR><bs><bs><bs><bs>}<up><tab>

Essentially, we are telling our IDE that if a user opens a bracket and then hits enter, and of course, when we press enter, we will jump to a new line with this command above the cursor will not begin at the beginning of the line but will left some space behind, the space here is represented by the .

In our case, we defined the space in the fifth line of our file as you can see below

set tabstop=4

Autocomplete

It is time to discuss autocomplete, so we can end this blog post.

You will need to create two new folders text and autoload in the text folder we will put all the possible worlds in our new programming language, in our case we created 4 files, one for decorators, another one for types, values, and iop.text file is for identifiers.

These four files are imported in autoload/iopcomplete.vim file, this last-mentioned file contains the autocomplete logic you can check the full file here at this link .

This was just a general description of what we did to create a new simple vim extension, for this new programming language, you can understand more by reading the code in theGithub repository.

Create A Vim Plugin For Your Next Programming Language, Structure, and syntax highlight.

Ez Pz Developement — Fri, 14 Jun 2024 22:09:33 +0000

Vim is a text-based editor, opensource, it is also an improved version of the old vi UNIX, Vim has so many features including multi-level undo, syntax highlighting, command line history, on-line help, spell checking, filename completion, block operations, script language, etc.

If we want to talk about compatibility, Vim runs under MS-Windows (XP, Vista, 7, 8, 10), macOS, Haiku, VMS, and almost every OS based on UNIX.

In today's post, I would like to show you how to write your own vim extension for a new programming language, I wrote this plugin with the help of my two coworkers Imen and Djamel.

Introduction

First, let me introduce you to IOP (Intersec Object Packer) it is a method to serialize data in different communication protocols, inspired by Google Protocol Buffers, IOP syntax looks like D language syntax, and all of this is according to IOP official documentation.

The Structure of a Vim Plugin

When we start working on our extension, we will create a root folder under the name vim-iop, which is exactly what we picked as a name for our vim extension.

This directory will contain three other important folders which are :

autoload: is a technique for blocking the loading of your plugin’s code until it’s required, in our case, we will implement the autocomplete feature in this folder.
ftdetect: or file type detection, has a clear purpose to figure out what file type a given file is.
ftplugin: contains scripts that run automatically when vim detect a file opened or created by a user, in our case this file will contain the logic to implement indentation.
scripts: contains a script that implements syntax highlight.

Detect File Type

In this section, we add the code to set file type for IOP files, but first, our root folder vim-iop must look like this :

vim-iop
------- ftplugin
------- ftdetect
------- syntax
------- autoload

In this part we need to create a new file ftdetect/iop.vim, add this code below to it:

" ftdetect/iop.vim
autocmd BufNewFile,BufRead *.iop setfiletype iop

Syntax Highlight

In this section, we will write some vim script in addition to some regex, so we can add the syntax highlight feature to our Vim extension.

Before we can start coding i want to mention that IOP has Basics types which are : int, uint, long, ulong, byte, ubyte ... and more, plus four complex types struct, class, union, enum , if you want to learn more about this types make sure to check this link.

So for the code below, we need add the logic to highlight the IOP types mentioned in this part of IOP documentation.

"syntax/iop.vim

**_syntax keyword_ iopComplexType** class enum union struct module **nextgroup** =iopComlexTypeName **skipwhite**
 **_syntax keyword_ iopBasicTypes** int uint long ulong xml 
**_syntax keyword_ iopBasicTypes** byte ubyte short ushort void 
**_sytanx keyword_ iopBasicTypes** bool double string bytes

" complex types name
**_syntax match iopComlexTypeName_**" **\w\+**" contained

as you can see we have iopComplexType , iopBasicTypes both of these variables contain the different complex and basic types of IOP, we also want to tell our extension that each complex type is followed by a name and that white space should be ignored, after this, we need to tell our vim extension to highlight this types by adding the code below in the bottom of syntax/iop.vim.

"syntax/iop.vim
**_highlight link_**  **iopComplexType** Keyword
**_highlight link_ iopBasicTypes** Type

in the end and after adding this extension to our vim ide, we will see something like this.

IOP syntax contains decorators also, we are going to write some regular expressions in order to highlight this, so just add the code below to our syntax/iop.vim file.

"syntax/iop.vim

syntax match iopDecorator /^ **\s*** @/ nextgroup=iopDecoratorFunction
syntax match iopDecoratorFunction contained / **\h** [a-zA-Z0-9_.]*/

In the first line of the code above we are telling our vim extension that this decorator can start with a zero or multiple white spaces followed by an “@” ( /^ \s *@/) , and the nextgroup keyword means that after “@” there is a name of this decorator, the name of this decorator can contain all alphabet letters whether it is upper or lower case, this decorator name can also contain numbers and the two special characters “_ “ and “.”.

After telling our vim extension to highlight the decorators.

"syntax/iop.vim

highlight link iopDecoratorFunction Function

This is an example of what we will see in our vim IDE.

if you want the complete implementation of vim-iop syntax highlight make sure to check this link.

That's it for now, in the next post, i will show you how to add autocomplete and indentation.

References:

Ivy the Angular Compiler.

Ez Pz Developement — Wed, 26 Jul 2023 22:42:19 +0000

I'm writing this blog post to summarize what I have learned about Ivy, the angular compiler, after reading some blog posts and watching talks by different software engineers and developers. Some of them are in the main team that works on angular.

A compiler for a frontend framework why?

When I first heard about the Ivy Angular compiler it was something weird that I could not understand, after doing some research I found out that there are a lot of other compiled frontend frameworks like Svelte, and Solid.

Based on what I have learned, we can filter these frameworks based on 3 factors which are:

Compiling approaches: some frameworks compile everything , while others require almost no compilation , the last mentioned approach makes the framework do most of the work on runtime, while some other frameworks use a hybrid of these two approaches.
The language used in compilation logic: some frameworks use Javascript or Typescript as logic for their compilation process, while others use Js alternatives like ELM or MINT.
Templating languages: some use an HTML-first approach, according to Ryan Carniato the language treats the source file as an enhancement of HTML, there is also another type of templating language which is JSX.

Another reason why it is a good idea to have a compiler for a frontend framework is the performance that comes with it, it produces more optimized and faster code, a faster dom-manipulation and startup when running compilation, and it also reduces the bundle size.

In addition to the last-mentioned cons, a compiler optimizes and reduces the possibilities for a framework at runtime.

No one can argue that even with all of this optimization, a framework can not be faster than vanilla js, but it is less verbose, and it is easier and faster to write and create complex applications using a framework than using Js alone.

Ivy compiler as an example

Ivy is another rewrite of the Angular compiler and runtime to achieve a better build time, and bundle size, and maybe to get rid of the zone.js when it comes to change detection, and more optimization in the generated code by using some principles like :

Tree shaking: the process of removing unused and unreferenced code using static analysis
Locality: an approach to make build and compilation go faster by compiling each component independently.

So the main goal of Ivy is to turn these templates which use the declarative coding approach and turn it into imperative JS/TS code, this includes generating a code from the templates documents, adding a change detection mechanism, and applying the changes when they happen.

Ivy can do his job using a JIT or AOT approach, both of these approaches have some advantages and disadvantages I recommend checking this post to understand more about JIT, for AOT I recommend this one.

Angular JIT Compiles your application in the browser at runtime it was the default approach until Angular 8, while AOT run _ngc c_ompiles your application and libraries at build time, it is the default starting in Angular 9.

How this compiler work

Ivy compiler architecture was inspired by Ts compiler Architecture, while the typescript compiler has 3 steps (Program creation, type checking, and Emit), and the Angular compiler has two additional steps Analysis and Resolve.

Program creation: in this step angular tries to discover all the file resources to understand the program (app or library), starting from tsconfig.json, and then expand and discover all the other files, in addition to what typescript compiler do angular team has customized this step by adding some more features and code to make it compatible with what an Angular dev need, it is because of this customization developers can use libraries like ng factory.
Analysis: the compiler will go through all the classes in the files, find all the ones that are decorated with an Angular decorator, and try to understand each component and directive in isolation, in this step the compiler does not have an idea about which modules this components belongs to.
Resolve: try to understand everything about the library or the app as a whole, in this step the compiler will try to find each component module, do some optimization, decisions, and handle errors, he also tries to understand the structure of the project.
Type Checking: its type script time to check if there are any errors in our code.
Emit: this step is the most expensive one where the Angular compiler generates code js for each class that has an angular decorator.

Another cool feature that Ivy has is template type checking, what can this feature do is to throw an error and show you the exact place of the error in the template ( by line and columns), according to Alex Rickabaugh this was something tricky to add.

in the end, I recommend watching Alex Rickabaugh's amazing talk about the ivy compiler, to get more about ivy, and go deeper into how the Angular compiler work.

references :

My blog post was inspired by the amazing references below, I did my best to summarize what I learned from these blog posts.

https://blog.ninja-squad.com/2019/05/07/what-is-angular-ivy/

What I’ve Learned About WebAPIS, An Introduction

Ez Pz Developement — Sun, 23 Jul 2023 23:05:18 +0000

In In this post, I’d like to discuss what i learned about web APIs while working on my master’s thesis, i wanted to write and summarize what I learned in my notebook or on Google Docs, but I decided that sharing and summarizing what i learned on the Internet would be more effective, as I may receive additional suggestions or criticism, it will provide an opportunity for me to learn new things.

The following post serves as an introduction; i’ll discuss the most important things I learned and what i’ll discuss in the following posts.

as the majority of us know (developers, software engineers …) web APIs or an application programming interface is like middleware or a messenger that handles requests and guarantee the interaction between application, devices, and systems.

Another definition is that web APIs is an online programming interface of an organization or an enterprise it enables other applications to interact with its backend systems, as mentioned on the Hcltech blog.

After talking about APIs definition another important term that creates a connection between the different two or more apps via their APIs, allowing those systems to share data, and it's obvious that we are talking about API integration.

As we said in the previous paragraph “a connection between the different two or more apps via their APIs”, some of these APIs can be public and nonprofits while others are designed with a business goal and a need of other external developers, who are a critical component in reaching the targeted goal which this API was designed and created for.

An API with aim of generating revenue usually use three archetypes is according to a recent research paper that i read¹, these archetypes are :

Professional Services: provide access to their APIs, software as a services or data to a third party consumer through a standarized interface and a clear pricing plans in order to generate direct incomes from this service.

Mediation Services: bring their services to light by making them available for external or third party developers, this developers enter another party for example this party can be represented by the clients of this developers, in order to be complementary to the services provided by the organization that adopts the mediation service.

Open Asset Services: expose their services for free an access to their data or maybe more is in the hand of third-party developers, in order to increase the interaction and remove the bariers with the developers communitys

After reading and searching in the field of monetization and planning phase i started reading more about how we can develop a webAPI , its here where i learned a lot about API no-code platforms, API integration platforms, openAPI…., and alot of APIs development leaders like Mulesoft, WSO2, and Apigee which provide alot of documentation about APIs design, management and more, their documentation was very valuable for me, i will summarize and talk about what i learned from each resource and each APIs provider in the next posts.

Another thing I’d like to mention is that we’ve all heard of code smells, but have you ever heard of test smells? I first heard about test smells five months ago while looking for a scientific paper discussing how to do unit tests², tests smell this was not the only new topic I learned about; there are many others as well, for example, TDD, Continuous development, Continuous integration and the difference between CI and feature branching and how to apply this two properly from the book⁷, blog³ and the videos⁴ of Dave Farely.

Conclusion

this post was just an introduction to along series about what i learned about web apis, and as I mentioned at the beginning, I wanted to write and summarize what I learned in my notebook or google docs, but I thought that sharing and summarizing what i learned on the Internet would be better, as I might get more suggestions or some criticism that would be a reason for me to learn new things.

References:

[1] Fostering Value Creation with Digital Platforms: A Unified Theory of the Application Programming Interface Design by Jochen Wulf & Ivo Blohm

[2] tsDetect: an open source test smells detection tool by Anthony Peruma et al

[3] Dave Farley’s Weblog ,

[4] https://www.youtube.com/channel/UCCfqyGl3nq_V0bo64CjZh8g

[7] Fostering Value Creation with Digital Platforms: A Unified Theory of the Application Programming Interface Design by Jochen Wulf & Ivo Blohm

Building a Serverless Application with NestJS and the Serverless Framework: Authentication and a…

Ez Pz Developement — Fri, 21 Jul 2023 19:09:15 +0000

Building a Serverless Application with NestJS and the Serverless Framework: Authentication and a custom lambda Authorizer.

Image for the blog building a Serverless Application with NestJS and the Serverless Framework: Authentication and a custom lambda Authorizer

Table of contents:

Intro
Handling authentication using lambda functions
Enabling CORS

In the previous post, I talked about the following :

How to setup the app following a mono repo approach https://medium.com/@ezpzdev/building-a-serverless-application-with-nestjs-and-the-serverless-framework-a-monorepo-approach-5460bb86a45.
Adding API portal https://ezpzdev.medium.com/building-a-serverless-application-with-nestjs-and-the-serverless-framework-api-portal-76f6bee8a8e.

after the last blog here is what our file structure looks like

apps
  users
    src
      app.controller.ts
      app.module.ts
      app.service.ts
      main.ts
      serverless.yaml
    tsconfig.app.json
  items
    src
      app.controller.ts
      app.module.ts
      app.service.ts
      main.ts
      serverless.yaml
    tsconfig.app.json
config
  serverless.yaml
nest-cli.json
serverless-compose.yaml
package.json
tsconfig.json
.eslintrc.jsya

in this blog post I will write about how I handled JWT authentication using a lambda function

Handling authentication with a lambda function

The first thing I did was to search in the serverless framework authentication to find if there is a way to handle authentication faster with the serverless framework and this led to this blog post: https://www.serverless.com/blog/strategies-implementing-user-authentication-serverless-applications/ which contains details about how to implement authentication using a lambda custom authorizer so my plane was to do the following

check aws documentation that talks about authorization.
write a lambda function that handles login and signup.
write a lambda function that handles refreshing tokens.
and writing a function that handles authorization where I will check if a request of the client contains a valid token.

after checking the aws documentation i decided to implement a simple version of the authorizer function.

the first thing that i did was to go inside my src folder and created a new folder with the name auth and added the following lines to my new yaml file

service: auth

plugins:
  - serverless-offline

provider:
  name: aws
  region: eu-west-3
  runtime: nodejs16.x
  stage: dev
  environment:
    DYNAMODB_TABLE: authors-${opt:stage, self:provider.stage}
    JWT_ACCESS_TOKEN_SECRET_KEY: __secret_key__
    JWT_REFRESH_TOKEN_SECRET_KEY: __refresh_token_secret_key__
    JWT_ACCESS_EXPIRES_IN_MINUTES: 30
    JWT_REFRESH_EXPIRES_IN_MINUTES: 10080
    PASSWORD_SALT_ROUNDS: 10
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:Query
        - dynamodb:Scan  
        - dynamodb:GetItem
        - dynamodb:PutItem
        - dynamodb:UpdateItem
      Resource: 
        Fn::Join:
          - ''
          - - "arn:aws:dynamodb:${opt:region, self:provider.region}:*:table/"
            - ${self:provider.environment.DYNAMODB_TABLE}
  apiGateway:
    restApiId:
      'Fn::ImportValue': MyApiGateway-restApiId
    restApiRootResourceId:
      'Fn::ImportValue': MyApiGateway-rootResourceId

custom:
    serverless-offline:
        httpPort: 3000
        websocketPort: 3001
        lambdaPort: 3002

functions:
  signin:
    handler: dist/main.signin
    events:
      - http:
          method: POST
          path: /signin

  signup:
    handler: dist/main.signup
    events:
      - http:
          method: POST
          path: /signup
  refreshToken: 
    handler: dist/main.refreshToken
    events:
      - http:
          method: POST
          path: /refresh-token
  authorizer:
    handler: dist/main.authorizer

resources:
  Resources:
    Authorizer:
      Type: AWS::ApiGateway::Authorizer
      Properties: 
        Name: ${self:provider.stage}-Authorizer
        RestApiId: 
          'Fn::ImportValue': MyApiGateway-restApiId
        Type: TOKEN
        IdentitySource: method.request.header.Authorization
        AuthorizerResultTtlInSeconds: 300
        AuthorizerUri:
          Fn::Join:
            - ''
            - 
              - 'arn:aws:apigateway:'
              - Ref: "AWS::Region"
              - ':lambda:path/2015-03-31/functions/'
              - Fn::GetAtt: "AuthorizerLambdaFunction.Arn"
              - "/invocations"
  Outputs:
   AuthorizerId:
     Value:
       Ref: Authorizer
     Export:
       Name: authorizerId

let’s explain more about this file, our serverless framework configuration file sets up an AWS Lambda service, called auth, intended for handling authentication in the application.

The environment subsection includes environment variables such as the JWT secret keys for both access and refresh tokens, the duration of the tokens, and the DynamoDB table name, which is dynamic based on the stage (dev in this case).

In the iamRoleStatements, it sets permissions for the Lambda functions to perform various actions on DynamoDB. The Resource subsection constructs the ARN (Amazon Resource Name) for the DynamoDB table.

The functions the section includes different AWS Lambda functions with respective HTTP events, such as signin, signup, refreshToken, and authorizer.

The authorizer the function doesn't have an events section or a dedicated path because it's not intended to be accessed directly through an HTTP request. Instead, the authorizer function is used as a custom authorizer for your API Gateway.

An authorizer is a Lambda function that performs authentication and authorization on requests before they reach the actual service endpoints.

When an incoming request triggers an AWS API Gateway event, the authorizer function is invoked first. This function examines the authorization token included in the request’s Authorization header and determines whether the request is allowed.

The resources section of the YAML file is where you set up the Authorizer as a custom authorizer for the API Gateway. It doesn't have an HTTP endpoint because it's not meant to be invoked directly by HTTP requests but rather as an intermediate layer by the API Gateway. The AuthorizerUri property specifies the Lambda function that API Gateway calls for the custom authorization.

The authorizerId output can be imported into other Serverless services users and items as the identifier of this authorizer.

Now it is time to write code for our functions: signin, signup, refreshToken, and authorizer .

Signup:

#apps/auth/src/main.ts
export const signup = async (
  event: any,
  _context: Context,
  _callback: Callback,
) => {
  const appContext = await NestFactory.createApplicationContext(AppModule);
  const appService = appContext.get(AppService);

  const { email, password, firstName, lastName } = JSON.parse(event.body);

  try {
    const result = await appService.signup(
      email,
      password,
      firstName,
      lastName,
    );
    return {
      statusCode: 201,
      body: JSON.stringify({
        success: true,
        data: result,
      }),
    };
  } catch (error) {
    console.log(error);
    return {
      statusCode: 500,
      body: JSON.stringify(error.response ?? error.message),
    };
  }
};

This function receives an event, context, and callback, but also extracts the firstName and lastName fields from the request body. These are used along with the email and password to call the signup method of appService. If the sign-up process is successful, it returns a 201 status code along with the result of the operation. If an error occurs, it responds with a 500 status code and an error message.

Now let’s take a look to our signup service which handles the logic that does the signup.

  async signup(
    email: string,
    password: string,
    firstName: string,
    lastName: string,
  ) {
    let user = await this.getUserByEmail(email);

    if (user) {
      throw new BadRequestException('Email already in use');
    }

    const hash = await this.hash(password);
    user = await this.createUser({
      email,
      password: hash,
      firstName,
      lastName,
    });
    console.log(user);
    if (!user) {
      throw new InternalServerErrorException();
    }

    const payload = {
      email: user.email,
      firstName: user.firstName,
      lastName: user.lastName,
      sub: user.id,
    };

    const accessToken = await this.jwtService.signAsync(payload);
    const refreshToken = await this.jwtService.signAsync(payload, {
      secret: jwtConstants.refreshTokenSecret,
      expiresIn: `${jwtConstants.refreshExpiresIn} min`,
    });

    this.updateRefreshToken(user.id, refreshToken);
    return {
      access_token: accessToken,
      refresh_token: refreshToken,
    };
  }

signup(): This method is used to create a new user account.

It first checks whether a user with the given email already exists by calling (the email must be unique for each user) getUserByEmail().
If the user already exists, it throws an error. If not, it hashes the provided password using hash(), creates a new user with the hashed password, and provided details using createUser(), and creates an access token and a refresh token for this user using jwtService.signAsync().
It then associates the refresh token with the user by calling updateRefreshToken(). Finally, it returns the access token and refresh token.

Below is the code and the explanation for the 4 helper functions used in the signup method, in these methods JWT (JSON Web Token) from nestjs is used to generate tokens, bcrypt is used for hashing passwords, andAWS DynamoDB from aws-sdk is used for storing user information.


  private async updateRefreshToken(id: string, refreshToken: string) {
    const params = {
      TableName: process.env.DYNAMODB_TABLE,
      Key: { id },
      UpdateExpression: 'set refreshToken = :refreshToken',
      ExpressionAttributeValues: {
        ':refreshToken': refreshToken,
      },
    };
    const res = await this.db.update(params).promise();
    if (res.$response.error) {
      throw new InternalServerErrorException(res.$response.error.message);
    }
  }

  private async hash(password: string) {
    const salt = await bcrypt.genSalt(jwtConstants.saltRounds);
    return await bcrypt.hash(password, salt);
  }
  private async getUserByEmail(email: string) {
    const params = {
      TableName: process.env.DYNAMODB_TABLE,
      FilterExpression: 'email = :email',
      ExpressionAttributeValues: {
        ':email': email,
      },
      ProjectionExpression: 'id, email, firstName, lastName, password',
    };

    const res = await this.db.scan(params).promise();
    if (res.$response.error) {
      throw new InternalServerErrorException(res.$response.error);
    }
    console.log(res.Items[0]);
    return res.Items[0];
  }

  private async createUser(user: any) {
    const { email, firstName, lastName, password } = user;
    const id = crypto.randomUUID();
    const data = {
      TableName: process.env.DYNAMODB_TABLE,
      Item: {
        id: id,
        email,
        firstName,
        lastName,
        password,
      },
    };

    const res = await this.db
      .put({
        ...data,
      })
      .promise();
    if (res.$response.error) {
      throw new InternalServerErrorException(res.$response.error.message);
    }

    return { id, email, firstName, lastName };
  }

getUserByEmail(): This private method queries the DynamoDB table to find a user with the given email. It returns the first user that matches the email, if any. If there is an error with the scan operation, it throws an error.
createUser(): This private method adds a new user to the DynamoDB table. The user’s id is randomly generated, and the provided email, first name, last name, and password are stored in the table. If there is an error with the put operation, it throws an error.
updateRefreshToken(): This private method updates a user’s refresh token in the DynamoDB table. If there is an error with the update operation, it throws an error.
hash(): This private method hashes a password using bcrypt. It first generates a salt using bcrypt.genSalt() with the number of salt rounds defined in jwtConstants, then hashes the password with this salt using bcrypt.hash().

Signin:

Now as we are done with the signup method and all the methods that we use in it, it is time to move to signin.

#apps/auth/src/main.ts

export const signin = async (
  event: any,
  _context: Context,
  _callback: Callback,
) => {
  const appContext = await NestFactory.createApplicationContext(AppModule);
  const appService = appContext.get(AppService);

  const { email, password } = JSON.parse(event.body);

  try {
    const result = await appService.signIn(email, password);
    return {
      statusCode: 200,
      body: JSON.stringify({
        success: true,
        data: result,
      }),
    };
  } catch (error) {
    console.log(error);
    return {
      statusCode: 401,
      body: JSON.stringify(error.response ?? error.message),
    };
  }
};

This signin function accepts an event, context, and callback. It extracts the email and password from the request body. These are utilized when invoking the signIn method of appService. If the sign-in procedure is successful, a 200 status code and the result of the operation are returned. Should an error occur, it responds with a 401 status code and an error message.

Now let’s take a look at our signin service which handle the logic that does the signin.

  async signIn(email: string, pass: string) {
    const user = await this.getUserByEmail(email);

    if (!user) {
      throw new BadRequestException('Wrong credentials');
    }

    const match = await bcrypt.compare(pass, user?.password);

    if (!match) {
      throw new BadRequestException('Wrong credentials');
    }

    const payload = {
      email: user.email,
      firstName: user.firstName,
      lastName: user.lastName,
      sub: user.id,
    };

    const accessToken = await this.jwtService.signAsync(payload);
    const refreshToken = await this.jwtService.signAsync(payload, {
      secret: jwtConstants.refreshTokenSecret,
      expiresIn: `${jwtConstants.refreshExpiresIn} min`,
    });

    this.updateRefreshToken(user.id, refreshToken);
    return {
      access_token: accessToken,
      refresh_token: refreshToken,
    };
  }

The function receives the email and password (pass) as parameters. and do the following

It retrieves the user’s details from the database using the getUserByEmail method.
If there’s no user found with the provided email, it throws a BadRequestException error with the message 'Wrong credentials'.
Next, it checks whether the provided password matches the user’s password stored in the database. The bcrypt.compare method is used to compare the hashed version of the input password with the hashed version stored in the database.
If the passwords don’t match, it throws an BadRequestException error with the message 'Wrong credentials'.
If the user is found and the passwords match, it prepares a payload with the user’s details.
The method then creates a JWT access token and a refresh token using the jwtService.signAsync method. The payload is used as the data for these tokens. The refresh token has a different secret and expires after a defined amount of time.
The updateRefreshToken the method is called to associate the new refresh token with the user in the database.
Finally, the method returns both the access token and the refresh token. These can be used by the client application to make authenticated requests and renew the access token when it expires, respectively.

Refresh Token:

Now let’s write the lambda function that takes of refreshing token when expired.

export const refreshToken = async (
  event: any,
  _context: Context,
  _callback: Callback,
) => {
  const appContext = await NestFactory.createApplicationContext(AppModule);
  const appService = appContext.get(AppService);

  const { refreshToken } = JSON.parse(event.body);

  try {
    const result = await appService.refreshToken(refreshToken);
    return {
      statusCode: 200,
      body: JSON.stringify({
        success: true,
        data: result,
      }),
    };
  } catch (error) {
    console.log(error);
    return {
      statusCode: 403,
      body: JSON.stringify(error.response ?? error.message),
    };
  }
};

This refreshToken function takes an event, context, and callback as inputs. From the request body, it extracts the refreshToken. This is then used when calling the refreshToken method of appService. If the refresh operation is successful, a 200 status code and the result of the process are returned. This would typically be a new access token. However, if an error occurs, the function responds with a 403 status code and the respective error message. The error could occur for various reasons, such as the provided refreshToken is invalid or expired.

Now let’s take a look at our refreshToken service which handle the logic that refreshes the token.

  async refreshToken(refreshToken: string) {
    let user = undefined;
    try {
      const payload = await this.jwtService.verifyAsync(refreshToken, {
        secret: jwtConstants.refreshTokenSecret,
      });
      user = payload;
    } catch (e) {
      console.log(e);
      throw new InternalServerErrorException(
        'Error while validating refresh token',
      );
    }

    const user = await this.getUserByEmail(user.email);
    if (!user) {
      throw new BadRequestException('Invalid refresh token');
    }

    if (user.refreshToken && user.refreshToken !== refreshToken) {
      console.log('refresh token does not match.');
      throw new BadRequestException('Invalid refresh token');
    }

    const payload = {
      email: user.email,
      firstName: user.firstName,
      lastName: user.lastName,
      sub: user.id,
    };

    return {
      access_token: await this.jwtService.signAsync(payload),
    };
  }

The refreshToken function accepts the refreshToken as a parameter and proceeds as follows:

1- It attempts to verify the refreshToken using the jwtService.verifyAsync method. If there's an issue with the verification, an error is logged, and it throws an InternalServerErrorException with the message 'Error while validating refresh token'.

2- Once the refreshToken is verified, the payload (which contains the user's information) is extracted from the refreshToken and used to retrieve the user's data from the database using the getUserByEmail method.

3- If no user is found, or the refreshToken stored in the database for the user does not match the provided refreshToken, it throws a BadRequestException with the message 'Invalid refresh token'.

4- If the user exists and the refreshToken is valid, it prepares a new payload with the user's details. The payload includes the user's email, first name, last name, and id.

5- Finally, it generates a new JWT access token using the jwtService.signAsync method, with the payload as the data for this token and returns this new access token. The client application can use this new access token for further authenticated requests.

This process helps ensure that the user is still valid and has the right to access the resources, even when the access token is expired but the refresh token is still valid. This reduces the need for the user to provide their credentials again, improving the user experience.

Authorizer:

With the main subject that we are going to talk about in this post which is writing an authorizer that going to be used in the other services and in our API gateway to check if a request is valid or not.

as a reminder, we already created the required configuration in our serverless.yaml file , below is the most important parts that your file have to make the authorizer work with the API gateway and with other services and lambda function in the different serverless.yaml files.

service: auth

provider:
  ...
  apiGateway:
    restApiId:
      'Fn::ImportValue': MyApiGateway-restApiId
    restApiRootResourceId:
      'Fn::ImportValue': MyApiGateway-rootResourceId

...

functions:
  ...
  authorizer:
    handler: dist/main.authorizer

resources:
  Resources:
    Authorizer:
      Type: AWS::ApiGateway::Authorizer
      Properties: 
        Name: ${self:provider.stage}-Authorizer
        RestApiId: 
          'Fn::ImportValue': MyApiGateway-restApiId
        Type: TOKEN
        IdentitySource: method.request.header.Authorization
        AuthorizerResultTtlInSeconds: 300
        AuthorizerUri:
          Fn::Join:
            - ''
            - 
              - 'arn:aws:apigateway:'
              - Ref: "AWS::Region"
              - ':lambda:path/2015-03-31/functions/'
              - Fn::GetAtt: "AuthorizerLambdaFunction.Arn"
              - "/invocations"
  Outputs:
   AuthorizerId:
     Value:
       Ref: Authorizer
     Export:
       Name: authorizerId

We can go back to our auth/main.ts and write some code for our authorizer, the provided code is an AWS Lambda function that serves as an “authorizer” in the context of AWS API Gateway. It will be invoked before your actual business logic function to verify that the incoming request has the necessary permissions to perform the intended action, let’s explain more about the function

The function then retrieves the accessToken from the event.authorizationToken property by removing the 'Bearer' prefix from it.
It also extracts the methodArn from the event.methodArn. methodArn is the Amazon Resource Name (ARN) of the incoming request. This is an identifier that AWS uses to identify individual resources. It represents the requested resource (like an API method) to be accessed.
It then calls the authorizer method from AppService bypassing accessToken and methodArn. This function should return a policy document that tells API Gateway what resources this token is allowed to access.
If the authorizer function is successful and the user is authorized, it returns a policy document by calling the callback function with null as the first parameter and authorized as the second parameter.
If an error occurs, it calls the callback function with null as the first parameter and the error response as the second.

now we can explain why are we using methodArn , and callback in the authorizer :

The reason for extracting the methodArn is to generate a policy document specific to the API method being accessed. A user may have different permissions for different API methods. The methodArn helps us identify which API method the user is trying to access so that we can generate the correct policy document.
The callback function is part of the AWS Lambda handler. In an AWS Lambda function, the callback function is used to signal the end of the function’s execution and return a response to the service that invoked the Lambda function. It’s essential to call the callback function once you’re done with your processing, or AWS Lambda will continue to wait until the function execution times out.

Now let’s dive deeper and discover our authorizer service and explain more about how the authorizer logic work, below is the authorizer service and the other two helper functions used in it.

  async authorizer(accessToken: string, methodArn: any) {
    if (!accessToken || !methodArn)
      return this.generateAuthResponse('None', 'Deny', 'None');

    // verifies token
    const decoded = await this.verifyToken(accessToken);
    if (decoded && decoded.sub) {
      return this.generateAuthResponse(decoded.sub, 'Allow', methodArn);
    } else {
      return this.generateAuthResponse('None', 'Deny', methodArn);
    }
  }

  async verifyToken(accessToken: string) {
    console.log('verify token', accessToken);
    try {
      return await this.jwtService.verifyAsync(accessToken);
    } catch (e) {
      throw new BadRequestException('Invalid access token');
    }
  }

  private generateAuthResponse(principalId, effect, methodArn) {
    const policyDocument = this.generatePolicyDocument(effect, methodArn);

    return {
      principalId,
      policyDocument,
    };
  }

  private generatePolicyDocument(effect, methodArn) {
    if (!effect || !methodArn) return null;

    const policyDocument = {
      Version: '2012-10-17',
      Statement: [
        {
          Action: 'execute-api:Invoke',
          Effect: effect,
          Resource: methodArn,
        },
      ],
    };

    return policyDocument;
  }

authorizer(accessToken: string, methodArn: any): This function is the authorizer for the API. It checks if an accessToken and methodArn (a unique identifier for the method to be authorized) are provided. If either is missing, it generates an authorization response denying access. If both are provided, it verifies the token. If the token is valid and contains a sub (subject) field, it generates an authorization response allowing access to the methodArn. If the token is not valid, it generates a response denying access.
generateAuthResponse(principalId, effect, methodArn): This function generates an authorization response. It creates a policy document (using generatePolicyDocument(effect, methodArn)) and combines it with the principalId to form the response. The principalId is the identifier of the user or role for which the policy is being created.
generatePolicyDocument(effect, methodArn): This function generates a policy document. A policy document is a structured policy that AWS uses to evaluate whether to allow or deny access to a specific AWS resource. This function accepts an effect (either 'Allow' or 'Deny') and a methodArn and constructs a policy document from them
verifyToken(accessToken: string): This function verifies if the provided accessToken is valid. It uses the jwtService.verifyAsync method to decode the token and confirm its legitimacy. If the token is invalid, it throws a BadRequestException error with the message 'Invalid access token'.

For more details about method arn, callback, and generating policy documents check the links below :

Enabling CORS

In order for our authorizer to work we need to import it in our serverless.yaml file and enable CORS for the function that accepts PUT and POST requests.

export const updateItem: Handler = async (
  event: any,
  _context: Context,
  _callback: Callback,
) => {
  const appContext = await NestFactory.createApplicationContext(ItemsModule);
  const appService = appContext.get(ItemsService);
  const token = appService.extractTokenFromHeader(event);
  const { id } = event.pathParameters;
  const { name, description } = JSON.parse(event.body);

  try {
    const res = await appService.updateItem(
      id,
      {
        name,
        description,
      },
      token,
    );
    return {
      statusCode: HttpStatus.OK,
      body: JSON.stringify(res),
      headers: {
        'Access-Control-Allow-Origin': '*',
        'Access-Control-Allow-Methods': '*',
        'Access-Control-Allow-Credentials': true,
      },
    };
  } catch (error) {
    console.log(error);
    return {
      statusCode: HttpStatus.BAD_REQUEST,
      body: JSON.stringify(error.response ?? error.message),
      headers: {
        'Access-Control-Allow-Origin': '*',
        'Access-Control-Allow-Methods': '*',
        'Access-Control-Allow-Credentials': true,
      },
    };
  }
};

One key thing to note in this function is the headers object in the response. This is related to the concept of Cross-Origin Resource Sharing (CORS). CORS is a mechanism that uses additional HTTP headers to tell browsers to give a web application running at one origin, access to selected resources from a different origin.

In this code, the headers 'Access-Control-Allow-Origin': '*' and 'Access-Control-Allow-Methods': '*' are set to ' * ' which means all origins and all methods are allowed. In a production environment, it's typically recommended to restrict this to only the origins and methods that need to be used for security reasons. For example, you could restrict the origins to 'https://yourwebsite.com' and the methods to 'GET, POST' as follows:

'Access-Control-Allow-Origin': 'https://yourwebsite.com',
'Access-Control-Allow-Methods': 'GET, POST'

That’s a quick overview of the code. As always, keep in mind to secure your applications by not exposing sensitive data in error messages and by implementing proper authentication and authorization mechanisms.

Now if you try to run npm run deploy and then check your aws api portal you will find this authorizer, trying to access the updateItem function will throw an error until you provide a valid access token.

That’s all for this post! I’ve shared my journey on learning to build a basic API using NestJS, AWS, and the Serverless Framework. I plan to write more about my experiences with these tools in future posts. I hope you find this helpful, and maybe even learn something new!

Building a Serverless Application with NestJS and the Serverless Framework: API portal

Ez Pz Developement — Sat, 15 Jul 2023 23:15:21 +0000

Table of Contents:

Introduction
Using A One API Portal
Th end

Introduction:

In my last blog post, we walked through the process of building a straightforward application with the Serverless Framework and NestJS. Today, we’re going to take a step further. Our primary focus will be on having all apps under a single AWS API Gateway, as opposed to each having its separate gateway.

A one API Portal:

As we said each of 3 apps was related to its own API portal what i first did is to go to the serverless documentation and I found this post talking about how to create a configuration for an API portal and expose it as an output for the other apps and serverless files to use.

So here is what I did, first I created a configuration folder configand added a new serverless file to it so my files structure now looks like this

apps
  users
    src
      app.controller.ts
      app.module.ts
      app.service.ts
      main.ts
      serverless.yaml
    tsconfig.app.json
  items
    src
      app.controller.ts
      app.module.ts
      app.service.ts
      main.ts
      serverless.yaml
    tsconfig.app.json
config
  serverless.yaml
nest-cli.json
serverless-compose.yaml
package.json
tsconfig.json
.eslintrc.js

I also added this config folder to the serverless-compose.yaml

services:
  users:
    path: apps/users
    dependsOn: config
  items:
    path: apps/items
    dependsOn: config
  config:
    path: config

After done with the file configuration part i moved back to my config/serverless.yaml file and added the following code to it

service: config

provider:
  name: aws
  runtime: nodejs16.x
  stage: dev
  region: eu-west-3

resources:
  Resources:
    MyApiGW:
      Type: AWS::ApiGateway::RestApi
      Properties:
        Name: MyApiGW
  Outputs:
    apiGatewayRestApiId:
      Value:
        Ref: MyApiGW
      Export:
        Name: MyApiGateway-restApiId

    apiGatewayRestApiRootResourceId:
      Value:
        Fn::GetAtt:
          - MyApiGW
          - RootResourceId
      Export:
        Name: MyApiGateway-rootResourceId

Now let’s break down this file and explain each line of it:

service: config: Here, we're simply naming our service "config".
provider:: This section gives the details about our cloud service provider. We specify we're using Amazon Web Services (name: aws), with Node.js 16.x as the runtime environment (runtime: nodejs16.x). We also denote we're working in the development stage (stage: dev) in the region 'eu-west-3' (region: eu-west-3).
resources:: In this block, we define the AWS resources used in our service.
MyApiGW:: We name our API Gateway "MyApiGW".
Type: AWS::ApiGateway::RestApi: We specify we're creating a RESTful API using AWS's API Gateway.
Properties: Name: MyApiGW: We set the name of the API Gateway under properties.
Outputs:: This part is where we define the output values from our serverless service. These are useful for cross-stack resource sharing or for sharing references with other services.
apiGatewayRestApiId:: We're exporting the ID of the API Gateway here. The value is a reference to the API Gateway we defined earlier (Ref: MyApiGW). The exported output is named "MyApiGateway-restApiId" (Export: Name: MyApiGateway-restApiId).
apiGatewayRestApiRootResourceId:: Similarly, we're exporting the Root Resource ID of the API Gateway. The value is the "RootResourceId" attribute from the API Gateway (Fn::GetAtt: - MyApiGW - RootResourceId). The exported output is named "MyApiGateway-rootResourceId" (Export: Name: MyApiGateway-rootResourceId).

For more details about this make sure to check :

Serverless API Gateway V1 documentation
There is also another way to achieve the same results by following Serverless API Gateway V2 documentation, API Gateway V2 is faster and cheaper than V1 according to the documentation.
Also, check this link for a simple explanation.
And this one contains better documentation.

We have now successfully set up our configuration file. The next step involves instructing our two applications users and items to use the API Gateway that we’ve established in our configuration folder. Let’s proceed with that

Now that we’re all set with the configuration, let’s navigate to users/serverless.yaml and items/serverless.yaml Within the provider section of these files, we’ll add the following lines of code:

# users/serverless.yaml and items/serverless.yaml
provider:
   ....
  apiGateway:
      restApiId:
        'Fn::ImportValue': MyApiGateway-restApiId
      restApiRootResourceId:
        'Fn::ImportValue': MyApiGateway-rootResourceId

Now let’s explain the simple changes that we added to the two files :

provider.apiGateway.restApiId - This tells Serverless Framework to use an existing AWS API Gateway, instead of creating a new one for this service. The existing API Gateway's ID is provided by importing the value from CloudFormation's output named "MyApiGateway-restApiId". This is useful when you want to manage the API Gateway separately from the Serverless application or share it among multiple Serverless applications.
provider.apiGateway.restApiRootResourceId - Similarly, this tells Serverless Framework to use an existing root resource in the AWS API Gateway, instead of creating a new one. The existing root resource's ID is provided by importing the value from CloudFormation's output named "MyApiGateway-rootResourceId". This is again useful when the root resource is managed separately or shared among multiple Serverless applications.

Note: Fn::ImportValue is a CloudFormation intrinsic function used to import values exported by other stacks in the same AWS account and region. This allows for cross-stack references, sharing resources and values among different CloudFormation stacks, which can be useful for managing larger architectures or when it is beneficial to have separate stacks for different concerns.

For a more comprehensive understanding of CloudFormation’s intrinsic functions and stacks, I highly recommend diving into the following resources:

now we simply need to run :

npm run deploy

Which was already added to our package.json file in the root directory of the app.

"scripts": {
  "build:users": "nest build --tsc users",
  "build:items": "nest build --tsc items",
  "build:all": "npm run build:users && npm run build:items",
  ...
  "deploy": "npm install && npm run build:all && npx serverless deploy",
  ...
},

Following these steps, you can navigate to your AWS console and locate API Gateway. There, you will discover a single, centralized API Gateway managing all of your application links. Its interface should resemble the screenshot provided below.

An illustrative screenshot showcasing a custom API Gateway, constructed with the method followed in this blog post, utilizing NestJS and the Serverless Framework.

So instead of seeing the links in the dashboard, you will see an items and books links.

In summary, here’s a concise roadmap to consolidate our multiple apps under a single API gateway:

Construct a configuration file containing the YAML setup for the API gateway, and ensure it exports the apiGatewayRestApiId and apiGatewayRestApiRootResourceId.
Incorporate these two exported values into the provider section of all our application-specific serverless.yaml files.
Deploy the app and validate our unified API gateway via the AWS console.

The end

I hope this post has offered some insight into my learning journey and the steps I’ve taken to create an app using NestJS and Serverless Framework. If you’ve made it this far, I want to extend my sincere gratitude for your time and interest.

Your feedback means a lot to me. If you have any suggestions or simply want to share your own experiences, I invite you to leave a comment or get in touch.

Stay tuned for my upcoming post, where i will explore adding authentication to the application. Thank you.

A Concise Guide to Utilizing HashiCorp Vault in Production

Ez Pz Developement — Mon, 26 Jun 2023 22:53:28 +0000

Introduction

In today’s digital landscape, the need to protect sensitive information has become paramount. Secrets management is the practice of securely storing, managing and distributing critical data such as passwords, API keys, database credentials, and encryption keys. By implementing robust practices and utilizing specialized tools, secrets management aims to safeguard these secrets from unauthorized access and misuse.

HashiCorp Vault

HashiCorp Vault is a popular open-source tool designed for secure secrets management and data protection. It offers a comprehensive solution for storing, accessing, and managing sensitive information, such as passwords, API keys, certificates, and encryption keys.

Key Features and Capabilities

Centralized Secrets Management : Vault provides a central repository for securely storing secrets, eliminating the need to store sensitive information in configuration files or source code. This centralized approach improves security by reducing the risk of accidental exposure or unauthorized access
Secure Storage: Vault encrypts secrets at rest using industry-standard encryption algorithms. It ensures that sensitive information remains encrypted when stored in backend storage systems, adding an extra layer of protection.
Dynamic Secrets: Vault can generate dynamic secrets on-demand for various resources such as databases, cloud providers, and more. These dynamic secrets have short lifetimes and are automatically revoked after a certain period or when no longer needed. This approach minimizes the risk of credentials being compromised or misused.
Auditing and Logging : Vault keeps detailed logs of all operations, including access attempts, secret retrieval, and modification. These audit logs enable organizations to track and monitor secret usage, aiding in compliance efforts and troubleshooting
Integration and Extensibility : Vault integrates with various authentication systems, identity providers, and cloud platforms. It provides a flexible API and SDKs for developers to build custom integrations and automate secrets management processes.
Encryption as a Service : Vault provides encryption as a service, allowing users to encrypt and decrypt data using various encryption algorithms. This feature ensures the confidentiality and integrity of secrets
Dynamic Secrets Revocation : Vault supports automatic revocation of dynamic secrets after a predefined period or when no longer needed. This helps mitigate the risk of unauthorized access to secrets.
Secrets Encryption and Decryption : Vault allows users to encrypt and decrypt secrets using encryption keys managed within the system. This feature ensures that secrets are protected and can be securely transmitted across different systems.

Common Use Cases For Vault

Database Credential Management : Vault can securely store and manage credentials for databases, such as usernames, passwords, and connection strings. Applications can retrieve these credentials dynamically from Vault, reducing the risk of hardcoded credentials and improving security
API Key Management: Vault can be used to store and distribute API keys securely. By centralizing API key management in Vault, organizations can enforce access control, monitor usage, and rotate keys when needed.
Encryption Key Management : Vault can generate, store, and manage encryption keys used for data encryption. This ensures that encryption keys are securely stored and protected, reducing the risk of unauthorized access to sensitive data.
Token Management: Vault can generate and manage tokens used for authentication and authorization. It provides fine-grained control over token creation, revocation, and expiration, allowing organizations to enforce security policies and manage access to resources.
SSH Key Management : Vault can act as an SSH key manager, providing a secure repository for storing and distributing SSH keys. This centralizes the management of SSH keys, simplifying access control and auditing.

Vault Installation

Note : Please note that the installation instructions provided below are specific to Linux. If you are using a different operating system, we recommend referring to the official documentation for the appropriate installation steps

Using the Debian package repository, HashiCorp provides a variety of Official Release Channels. see more here

GPG is required for the package signing key

sudo apt update && sudo apt install gpg

Download the signing key to a new keyring

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

Verify the key’s fingerprint, add the HashiCorp repo, update and install the vault

gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update

sudo apt install vault

Verify if everything is all right (vault installed correctly)

vault

Usage: vault <command> [args]

Common commands:
    read Read data and retrieves secrets
    write Write data, configuration, and secrets
    delete Delete secrets and configuration
    list List data or secrets
    login Authenticate locally
    agent Start a Vault agent
    server Start a Vault server
    status Print seal and HA status
    unwrap Unwrap a wrapped secret

Other commands:
    audit Interact with audit devices
    auth Interact with auth methods
    debug Runs the debug command
    kv Interact with Vault's Key-Value storage
    lease Interact with leases
    monitor Stream log messages from a Vault server
    namespace Interact with namespaces
    operator Perform operator-specific tasks
    path-help Retrieve API help for paths
    plugin Interact with Vault plugins and catalog
    policy Interact with policies
    print Prints runtime configurations
    secrets Interact with secrets engines
    ssh Initiate an SSH session
    token Interact with tokens

With Vault installed, the next step is to start a Vault server.

Starting the Server

you can test working with Vault with the “dev” server, which automatically unseals Vault, sets up in-memory storage, etc. (it's not covered here ) ⇒ see doc

Deploy vault in prod env

The /var/lib/vault/data directory that the vault uses must exist. (you can choose other placements)

sudo mkdir -p /var/lib/vault/data

Vault is configured using HCL files

Create the Vault configuration in the file config.hcl

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = true
}

api_addr = "http://127.0.0.1:8200"

storage "file" {
  path = "/var/lib/vault/data"
}

Let’s break down the configuration file:

The listener block defines the network interface and port where Vault will listen for API requests. In this example, it's set to 127.0.0.1:8200 , which means Vault will listen on the loopback address on port 8200.
tls_disable is set to true to disable TLS (Transport Layer Security) encryption for simplicity. In a production environment, it's recommended to use TLS.
The api_addr configuration specifies the base URL for accessing the Vault API. In this case, it's set to http://127.0.0.1:8200 to match the listener address.
The storage block configures the storage backend for Vault. In this example, it's set to a file storage backend with the path /var/lib/vault/data. You can modify this path based on your needs or use a different storage backend like "consul" or "etcd" for production environments.

Set the -config flag to point to the proper path where you saved the configuration above.

vault server -config=/etc/vault/config.hcl

Initializing the Vault

Initialization is the process of configuring Vault.

This only happens once when the server is started against a new backend that has never been used with Vault before.

Launch a new terminal session, and set VAULT_ADDR environment variable.

export VAULT_ADDR='http://127.0.0.1:8200'

To initialize Vault use vault operator init. This is an unauthenticated request, but it only works on brand new Vaults without existing data:

vault operator init

output:

Unseal Key 1: LhMnWcFuP/alHmtIlWTf8cp7ONbz0WFLiy28FG1IcQBK
Unseal Key 2: VU2lbINIzkovgTmHihSiTgxXUGsHnrJwLb/Q2V9FJ1Vc
Unseal Key 3: vo2lwvK8AqUHq87ASUDYoIpjO7jJnI2QJrOrQnRO9FGJ
Unseal Key 4: XMHENMUX8RPoI8AF2dIPx7/pJS+EeTywmJSHF/kt970B
Unseal Key 5: lE8hetFDRFoQ61NKJ3nNLRH/QKhD+sno5zlnKU/+z5sv

Initial Root Token: hvs.57tWqWNKEVgbyF1zPcETh4Qt

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated root key. Without at least 3 keys to
reconstruct the root key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Initialization outputs two incredibly important pieces of information: the unseal keys and the initial root token. This is the only time ever that all of this data is known by Vault, and also the only time that the unseal keys should ever be so close together.

For the purpose of this tutorial, save all of these keys somewhere, and continue. In a real deployment scenario, you would never save these keys together. Instead, you would likely use Vault’s PGP and Keybase.io support to encrypt each of these keys with the users’ PGP keys. This prevents one single person from having all the unseal keys. Please see the documentation here

Seal/Unseal

Every initialized Vault server starts in the sealed state. From the configuration, Vault can access the physical storage, but it can’t read any of it because it doesn’t know how to decrypt it. The process of teaching Vault how to decrypt the data is known as unsealing the Vault.

Unsealing has to happen every time Vault starts. It can be done via the API and via the command line. To unseal the Vault, you must have the threshold number of unseal keys. In the output above, notice that the “key threshold” is 3. This means that to unseal the Vault, you need 3 of the 5 keys that were generated.

vault operator unseal

To unseal the vault you must use three different unseal keys, the same key repeated will not work.

As you use keys, as long as they are correct, you should soon see output like this:

Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.7.0
Storage Type raft
Cluster Name vault-cluster-0ba62cae
Cluster ID 7d49e5fd-a1a4-c1d1-55e2-7962e43006a1
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address &lt;none&gt;
Raft Committed Index 24
Raft Applied Index 24

When the value for Sealed changes to false, the Vault is unsealed.

login

Now authenticate as the initial root token (it was included in the output with the unseal keys).

vault login

While token-based authentication is enabled by default, Vault also supports various other authentication methods, such as username/password, LDAP, OIDC, GitHub, etc. These methods provide additional flexibility for authenticating users and applications with Vault.

Using the HTTP APIs with Authentication

All of Vault’s capabilities are accessible via the HTTP API in addition to the CLI.

Start a new Vault instance using the newly created configuration. (stop the previous instance using ctrl+c )

vault server -config=/etc/vault/config.hcl

Launch a new terminal session, and use curl to initialize Vault with the API.

curl \
    --request POST \
    --data '{"secret_shares": 1, "secret_threshold": 1}' \
    http://127.0.0.1:8200/v1/sys/init | jq

output

{
  "keys": [
    "ff27b63de46b77faabba1f4fa6ef44c948e4d6f2ea21f960d6aab0eb0f4e1391"
  ],
  "keys_base64": [
    "/ye2PeRrd/qruh9Ppu9EyUjk1vLqIflg1qqw6w9OE5E="
  ],
  "root_token": "s.Ga5jyNq6kNfRMVQk2LY1j9iu"
}

This response contains your initial root token. It also includes the unseal key. You can use the unseal key to unseal the Vault and use the root token perform other requests in Vault that require authentication.

To make this tutorial easy to copy-and-paste, you will be using the environment variable $VAULT_TOKEN to store the root token.

export VAULT_TOKEN="s.Ga5jyNq6kNfRMVQk2LY1j9iu"

Using the unseal key (not the root token) from above, you can unseal the Vault via the HTTP API.

Example:

curl \
    --request POST \
    --data '{"key": "/ye2PeRrd/qruh9Ppu9EyUjk1vLqIflg1qqw6w9OE5E="}' \
    http://127.0.0.1:8200/v1/sys/unseal | jq

output

{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.7.0",
  "migration": false,
  "cluster_name": "vault-cluster-1b34e68e",
  "cluster_id": "2cccf342-091a-b060-900b-04c29bb71ed4",
  "recovery_seal": false,
  "storage_type": "file"
}

You can invoke the Vault API to validate the initialization status.

curl http://127.0.0.1:8200/v1/sys/init

output

{ "initialized": true }

Creating secrets

Vault’s secret engines are components that enable the secure storage, generation, and management of secrets within Vault. They provide a way to interact with and manage different types of sensitive data, such as passwords, API keys, certificates, and more. Each secret engine has its own purpose and functionality

some of the common secret engines in Vault:

Key/Value (KV) Secret Engine
Database Secret Engine
AWS Secret Engine:
PKI (Public Key Infrastructure) Secret Engine
Transit Secret Engine

for the sake of simplicity, we will go with Key/Value (KV) Secret Engine :

The KV secret engine is the most basic and versatile secret engine in Vault.
It allows you to store key-value pairs of arbitrary secrets within Vault.
It provides a simple CRUD (Create, Read, Update, Delete) API to interact with secrets.
It is often used for storing application configuration, database credentials, or other general-purpose secrets.

You can check the list of enabled secret engines by running the following cURL command:

curl --header "X-Vault-Token: $VAULT_TOKEN" http://127.0.0.1:8200/v1/sys/mounts

To enable the KV secret engine and mount it at the appropriate path, you can use the following cURL command :

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"type": "kv-v2"}' http://127.0.0.1:8200/v1/sys/mounts/secret

Create your first secret : create a simple secret with name mysecret and object {“key”:”value”}

curl --header "X-Vault-Token: $VAULT_TOKEN" --request POST --data '{"data": {"key": "value"}}' http://127.0.0.1:8200/v1/secret/data/mysecret

output

{"request_id":"ad28e5de-5eac-34bd-9252-fc9894da95cd","lease_id":"","renewable":false,"lease_duration":0,"data":{"created_time":"2023-06-23T00:27:21.7933063Z","custom_metadata":null,"deletion_time":"","destroyed":false,"version":1},"wrap_info":null,"warnings":null,"auth":null}

Retreive the created secret

curl --header "X-Vault-Token: $VAULT_TOKEN" http://127.0.0.1:8200/v1/secret/data/mysecret

output

{"request_id":"f3ec354c-cf71-b499-a8f4-90cc30d0eab6","lease_id":"","renewable":false,"lease_duration":0,"data":{"data":{"key":"value"},"metadata":{"created_time":"2023-06-23T00:27:21.7933063Z","custom_metadata":null,"deletion_time":"","destroyed":false,"version":1}},"wrap_info":null,"warnings":null,"auth":null}

Conclusion

In conclusion, this tutorial provided a straightforward and concise guide on how to initiate, initialize, and access the Harishcop vault using simple HTTP requests. By following these steps, you can easily gain access to the vault and leverage its benefits. We hope that you found this tutorial useful and that it assists you in your endeavors.

Things playing League of legends has taught me, a noob point of view

Ez Pz Developement — Thu, 15 Jun 2023 22:00:32 +0000

I've been playing League of Legends since the end of Season 9, and guess what? I started playing ranked once I reached level 30, and I got stomped, losing most of my games by 70% to 75% in that season and ending up in Iron 3.

If you want to know more about League of Legends, you can watch this [_video](https://youtu.be/u9JdENUoGik)._

After that, I started learning about how tofarm, wave management, and team fight positioning, and I managed to get to bronze 2 after 250 games.

Even after playing for 3 seasons, I'm still a hard stuck noob in bronze and silver XD, I tried my best to learn the fundamentals of these games, but I feel like I still have a lot to learn in order to get a better rank.

But I'm not writing this post to tell my story with league legends, instead, I want to write about how playing this game helped me approach life with a better view and understand life and people better, and below are some points:

For me, League of Legends is like a real-life simulator; you will play with different types of people in different situations, and I can say that this game will make you become the worst version of yourself during the matches.

A game like this will help you learn that winning or losing is about you and not other people. If you want to win more, then you should practice and do a better job.

It also helped me realize that in a lot of cases, blaming other people when they make mistakes will only make things worse, especially if you are on the same team as those people, I don’t just mean League of Legends teams when I say the same team; I mean every other team in every other aspect of our lives.

There is no such myth as super-smart people getting massive results by doing little work or no work, to get good at anything, you must practice, focus, plan, and dedicate some time to it.

League of Legends helped me to know more about my friends' personalities; how they think and play in the league is the same as how they think and live in real life, For me, here are the types of people that I find in a league of legends and in real life :

Cool people who never get angry, always stay calm and play for fun. Even if they have a 10-game losing streak, they will keep playing, Another important thing about these people is that they have no problem playing with other players who are at a lower skill level, they just play with them and try to teach them about the game and make them a better player.
Another type of people’s personality is the titled perfectionist who will get angry and lose his mind if he or one of his teammates miss played, and by miss playing I mean missing skill shots, dying in the early phases of the game, going into some stupid fights with the enemies team, or missing a smite on a dragon or a Baron, once that happens this type of players will start cursing, flaming everyone in the team, and asking them to forfeit because he doesn’t want to play anymore, personally I think that the problem with these players mentality is that they expect someone who just started playing and his rank is bronze or silver, to know everything and do no mistakes and play like a professional player, for me this type of persons are not realistic at all.
Finally, there are delusional players who believe that losing or being stuck in a low rank is due to their teammates and the Riot ranking system.

In the end, in my opinion, I think that the league and those competitive games are just like any other area of life, if you stick to a plan, practice, and dedicate a good amount of time to them, you will be better.

Data Transfer Objects (DTOs): A Comprehensive Guide

Ez Pz Developement — Wed, 14 Jun 2023 23:46:26 +0000

Introduction

Imagine 🤔✨ having a complex object or entity with countless properties and fields. But wait! In certain situations, not all of that data needs to be transferred or processed.

Welcome ✨, brave adventurer, to the realm of Data Transfer Objects (DTOs) 📦! These magical constructs allow you to choose precisely the data you seek from the vast depths of complexity.

Alright, let’s transition back to our world! So, what exactly are DTOs?

For instance, let’s consider an e-commerce application. When displaying a list of products on a webpage, you may only need to transmit basic information like the product name, price, and image URL. In this scenario, instead of transferring the entire product object with all its associated data (e.g., reviews, descriptions, status), you can create a ProductDTO that includes only the essential fields for displaying the product list efficiently.

Purpose of Data Transfer Objects (DTOs) in backend development

DTOs are used to encapsulate and transfer data between different layers of an application :

For example, when data needs to be sent from the client (such as a web browser) to the server, or vice versa, DTOs provide a structured way to package and transfer that data. The client can create a DTO, populate it with the necessary data, and send it to the server. The server, in turn, can receive the DTO, extract the relevant information, and process it accordingly.
By using DTOs, developers can define and control the data that is sent and received.
Minimizing unnecessary data transfer (reducing overhead, minimize bandwidth usage, reduce latency which lead to improvment in performance).
Reducing the risk of exposing sensitive information.
Enable the decoupling of the API contract from the internal representations, allowing for flexibility and evolution of the backend system without impacting external consumers.

Decoding DTOs: Unveiling Their Definition and Distinctions from Domain Entities and Database Models

Let’s break down the table:

Purpose : DTOs serve the purpose of facilitating efficient data transfer and communication between different components or layers of an application. Data models represent the structure and relationships of the underlying database. Domain entities capture the business logic and behavior of the application.
Data Subset : DTOs contain a subset of relevant data needed for specific use cases or communication scenarios. Data models encompass the complete set of data stored in the database. Domain entities also hold the complete set of data related to the business logic.
Abstraction : DTOs provide a layer of abstraction, separating the internal data representations from external interactions. Data models and domain entities do not necessarily involve abstraction as they directly represent the underlying data or business logic.
Immutability : DTOs can be designed as immutable objects, ensuring that the transferred data remains consistent and unmodifiable. Data models and domain entities are typically mutable and subject to modifications.
Transformation : DTOs involve data transformation or conversion to adapt the data from one format to another, facilitating interoperability between different components or systems. Data models and domain entities do not inherently involve transformation as they represent the original data structure and business logic.

The Creation of DTOs

The process of creating DTO classes or structures in a backend application involves the following steps:

Identify the Data Subset : Determine the specific subset of data that needs to be transferred or communicated between different parts of the application.
Design the DTO Class/Structure : Create a DTO class or structure that represents the identified subset of data. The class/structure should contain properties that mirror the fields in the subset, with appropriate data types.
Map Data to DTO : Implement a mapping mechanism to populate the DTO object with data from the source. This could involve manually assigning values from the source object or using mapping libraries/tools to automate the process.
Use DTO for Data Transfer : Pass the DTO object between different components or layers of the application, transferring the necessary data in a standardized format.

Here is an example of code showing the utilization of DTOs

class ProductDTO {
  constructor(id, name, price, description) {
    this.id = id;
    this.name = name;
    this.price = price;
    this.description = description;
  }

  getFormattedPrice() {
    return `$${this.price.toFixed(2)}`;
  }

  static fromProductEntity(productEntity) {
    return new ProductDTO(
      productEntity.id,
      productEntity.name,
      productEntity.price,
      productEntity.description );
  }
}

// Usage
const productEntity = {
  id: 1,
  name: 'iPhone ',
  price: 999,
  description: 'The latest iPhone model.',

};

const product = ProductDTO.fromProductEntity(productEntity);
console.log(product.getFormattedPrice());

In this example, the ProductDTO class represents a product with properties for id , name , price , and description. It has a constructor that takes these properties as arguments and assigns them to the corresponding class properties.

The getFormattedPrice() method is defined within the class. It returns the formatted price of the product by using the toFixed() method to round the price property to 2 decimal places and prepending it with the dollar sign.

The fromProductEntity() method is a static method that takes a productEntity object as an argument. It creates a new instance of the ProductDTO class and initializes its properties with the corresponding properties from the productEntity object.

In the usage example, a productEntity object is created with sample values for id , name , price , and description. The fromProductEntity() method is then called to create a new ProductDTO instance based on the productEntity. Finally, the getFormattedPrice() method is called on the product object to retrieve the formatted price and log it to the console.

Mapping Techniques for DTOs and Domain Entities/Models:

There are many ways /techniques for mapping , lets explore some of them

Manual Mapping : Writing custom code to map between DTOs and domain entities/models. Provides control and customization but can be time-consuming and error-prone for complex mappings.
Mapping Libraries like AutoMapper : Utilizing libraries like AutoMapper for automated and convention-based mapping. Reduces manual mapping code, improves readability, and simplifies maintenance. Requires initial setup and configuration.
Object-Relational Mapping (ORM) Frameworks: Leveraging ORM frameworks such as Entity Framework ( EF ) or *Hibernate … *. These frameworks handle mapping between domain entities and the database. They automate mapping, handle complex scenarios, and provide additional features like lazy loading and transaction management

Use Cases and Benefits (Node js example)

User Registration : When a user registers in your Node.js application, you can use a UserDTO (User Data Transfer Object) to encapsulate the user’s registration data, such as username, email, and password. The UserDTO allows you to validate the data, apply any necessary transformations or sanitization, and pass it to the appropriate services for further processing.

// UserDTO.js
class UserDTO {
  constructor(username, email, password) {
    this.username = username;
    this.email = email;
    this.password = password;
  }
}

// UserService.js
class UserService {
  registerUser(userDTO) {
    // Validate and process user registration data
    // Save user to the database
    // Perform any additional operations
  }
}

// Usage in Node.js app
const userDTO = new UserDTO('john', 'john@example.com', 'password123');
const userService = new UserService();
userService.registerUser(userDTO);

API Responses: When responding to API requests, you can use DTOs to structure and control the data sent back to the clients. For example, you may have a ProductDTO to represent the data returned when querying a product. The ProductDTO allows you to select specific fields, exclude sensitive information, and ensure consistent data formatting across different API endpoints.

// ProductDTO.js
class ProductDTO {
  constructor(id, name, price) {
    this.id = id;
    this.name = name;
    this.price = price;
  }
}

// API Endpoint
app.get('/products/:id', (req, res) => {
  // Fetch product data from the database
  const product = getProductById(req.params.id);

  // Map product data to ProductDTO
  const productDTO = new ProductDTO(product.id, product.name, product.price);

  // Send the productDTO as the API response
  res.json(productDTO);
});

Database Operations: DTOs can be useful when interacting with the database in a Node.js app. For instance, you might use a DatabaseRecordDTO to represent a specific record retrieved from the database, allowing you to manipulate and transform the data before presenting it to the user or passing it to other parts of your application.

// DatabaseRecordDTO.js
class DatabaseRecordDTO {
  constructor(id, data) {
    this.id = id;
    this.data = data;
  }
}

// DatabaseService.js
class DatabaseService {
  fetchRecordById(id) {
    // Fetch record from the database
    const record = getRecordById(id);

    // Map and transform the record data to DatabaseRecordDTO
    const recordDTO = new DatabaseRecordDTO(record.id, record.data);

    // Perform further operations with the recordDTO
    // ...
  }
}

DTO Best Practices

In order to effectively work with DTOs and ensure their optimal usage, it is important to follow a set of best practices. These practices will help you leverage the full potential of DTOs and improve the overall quality of your code.

Simplify DTOs: Keep DTOs focused on their purpose and avoid unnecessary complexity. Limit the fields and business logic within DTOs to ensure they serve their primary role of transferring data between components.
Use Clear Naming: Choose descriptive names for DTOs and their properties to enhance code readability and understanding.
Separate Validation: Handle data validation separately from DTOs using validation libraries or dedicated validation classes
Document Purpose: Provide documentation for DTOs, including their purpose, usage, and any validation rules or constraints
Framework Independence: Ensure that DTOs remain independent of specific frameworks or libraries for improved reusability

Conclusion

In conclusion, Data Transfer Objects (DTOs) play a crucial role in backend development by facilitating the transfer of data between different layers or components of an application. They allow for efficient and controlled data communication, validation, and transformation. By keeping DTOs focused, using meaningful names, and minimizing data, we can create simple and effective DTOs

Building a Serverless Application with NestJS and the Serverless Framework: A Monorepo Approach

Ez Pz Developement — Sun, 11 Jun 2023 17:41:35 +0000

Table of Contents:

Introduction
Serverless Configuration and DynamoDB Setup for Two Services
Deploying to AWS
Running It Offline

Introduction

This blog post explores the utilization of NestJS features in a monorepo mode to build a serverless application using a combination of AWS and the Serverless Framework. The goal is to address the challenge of combining a serverless framework with a monorepo, specifically in the context of creating multiple NestJS apps within the same repository, where each app handles the logic for different endpoints and plays a controller-like role.

To get started, make sure you have Node.js 16.x installed (you can use nvm for easy version management) along with the Serverless Framework and basic knowledge of how the Serverless Framework and AWS Lambda work, as well as familiarity with DynamoDB.

To install the Serverless Framework globally, use the following command:

npm install -g serverless

Next, install the NestJS CLI globally by running:

npm i -g @nestjs/cli

For more detailed information and a brief introduction on how to get started with these tools, refer to the following links:

Serverless Framework: https://www.serverless.com/framework/docs/getting-started
NestJS: https://docs.nestjs.com/first-steps

Now, let’s dive into the coding part. Start by creating a standard NestJS application structure using the following command:

nest new users

This will generate a folder structure similar to the one below:

node_modules
src
  app.controller.ts
  app.module.ts
  app.service.ts
main.ts
nest-cli.json
package.json
tsconfig.json
.eslintrc.js

To convert our project into a monorepo structure, use the command:

nest generate app items

This command will convert the project structure into a monorepo structure that looks like thi

apps
  users
    src
      app.controller.ts
      app.module.ts
      app.service.ts
      main.ts
    tsconfig.app.json
  items
    src
      app.controller.ts
      app.module.ts
      app.service.ts
      main.ts
    tsconfig.app.json
nest-cli.json
package.json
tsconfig.json
.eslintrc.js

At this point, we have set up the necessary NestJS structure for our project. Now, let’s introduce the powerful Serverless Compose file, which was recently introduced in version 3.15.0 of the Serverless Framework. The documentation here provides more insights into how to compose Serverless Framework services.

In the root of our project, create a new file named serverless-compose.yaml with the following content:

services:
  items:
    path: apps/items
  users:
    path: apps/users

To deploy our services to AWS, run the following command:

npx serverless deploy

To run any plugin associated with the users or items service, use the following format:

npx serverless items:<plugin-name>

With this setup, you can utilize the power of NestJS and the Serverless Framework in a monorepo structure to build your serverless application on AWS.

Serverless Configuration and DynamoDB Setup for Users and Items Services

Serverless Configuration

Let’s now dive into the “apps/users” directory and embark on the journey of crafting the serverless configuration for our users service. Below, you’ll find an upgraded version of the configuration file, accompanied by a more detailed technical explanation:

#apps/users/serverless.yaml
service: users

plugins:
  - serverless-offline

custom:
    serverless-offline:
        httpPort: 3003
        lambdaPort: 3005

functions:
  getUser:
    handler: dist/main.getUser
    events:
      - http:
          method: GET
          path: /users/{id}
          request: 
            parameters: 
              paths: 
                id: true
  getUsers:
    handler: dist/main.getUsers
    events:
      - http:
          method: GET
          path: /users

provider:
  name: aws
  region: eu-west-3
  runtime: nodejs16.x
  stage: dev

Below is a comprehensive breakdown of each section in the serverless configuration file:

service: This denotes the unique identifier for the service that will be created. In this instance, the service is aptly named "users".
plugins: This section enumerates the plugins employed within the service. For this case, the "serverless-offline" plugin is utilized. The "serverless-offline" plugin emulates the behavior of AWS λ and API Gateway on your local machine, significantly expediting the development process. It initializes an HTTP server that manages the lifecycle of requests and calls your handlers.
custom: This section comprises customized configuration options for the service. Specifically, the custom configuration pertains to the "serverless-offline" plugin. The properties "httpPort", and "lambdaPort" allow for the configuration of ports utilized when running the service in offline mode.
functions: This section catalogs the individual functions to be created within the service. In this scenario, there are two functions: "getUser" and "getUsers".
handler: This property specifies the file housing the code for a given function. In this instance, the function "getUser" is defined in the file "dist/main.getUser", while "getUsers" resides in "dist/main.getUsers".
events: This section enumerates the events that trigger the execution of functions. Both functions in this case are triggered by HTTP requests. The "getUser" function is triggered by HTTP requests to the path "/users/{id}", wherein "id" represents a path parameter. Similarly, the "getUsers" function is triggered by HTTP requests to the path "/users".
provider: This section specifies the cloud provider to be utilized for deploying the service. In this case, the chosen provider is AWS (Amazon Web Services).
name: This property designates the name of the AWS service that will be generated. In this case, the AWS service will be named "users".
region: This property signifies the AWS region where the service will be deployed. In this instance, the chosen AWS region is "eu-west-3".
runtime: This property defines the runtime environment for the service. In this case, the service will be executed within the Node.js 16.x runtime environment.
stage: This property denotes the stage of the service. In this instance, the stage is designated as "dev".

Users Service Code Implementation

Let’s examine the implementation of the Users service code. First, we have the main.ts file located in the apps/users directory:

// apps/users/main.ts
import { HttpStatus } from '@nestjs/common';
import { NestFactory } from '@nestjs/core';
import { Callback, Context, Handler } from 'aws-lambda';
import { UsersModule } from './users.module';
import { UsersService } from './users.service';

export const getUser: Handler = async (
  event: any,
  _context: Context,
  _callback: Callback,
) => {
  const appContext = await NestFactory.createApplicationContext(UsersModule);
  const appService = appContext.get(UsersService);
  const { id } = event.pathParameters;
  try {
    const res = await appService.getUser(id);
    return {
      statusCode: HttpStatus.OK,
      body: JSON.stringify(res),
    };
  } catch (error) {
    console.log(error);
    return {
      statusCode: HttpStatus.BAD_REQUEST,
      body: JSON.stringify(error.response ?? error.message),
    };
  }
};

export const getUsers: Handler = async (
  _event: any,
  _context: Context,
  _callback: Callback,
) => {
  const appContext = await NestFactory.createApplicationContext(UsersModule);
  const appService = appContext.get(UsersService);
  try {
    const res = await appService.getUsers();
    return {
      statusCode: HttpStatus.OK,
      body: JSON.stringify(res),
    };
  } catch (error) {
    console.log(error);
    return {
      statusCode: HttpStatus.BAD_REQUEST,
      body: JSON.stringify(error.response ?? error.message),
    };
  }
};

In the main.ts file, we define two AWS Lambda handlers, getUser and getUsers, which are responsible for handling the corresponding API endpoints. Here's a breakdown of the code:

getUser handler: This function receives an HTTP event containing the id parameter in the path. It creates an application context using NestFactory.createApplicationContext with the UsersModule. Then, it retrieves an instance of the UsersService from the application context. The getUser method of the service is called with the id, and the result is returned as an HTTP response with a status code of 200 (OK).
getUsers handler: This function doesn't require any parameters. It follows a similar process to the getUser handler. It creates an application context, retrieves the UsersService, and calls the getUsers method. The resulting array of users is returned as an HTTP response with a status code of 200 (OK).

Next, let’s take a look at the UsersService implementation in the users.service.ts file:

// apps/users/users.service.ts
import { Injectable, InternalServerErrorException } from '@nestjs/common';
import { DynamoDB } from 'aws-sdk';

const db = new DynamoDB.DocumentClient({
  convertEmptyValues: true,
  paramValidation: true,
});

@Injectable()
export class UsersService {
  async getUser(id: string) {
    const res = await db
      .get({
        TableName: process.env.DYNAMODB_TABLE,
        Key: { id },
        AttributesToGet: ['id', 'email', 'firstName', 'lastName'],
      })
      .promise();
    if (res.$response.error || !res.Item) {
      throw new InternalServerErrorException(res.$response.error);
    }
    return res.Item;
  }
 async getUsers() {
    const res = await db
      .scan({
        TableName: process.env.DYNAMODB_TABLE,
        AttributesToGet: ['id', 'email', 'firstName', 'lastName'],
      })
      .promise();
    if (res.$response.error) {
      throw new InternalServerErrorException(res.$response.error.message);
    }
    return res.Items;
  }
}

The UsersService class provides the business logic for handling user-related operations. Here's a breakdown of the code:

getUser(id: string): This method retrieves a user from the DynamoDB table based on the provided id. It uses the get method of the DynamoDB.DocumentClient to fetch the user's data from the table. If there is an error or the item is not found, an InternalServerErrorException is thrown. Otherwise, the retrieved user is returned.

getUsers(): This method retrieves all users from the DynamoDB table. It uses the scan method of the DynamoDB.DocumentClient to perform a scan operation on the table. If there is an error during the scan operation, an InternalServerErrorException is thrown. Otherwise, the array of retrieved users is returned.

Finally, let’s look at the UsersModule defined in the users.module.ts file:

// apps/users/users.module.ts
import { Module } from '@nestjs/common';
import { UsersService } from './users.service';
@Module({
  imports: [],
  providers: [UsersService],
})
export class UsersModule {}

The UsersModule is a basic NestJS module that imports no other modules (imports: []) and provides the UsersService as a provider (providers: [UsersService]).

In summary, the code provided demonstrates the implementation of the Users service in the NestJS monorepo. It includes Lambda handlers for handling API requests, a UsersService class for performing user-related operations using DynamoDB, and a UsersModule that defines the service as a provider.

In addition to that, a typescript configuration file needs to be added that provides compiler options for the Users service. It extends the root tsconfig.json file located in the monorepo root directory and provides specific compiler options and file inclusion/exclusion rules for the Users service within the NestJS

//apps/users/tsconfig.app.json
{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "declaration": false,
    "outDir": "./dist",
  },
  "include": ["src/**/*", "src/*"],
  "exclude": ["node_modules", "dist", ".build", "test", "**/*spec.ts"]
}

DynamoDB Setup

As you can see or notice we are using DynamoDB as the main database for our application and it needs some setup with a serverless framework to work properly so below you can see how our file will look when we add Dynamodb.

#apps/users/serverless.yaml
service: users

plugins:
  - serverless-offline

custom:
    serverless-offline:
        httpPort: 3003
        lambdaPort: 3005

functions: 
  getUser:
    handler: dist/main.getUser
    events:
      - http:
          method: GET
          path: /users/{id}
          request: 
            parameters: 
              paths: 
                id: true
  getUsers:
    handler: dist/main.getUsers
    events:
      - http:
          method: GET
          path: /users

provider:
  name: aws
  region: eu-west-3
  runtime: nodejs16.x
  stage: dev
  environment:
    DYNAMODB_TABLE: ${self:service}-${opt:stage, self:provider.stage}
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:Query
        - dynamodb:Scan  
        - dynamodb:GetItem
        - dynamodb:UpdateItem
      Resource: 
        Fn::Join:
          - ''
          - - "arn:aws:dynamodb:${opt:region, self:provider.region}:*:table/"
            - ${self:provider.environment.DYNAMODB_TABLE}

resources:
  Resources:
    UsersTable:
      Type: AWS::DynamoDB::Table
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
        TableName: ${self:provider.environment.DYNAMODB_TABLE}

So as we can see our serverless file has been updated to add the following things:

IAM permissions: The IAM role for the users service has been updated to allow the following DynamoDB operations:

Scan
GetItem

DynamoDB table: The DynamoDB table is used to store data for the users service. The table name is a concatenation of the service name and the stage. This allows you to have multiple tables for the same service, each with a different stage.

Resources: A new DynamoDB table resource has been added to the users service. This resource defines the DynamoDB table that will be created for the service.

Items Service Implementation

Now let’s focus on the implementation of the Items service. We’ll take a closer look at the files that describe the Items service, its model, main file, and serverless.yaml. We’ll also cover the setup of DynamoDB and the usage of the “serverless offline” plugin

#apps/items/serverless.yaml
service: items

plugins:
  - serverless-offline

custom:
    serverless-offline:
        httpPort: 3006
        lambdaPort: 3008

functions:
  createItem:
    handler: dist/main.createItem
    events:
      - http:
          method: POST
          path: /items
  getItem:
    handler: dist/main.getItem
    events:
      - http:
          method: GET
          path: /items/{id}
          request: 
            parameters: 
              paths: 
                id: true
  getItems:
    handler: dist/main.getItems
    events:
      - http:
          method: GET
          path: /items

provider:
  name: aws
  region: eu-west-3
  runtime: nodejs16.x
  stage: dev
  environment:
    DYNAMODB_TABLE: ${self:service}-${opt:stage, self:provider.stage}

  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:Scan  
        - dynamodb:GetItem
        - dynamodb:PutItem
      Resource: 
        Fn::Join:
          - ''
          - - "arn:aws:dynamodb:${opt:region, self:provider.region}:*:table/"
            - ${self:provider.environment.DYNAMODB_TABLE}

resources:
  Resources:
    ItemsTable: 
      Type: AWS::DynamoDB::Table
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
        TableName: ${self:provider.environment.DYNAMODB_TABLE}  

//apps/items/items.module.ts
import { Module } from '@nestjs/common';
import { ItemsService } from './items.service';

@Module({
  imports: [],
  providers: [ItemsService],
})
export class ItemsModule {}

//app/items/items.service.ts
import { Injectable, InternalServerErrorException } from '@nestjs/common';
import { v1 } from 'uuid';
import { DynamoDB } from 'aws-sdk';

const db = new DynamoDB.DocumentClient();

@Injectable()
export class ItemsService {
  async createItem(item: any) {
    const { title, description } = item;
    const createdOn = new Date().getTime();

    const data = {
      TableName: process.env.DYNAMODB_TABLE,
      Item: {
        id: v1(),
        title,
        description,
        createdOn,
      },
    };

    try {
      await db.put(data).promise();
      return item;
    } catch (error) {
      throw new InternalServerErrorException(error.message);
    }
  }

  async getItem(id: string) {
    const params = {
      TableName: process.env.DYNAMODB_TABLE,
      Key: { id },
    };

    try {
      const result = await db.get(params).promise();
      return result.Item;
    } catch (error) {
      throw new InternalServerErrorException(error.message);
    }
  }

  async getItems() {
    const params = {
      TableName: process.env.DYNAMODB_TABLE,
    };

    try {
      const result = await db.scan(params).promise();
      return result.Items;
    } catch (error) {
      throw new InternalServerErrorException(error.message);
    }
  }
}

The tsconfig.app.json file for the Items service is the same as the one used for the Users service.

Deploying to AWS

To deploy our serverless application to AWS using the Serverless Framework, we can start by adding the following scripts to our package.json file:

"scripts": {
  "build:items": "nest build --tsc items",
  "build:users": "nest build --tsc users",
  "build:all": "npm run build:users && npm run build:items",
  "deploy": "npm install && npm run build:all && npx serverless deploy",
  ....
 },

These scripts provide a convenient way to build and deploy the application, allowing you to easily compile the TypeScript code and package it for deployment using the Serverless framework.

"build:api": "nest build --tsc api": This script builds the API module using the NestJS CLI. It compiles the TypeScript code in the api module and generates the corresponding JavaScript files.
"build:items": "nest build --tsc items": This script builds the items module using the NestJS CLI. It compiles the TypeScript code in the items module and generates the corresponding JavaScript files.
"build:users": "nest build --tsc user": This script builds the users module using the NestJS CLI. It compiles the TypeScript code in the users module and generates the corresponding JavaScript files.
"build:all": "npm run build:items && npm run build:users": This script is a convenience script that runs the build scripts for all the modules (items, and users) sequentially. It ensures that all the modules are built before proceeding to the deployment.
"deploy": "npm install && npm run build:all && npx serverless deploy": This script handles the deployment process. It first installs the required dependencies (npm install), builds all the modules (npm run build:all), and then deploys the application using the Serverless framework (npx serverless deploy).

Adding the necessary packages and deploying to AWS:

package:
  exclude:
    - ../../node_modules/**
    - ./src/**
    - ./test/**
  include:
    - '../../node_modules/@nestjs/common/**'
    - '../../node_modules/@nestjs/core/**'
    - '../../node_modules/@nestjs/schematics/**'
    - '../../node_modules/@nestjs/testing/**'
    - '../../node_modules/tslib/**'
    - '../../node_modules/reflect-metadata/**'
    - '../../node_modules/uid/**'
    - '../../node_modules/rxjs/**'
    - '../../node_modules/iterare/**'
    - '../../node_modules/@nuxtjs/**'
    - '../../node_modules/fast-safe-stringify/**'
    - '../../node_modules/path-to-regexp/**'
    - '../../node_modules/cache-manager/**'
    - '../../node_modules/class-transformer/**'
    - '../../node_modules/class-validator/**'
    - '../../node_modules/cache-manager/**'
    - '../../node_modules/@angular-devkit/**'
    - '../../node_modules/jsonc-parser/**'
    - '../../node_modules/pluralize/**'
    - '../../node_modules/body-parser/**'
    - '../../node_modules/cors/**'
    - '../../node_modules/express/**'
    - '../../node_modules/multer/**'
    - '../../node_modules/@vendia/**'

In the provided code snippet, the package key is added to the serverless files for both the items and users services. This key is used to configure the packaging process of the Lambda functions, determining which files and directories should be included or excluded in the deployment package.

The exclude property within the package the configuration specifies patterns to exclude files and directories from the deployment package. In this case, the patterns ../../node_modules/**, ./src/**, and ./test/** are used. The ../../node_modules/** pattern excludes all files and directories within the node_modules directory, ensuring that dependencies installed via npm or yarn are not included in the deployment package. The ./src/** and ./test/** patterns exclude the source code and test files, respectively, as they are not required for the deployed application to function properly.

On the other hand, the include property lists specific packages that are necessary for the Lambda functions to execute correctly. The packages specified in the include property are included in the deployment package, ensuring that the functions have access to their required dependencies. In the provided code snippet, various NestJS packages (@nestjs/common, @nestjs/core, @nestjs/schematics, @nestjs/testing), as well as other essential packages (tslib, reflect-metadata, uid, rxjs, iterare, @nuxtjs, fast-safe-stringify, path-to-regexp, cache-manager, class-transformer, class-validator, @angular-devkit, jsonc-parser, pluralize, body-parser, cors, express, multer, @vendia) are included.

By configuring the package property in this way, we optimize the deployment package size by excluding unnecessary files and only including the required packages. This results in faster deployment times and reduces the memory footprint of the Lambda functions.

Once the package configuration is in place, and running the command npm run deploy in the root directory triggers the deployment process. The Serverless framework packages the Lambda functions along with the specified packages and deploys them to AWS.

Overall, the package configuration helps address two important issues: including the required packages in the deployment and excluding unnecessary files to optimize the package size.

Running it offline

Before we dive into the technical details, let’s address a fundamental question: why should we run our serverless functions offline in the first place? By doing so, we unlock several advantages that significantly enhance our development process.

Ease of Development:

Running serverless functions offline provides developers with the freedom to work on their code locally, without the need for a live serverless infrastructure, resulting in a faster and more efficient development workflow.

Cost Savings: Optimizing Your Budget

Serverless functions are billed based on their usage, such as the number of invocations and execution duration. Deploying and testing functions in a live environment during development can lead to unexpected costs.

Rapid Iterations: Accelerating Your Progress

Offline execution empowers developers to iterate rapidly and test their serverless functions without any deployment delays. With the ability to make code changes, run functions locally, and observe immediate results.

Isolation and Debugging: Mastering the Art of Troubleshooting

Running serverless functions offline provides a controlled and isolated environment for debugging, this isolation simplifies the troubleshooting process, enabling us to identify and fix issues without the complexities introduced by cloud-based environment.

Now, let’s explore how we can implement offline execution for our serverless functions using the Serverless Framework. Going back to our package.json file, we can add the following scripts to enable offline execution for specific services:

"scripts": {
  "start:dev:api": "npm run build:api && serverless api:offline",
  "start:dev:items": "npm run build:items && serverless items:offline",
  "start:dev:users": "npm run build:users && serverless users:offline",
  "start:dev": "concurrently \"npm run start:dev:items\" \"npm run start:dev:users\"",
  ...
}

The added scripts in the package.json file allows us to run the items and users services offline for development and testing purposes. Here's an explanation of each script:

"start:dev:items": "npm run build:items && serverless items:offline" azeazthis script builds the items service by running npm run build:items, which compiles the TypeScript code into JavaScript, after the build process, it starts the items service in offline mode using the serverless items:offline command, running the items service offline allows us to test and work with it locally without the need for a live serverless infrastructure.
"start:dev:users": "npm run build:users && serverless users:offline" This script builds the users service by running npm run build:users, which compiles the TypeScript code into JavaScript, after the build process, it starts the users service in offline mode using the serverless users:offline command, running the users service offline enables us to test and interact with it locally without deploying it to a production environment.
"start:dev": "concurrently \"npm run start:dev:items\" \"npm run start:dev:users\"" This script uses the concurrently package to run multiple scripts concurrently, It starts the items and users services simultaneously in offline mode running both services together allows us to test the integration between the items and users services locally, mimicking the behavior of a real production environment.

By running these scripts, we can easily start the items and users services locally and test them in an offline mode. This setup enables faster development iterations and facilitates debugging without the need for a fully deployed infrastructure. It provides a convenient way to work on and test specific services independently or together as part of a larger system.

In this post, we have built our application using a serverless framework combined with Nestjs, there still be a lot of features in both of these tools that we will discover in the upcoming posts, and we will also fix some mistakes and improve our app by adding more features to it.