Forem: Gaurav Keswani

Event-Driven Serverless Computing

Gaurav Keswani — Fri, 31 Jul 2020 20:53:36 +0000

In a previous post, I wrote about serverless computing and why it should be on your mind while building software applications in today's world.

What is Serverless Computing? 🤔

Gaurav Keswani ・ Jul 31 ・ 3 min read

#serverless #aws #azure #googlecloud

Wikipedia defines serverless computing as:

Serverless computing is a cloud-computing execution model in which the cloud provider runs the server, and dynamically manages the allocation of machine resources.

Simply put, serverless computing is a paradigm of computing that entirely manages the deployment and execution of the code while allowing the developer to focus on their application logic.

Serverless Design Patterns

Serverless computing has three major design patterns:

Event-driven data processing
Web applications
Event workflows

In this post, I'll go into details for the "Event-driven data processing" pattern and follow up with posts for the other two at a later time.

Event-Driven Data Processing

One of the most common applications for serverless computing is to trigger actions after an event occurs. This pattern is known as the event-driven data processing pattern.

Use Case

Allow client applications to upload images to AWS S3 and compress the image.

Solutions

One way of building a system that does this would be to use a batch process that continuously polls our S3 bucket for newly uploaded images. Once our batch process finds a new image, it can compress it and store the compressed image in the destination S3 bucket.

While this would do the job, one can argue that there is a better way to do it. Instead of having a batch process continuously poll the S3 bucket (which is a massive waste of resources😔), we can have our application submit the image to a message queue (like AWS SQS) and have the queue workers do the job of compression and loading into AWS S3.

While the second solution with the message queue is better than the first one, there are two issues with it:

We must have the queue workers (batch processes that poll for messages from AWS SQS) running at all times😔 irrespective of whether a client is submitting images or not.
Our client application now has to submit to SQS instead of doing what it was originally supposed to do i.e. store images in S3. While this isn't a huge deal, it leaks unnecessary information to the client.

This is where serverless and the event-driven data processing pattern comes into the picture. The serverless solution would let the user upload the image to S3 and have a lambda function triggered every time a new image is uploaded. This lambda function would have to contain the application logic to compress the image and store the compressed image back into the destination S3 bucket.

Apart from being straightforward, using a serverless solution in this scenario is also much cheaper🤑 . This is because you only get charged for compute resources when an image is uploaded and compressed compared to the previous "server-full" solutions where you need to have a process running at all times.

Summary

At a very high level, event-driven serverless solutions have the following benefits when used in the right scenarios:

Developer Agility: Developers can focus their efforts on application logic and deliver business value quicker rather than focussing on infrastructure issues
Elasticity: The infrastructure scales down all the way to 0 if there are no incoming requests with little to no configuration.
Reduced Cost: We pay nothing for the periods when no requests coming in.

Thank you for reading! ❤️

What is Serverless Computing? 🤔

Gaurav Keswani — Fri, 31 Jul 2020 20:44:06 +0000

Please note that this blog post is a compilation of my thoughts after working with different serverless solutions and reading the material provided by Amazon Web Services (Lambda Functions), Microsoft (Azure Functions), Google (Cloud Functions) and Twilio (Twilio Functions). The aim is to introduce you to the concept of serverless to pique your interest around this new computing paradigm.

Okay. So what is Serverless?

Serverless architecture (also known as serverless computing) is a computing pattern where the application is hosted by a third-party service in a way where the developer does not need to manage the server software or hardware😍.
Simply put, serverless computing is a paradigm of computing that entirely manages the deployment and execution of the code while allowing the developer to focus on their application logic.

And how does it differ from a PaaS (Platform as a Service) solution?

In the world of PaaS, your applications are deployed as a single unit using some kind of web framework like Ruby on Rails, Django, etc. These deployments can only be scaled at the application level and you have no choice but to run multiple instances of your entire application to handle any additional load😔.

With Serverless, also known as FaaS (Function as a Service), you can compose your application into individual, autonomous functions. Each function can be scaled individually and automatically as the function call frequency increases or decreases😃.

But why is this important?

Being able to scale your functions individually and independently of the overall application is a very cost-effective🤑 way of paying for compute resources. This is because you will only pay for the number of times your function was called and the amount of time it spent performing the computation rather than paying to have your application always on and waiting for requests to come in.

So when should you consider using Serverless?

When you have a high volume of activities (like web requests, kew messages, transactions) and you want to isolate and scale tasks independently.
Dynamic workloads: If you have dynamic workloads or run something infrequently, it might not make sense paying for compute 24 x 7. Using the serverless pattern ensures that you don't get charged until your functions are called.

Wait. Does my cloud vendor even support this?

All major cloud vendors have an implementation of serverless computing in 2020 and so there is really no excuse to ignore it😜.

Amazon calls their implementation AWS Lambda, Microsoft calls their Azure Functions while Google calls it Cloud Functions. And while these names might differ they all pretty much do the exact same thing😂.

AWS Lambda
Azure Functions
Google Cloud Functions

Shouldn't we be building vendor-agnostic solutions though?

If you really want to, sure. If you're someone that is on the multi-cloud train and aims to build a vendor-agnostic solution, there is something for you as well😉

The Serverless Application Framework

Why don't we migrate everything to serverless then?

Serverless computing has some major benefits if used in the right way:

Reduced service costs 💵 since you only pay for what you use and when you use it.
Simplified scalability management - The cloud vendor (AWS in this case) manages uptime, availability, reliability, etc and most companies in the world would be better off letting the cloud vendor manage this for them.
Quicker and smaller deployments since the code is now decoupled into smaller units

While there are clear benefits to using serverless, it comes with its own risks that need to be weighed before committing to it:

Immaturity of the ecosystem and lack of tooling
It is difficult/expensive to migrate current legacy systems/monoliths (data migration, people training, code rewrites, etc)
The decoupling can lead to too many small parts to manage 😕 if things are not planned well. This is something we're seeing today in the discussion of monoliths and micro-services as well.

Like most things in life, serverless comes with its own pros and cons. That being said, if used in the right scenario and built thoughtfully, the pros quickly outnumber the cons.

Thank you for reading! ❤️

Why code reviews should be top of mind for all engineering teams?

Gaurav Keswani — Fri, 31 Jul 2020 20:26:40 +0000

Code reviews are a great opportunity for developers to learn from each other and improve their skill set. At the same time, it's our last chance to avoid shipping bugs to production. Make sure you make good use of them instead of falling into a common trap

Why are code reviews important?

The value it brings to the team

Code review is one of the most crucial processes for ensuring knowledge transfer and best practices throughout a development team. As important as code reviews are, we typically leave it up to the reviewer to improve their process and this usually results in inconsistent review quality across a team. The poor reviewers become known and folks looking for an easy approval know who to go to.

Your team is only as good as your weakest reviewer.

Catching errors early

One part of managing a software-engineering process is catching problems at the "lowest-value" stage - that is, at the time at which the least investment has been made and at which problems cost the least to correct. To achieve such a goal, developers use "quality gates", periodic tests or reviews that determine whether the quality of the product at one stage is sufficient to support moving on to the next.

Knowledge Sharing

A code review has 4 purposes:

To make everyone on the team better
To make everyone on the team better
To make everyone on the team better
To catch errors or code smells before they become bugs or tech debt

That's not me forgetting to change the 2nd and 3rd reasons. The first 3 reasons are the same because it's that much more important than anything else you do in a code review.

Making knowledge communal makes everyone better.

Making everyone on the team better means that everyone understands the codebase. Everyone understands the data flow. Everyone understands the reasoning behind not just the mechanics of why something is the way it is. At least someone at the code review learns something and this does not necessarily have to be the person whose code is being reviewed. While this may not always happen, it should always be the goal.

Promoting Collective Ownership

The original intent and a continued motivation behind the code review process is that they help us take collective ownership in the creation of our software. In other words, we're all stakeholders in the development process by having a hand in controlling the quality of our products.

Code Review Best Practices

Keep reviews small

Giving a valuable code review requires that pull requests are small. It's unfair to give a reviewer a diff consisting of hundreds or thousands of lines of code changes. They'll end up spending anywhere from 30 to 60 minutes reviewing that diff and then have to do re-reviews when the reviewee makes the suggested changes.

A larger review also leads to a lot more time spent in the process and a lower quality output than desired. This effect has been measured by multiple companies and is documented below as the inverse correlation between lines of code and defect density. (i.e. number of defects discovered in the code review stage)

Always discuss architecture

Making sure that things are architected in a consistent and correct way is really important. This doesn't just benefit the piece of code being added, but also future hires. If things are consistent in terms of architecture it's easier for new people to understand how things fit together.

Link to relevant style guidelines

If you find code that needs to be changed to conform to your team's style guidelines or language guidelines (like PEP8 for Python), link to a document that outlines this.

If you find yourself commenting on style frequently, you should automate code style through hooks. The idea should be to focus on reviewing the code for correctness rather than style!

Constructive criticism

It is very important to cultivate a constructive criticism culture in your team and value it the most.

To have someone say all kinds of stuff about your code can be tough. We learn that criticism means you did something bad, you screwed up. And we tend to take it personally, to get offended, to feel not good enough. This is one attitude each and every one of us needs to get over. To not take criticism personally but as a suggestion of how to become better and constantly improve.

It is very important to have someone that challenges your code constantly. There is no faster way of becoming a better programmer.

As a senior developer, should I be doing code reviews?

Absolutely. In fact, you should be doing more code reviews as your responsibility on the team grows. There are multiple reasons around this:

There's always something to learn, even from more new developers.
You can understand the problems that particular developers face and help them out.
Spotting repeating problems allows senior developers to start thinking about raising those issues in team meetings, creating more guidelines around them, giving presentations or writing articles, or possibly writing custom static analysis rules to catch these problems before code reviews.

Communication is key

As a code reviewee, it's fine to conduct a drafting/work in progress review to solicit preliminary feedback (especially for large changes), but make sure to explicitly communicate this with the reviewer to avoid confusion. Also, make sure to communicate with your reviewer when your review has left the "drafting" state.

As a code reviewer, remember that reviews don't need to be finished in one sitting (especially large ones). If the review is large, review a chunk of code at a time and communicate your progress. Communication is important to avoid multiple back and forth and to avoid giving a false sense of being near to completion to the code reviewee. This will then allow the code reviewee to plan their day and other tasks accordingly while also communicating the correct estimations to the project lead and manager.

Point out good lines of code

It is good practice to let the code reviewee know when you were impressed with a particular code snippet or learnt something new from their code review. This behaves like a morale booster for the reviewee and also lets them know that the approach they tried was better than the status quo of the code base.

A shout out like this also leads to a permanent registry of this new way of doing things in their mind (and the team's mind!) and might encourage them to try more of their ideas in the future instead of sticking to the way things are usually done on the team.

Give shoutouts when you learn something new because it is in those moments that you level up from your local maxima as an engineering team.

Thank you for reading❤️