Forem: Renato Byrro

How about an AWS Lambda Spot with 70% savings?

Renato Byrro — Thu, 14 May 2020 13:39:38 +0000

A couple of days ago I published this tweet:

// Detect dark theme var iframe = document.getElementById('tweet-1260296325456891905-310'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1260296325456891905&theme=dark" }

Some people confused my idea saying that Lambda is already Spot. I understand the position they come from. Lambda only charges per usage, it's not 'rented'.

But the idea I have for Spot is the situation when AWS has idle servers lingering around their data centers, in which they incentivize low-value workloads by lowering prices. Low revenue is better than nothing in those cases.

Workloads that wouldn't be cost-effective to run on EC2, become interesting on Spot instances.

Spot has particularities:

AWS can terminate the instance at any moment (when someone else is willing to pay more for it), so your workload must be super fault-tolerant;
We can bid how much we'd like to pay for each workload, so we'll use the Spot instances only when the price is low enough;

The same could happen for Lambda. We store invocations in a particular queue and integrate it with a Lambda Spot, setting a price per invocation and per GB-seconds. When the price is low enough, the Lambda platform will start pulling messages and processing them.

Under the hood, Lambda runs on Firecracker, which runs on EC2 instances. What we need is a Firecracker running on Spot Instances. They could go away on a snap, which would halt all Lambda executions, but that's fine.

Leave your suggestions in the comments. Like ❤️ the article if you like the idea 💡

Why Serverless Apps Fail and How to Design Resilient Architectures

Renato Byrro — Tue, 21 Apr 2020 12:48:14 +0000

By observing 100,000's of serverless backend components monitored for 2+ years at Dashbird, experience shows that Serverless infrastructure failure boils down to:

Throughput and concurrency limitations due to scalability mismatches;
Increased latency & associated timeout errors;
Inadequate management of resource allocation;

Isolated faults become causes of widespread failure due to dependencies in our cloud architectures (ref. Difference of Fault vs. Failure). If a serverless Lambda function relies on a database that is under stress, the entire API may start returning 5XX errors.

You may think this is just a fact of life, but we can dodge or at least mitigate these failures in many cases.

Serverless is not a magical silver bullet. These services have their limitations, especially to scalability capacities. AWS Lambda, for example, can increase concurrency level up to a certain level per minute. Throw in 10,000 concurrent requests out of thin air and it will throttle.

A typical architecture looks like this:

It usually works well under a low scale. Put in more load a single component's fault can bring the whole implementation to its knees.

Consider this scenario: due to market reasons, API Endpoint 1 starts receiving an unusual amount of requests. Your clients are generating more data and your backend needs to store it in the RDS instance. Relational databases usually don't scale linearly to the I/O level, so we can expect an increase in query latency during this peak demand. API Endpoint 1 or Lambda function 1 will start timing out at some point due to the database delays.

Another possible fault scenario is throttling from Lambda function 1 due to a rapid increase in concurrency.

Not only API Endpoint 1 will become unavailable to clients, but also the second endpoint. In the first scenario, Endpoint 2 also relies on the same RDS instance. In the second scenario, Lambda function 1 will consume the entire concurrency limits for your AWS account, causing Lambda function 2 to throttle requests as well.

We can avoid this by decoupling the API Endpoint 1 and Lambda function 1. In the example, our clients are only sending information that needs to be stored, but no processing and customized response are needed. Here is an alternative architecture:

Instead of sending requests directly from API Endpoint 1 to the Lambda function 1, we first store all requests in a highly-scalable SQS queue. The API can immediately return a 200 message to clients. The Lambda function 1 will later pull messages from the queue in a rate that is manageable for its own concurrency limits and the RDS instance capabilities.

With this modification, the potential for widespread failure is minimized by having a queue absorbing peaks in demand. SQS standard queues can handle nearly unlimited throughput. At the same time, all components serving Endpoint 2 can continue to work normally, since data consumption by the Lambda function 1 is smoothed out.

This is a simplified example, there are several aspects to consider in terms of potential failure points and architectural improvements. We hosted a webinar to cover these topics in much more depth, which you can watch here or below:

Monitoring 101: AWS CloudWatch vs Serverless-oriented

Renato Byrro — Thu, 26 Mar 2020 22:33:07 +0000

At this moment, billions of people are rushing to the internet for work, entertainment, shopping - everything, really. It's great that we developed this virtual world and can keep the lights on, despite what’s happening outside.

On the other hand, cloud systems and developers are under pressure to meet an unparalleled demand. At Dashbird, we have always thought developers deserve the most efficient tools to discover and resolve issues in order to keep cloud apps running smoothly.

For applications running on AWS, CloudWatch might have been enough so far. In our research, though, we identified that teams relying solely on CloudWatch tend to lag behind on issue discovery and resolution time.

We have compiled this feature comparison list between Dashbird and AWS CloudWatch to help you decide on the most efficient tool for your specific needs:

If you haven't yet, you can start your free 14-day trial with Dashbird today and test out the premium features yourself. No credit card required.

The Serverless Trilemma: Build High-Quality Architectures

Renato Byrro — Mon, 23 Mar 2020 17:21:30 +0000

Hey, we just published a video in the Serverless Well-Architected series. It covers three essential concepts to high-quality architectures:

Black-box
Double-billing
Substitution

Let me know your thoughts in the comments. Subscribe to receive updates on future content. Try a serverless monitoring tool that can help you improve your architecture.

The Serverless Trilemma

Renato Byrro — Wed, 26 Feb 2020 13:51:20 +0000

Quick Intro

The Serverless Trilemma is a framework developed by researchers in IBM a couple of years ago (full academic paper). It helps us evaluate the quality of architectural designs and make better decisions by understanding the trade-offs involved.

This is an introductory article, we'll publish more hands-on guidance on how to apply these principles in next articles on this series. To stay tuned, please subscribe here.

Basic Constraints

There are three desired properties we should look for in a serverless architecture:

Black-box
Double-billing
Substitution

Black-box

The Black Box principle indicates that each function should work
independently from the rest of the system and no implementation details should be leaked.

Consider two functions running on AWS Lambda: A and B.

Function A uses the AWS SDK to invoke Function B.

What happens if we need to move the Function B code to another infrastructure? Perhaps the AWS Lambda memory limits aren't enough anymore.

The AWS SDK invocation process will not work anymore. In order to migrate Function B, a concomitant modification is required in Function A.

This type of architecture makes it difficult to introduce changes and increases the likelihood of unintended consequences.

The problem here is that Function B is leaking its implementation details, thus not working as a Black-box. The leakage is having Function A aware that Function B is deployed in AWS Lambda.

To satisfy the Black Box principle, we could create an HTTP API. By routing requests through the API, we can decouple the functions. Later we can just update the API routing to the new infrastructure when migrating Function B.

This simplifies change, which can be extremely valuable as the system and development team grows.

Double-billing

This principle is relatively simple: two functions should not run at the same time while one is in idle state waiting for the other.

In Serverless, we pay for the function execution time. Consider Function A starts processing and being billed. At some point, it invokes Function B and waits for a response.

Problem is: While B is processing, A continues to be charged, despite being idle. Function B processing time is effectively being charged twice, which is arguably undesirable.

Substitution

Substitution is about having a composition of functions behaving just like any normal function.

But what is composition?

Consider a user requests an invoice from a billing system, which is served by a serverless function. This function may rely on a second one to apply any discounts available to this user. Yet another function is responsible for calculating Sales Tax, for instance.

All three functions are composed to work as a group, hence the name "composition".

The substitution principle indicates that all three functions
together should behave like a function. From invocation methods to timeout to dead-letter-queue, everything behaves as if they were a single function.

The requester doesn't actually know this is a composition of functions. Any developer can rely on this composition without worrying about how it behaves, they just apply everything they already know.

This property improves productivity and overall quality by reducing errors and unintended consequences.

Trade-offs involved

Most of the time, it is not possible to have all the properties together. Applying the Black box principle is usually easy. When it comes to the other two, there is usually a trade-off.

When we prioritize one, such as Substitution it becomes difficult to apply Double-billing.

The Serverless Trilemma framework helps to make architectural decisions
and identify what needs to be prioritized:

Black Box improves maintainability, scalability, and resilience by decoupling components
Double-billing enables cost-efficiency and reduces the overall concurrency level of the system
Substitution enables code reusability and improves the quality

What should be prioritized will really depend on each case. We'll dive into more details about this decision-making process in the next articles coming on this series.

To stay tuned, subscribe here for free to receive announcements when we publish again.

Production-ready: Microservices Composition and Orchestration

Renato Byrro — Mon, 20 Jan 2020 11:15:30 +0000

Originally published in the Cloud Knowledge Base

A strong and mature trend in modern cloud software development is to implement components that are:

Single-purpose
Small
Deployed independently
Easily testable

These component services are then composed (or orchestrated) into a cohesive and organized workflow to fulfill higher-level tasks and accomplish the business or practical purposes expected from the system.

Having independent components enables software reusability and decoupling of the software architecture. As a result, the entire system can be easier to test, extend and maintain.

How about composition in Serverless?

Composing serverless functions, such as AWS Lambda, is not as easy as it may sound. Many benefits that developers appreciate in serverless functions are dependent on certain properties.

For example:

High scalability of functions depends on its stateless property
Pay-per-use model (no idle resource waste) is only possible because functions are ephemeral, run on-demand, with no long-running processes

The same properties that make serverless so attractive are also what poses some challenges for services composition.

In function composition architectures, we would want at least three principles met:

Substitution: being able to replace (or perhaps extend/modify) a component without breaking the rest of the system;
Isolation: each component should be a black-box, no service should be aware of other services' implementation details;
Zero-waste: since serverless is about pay-per-use, we would like to avoid a double-billing situation, for example (more on that below)

Serverless composition challenges

Let's consider a trivial situation: a user creates a free account (no payments involved) and subscribes to an online service. Some tasks should be performed as a result:

Persist the user data in a database store
Send a message to validate the user e-mail address
Notify a webhook to track marketing campaign performance (conversion from lead to a subscribed user)

If each of these tasks should be performed by isolated and independent components, how should we coordinate them together?

A flat, simple implementation would be having a function running behind an API and coordinating all other tasks sequentially. We will see next why this implementation is sub-optimal, but please check the diagram below first:

Recovering the three principles we are looking to meet:

Substitution
Isolation
Zero-waste

The implementation above does not meet any of these desired properties.

Breaks Substitution

First, it makes it difficult to replace and extend components. Suppose this system has been running for a while. Now there is a requirement to also validate the user mobile phone number with an SMS. When CreateUser invokes the ValidateEmail function, it only provides a recipient e-mail address, not a phone number. We cannot just replace ValidateEmail with a more general-purpose ValidateContacts, for example.

Breaks Isolation

Second, the CreateUser function must be aware of other services and at least some of their implementation details. For example, it must be aware that these services are implemented as a Lambda function. This breaks the isolation property.

Breaks Zero-waste

Finally, there is a lot of double-billing going on. While the StoreUserData is communicating with the backend database, for example, the CreateUser keeps waiting in idle state before moving to the next task. Even if the code is implemented asynchronously, there will still be some latency in which multiple functions will be billed at the same time. This breaks the zero-waste property.

Considering options

Let's find out whether we can meet all three desired principles while orchestrating our functions. We will cover several different approaches to function composition below, both in the client and in the backend.

Some of these patterns should be avoided, others are recommended, but have their strong sides and weaknesses and should be considered depending on the context and use case.

Client-side

For processes that have some sort of interaction with a UI (user interface), the orchestration logic could be embedded in the client-side.

In the user subscription example above, this could be structured as such:

The client-side software is responsible for invoking each of the endpoints needed for fulfilling the user creation process. All invocations can be executed in parallel to speed up the overall processing time for the end-user.

There are a few downsides and caveats to this model, though:

Lacks reusability

The client-side model makes code reusability harder. Since enabling reusability is one of the main purposes of designing modular component architecture in the first place, it might not be a good fit for our goals.

The orchestration process is implemented for one single type of client: a web browser. The code implemented for this client cannot be easily transported into a backend serverless environment.

What happens if, for instance, we would like to support programmatic account creation through an API or CLI (Command Line Interface), instead of a browser UI?

One could argue: the same Javascript code running in the browser could be used in a NodeJS serverless function. While that is technically true, from an architectural perspective it may lead to bad implementation.

By moving this client-side diagram architecture to a backend serverless function, we would be back in a situation similar to the sequential model described above. In this model, our composition fails to meet all of the three design principles we are looking for.

Difficult to handle consistency

What happens if the StoreUserData function could not reach the database at the moment the account was being created? The user data would remain pending for storage. Database-wise speaking, the account wouldn't have been created yet.

Should the client still request the validation e-mail?
How about the webhook, should it still be notified?

These decisions will all have to be made in the client, increasing the complexity of the software running on the client-side. Depending on the answers to the questions above, it won't be possible to request all three endpoints in parallel. But responding to the user only after a three-step sequential API interaction might be too long.

Exposes internal business rules

Some business rules that would ideally be kept private have to be exposed on the client-side for the orchestration to run there. This is inherent to running code on the client-side and can't really be solved.

Asynchronous Messaging

We covered asynchronous messaging more extensively in another page, so we won't go into details about the concept here.

It is possible to satisfy all three principles with this architecture: substitution, isolation, zero-waste.

The disadvantages of this approach are when we need more fine-grained control over the workflow logic. The same questions related to handling consistency in the Client-side model also apply here. If one step fails, for example, it is very difficult - if not impossible - to control how other steps of the process should behave.

That doesn't mean this pattern shouldn't be used. As a matter of fact, it is perhaps one of the most commonly and successfully used patterns for a serverless composition strategy that is both scalable and maintainable. It's just not a good fit for all use cases.

Chaining

Each function could be wired up, having one invoking the next after the end of each task execution. The end-user waiting time could be reduced by having the API responding early with a confirmation message that the request was received and is being processed.

This model unfortunately also breaks some of the design principles we are looking for.

It is not possible to easily replace a component without affecting other parts of the system. Substitution is broken.

Different components must be aware of each other and some implementation details are leaked to one another. Isolation is broken. This could be solved by using an internal API Gateway or a Message Queue as an interface between components, though.

The last principle, Zero-waste is satisfied, since there would be no double-billing in this model.

Asynchronous Orchestrator

This pattern involves the usage of a concept called Finite-state Machine (FSM). If you are not familiar, please take a look at this introductory article we published recently before moving forward.

In an FSM, it is possible to:

Easily replace components without breaking the rest of the system
Have components acting as complete black-boxes, without knowledge of each other or implementation details leakage
Avoid double-billing

Some additional advantages to this pattern:

Run multiple tasks in parallel
Easily handle retry in case of failure in a specific step of the process
Handle complex logic such as:

if Step_1 returns True:
    run Step_2
else:
    run Step_3

Along with Asynchronous Messaging discussed above, this is one of the most commonly and successfully used patterns for serverless functions composition.

Although it is a bit more difficult to implement, deploy and test, it meets the needs of more complex workflows while still delivering a scalable and maintainable architectural design.

Most cloud service providers will have managed/serverless offerings of an FSM, which greatly simplifies the implementation and maintenance. AWS, for example, has the StepFunctions service.

References

Leveraging Serverless Cloud Computing Architectures (Master Thesis), by R.T.J. Bolscher

Composition of Serverless Functions, by Olivier Tardieu

Comparison of FaaS Orchestration Systems, by Pedro García López, Marc Sánchez-Artigas, Gerard París, Daniel Barcelona Pons, Álvaro Ruiz Ollobarren and David Arroyo Pinto

The serverless trilemma: function composition for serverless computing, by Ioana Baldini, Perry Cheng, Stephen Jason Fink, Nick Mitchell, Vinod Muthusamy, Rodric M. Rabbah, Philippe Suter, Olivier Tardieu

Solve The Cold Start Issue with Lambda Provisioned Concurrency

Renato Byrro — Mon, 09 Dec 2019 12:57:47 +0000

Originally published in Dashbird Blog

Cold Starts have been a massive issue with FaaS. In summary, it makes functions slower to startup in some cases. That's in the opposite way of every effort to improve web applications performance.

Many efforts have been made in the recent years to solve AWS Lambda cold starts or educate on handling them. Many have mitigated the issue, but none really solved it.

AWS has just made a great progress on the area with the Provisioned Capacity feature announcement. As the function scales up, instead of waiting new requests to come in before provisioning resources to serve them, AWS will proactively provision new instances of the function in advance.

This behavior guarantees the performance of every request will stay within double digit milliseconds, up to the Provisioned Concurrency threshold set to the function. There are some caveats that developers should be aware though. For example: it makes your functions inelligible to the Lambda Free Tier.

Here is a detailed and step-by-step guide to learn everything about this feature

Lambda Provisioned Concurrency is generally available in several regions and already integrated with AWS SAM, CodeDeploy and other serverless frameworks.

Image credits to Denys Nevozhai on Unsplash.

If You Learn to Build Scalable Applications, You Can Change Your Career

Renato Byrro — Mon, 25 Nov 2019 14:40:31 +0000

What we're up to

The web is huge, and it's getting bigger. Every. Single. Minute.

More people. Doing more things. Using more devices. On faster connections.

Everything indicates that load pressure over all kinds of applications is only up to increase: from small to big ones, from B2C through B2B. More companies will need teams that can deliver on the scalability promise.

Likely, only a handful out of every 100 developers nowadays is really into building things that can scale. You can be at the top of the market.

First Steps

A scalable system is one that can continue to perform in a reliable manner under highly variable and often increasing levels of load.

A system's scalability problem is rarely a single variable analysis. It usually involves at least two-dimensional thinking: a load metric and a time-period. More often than not, it's a multi-dimensional beast.

Some two-dimension problem examples are:

How does the database system scales when IOPS¹ increases from 1,000 to 10,000 over one second?
How load time is affected when website pageview requests grow from 200 to 5,000 over one minute?

Bear with me along the next lines to understand the multi-dimension ones.

Load Profile

What we first need is to express what load means for each of our systems.

For a website, load can be online visitors or pageviews per second. For a database, it could be concurrent queries, the number of I/O operations, or the amount of data getting in and out.

How load is described also depends on the system architecture and the business case. This is where things start to get complicated.

In an e-commerce website, for example, the system may scale well to serve 10,000 people shopping at the same time across a thousand-item catalog. But what happens if a huge group is shopping for a single item?

Perhaps a very positive review by popular social-influencer just went viral on social networks.

We don't wanna be the ones explaining why the hot moment was missed due to system restrictions.

When scalability collides with consistency

Take a scalable database system. It will employ some sort of multi-node replication. Think of it as the product description, price, etc. replicated across different servers to handle a huge amount of read requests.

If each server can manage 1,000 RPS (req. per sec.) and we need to handle 10,000 RPS, we'll need at least 10 of those servers. The same data will be replicated in all of them. One will act as the main server to receive updates or delete requests. Once a data point is modified, the main server will notify the other servers to follow along with the update.

This is usually called a Master-Slave replication system² and is very common in database setups.

Think about consistency now. You know "over-booking" in airline seats? That's sort of a consistency issue. Airlines do it consciously. But we don't want two shoppers ordering the same physical item, because the store won't be able to fulfill both orders.

When a purchase is confirmed, the system will decrement the number of items available in stock. And here comes the problem: the DB might be able to handle decrementing hundreds of different items at once. But what happens if thousands try to decrement the same value in a very short period?

This is the sort of circumstance that happens due to market trends and human behavior. Developers must account for these factors when thinking about load and scalability.

Handling Load

The more we strive to anticipate possible challenging load scenarios for the system, the better it will perform in reality.

It is necessary to consider:

The load profiles and metrics
How much and how fast load can vary
Which resources are needed to cope with these variations without hurting performance or reliability

Thinking about load profile

Let's dive deeper into the e-commerce example.

Say the shopping cart module is responsible for checking an item's availability before confirming a purchase. A naive approach is to read the number of items available in the product's profile stored in the database and decrement it right after confirming its availability.

That strategy can lead to bottlenecks and crash under increasing loads because all order requests will be rushing to decrement the same value consistently. Not only the master node but all replicas must be updated at the same time to ensure consistency.

Better approaches

One solution would be using Multi-Master³ replication. This type of system provides the logic to handle concurrent requests. It is usually not trivial to implement, though.

Some database services will provide multi-master out of the box. This is the case of the serverless DynamoDB service, by AWS.

Using this type of service can save a lot of time. Instead of solving the infrastructure cluster and low-level replication issues, we can focus on the scalability issues specific to the user problem at hand.

Another possibility is using a message queue as a buffer, instead of having the shopping cart rushing directly to the database. All the ordered items are placed into a queue. A background job is responsible for pulling orders from the queue and processing them (check availability, decrement stock, etc).

Once all items from an order are processed, another background job can confirm the customer's purchase.

The queue buffer decouples the front and backend systems from the database and allows each to scale separately. It makes tackling scalability challenges much easier. Even when using a Multi-Master database, the message buffer is usually a good architectural pattern to consider.

How load is profiled and scalability challenges are tackled will depend a lot on the use case. There is no one-size-fits-all strategy here.

And here is where you can differentiate yourself in your software career. For many companies (not only larger but especially startups that need to grow fast), it is a must-have for the DEV team to think about scalability, anticipate challenges and build a resilient and scalable system.

Learn how to scale systems, and you can change your career. At the end of this article, there is an indication of a great book, in case you'd like to dive deeper. It's not easy and you can never study the topic too much.

Thinking about resources

Resources can scale:

Vertically (scale-up): increasing CPU power or RAM, for example
Horizontally (scale-out): adding more servers to a cluster, for instance

A great number of healthy architectures will mix both approaches. Sometimes, having many small servers is cheaper than a few high-end machines, especially for highly variable loads. Large machines can lead to increased over-provisioning and wasted idle resources.

When load scales down, it's much easier to kill a couple of idle small machines than to frequently move a system from big to smaller machines.

In other cases, perhaps a big machine would perform faster and cheaper than a cluster of small ones.

It depends on the case and developers must try different approaches to find one that suits both performance requirements and project budget.

Using serverless systems greatly simplifies the level of responsibility developers have over how systems cope with load. These services abstract away decision-making about scaling-up or out, for example, and also provide SLAs that the development team can rely on.

As mentioned above, one great database service is AWS DynamoDB. It solves the lower level scalability and availability issues out of the box.

For small and mid-sized teams, projects that need short time-to-market and fast iterations, using services like DynamoDB can be a great competitive advantage. It allows us to abstract away undifferentiated infrastructure issues to focus on the scalability challenges of the business case at hand.

Mastering these types of services is also a great knowledge acquisition in a developer's scalability belt. Check this Knowledge Base to learn more about scalable databases, compute engines and more.

Load Metrics and Statistics

Metrics will need some sort of aggregation or statistical representation. Average (arithmetical mean) is usually a bad way to represent metrics because it can be misleading. It doesn't tell how many users experienced that level of performance. In reality, no user might have experienced it at all.

Consider the following application load and user base:

User	Response time
A	210 ms
B	250 ms
C	240 ms
D	20 ms

The average response time would be 180 milliseconds. But no user experienced that response time. 75% of the users experienced a performance that is worse than average. The arithmetic mean is highly sensitive to outliers, which is the case of the distribution above.

This is why percentiles are more commonly used to express systems performance. They are also the basis for service level objectives (SLOs)⁴ and agreements (SLAs)⁵.

The most common percentiles are 95th, 99th, and 99.9th (also referred to as p95, p99, and p999).

A p95 level is a threshold in which at least 95% of the response times fall below. In the example above, our p95 would be 250 ms. Since we have only a handful of request samples, it would be the same threshold for all percentiles. If we were to compute a p75, it would be 240 ms, meaning: three out of four (75%) of the requests were served within 240 milliseconds.

Wrapping up

Thinking about a system load and preparing it to scale smoothly under different load profiles is no easy task. That is precisely why you should get better at it.

Gone are the times when only a handful of applications on the web needed to worry about scale. The web is already huge, and it's getting bigger.

More and more people are joining it every day. With more devices: laptops, smartphones, tablets, smartwatches, smart glasses, smart rings... Who knows what else they'll smart'ify next!?

IoT devices are exploding. Networks are only getting faster.

As I said in the beginning: everything indicates that load pressure over all kinds of applications is only up to increase. More and more companies will need teams that can provide scalable systems.

You can be one of the few, top 1% of the market delivering it.

Subscribe to our Knowledge Base and receive notifications when we publish content similar to this one.

Acknowledgements and Footnotes

This article was heavily inspired by the Designing Data-Intensive Aplications book, by Martin Kleppmann. I strongly recommend the read if you are interested in getting deeper into the topic.

Cover Image by Katee Lue on Unsplash
Holding plants image by Daniel Hjalmarsson on Unsplash
Hands with elastic image by NeONBRAND on Unsplash
Woman stretching image by Andrés Gómez on Unsplash
Metrics dashboard image by Luke Chesser on Unsplash
Bull collision image by Uriel Soberanes on Unsplash
Vertical vs. Horizontal scaling image by AccionLabs
Message queue buffer image by Christopher Demicoli
Serverless architecture image by Ritchie Vink

IOPS: Input/Output Operations Per Second ↩
Master/Slave (technology) and Database Replication ↩
Multi-Master Replication ↩
SLO: Service Level Objective ↩
SLA: Service Level Agreement ↩

In a RollerCoaster-Driven Development Era, We Must Connect the Dots

Renato Byrro — Fri, 15 Nov 2019 11:46:26 +0000

🎢 RollerCoaster-Driven Development

Software development has evolved at a thrilling pace in the past years. Every developer has felt genuinely overwhelmed at some point, undoubtedly.

You might be feeling overwhelmed right now with so much to learn and account for in your work. 😟

How many frontend frameworks have we had to learn in five years? How many more there will be in the next ones?

A big stone in the developer journey is finding and organizing trusted sources for learning and advancing its knowledge and capabilities.

Am I doing this right? Is there a better way? 😰
Where can I find a trusted reference for good practices for this?
Is this the most accurate and up-to-date resource for learning XYZ?

Too-much, Too-little

We never had so much information and knowledge readily available like today.

That is absolutely true. 👏

But all this content is scattered. Disorganized. No time-relevancy ordering. Difficult to navigate and reason. It's hard to connect the dots.

There only so much that search engines can do to help ourselves.

Javascript. Typescript. Node.js. Python. SPA. Cloud. Serverless. Less than 20% of technologies available dominate +80% of modern web development stack.

So why does everyone have to go through the very same learning pains over and over? Thousands of developers have to learn the same things. But all. Same. Pain! Why? ❓

✨ Connecting the Dots

We've suffered enough with these pains at Dashbird while cracking our minds around finding the best ways to successful cloud serverless implementations.

We decided to stop. Start to organize everything we find valuable for developers. Compile it in an open repository. A single one. Always up-to-date. :neckbeard:

We called it Cloud Knowledge Base. It is about connecting the dots. So that you don't have to.

It aims to cover everything related to web and mobile development. From basic concepts to productivity tooling to advanced architectural patterns. As well as reliable cloud services that enable successful implementations.

The MVP is live now. 🚀 We will keep working consistently. Perfecting what's published. Expanding with new content. Every day, week, month.

As in any startup endeavor, we kicked off an MVP with a minimal set of topics covered. Subscribe to receive updates 🔥 (by email or Atom feed) as we push forward.

Suggestions and contributions from the development community are more than welcome. Let us know which topics you'd like to see covered, or if you have anything to share or ask: renato@dashbird.io.

Cover image credits: Stephen Hateley on Unsplash

Last Chance to Join: Design Patterns for Secure and Scalable APIs on AWS Lambda

Renato Byrro — Thu, 31 Oct 2019 13:38:53 +0000

Join our Tech Talk today with Eoin Shanaghy (CTO @ fourTheorem) on:

Securing APIs on AWS Lambda

Learn proven and scalable architectural patterns to secure internal APIs on AWS Lambda 🔐

Hands-on presentation, from technical implementation to architectural decision factors 🙌

Interact with panelists to address your questions in real-time 💬

Reserve your seat now.

Is There a Cat in The Code!?

Renato Byrro — Thu, 24 Oct 2019 19:32:22 +0000

Just saw this on Twitter:

😆 😆

How I Fixed JWT Security Flaws in 3 Steps

Renato Byrro — Mon, 14 Oct 2019 08:31:57 +0000

There are just too many ways to do JWT wrong. 😢

And I fell for some... Don't panic, but it's likely to be your case as well.

Check these 3 commonly overlooked security areas on JWT implementations. It will take only a few minutes.

1) Broken libraries

There are +1,600 libraries matching "jwt" on npm. 😳

And +300 on Pypi. 😲

Do we need them all? Certainly not. Are they all secure? I won't trust. 😖

There are several ways your JWT library of choice might be compromised.

Can we cut to a simple solution?

Yes, I am also bored with security blah, blah, blah. 💤

Go to this resource and double-check which libraries follow practices proven to be safe. Most will by now. But better safe than sorry.

2) Unsafe token generation and distribution

The joyfull implementation: 🍀

a. Frontend requests user authentication

b. Backend authenticates and generates JWT

c. JWT is sent in the response body payload

d. Frontend store JWT in the localStorage

Ah, yes... The world would be beautiful without bad guys and if ugly things could not happen. 😇

Well. Back to the real world. 😎

Avoid following the above outline.

To help with items (a) & (b), make sure you selected a JWT library that follows best practices. Or that you implemented correctly on your own. Not that difficult really. Just care enough.

It's also good practice to log every authentication attempt (success and failures) and all contextual data you may possibly have.

JWT is frequently used in Serverless environments (because both are stateless, niiice!).

If that's your case, make sure you have professionals monitoring your logs and alerting you proactively. If that's not your case, the advice still holds. 😉

To address items (c) & (d):

Do not send JWT in the response body payload

Do not store JWT in localStorage

Problem is: any JavaScript code in your frontend is able to access the JWT. And do whatever it wants.

And that's dangerous.

Imagine what can happen if someone manages to inject malicious code in your frontend... and get all your users' JWTs?... Hum... Houston...

No. Instead, the backend should set the JWT as a cookie in the user browser. Make sure you flag it as Secure and httpOnly cookie. And SameSite cookie. Boy, that's a multi-flavor cookie.

This way, the JWT will be available to the backend in all subsequent requests, while remaining outside the reach of potentially dirty JS hands.

In your response body payload, send only what's necessary for the frontend to provide the features expected by the user. Did I mention to not include anything sensitive here? Should not.

I know. A cookie is not as cool as localStorage. But, look, they can be colorful! And SAFE. He's our friend. Let's treat him well. Deal? 🙌 🍪

3) Not handling secret keys securely

Any JWT implementation will involve some sort of secret data. Regardless of using a symmetric (one secret key) or an asymmetric (public/private keys) way to sign tokens.

Personally, I prefer symmetric implementations with HMAC. Simplifies things. But sometimes I use asymmetric RSA. Lately, I have been using the latter only. Well, they'll never know which one I really use. 😜

No one should ever know how YOU implement JWT either. Not to mention your secret or private keys.

Things you should avoid doing with your secret/private key when possible:

💻 Storing in a config file and committing to your Git repo
📣 Sharing with your team on your Drive, Dropbox, Slack, whatever
♻️ Having the same keys for local, test, and production environments

Instead:

✌️ Distribute keys for your development team to use only in local and testing environments
👍 Store production keys in a safe place, only available to the production app
🔐 Keep the production keys away from prying eyes, load them as environment variables, on-demand, protected against unintended access