Forem: Wissam A

My Best Advice for New Hires

Wissam A — Wed, 24 Mar 2021 02:28:38 +0000

I recently hired four new engineers for the API team at GitHub. Starting a new job is never easy, and I feel like it is especially harder when you land at one of your dream jobs.

As part of their interview process and onboarding, I've been thinking a lot about how to help our new teammates be as successful as possible and what advice I can give without overwhelming them with information. It ended up boiling down to this:

Do not rush to show value.

What does that even mean?

I can't speak for everyone, but I and many other new comers had pretty big imposter syndrome when we joined GitHub.

These questions were always top of mind for me: How did I earn this job? Will I live up to expectations? Will they figure out that I'm a fraud? How can I prove myself as fast as possible?

This mentality is more harmful than anything. Rushing to show value is not helpful, and most people tend to do mistakes when they feel rushed.

Furthermore, they do not have to prove themselves. The fact that they successfully went through our interview process and that we extended an offer out to them is proof in and of itself that they deserve it.

In fact, they should trust our decision-making process that led to their hiring, and hopefully remove some pressure off of their shoulders.

Learn to walk before you run

Every step in the onboarding process is important, and the goal is not to skip them in the spirit of moving fast. New hires should focus on learning to walk before they can run. I understand that it is challenging to not be as productive as you know you can be, but the onboarding process is a great opportunity to solidify your foundation and understanding of your new "company's system and architecture.

These steps will serve you well in the future:

Taking the time to digest all the information you're reading.
Asking questions.
Digging deeper into the code to better understand how the system works.
Thinking about the big picture, and how the different parts of the system interact with each other.
Bookmarking important links that you'll want to refer back to in the future.
Pairing with other engineers.
Listening and participating in meetings.
Taking notes.

The goal is to set yourself up for future success. If you skip a bunch of important steps, you (and others) will quickly notice the gaps in your understanding.

Conclusion

Imposter syndrome is real, and a big topic on its own, but ultimately it is important to acknowledge that it exists and to also understand that reacting to it by rushing your onboarding is not the key to success.

Instead, take a more steady approach by solidifying your foundation and understanding of your new company, and you will be better set up to show all the value that you can bring to them.

Photo by Bonneval Sebastien on Unsplash

PS: If you enjoyed this article, you may enjoy some of my other articles on designingforscale.com

Why Management May Not Be Right For You

Wissam A — Sat, 16 May 2020 20:51:11 +0000

I often talk to engineers who are interested in making the big jump to management and becoming engineering managers, but they're not sure if it's the right fit for them. They are (understandably so) concerned if they will like it or not, and what their day to day will look like. Here a few points you should consider before making the big jump into management.

It is not a promotion

Switching to management is not a promotion. Rather, it is a career transition and you'll be starting at the bottom of the career ladder where you'll feel like a novice at the beginning. Depending on your previous position as an Individual Contributor (IC), this might not even come with a raise. However, you'll have an opportunity to increase your impact on the company and business. You'll be learning something new every day. You'll help people grow in their career and as individuals. There are a lot of pros to being a manager, but don't make the switch if your main motivator is to climb the career ladder and make more money. In this case, you might want to stay on the technical track and grow there.

Get ready to code much less

As a manager, you'll be coding a lot less (read: almost not at all), and you have to be okay with that. I personally work on side projects a few hours per week to get my coding fix and keep learning new technologies, but I do not contribute meaningful code at my day job and that's 100% fine with me because my focus is on delivering more impact to the team and business by focusing on higher-leverage areas. A key concept you should also aspire to is to stay involved with code, but stay out of critical paths such as being a required reviewer or a blocker for shipping a feature. Instead, you could work on a bug fix or internal tooling to help improve the team's velocity.

Pictured above are my GitHub contributions as an engineer VS manager

Your definition of success will change

As a manager, your definition of success will change. Successes are far more spread apart than as an Individual Contributor, and you will not be getting the same adrenaline rush. As an engineer, you might be getting multiple successes per day when you ship a fix or merge a PR — that adrenaline rush that feels good. As a manager, your successes might be helping the team re-align on a project, hitting a quarterly objective, helping an engineer get promoted, or writing a proposal that gets approved by the VP. As you can imagine, these successes take longer to build and are more spread out, and the adrenaline rush you get will also be different.

Meetings, meetings, meetings

Your days will mostly be spent in meetings, despite how much you try to reduce the number of hours spent in meetings. I've found certain tricks to help me take better control of my calendar — and it helps a lot — but the bottom line is that you have to attend and even run a lot of important meetings. These meetings include (ideally weekly) 1:1s with your direct reports, team planning meetings, leadership and organization meetings, etc. Because of that, you'll also have less uninterrupted chunks of time so you will be less likely to "be in the zone" unless you are very defensive with your calendar. For instance, I block out large chunks of time on my calendar to ensure that I get regular focus work done and don't jump from a meeting to another.

Finally, this is not a one-way door

It's important to note that making the switch to management should not be a one-way door. I suggest trying it out for 1-2 years to gain experience, a different perspective, and gain insights into how your manager thinks, and then deciding if you want to pursue it further or would prefer switching back to being an engineer. On average, it takes about 2-3 years before your skills erode so you should still be in a relatively good place to jump back to engineering after 1-2 years.

If these do not sound like blockers to you and are motivating instead, then you should be well prepared for making the switch to management. I hope I was able to provide more insight into what a manager's role entails and that I brought up some good points for you to consider if this is the right fit for you or not.

Understanding Redux Middleware And Writing Custom Ones

Wissam A — Wed, 06 Sep 2017 16:26:53 +0000

If you use Redux, you’ve most likely used redux middleware before through - for example - redux-thunk, redux-promise-middleware, redux-saga or redux-logger. These are important middleware in most apps but yet some of us (including me) take this concept for granted without digging any further.

With that said, I recently had to implement a caching solution in a large application and, after doing some research and poking one of my colleague’s brain (thank you Rec!), decided that implementing a custom middleware was the best approach for this, mainly because:

It makes for cleaner code
It makes for more maintainable code (think separation of concerns)
It groups all the caching logic in one location

In this article, I’d like to explain what a Redux middleware is, and how I implemented a custom middleware.

What is a Middleware?

For backend developers, a Redux middleware is similar to a middleware in Express or in ASP.NET. Although it is not exactly the same thing, it’s similar and represents a good way of thinking of this concept.

In Redux, a middleware is used to intercept dispatched actions before they make it to the reducer. This means that when you call dispatch on an action, the action goes through a (or many) middleware before hitting the reducer - if it even makes it that far, but more on that later.

You can apply multiple middleware to a Redux store, which means that the action will have to go through all the middleware before making it to the reducer. The order of execution is actually the order in which you pass the middleware to the store. Also, at any point in a middleware, you can chose to stop forwarding the action, which will end the cycle.

For instance, in my caching middleware, I first check if the same action is already in progress. If it is, I cache the latest payload and interrupt the flow by returning out of the middleware. Since I’m not calling next or dispatch, the action flow will not continue.

Why Use a Middleware?

As expressed above, actions go through middleware before getting to the reducers, which gives us a great way of applying logic or filters to all actions. This means that the logic is grouped in one place instead of being spread across reducers, that we can easily identify where to investigate if a bug occurs, and we can swap that code out if we ever need to.

Some use-cases that benefit from using middleware:

Logging: every action goes through this middleware, so we can log its type and payload for debugging or tracking purposes.
Error tracking: if any asynchronous action returned an error, this middleware can display a notification.
Caching: Only call your API for the same action once, and cache the result for future calls.
Auth requests: For API calls, apply an authentication token before sending out the request.
So much more :)

Writing a Middleware

To define your own middleware, you need to write a function with the following signature:
store => next => action => result

This looks very confusing at first glance - I hear you - so let’s break it down a little bit:

store is the Redux store instance that will be passed to your middleware.
next is a function that you need to call with an action when you want to continue the flow execution, which means passing the action to the next in line: either the following middleware or a reducer.
action is the action that was originally dispatched so that you can access it, apply logic based on the action, and eventually pass it on using next.
result is the value used as the result of the dispatch call.

Finally, to apply this middleware to the Redux store, you need to call applyMiddleware when creating the store through createStore(). Here’s a nice example from the official Redux documentation:

import { createStore, combineReducers, applyMiddleware } from 'redux'

let todoApp = combineReducers(reducers)
let store = createStore(
  todoApp,
  // applyMiddleware() tells createStore() how to handle middleware
  applyMiddleware(logger, crashReporter)
)

In the example above, the middleware logger will be called first, followed by the crashReporter middleware since this is the order in which they were passed to applyMiddleware.

The Caching Middleware

As mentioned in this article’s outline, I implemented a caching middleware recently to solve a very specific issue. I know that there are existing caching middleware out there, but I needed something small and specific to the issue at hand, so I wrote a few lines of code instead of using an existing library.

For this issue, I had to make sure that a WYSIWYG editor only called the backend sequentially when saving the content. For instance, if auto-save kicked in while a save was already occurring, I did not want to send the text to the backend until the previous call completed. The same concept also applies if the user hit the Save button multiple times.

Here’s what my middleware looks like:

export default function textUpdatesMiddleware () {
  return store => next => action => {
    if (action.type === UPDATE_TEXT) {
      // Check if the new text in the payload is different from what we already have in the store
      if (!shouldSaveText(action.payload, store)) return

      // Are we currently saving?
      // isUpdatingText is set to `true` in a reducer
      // A reducer listens to CACHE_TEXT_UPDATE and will store the payload into `pendingTextUpdate`
      // We only cache the latest content, not all of them
      if (store.getState().isUpdatingText) {
        return store.dispatch({
          type: CACHE_TEXT_UPDATE,
          payload: action.payload
        })
      } else {
        // This uses `redux-promise-middleware`
        return store.dispatch({
          type: UPDATE_TEXT,
          payload: {
            promise: http.patch(apiEndpoint, content)
          }
        })
      }
    }
    // This uses the `redux-promise-middleware` convention of _PENDING, _FULFILLED, _REJECTED
    if (action.type === UPDATE_TEXT_FULFILLED) {
      const pendingTextUpdate = store.getState().pendingTextUpdate
      // If we had a pending update
      if (pendingTextUpdate) {
        // A reducer listens to UNCACHE_TEXT_UPDATE and will clear `pendingTextUpdate`
        store.dispatch({ type: UNCACHE_TEXT_UPDATE })
        // Allow the fulfilled action to continue on to the reducers
        next(action)
        // Dispatch the update with the cached content
        return store.dispatch({
          type: UPDATE_TEXT,
          payload: pendingTextUpdate
        })
      }
    }
    // Nothing to do here - keep calm and carry on
    next(action)
  }
}

Based on the code above, it’s worth noting that Redux applies some magic when you call store.dispatch from within a middleware and the action will travel through all the middleware again, including the current middleware that dispatched it. However, when you call next, the action moves on to the next middleware in the flow.

Conclusion

This middleware solves a specific issue I was experiencing, but we could just as well make it more generic so that it applies the same concept to all (or a subset of) actions. At this point, I don’t have any need to make it generic so I did not want to over-engineer it, but it’s worth noting that it’s definitely doable.

If I hadn’t applied this logic in a middleware, I would have had to validate that an API call is not currently in-progress from a reducer, then dispatch calls from the reducer to cache the content, and also listen for the FULFILLED action from the reducer or the then on the http call, and then re-dispatch the action. This gets messy real quick and doesn’t scale well if we need to make it more generic.

I hope this was a good introduction to middleware and that it covered enough of the basics to get you started if you ever need to write your own custom one.

My final piece of advice is that research and discussions are very valuable. I am very glad I decided to not go with my original (bad) approach because something felt wrong and that I did more research, discussed it with a colleague, and ended up settling on using a middleware because the final result is a better solution.

Credit

The official Redux documentation section on Middleware.
You Aren’t Using Redux Middleware Enough.

This post was originally published on Designing for Scale

Business Rules Must Be Enforced by the API

Wissam A — Tue, 15 Aug 2017 18:38:27 +0000

An important concept to keep in mind when designing web applications is that business rules must be enforced on the backend, by the API. In this article, I will demonstrate why this is crucial and how avoiding to do so will open up your application to security issues and bugs.

What Are Business Rules

Before jumping in to the technical stuff, let me clarify what I mean by business rules so that we are all on the same page.

Business rules, sometimes referred to as business logic or application logic, are what governs how your app should function and what validations to apply. For instance, a business rule at Twitter is that tweets cannot contain more than 140 characters (if we dumb it down for the sake of this example and remove logic such as URL shorteners, replying to someone, etc). This means that before sending a tweet, Twitter needs to ensure that it respects the character limit.

I'm over the character limit ðŸ‘†.

Where this validation should be made is exactly the topic of this article. Let's dive in!

Client-Side Validations

Client-side validations are useful in order to provide immediate feedback to the user, which makes for better user experience. For instance, when writing a tweet, we can instantly see if we went over the 140-character limit or not.

However, client-side validations are not enough on their own since they are easily bypassed. Browsers (and mobile apps for that matter) can be "hacked" or toyed with in order to bypass certain restrictions put in by the application's developers. In fact, browsers allow you to inspect and modify the html and JavaScript quite easily, so it's a bad idea to count only on frontend validations.

For instance, it's easy to re-enable the Tweet button in the browser even if we went over the 140-character limit. By doing so, we are bypassing the client-side validation and sending the longer tweet to the backend, as so:

Re-enabling the Tweet button:
Sending the long tweet to the backend, as seen in this POST request:
This is gracefully handled on the server and a friendly error message is displayed to the user:

To protect the integrity of your data and app, it is essential to enforce your business rules on the backend as well.

Backend Validations

Backend validations allow you to take another look at the data - now that it's made its way from the client to the server - and ensure that it respects the business rules. In our previous example, this is where you would ensure that the tweet being sent is indeed 140 characters or less. If not, the server should return an appropriate error to the client so that it is handled gracefully and the user knows what to do next.

The backend is also where you can ensure data integrity by validating the information being saved to the database, making sure it does not contain any ill-intented information, that the types are good, etc. This should also be done at a lower level than the API, such as an ORM or even the database, but I include those in the big backend bubble. Separating concerns and where exactly to apply these validations is a discussion for another article.

One of the reasons why validations on the backend are better suited for business logic is that you have control over your server and the code executed on it, as opposed to client-side code. This is not being run in a browser or on a mobile device and is not being manipulated by a user. It is code that you wrote and know how it will behave.

Conclusion

One important thing I'd like to clarify here is that both frontend and backend validations are important but their main goal is different.

Frontend validations are great for giving feedback to the user. Imagine having a user filling up a long form and submitting it, only for it to be rejected by the server because of an empty input, and prompting the user to re-fill the form. This would obviously be frustrating to users and is avoidable by using frontend validations.

Backend validations, on the other hand, are meant to ensure that the business rules are indeed applied and respected, and for data integrity.

I hope this was helpful and clarified the differences between the two parties. Let me know if you have any questions or comments. Thank you for reading.

This post was originally published on Designing for Scale

Understanding Hoisting in JavaScript

Wissam A — Fri, 28 Apr 2017 17:34:00 +0000

You know JavaScript, but do you really know JavaScript? It's a great language, even though some may argue otherwise. Sure, it's got some bad parts, but it has improved a lot in past years and developers are getting much better at using JavaScript correctly and at following best practices. Strict mode is also getting better at preventing newer developers from making some bad JavaScript mistakes and unfortunately running into unwanted behaviours.

However, not everyone has heard of the term Hoisting or knows what it means. In this article, I'll explain what hoisting is and show different examples so that you can better understand what it's all about.

The JavaScript Interpreter

When you execute your JavaScript code, the interpreter goes through the code twice.

The first run through the code is where it does a safety check and small optimizations of your code. Safety checks such as making sure that the syntax is right, if there are any calls to eval or with, etc. Then, it optimizes the code as best as it can to ensure better performance when it is executed. This is also where hoisting occurs (more on this soon), and is referred to as the compile run.

The second run is where it actually executes your code by going through it line by line, doing the assignments, calling the functions, and so on.

What is Hoisting?

Hoisting is when the JavaScript interpreter moves all variable and function declarations to the top of the current scope. It's important to keep in mind that only the actual declarations are hoisted, and that assignments are left where they are.

Hoisting is done during the interpreter's first run through the code.

Variable Declarations

Let's start with a basic example and look at the following code:

'use strict';

console.log(bar); // undefined
var bar = 'bar';
console.log(bar); // 'bar'

At first, you may think that the sample code would throw a ReferenceError on line 3 (console.log(bar);) because bar has not been declared yet. However, with the magic of hoisting, it won't throw a ReferenceError but the value of bar will be undefined at that point. This is because the JavaScript interpreter does a first run through the whole code and declares all variables and functions at the top of the current scope, and then, on the second run, will execute the code.

Here's what the same code would look like after the interpreter's first run:

'use strict';
var bar;
console.log(bar); // undefined
bar = 'bar';
console.log(bar); // 'bar'

Notice how bar is now declared at the top (var bar) but is not yet assigned at that point? It's a subtle but important difference, and this is why bar is logged as undefined instead of throwing a ReferenceError.

Function Declarations

Hoisting also applies to function declarations (not function expressions). Let's analyze the following sample code:

'use strict';

foo();
function foo() {
    console.log(bam); // undefined
    var bam = 'bam';
}

console.log(bam); // ReferenceError: bam is not defined

In this sample code, we are able to successfully call the function foo since it's a function declaration and therefore it is hoisted as-is to the top of the current scope. Then, foo will output undefined when calling it since, as in the previous example, bam is hoisted to the top of its current scope, which is function foo(). This means that bam was declared before calling console.log(bam) but it has not yet been assigned a value (bam = 'bam').

However, the important thing to note here is that bam was hoisted at the top of its current scope. This means that it was not declared in the global scope but in the function's scope instead.

Here's what the same code would look like after the interpreter's first run:

'use strict';

function foo() {
    var bam;
    console.log(bam); // undefined
    bam = 'bam';
}

foo();
console.log(bam); // ReferenceError: bam is not defined

Notice how foo() was moved to the top, and bam is declared in foo()? This means that, when you call console.log(bam) on line 10, it will not find the variable bam in the general scope and will throw a ReferenceError.

Function Expressions

Next, the third use case I'd like to cover is how function expressions are not hoisted as opposed to function declarations. Instead, it's their variable declarations that are hoisted. Here's some sample code to demonstrate my point:

'use strict';

foo();
var foo = function () {
    console.log(bam); // undefined
    var bam = 'bam';
}

This code throws a TypeError: foo is not a function error since only the variable declaration var foo is hoisted to the top of the file, and the assignment of the function to foo is done on the interpreter's second run only.

Here's what the same code would look like after the interpreter's first run:

'use strict';

var foo;
foo(); // `foo` has not been assigned the function yet
foo = function () {
    console.log(bam);
    var bam = 'bam';
}

What Takes Precedence?

Finally, the last use case I'd like to cover is that function declarations are hoisted before variables. Let's look at the following code:

'use strict';

console.log(typeof foo); // 'function'

var foo = 'foo';

function foo () {
    var bam = 'bam';
    console.log(bam);
}

In this example, typeof foo returns function instead of string, even though the function foo() is declared after the variable. This is because function declarations are hoisted before variable declarations, so foo = 'foo' is executed on the second run, after we call typeof foo.

On the first run, the interpreter will hoist foo() at the top of the current scope, and then will get to the var foo = 'foo' line. At that point, it realizes that foo was already declared so it doesn't need to do anything and will continue its first run through the code.

Then, on the second run (which basically executes the code), it'll call typeof foo before it gets to the assignment foo = 'foo'.

Here's what the same code would look like after the interpreter's first run:

'use strict';

function foo () {
    var bam = 'bam';
    console.log(bam);
}

console.log(typeof foo); // 'function'
foo = 'foo';

ES6

ES6 is the future and is what most developers will be using moving forward, so let's see how hoisting applies for ES6 code.

Hoisting doesn't apply the same way for let and const variables compared to var variables, as we saw above. However, let and const variables are still hoisted, the difference being that they cannot be accessed until the assignment is done at runtime.

From ES6's documentation:

The variables are created when their containing Lexical Environment is instantiated but may not be accessed in any way until the variable’s LexicalBinding is evaluated.

At the end of the day, it's a small technicality where the interpreter applies hoisting to these variables on the compile run but they'll throw reference errors when accessed before the assignment happens, essentially preventing us from accessing these variables before their assignment.

Conclusion

I hope this clarifies how hoisting works in JavaScript. It's definitely not as tricky or complicated as it sounds, but it does require us to breakdown the different use cases and trying different scenarios to understand how things work under the hood.

Do not hesitate to leave me comments or questions if you have any - I'd love to hear your feedback.

This post was originally published on Designing for Scale

How Facebook Live Scales

Wissam A — Wed, 26 Apr 2017 01:33:07 +0000

I was recently watching Facebook's F8 conference - which was streamed on Facebook Live - and it got me curious as to how Facebook Live scales so seamlessly to millions of people around the world.

How does Facebook Live scale? Let's dig in and analyze!

F8 2017 was focused mostly around AR/VR

The Challenges

Before analyzing the architecture and how Facebook Live can scale to that extent, let's look at some of the challenges that building such a product comes with.

Chris Cox, the company's Chief Product Officer, had an interview with Wired¹ back in April of 2016, where he enumerated the following challenges, considering that there were over 1.86 billion monthly active users on Facebook by the end of 2016:

Needs to be able to serve up millions of simultaneous streams
Needs to be able to support millions of users on the same stream

To quote Chris Cox, this "turns out it’s a really hard infrastructure problem", which makes it even more exciting to deconstruct and analyze.

Another unique challenge with live video is that traffic comes in spikes, and your infrastructure must be able to handle huge traffic spikes. For instance, when a celebrity starts a live video stream, there will be a huge traffic spike when users join and while the video is still streaming live, and then it'll slow down drastically when the video ends but it will still by viewable by users, even if it is no longer streaming live. The main challenge will be to gracefully handle the spikes while it's streaming live.

The Architecture

For the purpose of this article, I'll be exclusively focusing on the high traffic spikes that occur when a user with millions of followers (e.g. a celebrity) starts a live broadcast. In cases like this, more than a million people could be watching the broadcast at the same time, and when they all join around the same time, this causes major stress on the servers and infrastructure.

When all these requests come in at once - which is called the thundering herd problem² - it can cause streaming issues such as lag, dropped packets, or even preventing users from joining the broadcast.

When there's a possibility you may run into such a problem, the first thing you should do is prevent all the requests from hitting your streaming server directly, which could be fatal and drop the server all together. Instead, you should aim at adding multiple layers that will filter these requests and ensure that only the necessary requests make it to the streaming server.

This can be achieved in many different ways, depending on the size of your product and the number of users. In Facebook's case, they chose the following architecture:

It's important to note here that Edge Cache servers are spread out across the globe to support Facebook's global reach. There's also a one-to-many relationship between the Origin Server and the Edge Cache, where multiple Edge Cache servers can send requests to the same Origin Server.

The flow works as follow:

The requests first hit the Edge Cache server closest to the user in order to reduce the latency between the user and the server. The Edge Cache servers are essentially a cache layer and do not do much processing.
If the requested packets are in the Edge Cache, then they are returned to the user.
If they are not in the Edge Cache, then the request is forwarded to the Origin Server, which is another cache server and has a similar architecture to the Edge Cache server.
If the requested packets are in the Origin Server, then they are returned to the Edge Cache who caches them before returning them to the user.
If they are not in the Origin Server, then the request is forwarded to the specific Streaming Server handling that live broadcast. The Streaming Server then returns the packets to the Origin Server, who caches them before returning them to the Edge Cache. The Edge Cache caches the response and returns the packets to the user.
Future requests for the same packets will then simply be handled by the Edge Cache and will not travel further than that layer, which speeds up the process incredibly and reduces the load on the Streaming Server.

As we can see, this architecture greatly reduces the number of requests that make it to the Streaming Server. For instance, if 5 requests make it to the Edge Cache sequentially, only the first one will go all the way to the Streaming Server and back, while the 4 others will immediately get a response from Edge Cache since it'll have cached it already.

However, this is still not enough for Facebook's reach. In fact, according to an article released by Facebook³, this architecture still leaks about 1.8% of requests to the Streaming Server. At their scale, with millions of requests, 1.8% is a huge amount to leak and puts a lot of stress on the Streaming Server.

Facebook's Global Reach: Edge Cache servers are in pink, and Origin Servers are in yellow. Source: http://cs.unc.edu/xcms/wpfiles/50th-symp/Moorthy.pdf

Why is it Leaking so many Requests?

The first flaw in the architecture above is that simultaneous requests to the same Edge Cache will make it through to the Origin Server. Since the Edge Cache forwards all requests for missing packets to the next layer, if simultaneous requests ask for packet A and it is not in the cache, all those requests will be forwarded. Extrapolating this at Facebook's scale, we can quickly see how so many requests are leaked.

The second flaw in the architecture is that multiple Edge Cache boxes can send the same packet request to the Origin Server, which will in turn forward all those simultaneous requests to the Streaming Server since the packet is not in its cache. This is similar to the first flaw but it's happening at the Origin Server level.

Again, 1.8% may not seem like a lot for normal products, but at Facebook's scale, it is indeed an important issue that needs to be resolved.

Adjusting the Architecture for High Scaling

Facebook's solution to this problem is creative, yet simple to understand. When multiple requests for the same packet hit the Edge Cache, they are grouped together in a request queue and only one goes through to the Origin Server. This is called request coalescing. Once the response comes from the server, it is stored in the cache and then the requests in the queue are responded from the cache.

For instance, when 10 simultaneous requests make it to the Edge Cache, they are all added to the same request queue and only one goes through, which helps reduce the leaking of requests. If we compare this scenario to the original setup, where all 10 simultaneous requests would have made it through, we see a huge increase in performance.

The same concept was also applied at the Origin Server level, where all requests for the same packet - coming from one or many Edge Cache endpoints - are grouped together and only one of them makes it through to the Streaming Server. Needless to say that this greatly reduces the overhead on the Streaming Server, especially consider the amount of requests a live broadcast may receive.

Load Balancing

Another important tweak that Facebook implemented was load balancing for the Edge Cache servers. In some cases, with huge traffic spikes, Edge Cache servers can get overloaded and no longer perform consistently, despite the use of request coalescing. To avoid such situations, the load balancer redirects requests to the closest Edge Cache with availability for the request. For instance, if the Edge Cache closest to you is already serving 200 000 requests, then the load balancer might forward you to an Edge Cache a little further but that is serving half the number of requests, which would mean you would still be getting a response much faster.

Facebook determines which Edge Cache is available by constantly measuring the load on these servers and keeping track of its performance. It goes even further than that and uses an algorithm to predict the load before it happens, which is incredible. The basis of the algorithm is complex enough that it would require its own article so I will not be getting into that at the moment, but it is a very interesting topic that I may cover in the near future.

Request Coalescing Using Nginx

For those of you using Nginx, request coalescing can be enabled by setting the proxy_cache_lock config to on (default value is off), as explained in the docs here:

When enabled, only one request at a time will be allowed to populate a new cache element identified according to the proxy_cache_key directive by passing a request to a proxied server. Other requests of the same cache element will either wait for a response to appear in the cache or the cache lock for this element to be released, up to the time set by the proxy_cache_lock_timeout directive.

Conclusion

I had a lot of fun researching and dissecting Facebook Live's architecture and how the engineers were able to make the product scale to millions of users. Although we may never build products that reach such a high amount of users, there are great lessons here that we can use in our own products, such as how adding multiple layers in the architecture helps filtering the number of requests that make it through to the processing server. This, in turn, improves the performance and availability of the server, which provides a better product to users and helps scaling it.

This post was originally published on Designing for Scale

Credits

Scaling Facebook Live at @Scale Conference
Scaling Facebook Live (2) at @Scale Conference