Forem: Harry Lucas

What every developer needs to know about TCP

Harry Lucas — Mon, 25 May 2020 17:27:14 +0000

This article was originally posted on Breadth a weekly newsletter aimed at helping you increasing your technical (and non-technical) knowledge to help you become a more effective engineer

TCP / IP - the real MVP of the internet.

Nearly every website/application uses them.

It is literally the fundamental building blocks that pretty much powers everything, yet many developers who build upon these foundations have no idea how they work.

And you know what? That can be ok - the best part about it being around so long is that it has proven to be a trustworthy and semi-reliable set of protocols.

You don't have to know in-depth how it works, but it is good have a basic foundational knowledge.

Let's dive in.

👇 The low down

Firstly, let's clear something up.

TCP / IP isn't a single thing - it's two separate protocols. Well, sort of. Really, it's one protocol built on top of the other. IP is the base level protocol. It stands for Internet Protocol.

Aptly put by Cloudflare:

The Internet Protocol (IP) is a protocol, or set of rules, for routing and addressing packets of data so that they can travel across networks and arrive at the correct destination.

Essentially, it is pretty much saying that it's a set of rules that define how a piece of data (packet) is sent across the internet.

Now you might be thinking - is that where an "IP Address" comes from? And you'd be right.

IP protocol basically tells computers how to get a packet of data from one IP address to another.

However, it isn't quite that simple.

To illustrate. Let's say, for example, you wanted to send a letter to your friend Bob.

Now Bob lives on the other side of the country, and your mail system can only allow letters 5cm by 5cm to be delivered.

Unfortunately, the letter you've written is much larger. Hence, you decide to cut up the letter into three parts and mail each piece individually.

Now when you send your letter, it gets sent with three different mailmen.

Mailmen 1 - has a few other deliveries to do on the way, so it takes him a bit longer to reach your friend.
Mailmen 2 - can take the letter straight there.
Mailmen 3 - unfortunately, get's into an accident and so the third part of the letter doesn't make it.

Now Bob has got 2 parts to a 3 part letter. Even worse, Bob isn't the brightest of the lot and can't figure out how to put pieces 1 and 2 back together in the right order. Since part 2 arrived before part 1, he reads in that order.

So here the IP is what tells you how to get a letter from A -> B. It says that you must use a mailman, you must give the mailman an address and you must only send small pieces of data at a time. You need to trust that they will get there time.

However, as we have trust seen, this isn't always that reliable. In fact, it's turned into a bit of a mess.

To illustrate the above example using actual packets (pieces of a letter), imagine you want to send some data from LA -> Melbourne.

One packet might get routed from LA -> LONDON -> MELBOURNE.

Another might got straight from LA -> MELBOURNE.

And the last packet might get dropped along the way.

Clearly I struggle with drawing countries.

Now because of this, if the receiving server attempts to read the packets as they come in, they will get B -> A and no C.

Basically, they will get rubbish and have no idea what it means.

Because of this, we call IP an unreliable protocol. It will try to get your data from A -> B as best it can. Still, it makes no guarantees about the order or deliverability of those packets.

So how does the internet work at all? How do we make this unreliable protocol, reliable?

Enter TCP

TCP (Transmission Control Protocol) is a protocol built on top of the IP protocol (yes, I am aware that's the same as saying ATM machine, no, I am not changing).

It attempts (and does a reasonably solid job) at making IP reliable.

♤ Digging Deeper

So how does TCP/IP work?

Well, let's go back to the letter example.

So to start with, before we actually even send the data, with TCP, we have to open a connection. We do this to tell the receiver we are going to send some data.

So in our example, instead of just sending the letter with our message, we instead send him a letter telling him we are going to send a letter. Bit meta right. We also ask him to send a confirmation that he got this letter.

This tells us a few things. If Bob sends us a confirmation back:

We know that his address was correct
We know his mailbox works and he can receive letters.
We know he has the basic writing skills of a 5 year old.

Now if Bob doesn’t reply we know:

He can’t receive letters or his address was wrong.
We shouldn’t bother sending him our real letter because he probably won’t receive it.
Or he just doesn’t want to hear from us.

This lets Bob know how to put the letters back in the same order as they were sent. Now it doesn't matter how slow the letters were or which order they arrived, Bob would still be able to reconstruct the original letter.

This solves one problem. We still have to figure out a way to handle the case of missing letters.

To do this, we also tell Bob that every time he receives a letter, he again needs to send us a letter back confirming what number-letter he received.

Now from when you send a letter, you start a timer. You know it will take a maximum of 3 days to send a letter to Bob and for him to send one back.

So if after three days, you don't receive a letter from Bob confirming he received it, then you will resend that part of the letter again.

That's the general gist. Obviously, there is much more that goes on in the internals. Still, for now, you and Bob have a working system for delivering messages.

The first part of any TCP connection is the handshake.

With a TCP connection, this handshake is comprised of three distinct parts.

Syn (short for synchronize)
Syn-Ack(short for Acknowledgement)
Ack

Because it's three parts, this is often referred to as a three-way handshake.

Essentially what goes on during the handshake is synchronization of sequence numbers (syn) and a set of acknowledgments (ack). This lets the other party know how to arrange the packets. It will also allow it to know if there is a packet missing.

For example, if you open a connection to a server and tell it that your sequence number starts at 123. When it acknowledges with 124, you can know that the server has received all packets up to, but not including, packet 124.

The server will also send you it's synchronization number, for example, 432. Now to let the server know you have received all messages up to 432, you send and ack with 433.

Now that everyone is on the same page, the transmission of data can begin.

Now the client and server can communicate, using seq numbers and acknowledgments, to transmit packets reliably.

The sequence numbers aren't actually incremented by 1 each time, however, and instead by the number of bytes being sent. This adds another level of reliability. Now not only do you know the order of the packets being received, but you can also tell if any bytes are missing.

Of course, there are still going to be many issues along the way. Still, there is a reasonably robust framework in place for dealing with and mitigating a lot of these errors.

🤷‍♂️ When would you use it?

Most likely, you are already using TCP/IP.

TCP is the best used where reliability is needed. Where, if any missing pieces of data can have a negative effect on the experience of the client.

For example, if you missed 10% of packets when loading a webpage - then chances are you are going to receive a hot mess, and the page will fail to render.

However, there are cases where reliability and error checking isn't really vital. In fact, it can be detrimental.

Gaming and Video streaming are two areas where TCP isn't actually the best choice. Instead, UDP is a much better option because UDP has a stronger focus on speed then reliability. Ensuring every single packet in a video call is received is nowhere near as important as making sure the participants can see and hear each other.

Does it really matter if 5 pixels in the corner are missing for 1 frame? Will anyone really notice? Most likely not. Here speed is king, and a few dropped packets aren't going to be a big deal.

💯 Gimme More

Hopefully, you now know a bit more about TCP then before you started reading. If you want to dive deeper and not only increase your breadth but also your depth then check out the below resources 👇

This weeks puzzle

This weeks puzzle is based around the Collatz conjecture.

The Collatz conjecture is quite simple. It states that:

Given a number (n):

If (n) is === 1 👉 return 1 and finish

If (n) is even 👉 return n / 2

If (n) is odd 👉 return 3n + 1

Repeat

This simple set of rules will always return 1 (well maybe not always but it hasn’t been proven otherwise).

The path from N -> 1 is called the sequence. E.g.

n = 20

20 -> 10 -> 5 -> 16 -> 8 -> 4 -> 2 -> 1

So if n = 20, the sequence is 8 numbers long.

Your challenge is to find the longest sequence where n < 1,000,000.

Please let me know if you come up with a solution in by responding to this tweet:

Breadth

@getbreadth

Our first article is out 💯 This one explains what TCP/IP is and why we use it to power well... nearly everything.

Check it out here 👇

breadth.substack.com/p/the-low-down…

We also have first puzzle at the bottom - this ones to do with the Collatz Conjecture #100DaysOfCode #programming

19:16 PM - 25 May 2020

0 0

Alternatively if you liked this article and want to show some love then above tweet is the place to do it ☝️

If you want to subscribe to Breadth and get posts the like above straight to your mail box - then click here.

What is Breadth?

Harry Lucas — Sat, 23 May 2020 17:21:09 +0000

This article was originally posted on Breadth a weekly newsletter aimed at helping you increasing your technical (and non-technical) knowledge to help you become a more effective engineer

One of the worst qualities a software engineer can possess is complacency.

Software engineering is a complex minefield, with ever changing goal posts.

You can literally take a nap and miss 5 new waves of front end frameworks.

Being complacent as a software engineer is the same as being ignorant. One of the biggest mistakes you can make is telling yourself that “You know what, I know every thing I need to know to do my job”.

Why is this such a problem?

Because once you start being complacent, you stop learning. When you stop learning you slowly keep out of touch with the industry. You accept that the knowledge and tools you have today are all you are going to ever need to do your job.

Never will you find a better way to perform X task, use new abstractions, solve problems in different ways, increase productivity, improve communication and most importantly progress as an engineer.

Breadth ↔️

So what is breadth? Breadth is the opposite to depth.

Breadth is the extent of how broad or wide your knowledge base spans in a particular area.

Whereas depth on the other hand is the extent or expertise you know about a particular topic in that area.

For example, breadth might be about how much you know about the automotive industry.

You might know a little bit about cars, the sales process, the manufacturing process, who the big players are - you get the picture.

That’s the breadth of your knowledge - you might not know a lot about all those things, but you know enough about them to hold a conversation or understand their basics.

Depth on the other hand, would be how much you knew about anyone of those individual topics. For example, you might have a great deal of depth on the manufacturing process of cars. You might be an industry expert.

But on the other hand you might have a small amount of depth of knowledge on the sales process.

Why is breadth important?

Now obviously as an engineer depth is important, but I’d argue that breadth is equally, if not more important.

Depth is probably what got you the job in the first place write. Your ability to write code in a particular environment.

It might be backend node.js web applications, it might be a react engineer or a C++ embedded device engineer. Most of that is going to be depth.

But what really helps you shine as an engineer is breadth.

To put it simply - your ability to solve a problem is limited by the amount of solutions you know. If you only know one tool, one language or one solution exists, then you are going to most likely use that.

As the famous quote goes - ‘Everything is going to look like a nail if all you have is a hammer’.

Knowing a lot about one small subset of an area is great. But that is the only time you will be useful. Outside of your comfort zone you’ll be lost.

A contrived analogy

Let’s there’s a guy named Bob. Now bob is quite the node.js + postgres aficionado. He’s been using them for years and knows them inside out.

Now Bob has gotten a task, implement a full text search for their entire catalogue of medical records. Just a measley 840gb of data.

Now as someone with a single hammer at his disposal (postgres) he’s most likely going to jump at using it’s full text search capabilities to solve the problem. Whilst he know’s it doesn’t really scale that well at those volumes - it’s all he knows so it’s all he goes for.

Had he had greater breadth of knowledge - he’d probably realise that hey - there is a better tool for solving this problem. In fact, he might have been aware of 5 seperate tools, and the tradeoffs associated with each.

Now he can evaluate each of these tools and come to the conclusion of the best tool for the job.

Suddenly he’s not trying to put humpty dumpty back together with a hammer, but instead tape and glue.

So to answer the original question - why is breadth important?

Because it gives you more options.

The more you step outside of your comfort zone, the more you learn and expand your technological breadth the more effective of an engineer you’ll be.

There's more than code

Harry Lucas — Sat, 16 May 2020 12:45:07 +0000

I would love your feedback if you have any on what other topics you'd like to see - you can reach me on twitter: @harryblucas

Let me paint you a picture.

Imagine Bob. Friendly guy, knows his stuff. Knows how to get things done.

Lone wolf type. Headphones on. Slack = Do not disturb.

Moves tickets from to do -> in progress -> review before you've even finished you're coffee.

You give him a feature and you know it's going to get done. And done well. Pull requests are a bit big and his Git hub profile is showing a strong gravitation towards the commit axis, but hey, it's all worth it in the end.

Dream developer right?

✅ Finishes features on time
✅ Focuses throughout the day
✅ Quality code

Whilst Bob seems to be doing some solid, A+ "pat on the back" work, he needs to realise there is more to being a good developer than just pumping out features.

Developers work in teams. Teams move together.

For every 700+ line PR he pumps out, he's just taken an hour out of a team mates day.

Every PR he doesn't review, has just fallen onto the lap of another team member. Now they can't focus on their features.

Whilst the team is discussing an important architectural change, Bob's got his headphones on.

Worst of all, is the opportunity cost that's lost with the way Bob works. By himself he can contribute 2x as any other team member. That's fantastic.

But what if Bob spent time helping and leveling up is other team members. Helping them fill in gaps in their knowledge. Showing them better patterns, abstractions. Setting up tools to help them work faster, leaving insightful feedback on their PRs.

What if Bob helped his whole team become just as effective as him. Suddenly, instead of 1 person contributing 2x, we have 5 people contributing 2x.

Now you don't have a 10x developer.

You've got a 10x team, and that is 10x more valuable.

Take aways

Think of your team like a rowboat. When someone stops rowing (PR reviews, stuck without help etc.) the boat slows and goes in circles.

Understand that there is more than just number of lines your write. Being apart of a team means exactly that, being apart of a team.
Smaller PRs are exponentially faster to review than larger once. Reduce the burden on your team.
Review your team members PRs as soon as soon as you get a chance. Get them rowing again.
Become a force multipler for your team. One person being 3x is less effective than 2 x 2x person.

Explained - Function composition

Harry Lucas — Mon, 04 May 2020 14:06:58 +0000

I would love your feedback if you have any on what other topics you'd like to see - you can reach me on twitter: @harryblucas

What is it?

This is another one of those terms that sounds fancy, but really isn't.

Function composition is process of combining two or more functions together into a single function, that produces the same result as if the functions were called independently and the result was passed from one to another.

That's it.

To break it down - we take two (or more) functions, and pass the result of calling the first function with the supplied argument to the second function and so forth.

const func3 = (value) => func2(func1(value))

An Example

To simplify it, you can think of a function as an individual Lego block.

Function composition is then the process of combining one or more of these Lego blocks in to a larger structure.

For an example, let's pretend we want to take a string, and count how many words start with the letter s.

First we create our individual Lego blocks (functions) that we are going to use to build the bigger block (function).

Which in code would be defined as:

const split = (string) => string.split(" ")

const downCase = (list) => list.map((s) => s.toLowerCase())

const countS = (list) => list.reduce((count, word) => word[0] === 's' ? count + 1 : count, 0)

Since these blocks are all the same shapes and have matching side - we can combine them together 💪

To do this in code:

const wordsStartWithS = (string) => countS(downcase(split(string)))

To read a function composed like this, you read it inside out.

However this isn't as clear as it could be, so instead to make it a bit clearer, what a lot of functional programmers will do is introduce a compose or pipe function, which is super simple and makes the above, even clearer. To read an example of what a pipe function is you can click here.

const pipe = (...fns) => (initValue) => fns.reduce((val, func) => func(val), initValue)

Then we would use it like this:

const wordsStartWithS = (string) => pipe(
  split,
  downcase,
  countS
)(string)

Which is now a very readable function declaration that reads like a recipe when cooking (split then downcase then count), all without a variable assignment.

Again just to reiterate what we've done here is combine multiple, smaller functions, into a single larger function.

Now of course this isn't the most robust example, but it is a start for getting your head around exactly what function composition is.

When should I use it?

You can use function composition to compose your entire application together. You can take your smallest functions and combine them together into larger business cases, you can then combine these functions together with other business cases to build a use case - you get the picture - it's just functions all the way down.

The great thing about adopting more of a function composition mindset is that it forces you to break apart your functions into their smallest component pieces.

This promotes composability, reuse and is in my opinion much easier to read.

For example, if we wrote this the traditional way:

const wordsStartWithS = (string) => string.split(" ").map((s) => s.toLowerCase).reduce((count, word) => word[0] === "s" ? count + 1 : count, 0)

Whilst we have gained the benefit of having it all in one line, we have:

Made it harder to read (lots of extra noise)
Harder for the reader to grok intent (they have to parse every single method call to figure out what it's doing)
Removed any possibility of reusing those component parts (which leads