Forem: Joshua Gilless

Generating Responsive Images for Your Small Website

Joshua Gilless — Mon, 16 Mar 2020 20:18:11 +0000

I love when things are easy in web development. For example, a responsive image is simple to create. You write your <img class="img-responsive" src="..." alt="..." /> tag, throw it in your html and then write 4 lines of CSS:

.img-responsive {
  max-width: 100%;
  height: auto;
}

Great! And you've even made it accessible because you wrote your alt text that described the image in a way that screen readers and people with slow connections will appreciate... Right?

Motivation

My wife, Emma, is an animator and character designer, and I sometimes help her with her website. A big part of her website is her portfolio, which consists of images. Those images need to load for people who want to view them. I also care about hosting costs and would like to keep my bandwidth down. So when she sends me a 1.3MB, 5100x3300 pixel image, I need to make sure that I'm not sending that to a phone that's 414 pixels wide.

Devices

I have a Pixel 2, it's a great phone, I'd call it high end. I get that the Pixel 4 is out these days and there's several generations of Samsung galaxy out there that are the current top of the food chain in the Android market. As far as iPhones go, they're even better in terms of chip power. Looking at new phones right now (black friday sales), the lowest I can find an new iPhone 8 for is ~$350, the lowest new Pixel 3a is $250. These aren't the newest models by any means (iPhone is on 11 I think), these are the low end of what we can call great devices.

But these are the bottom of the high end. When you go a tier down, you get cheaper and less performant devices. When you go to the top of the high end tier, you're getting devices that blow out the several year old laptop I have in terms of processing power and RAM.

You could send the high end devices the 1.3MB image, and they'd load it fine. It would even be responsive because of our fancy 4 lines of CSS. The device could display it and scroll and handle all of it and we wouldn't have to give it a second thought.

My wife and I have a phone plan where we pay by the Gigabyte up to a certain amount and then after that it's free. We usually use around 2GB per month and it's affordable. We bought our phones outright and so it keeps our monthly bill very low compared to a lot of other phone bills I've heard of. But again, we're at the bottom of the high end phone market. I know people using phones that nobody makes commercials for, phones that wouldn't fit the definition of a status symbol.

A 1.3MB image works great on an iPhone 8 or a Pixel 2, but what about a 4 year old Motorola? It might be a little slow. When you're hosting images, you're paying for storage and bandwidth, but the person viewing them is also paying for an internet connection. It's a component of accessibility to make sure that you're costing the people viewing your site as little as possible, while still giving them the best quality image.

The people viewing Emma's web site are mostly in America, people looking at it on relatively high end devices, with good internet connections. But if you're getting any traffic from Asia or Africa or South America, you're getting traffic from people who are using $30-$40 phones. The performance of those devices is not close to a late model iPhone. You have to consider accessibility as more than "Does this work on a screen reader?"

The Web Is Accessible

Fortunately, accessibility is built in to the web, we just love to break it by doing things in a way that is inconsiderate to the people viewing the website.

There's a lot of nuance for deciding how to present your images. A web site with lots of user generated content may decide to apply more compression to their images than a website that is an artist's portfolio hoping to get hired for making high quality art.

Image format is also important to consider, a PNG compresses differently than a JPEG for example. Aspect ratio is also important to consider, a phone held vertically can display a taller image and keep all of it in the viewport more natively than a very widescreen monitor that might cut off the top and the bottom.

Fortunately, because the web starts out accessible, we can get all the way there with that original block of CSS, and just using either the <picture> element, or with adding to our <img> element. We don't need to write JS around it until we want to add lazy-loading (another post for another time).

Starting Questions

Do your users generate the image, or is it static content delivered by an artist?
Do you need a different image to show up for different sized screens (art direction)?
Does the layout of your site change around the image as you change screen sizes?
Do we need to use the same image at different aspect ratios in different places?

What you end up doing will depend on each of these questions, and the grey area grows larger depending on the cases as we go along, you'll also find more nuanced questions to ask as follow up to each of those questions.

Intelligently cropping is a hard problem. For example, let's say you want to use an image that a user has uploaded as a profile picture and display the full thing in one page, but if you shrink to mobile it changes to a photo of just their face. You'd need some kind of facial recognition to crop properly to handle doing that at scale. You can get very close with an upper-crop rather than a center-crop if it involves people, but you're still shooting fairly blind.

For the rest of this post, let's go through something I did for a very specific case of a portfolio website.

Example: Portfolio Website

I want to start off with this example by saying that I don't think there is a perfect answer to how to do this. The best thing we can do is understand the tradeoffs we're making and do our best to make decisions that compromise the things we care about as little as possible.

Let's imagine that our web site changes from having a hamburger menu on mobile, to having a 300px wide side menu on desktop. The images on mobile will be 100% of the device width as you scroll, but on desktop, images will be 100% - 300px wide.

Funny enough, we've already hit the barest possible minimum with the first paragraph of this blog post.

What size images do we need to serve at different screen sizes?

Let's start off with some assumptions:

Giving a unique image size per phone screen size is beyond our abilities. So we won't have to generate a 320px wide, 375px wide, 414px wide, 640px, 828px wide etc images to satisfy every phone screen size and pixel density. We should be able to come up with a set of image sizes that covers the phone screens and pixel density fairly evenly.
The website viewed on a huge screen needs a huge image, but we should come up with a sensible max size.
The website viewed on a tiny screen needs a tiny image. Still keep in mind, a 414px wide screen at 3x pixel density is still 1242px wide.
We have some wiggle room with showing slightly larger images because the browser can render a larger image smaller and have it look nicer than it can if it's trying to render a smaller image larger.
We are not getting into art direction with the picture element in this case. We're going to render the same image at every screen size, just a different dimension and size file.

Given these assumptions, we can reason about some image sizes that we think we should deliver.

500px wide. This covers small phones rendering at single pixel density.
750px wide. This covers small desktops since remember, the sidebar is 300px wide.
1000px wide. This covers high density phone screens and fairly typical laptop screens.
1500px wide. This will take you up to 3x pixel density phones, larger laptops and some smaller desktop monitors or people who don't fullscreen their browser.
2500px wide. This is our semi-arbitrary sensible-maximum. It wont cover the 4k monitors, but if I find that screen size is popular, I can just add a key for the original.

How do we generate the images from the original file?

Like all questions, there's the robust answer for scale, and the answer for a small static site. A few years ago I had a rails site that used an image magic gem to automatically generate the right sized images based on what was uploaded. At work, we have dynamically generated and then heavily cached images from a single source file. For my wife's website, we're just going to use ImageMagick because everything else is such gross overkill.

I dumped all of her images into an "originals" folder so I didn't overwrite them, and then I figured that I could write some bash for loop and mess with it until it was generating all the images. But it was slow. ImageMagick has a lot of options that let you build something a little better than a bash for loop. Let's take a look at my end script.

It's important to note that this is a throw away script, it works within the project structure and workflow that Emma and I have. You can pass it individual files, or loop over a glob and pss those into it, but it does just take an individual file as the parameter. You probably want to do something different for your project.

#!/usr/bin/env bash

PARENT_DIR=$(dirname "${1}")

for f in ${1}*.png; do
    convert ${f} \
    \( +clone -resize 2500 -compress JPEG -quality 90 -background white -flatten -write $PARENT_DIR/$(basename ${f%.*})-2500.jpg +delete \) \
    \( +clone -resize 1500 -compress JPEG -quality 90 -background white -flatten -write $PARENT_DIR/$(basename ${f%.*})-1500.jpg +delete \) \
    \( +clone -resize 1000 -compress JPEG -quality 90 -background white -flatten -write $PARENT_DIR/$(basename ${f%.*})-1000.jpg +delete \) \
    \( +clone -resize 750 -compress JPEG -quality 90 -background white -flatten -write $PARENT_DIR/$(basename ${f%.*})-750.jpg +delete \) \
    \( +clone -resize 500 -compress JPEG -quality 90 -background white -flatten -write $PARENT_DIR/$(basename ${f%.*})-500.jpg +delete \) \
    null:
done

It loops over all the png files in a folder, takes the file, builds something out of it, outputs it a level up from the originals directory and creates a new filename off of the old one with the size appended to the file name. I'm also using JPEGs since they compress well on web and are supported in all browsers. Since it's a portfolio site, the quality I'm using is 90 since quality is important. If we cared about image quality less, we could probably get away with going down to 70.

+clone clones the last option.
+delete deletes the last option.

Running them at the beginning and end of each operation means we're always using the original image to generate the other sizes, so there's no chance of recursive quality degradation. the last option is null:, and that's just so I can keep every line of generating the size looking the same. Normally on the last option you'd not need to clone or delete or even -write I think. The script just looks weird and is less clear without it.

Now that we have our images generated:

How do we pick the best image to serve at different screen sizes?

Funny enough, our browser can actually do this for us, it just needs a little guidance.

HTML Image tags can have a couple of attributes aside from src, we should explore how to have a single image tag select an image to show using srcset.

Let's look at the HTML spec for srcset:

If the srcset attribute is present and has any image candidate strings using a width descriptor, the sizes attribute must also be present, and is a sizes attribute. The sizes attribute contributes the source size to the source set (if no source element was selected).

That leads us to 3 new questions:

What is an image candidate string?
What is a width descriptor?
What is a sizes attribute?

We can answer these very quickly by reading the spec.

An Image candidate string, has a valid url, whitespace char(s), and then either a width descriptor or a pixel density descriptor. In the set, we can't have duplicate width descriptors.
A Width Descriptor has a valid number and a "LATIN SMALL LETTER W".
A sizes attribute is a source size, a source size list or a source size value. It can have media queries and values for those media queries.

We can put all of this together to write a pretty cool image element.

First, let's generate our srcset. We'll pretend our original image was named img.jpg

<img
  ...
  srcset="
    img-500.jpg   500w,
    img-750.jpg   750w,
    img-1000.jpg 1000w,
    img-1500.jpg 1500w,
    img-2500.jpg 2500w
  "
/>

Now, for our sizes, we need a little more information. We know we have a few constraints such as the 300px static width sidebar on desktop. We know vertical tablet and under is going to be 100vw, but the static sidebars mean we sorta have to hint at conditions above those widths.

Since we're giving hints and not targeting specific devices, we can actually just be fairly imprecise with our device matching, we don't have to say "Well, iPads have this... desktops start here, large desktops start here..."

So for example, how much screen space will the image take up on 1200px wide? Well, it'll be 1200px - 300px = 900px. Which is 75vw.
Let's add in a few more, on a 900px screen, the image will be 66.6...% of the width of the screen, so let's just call it an even 67%. How about another one at 1800px screen? 83vw after we do the math. Anything above that, we want to say it's close enough to 100% that we just want to load the largest image, so we just say 100vw.

This can be done better, and stylistically you may want to build it from a min-width perspective than a max-width perspective. But the point is, for this specific project, as I resize the screen and look at it on different devices, the best image is served at the best time.

<img
  ...
  sizes="(max-width: 768px) 100vw, (max-width: 900px) 67vw, (max-width: 1200px) 75vw, (max-width: 1800px) 83vw, 100vw"
/>

If we put all of it together, we've got some html:

<img
  src="img-500.jpg"
  sizes="(max-width: 768px) 100vw, (max-width: 900px) 67vw,
(max-width: 1200px) 75vw, (max-width: 1800px) 83vw, 100vw"
  srcset="
    img-500.jpg   500w,
    img-750.jpg   750w,
    img-1000.jpg 1000w,
    img-1500.jpg 1500w,
    img-2500.jpg 2500w
  "
  alt="A super cool image"
  class="img-responsive"
/>

And some css (this plus whatever is required to have the image on the right and the sidebar navigation on the left taking up 300px and then the mobile version):

.img-responsive {
  max-width: 100%;
  height: auto;
}

You'll generate different sizes if your website is designed differently, you'll write different breakpoints and widths that images take up a those breakpoints. But without a single line of javascript, we've built an image tag that loads the right image for the right screen size.

Fin

HTML starts off accessible, it's up to us to keep it that way, and dealing with images is one of the fastest way to get off track. This hasn't covered the picture element, which lets you do things like serve different image formats to different browsers, for example .webp isn't supported by every browser yet. We've just scratched the surface of images on the web.

As a developer, you can do your part by understanding your users, making good decisions based on the available information, and executing in an accessible way. If you pull it off, you can save on bandwidth, save other people on bandwidth, make your site faster and more accessible.

Getting Started With Web Workers

Joshua Gilless — Thu, 13 Feb 2020 02:54:27 +0000

Hello all, today I'm taking a look at web workers. The general idea is that I'd like to implement a tiny one that just sends a message back and forth between the main index.js and the worker in worker.js. I'm hopeful that I can show you how I explore something that I don't really know anything about. After we implement a tiny one, we'll look at further use cases for "real" ones.

I start a lot of my projects the same way, just a HTML file, a JavaScript file, and a CSS file. For this project, we're not even going to need a CSS file.

I have several folders at my root, personal stuff goes in a folder called "projects". When I open my terminal, I often run a few commands.

Here they are, for brevity's sake, concatenated:

cd projects/tmp && mkdir webworkers && cd webworkers && touch index.html index.js && echo "console.log('hello');" > index.js

It changes me to a temporary directory, where I keep a lot of little silly things that I haven't build a project out of and can delete at any time. Any time a project is in this tmp folder, it is not worth caring about. If I start to care about the project, I move it up a directory.

The commands make a folder called webworkers and then adds two files, index.html and index.js. The file index.js starts off with a console.log('hello'); so that I can just verify that it's all connected when I first open it all up.

I use VSCode as my editor. VSCode comes with the Emmet toolkit that lets you just type html:5, hit tab, and it outputs a boilerplate html document.

Before the closing body tag, I manually typed:

<script src="index.js"></script>

And now I've got a very basic static site.

Simple Web Server

I use a python module to run a static web server in a directory. If you have python 2 run (in your terminal):

python -m SimpleHTTPServer 8000

If you're using python 3 run:

python -m http.server 8000

If you're lazy, and you do this a lot, you can also add an alias to your .bash_profile or whatever file your shell executes on load.

alias pserve="python -m http.server 8000"

Once your web server is started, open your browser, navigate to localhost:8000 and open the console. If you've done everything right you should see a "hello" waiting for you.

Web Workers!

Ok let's read a little documentation. So it looks like we want to create a worker from a separate file and that worker will run on a background thread. Cool, let's make the file.

touch worker.js

It has an event handler called onmessage, so we can make our whole file just that handler for now.

worker.js:

onmessage = function(e) {
  console.log(e);
};

I should explain what I'm doing here. We can use onmessage directly here because in the worker context, self and this both point to the global worker scope. The scoping is very similar to how you may use something like fetch or console directly in the main thread, even though they're on the window object because the main thread's global scope is Window.

If you're feeling verbose, you could write this instead:

self.addEventListener("message", function(e) {
  self.console.log(e);
});

The advantage of using addEventListener is that you don't override existing listeners on the event. If you use the sort-of-inline onmessage, you only ever get one event listener at a time, and if you ever declare a new onmessage handler, you override the first one. With addEventListener you can add as many as you want, but you have to worry about discarding the listeners when you're done with them. It's a small tradeoff to be aware of.

It's probably overall better to just stay consistent with addEventListener. And if I'm using addEventListener, it means I should really separate my function out in case I need to remove the listener later on. Saying out loud as I do this: "Premature optimization is the root of all evil, but building good habits is not."

function handleMessage(e) {
  console.log(e);
}
addEventListener("message", handleMessage);

Ok, so that's our worker.js, let's create it and send a message.

index.js:

const worker = new Worker("worker.js");

worker.postMessage("Hello");

Save and refresh the page and see what happens in the console.

> MessageEvent {isTrusted: true, data: "Hello", ...}

Very nice! There are a lot of properties on this MessageEvent object. Let's take a closer look at data, since I see that has "Hello" in it, and means data is what we sent.

Let's try posting an object! The HTML specification says attribute any data, so it can be more than a string.

index.js:

const worker = new Worker("worker.js");

worker.postMessage({ your: "face", is: "cool" });

Change what we're logging in worker.js to be just the data property of the event:

function handleMessage(e) {
  console.log(e.data);
}
addEventListener("message", handleMessage);

> {your: "face", is: "cool"}

Well, that's what I wanted to see. Now we know how to post data to the worker, what about receiving it and bringing it back?

Let's go back to index.js and have our worker receive a message:

const worker = new Worker("worker.js");

worker.postMessage({ your: "face", is: "cool" });

function handleMessage(e) {
  console.log(e.data);
}

worker.addEventListener("message", handleMessage);

And let's have our worker.js send a message when it receives one.

function handleMessage(e) {
  if (e.data.is === "cool") {
    postMessage("yes");
  } else {
    postMessage("how dare you?");
  }
}
addEventListener("message", handleMessage);

Save and refresh:

> yes

Excellent.

Let's change index.js one more time, to make sure all the data can be evaluated:

const worker = new Worker("worker.js");

worker.postMessage({ your: "face", is: "not cool" });

function handleMessage(e) {
  console.log(e.data);
}

worker.addEventListener("message", handleMessage);

Save, run again:

> how dare you?

There you have it, we can send messages with data back and forth from the index to the worker and back.

Tidying Up

You can remove the event listeners when you're done with them, both on the worker object in the main thread, and on the global scope in the worker thread.

Main thread:

worker.removeEventListener("message", handleMessage);

Worker thread:

self.removeEventListener("message", handleMessage);

That's an important thing to care about. Your code is probably running somewhere on less expensive hardware than your 2014 MacBook Pro that you're still holding on to because you don't want one of the new terrible keyboards.

Additionally, you might want to kill the worker thread when you're done with it. No use keeping it around after it's done. This can also be done in the main thread, or in the worker thread.

In the main thread, it looks like this:

worker.terminate();

And in the worker thread, it looks like this:

self.close();

The Browser Check

Ok, one last thing. Almost all browsers support Web Workers, but Opera Mini exists and people still use IE6 for whatever reason. You can verify that the browser trying to run your code supports web workers with a simple check:

if (window.Worker) {
  ...
}

Use Cases

Imagination is key here. This gets you some actual multithreading capabilities in the browser. It's not a JavaScript feature, JavaScript is single-threaded. This is an HTML feature.

What if you put your apps state management into a background worker? You could have all your expensive state calculations run in the background and have the main thread to do all the UI work. That would be pretty sweet.

What if you put all your data fetching in a worker? Your main thread wouldn't have to block anything while you're waiting on a response from the server.

What if you were building some kind of data-heavy app and wanted people to be able to upload a CSV and then parse that data into your application without having to send it to a server? A Web Worker could really help with offloading file processing off of the main thread.

Conclusion

I think this is the simplest worker we can make. It turns out that after what we've done here, it's just writing more javascript.

Each worker has a global context, but can't access the DOM, window, or document. You CAN access a few things that you would think of as under the normal window context that are also under the worker context.

There's a lot more to explore here. I've built a tiny example, just a proof of concept that says "Yep, this thing works." The next thing to do would be to build something non-trivial, like that thing that parses a CSV file without sending it to a server.

I think that exploring things you don't understand is really important. In this case, I learned how easy web workers are to set up and use. Any time you learn something small like this, it compounds on your ability to make good decisions when you're building things later on.

I hope you picked up something cool from this blog post. If you're using web workers in production, I'd love to hear about what you're using them for!

Fighting FUD

Joshua Gilless — Tue, 29 Oct 2019 16:53:23 +0000

On Friday evenings, I like to make cold brew coffee for the next morning. I just take a mason jar, dump in some coffee grounds and water, put the lid on and then wait for 12 hours. Saturday morning comes around, I filter it and then my wife and I go to the park and sit under a tree and have a nice coffee date. I love those early weekend mornings, there aren't a lot of people out and about. Sometimes we even get most of the park to ourselves! The deadlines from the rest of the week take a break until later, and we just have a morning to chat and sip some tasty coffee.

We did this last week, and I gotta tell you, I love being alive. It's pretty cool, I have so many things that I like doing that mostly require being alive. I like being alive with my wife, I'd like to keep it going at a high quality for as long as I can. I haven't always taken the best care of myself, but I have a lot more motivation to do so these days.

I just googled the average life expectancy in the US. A baby born in 2017 is expected to live to 78.7 years old. Ok, that's interesting, but now I just want to know more, how long will someone who was born in a different year live? The SSA has some actuarial tables about life expectancy, which is hopefully the least exciting sentence you'll read today. It looks like I can expect another 52 years of life! That's pretty good, but that means I've probably already lived over a third of my life.

None of those potential remaining days are promised though. I could develop a heart anomaly, get hit by a car or murdered by an extremist. Whatever remaining days I have left, I want to make them count. I want to spend time with loved ones, read good books, eat good food, plant some trees, make cool things.

"You gotta get 10,000 steps a day to stay healthy!"

I've had a hard time picking and sticking to an exercise/diet routine in the past. I have a tendency to try to min-max everything, which is great for software or role-playing games, but bad for a lot of other real normal life things. I've done a few things that I stuck with, P90X-3 and Couch-to-5k come to mind. After I finished P90X-3, I stagnated a little bit. I'd finished the program, I was ready to move on to something better. There had to be something better.

I spent a lot of time reading about different exercises that I could try to learn to do and add. I'm not into going to the gym or anything, so I found some beginner routines that I could work my way up the progression with. And then all these Internet People were saying that I'd never get stronger without using a barbell. So I kept reading, trying to find an exercise routine that I could do with just a couple of small dumbells and a pull-up bar.

While I was doing all that reading, I wasn't exercising. I was spending so much time trying to find the right routine and diet that I wasn't doing anything.

The health department has a few recommendations for adults. Key takeaways are that some activity is better than none. Substantial health benefits seem to happen at 2.5 hours a week of moderate-intensity. If you want to think of it as a per-day quota: less than 30 minutes a day.

The reality is, you probably don't even need 10,000 steps, older women seem to have the mortality rate curve down to flatten at around 7,500 steps per day. Now, I'm not a 72 year old woman, and this study isn't about measuring quality of life, but it seems like this is one of those situations where a little effort goes a long way. Don't worry about that 10,000 number, just do a little.

Instead of going for jogs or walks, I was worried that I'd hurt my knees if I wasn't doing the right routine. Instead of doing pull-ups, I worried about doing the right amount of repetitions (should I do sets of 8 or 10 or 12?). The thing is, just going for brisk walks for 30 mins a day can reduce the risk of all-cause mortality. I should have been just doing something instead of worrying that I was somehow not going to min-max my muscle gain.

FUD - Fear, Uncertainty, and Doubt

Fear, Uncertainty, and Doubt. It's a marketing strategy used for exercise routines, advertising, public relations, propaganda, politics, and more. It came about into our English lexicon about a hundred years ago, but now you hear it a lot in conversations about software. For example, the term: "Nobody ever got fired for buying IBM." was used by companies to justify millions of dollars to purchase IBM computers. What that phrase is really doing is planting a little seed of doubt that any other vendor won't be as reliable as IBM.

I had a lot of FUD around Go when I started trying to write things in it. Everything I'd read was "You just need the standard library". I'd see it everywhere, but then blog posts would often use the mux from Gorilla. The comments would all be something along the lines of "Real Go developers use the standard library." And there's a certain sort of point to that attitude, you can get REALLY far with just the standard library in Go.

All of that FUD just held me back from doing cool things with Go. What I should have done was use Gorilla and replace it when I knew more and could make a more educated decision. I was struck by uncertainty into inaction. Every time I started, I would doubt myself and use that as an excuse to start over.

Lots of people on twitter seem to ask about "Performance" in React. Should we write arrow functions in the Render method? Isn't that bad for performance? React and Web Components will never work, surely! They're two things that try to do the same thing differently. Right? Nope. It's a lot of FUD.

Which framework should I use? Well React is owned by Facebook and they just had some data issues a few years ago, so maybe I want to try Angular? What if Google shuts down Angular like they do other projects? Ok, what about Vue? Come on, they're not even backed by a huge company, so how can I know they're going to continue to have support!? It's crazy! If you listen to questions like these, all you'll find is a lot of decision paralysis.

So what if Google kills Angular? They've killed some of my favorite projects that other people also loved and used (RIP, Inbox). I doubt that they consider Angular sacred. If they kill it, it's fine! Let's think about it: Will your application stop working? Will you still be able to write new Angular code? What if Google kills Go? The compiler is still there, you still write binaries and can write and deploy new code. How is that any better of a question than: "What if Sun kills Java?" was 10 years ago?

Types of FUD

FUD is spread in different kinds of ways. There's the common "I read that inlining a function is bad for performance..." crowd. Then there's the "Look, I used React Router four years ago and Never Again." crowd. Then there's the worst of all, those who are actively spreading disinformation to drag a "competitor's" things down. They're so different in how you deal with responding to them.

The first is great, these people are parroting something they read and can learn where to put priorities. How do you measure performance? Are you actually profiling? Until you can run some benchmarks and actually show the difference in memory usage or clock cycles it's not really worth worrying about. I've fallen for this kind of uncertainty time and time again, and this is the kind of thing that leads you to worry about bottlenecks that are not even close to your real bottlenecks. Developer speed, typing speed, problem-solving speed are all speeds that you probably need to worry about before you worry about performance in React.

The second crowd isn't likely to be persuaded. Honestly, I don't blame anyone in this camp. If you've been burnt by something, it's hard to want to get back on the wagon. Sometimes these people do come around, but they'll do it in their own time. The best thing you can do here is to just do something kinda cool that catches their attention.

The third crowd is actively doing something destructive. Microsoft viewed Linux as a competitor to Windows. Linux is used by computers and servers all over the world, and so is Windows. Back in the early 2000s, there wasn't as much knowledge available to the general public about the differences between the two operating systems. Windows ran an advertising campaign called "Get the Facts" that claimed that the TCO (Total Cost of Ownership) for Linux was as much as 10 times higher than the TCO of Windows. They even had a totally compelling graph, unless you knew the price of the hardware they were running it on. The BBC covered the Get the Facts campaign, back in 2004. That misleading claim was intended to stir up some FUD and increase the Windows market share.

Combatting Your Own FUD

Examine your FUD. The Socratic method uses a cooperative argument of asking questions to draw out underlying assumptions. There's not necessarily a requirement that you have to have someone else to argue with though. You can ask yourself questions. You can find out where your fear, uncertainty or doubt came from.

My FUD with Go was combatted by just asking myself a few questions: "What if I write this router wrong?" Well, web requests won't go where you expect. "What if I don't know about pointers and write a memory leak?" What a doozy of a question, eh? You'd have a bug and your software would crash. Eventually, I asked myself enough questions to where I just had the confidence to Just Do It. Then I wrote and deployed software, and none of my fears were realized. It works fine.

Ground yourself in reality. Trace your doubts, find out not only what you believe, but why you believe it. Perform experiments! If you can suspend your pre-judgment of something, you can usually find a way to actually perform the experiment that you think isn't going to work.

Your browser has a performance api that you can use to profile your functions. Think you should refactor your app to remove all the inline arrow functions? Measure if you should. You probably gain a lot more from correctly writing your shouldComponentUpdate life-cycle functions or the comparison part of the useEffect hook.

The truth is, you'll sometimes find your fear, uncertainty or doubt is justified, but as long as it's amorphous it'll just kinda sit there, leading to wasted time worrying or wasted time stalled. Being honest with yourself is incredibly important. Being honest with yourself is a huge way to combat your own FUD. The biggest victory over FUD is not letting it interfere with you accomplishing the things you want to do.

So get out there and go for a jog, use a library in Go, write a React app with inline arrow functions. The cost of doing something slightly less than 100% perfect is a lot less than the cost of doing nothing.

Skills For The Programming Job Seeker

Joshua Gilless — Fri, 25 Oct 2019 14:29:10 +0000

I recently received a question from someone (paraphrased and anonymized):

I've got 6 months to brush up on job skills and knowledge to get ready for the job market. I was hoping you could tell me the stack and skills that Clique uses? I want to focus on those tools and be best prepared for any openings at Clique in the future.

It's a fun question, a lot of the advice out there is to "Just study algorithms and data structures". But the truth is there's a lot more to being a programmer than programming. The technology stack often doesn't matter that much, but skills matter a lot.

So what does Clique use?

Well, a lot of everything. There's the stuff that everybody uses: Javascript, HTML, CSS, and a CDN. These technologies aren't going anywhere, the only thing that changes is how you use them. It turns out once you use them for a few years, you get used to the happy path. Doing things with them doesn't take nearly as much time as it did in the beginning. It'll take some time to get there, but continuously building things compounds learning.

There's also the stuff that fewer people use: Golang, Appengine, Google Datastore. These are great technologies, but Appengine doesn't take long to get used to. Datastore is a lot like every other non-relational database. Go feels a lot like if you took Javascript, and mashed it with C. Once you've learned one language, others are easier to pick up.

In my time here, I've written programs in PHP, Go, one random Ruby script, and every flavor of Javascript under the sun. Right now, we're a Javascript and Go stack. Our current favorite JS library is React, but we have an old SPA written in JQuery that we still maintain. And to be honest, most of our JQuery could be replaced by vanilla Javascript these days.

So what's actually important to working here? I think I can give 5 things that I think are important to working at Clique. These skills don't go away from job to job, you'll use them everywhere you go.

Have "Soft Skills"
Be Able to Learn New Things Quickly
Know Your Tools
Have Business Skills
Know Data Structures and Algorithms

Have "Soft Skills"

I put this at the top because it's the most important. The programming industry is small, you will meet a lot of people and you'll run into them again and again. First impressions matter, but consistent good relationships last a lot longer than jobs. "Soft skills" is buzzword that doesn't do justice to the importance of these skills. Empathy is not "soft", it's how you build relationships. Self-management is not "soft", it's how you get things done.

We all have heard stories about the brilliant jerk, who writes great code, but everyone dreads interacting with. Don't be that person. Instead, be the person who can talk empathetically to everyone, from the senior managers to the people fresh out of school, in any department.

Communicating technical things to non-technical people is an incredibly important part of being a programmer. Don't use jargon and do be patient. You didn't learn everything you know in a few seconds of explanation, so don't expect that of others. You're a person first and a programmer second.

It's almost a guarantee that you'll screw up some times. Appreciate when people forgive you and give you room to grow, and remember to do the same for them. I haven't mastered this at the moment at all times, but I am always trying to do better. It's a good thing to learn from your mistakes and do your best to correct them in the future. Try to always treat others the way you want to be treated. This isn't something that you one day become the master of, you can always do better.

Be Able to Learn New Things Quickly

Programmers are people and people don't know everything. I write a ton of code and I can get really really far without looking anything up. The longer I've been doing it the less often I have to look things up. That can change quickly, I just started learning the Go programming language. Every core programming topic I know from Javascript transfers easily, but I end up googling the standard library a lot and using the docs to figure out which method does which thing. I'm slower writing Go than I was Javascript, but I'm picking it up quickly because I have a good foundation.

Part of your job will be getting new tasks of the form "Integrate with this...". They're usually quick, easy and simple. You still have to learn how to do them. Whether it's integrating your ads in with some header bidder or speeding up your server logging, you'll have to learn how to do it and then do it.

Know Your Tools

You'll spend a lot of time in places you don't expect and a lot of time in places you do expect. You can predict a few tools that you need to know your way around:

Your text editor of choice. VS Code, Atom, Sublime, Eclipse, Jetbrains, Visual Studio, whichever. Learn it and learn it well. Get to know the shortcuts and the editor.
Vim or Emacs. You'll end up using them more than you think. Learn to navigate, learn to write, learn to save.
The Command-Line. Linux/Mac, learn to navigate, learn to command things to do what you want. Understand the Unix Philosophy. I can't offer much advice on Windows, but I'd spend a lot of time learning the command line there too. Harness it, customize it, it will make you powerful.
Developer Tools of your environment of choice. Browser, compiler, whatever, learn to step through code, learn to read the errors and messages.
Scripting. I write tons of little scripts for things that I'll use over and over again. You get some practice in and then you write little things in a few seconds and just start using them. They can easily be discarded when you're done, but they'll save you lots of time. Scripting can be done in any language you want, some are easier than others. Check out Bash for a good one to start with.
Typing. I joke with people that I type for a living, most of my time is in front of a keyboard, and a lot of yours will be too. Practice good ergonomics, work your speed up and learn to love your keyboard.
Source control. I use Git, there are other options and every one of them is better than none.
Testing. It's worth learning a good testing philosophy. Test frameworks come and go, but learning to test the right things transcends frameworks. Testing is a tool you can use in any project to move quickly and release with confidence.
Continuous Integration (CI). Worth knowing, but may not be required. You can set it up to do tests, run builds, deploy your code with zero downtime. It's great, know it at a high level and be able to learn how to set it up if you need to.

There's plenty more, I'm sure, if you find yourself using a tool often, learn it well. The dividends of knowing your tools add up quickly.

Business Skills

You're a business asset! Hopefully, you're an asset. If you're not an asset, you're probably a liability. Don't be a liability.

Selling ideas, and sometimes yourself isn't really a skill that most programmers expect to need in their careers. You will have a lot of ideas that you'll have to sell people on. You'll have to negotiate timelines and priorities. To sell things, you need to be able to communicate ideas clearly and thoughtfully.

Business skills aren't really taught in engineering school, but understanding the business you work for will help make you more valuable to the company you work for. Much of your value will be in writing code, but you'll find out how to write the right code by having good business skills. Find out the requirements before you start, discuss things and then build the right things together with the business.

Data Structures and Algorithms

For a long time, I was sorta scared of all things Computer Science, I had a few classes in college, but there was a lot out there that just sounded intimidating. What's a red-black tree? What's a trie? When should I depth-first search, when should I breadth-first search? What's a Dijkstra?

What I've found recently though is that I can understand these algorithms a lot better than I could even a few years ago. You come across a lot of random knowledge while programming. All of that knowledge builds and compounds and you end up knowing data structures and algorithms better than you expect. DOM traversal is tree traversal. document.querySelector is just DFS. After you've been programming for a few years, you'll pick up more on this than you expect.

Lots of people say it in the reverse order: "Algorithms and Data Structures", which is nice and alphabetical. In my opinion, algorithms often tend to show themselves based on the underlying data structures. You'll write better algorithms by understanding the data structures available, what they do and when to use them. You might not need a course on it, but it helps to at least understand the basics.

I hesitated to include this section due to how broken tech-hiring is. Interviewing in the HackerRank style doesn't promote diversity, it self selects people who already have access. To pass an interview like this requires studying. Interviews that require 5 hours of commitment on-site select for people that can afford to take that off of work and sometimes afford a babysitter. Hiring like that makes access one of the strongest signals.

With tech hiring as bad as it is, we still are operating within the system. Plenty of companies model their hiring practices off of Google or Facebook. You can choose to not work for companies that hire with these practices, but I'm not sure that I'm one for protest votes.

Why do Stanford students pass Google interviews at a higher rate than Howard students? Here's an excellent thread on Twitter by Mekka Okereke that shows this better than I can. Basically, Stanford students have a class on how to pass the interview, CS9. You can probably guess what happened when Howard students were given the same access. For now, we do have access to a course on Coursera, Mastering the Software Engineering Interview. Which is probably a good refresher, I haven't taken it, but I would if I was interviewing in 6 months.

There's also sites like LeetCode or HackerRank that you can use for practice. These problems are nothing like what you'll likely end up solving at work, but if you approach them as a game, like Sudoku or Set, you can get a lot of programming practice from these. You can even solve them with a pencil and paper for bonus whiteboarding practice.

Conclusion

If I knew I was going to be looking for a job in 6 months, I would build a few things. Put a few random little projects online. Make up your own projects. Remember that everyone builds a todo list, but just 'cause everyone does it doesn't mean you can't too! Clone a popular site, Reddit and Hacker News get cloned all the time. Clone flappy bird or some other simple game.

I would spend time reading. Read up on anything software related. Learn cool things, like graph traversal, linear algebra, geometry. Learn random things that you'll probably never use like how to turn on your webcam with OCaml or how to animate a bouncing ball in CSS. Every piece of knowledge you gain compounds and you'll eventually end up knowing more than you even knew was possible when you first started.

I would spend some time writing my resume. I'd pick what I think needs to be on my resume, learn it, build a project with it and then put it on my resume. It might even be good to keep your LinkedIn up to date.

I would study data structures and algorithms. I would join a site like LeetCode or HackerRank and pick a language. If the jobs I wanted to go for were backend, I'd pick something like Go or Python, if the jobs were for frontend, I'd pick Javascript/Node. Then I'd do lots of problems in that language.

When the time came, I'd apply, I'd go in with some confidence that I have the basics and can learn what I need to.

Shave That Yak!

Joshua Gilless — Wed, 28 Aug 2019 01:54:04 +0000

One of my all-time favorite expressions is "Yak Shaving". It makes me laugh. It's super visual to imagine myself out there with the same electric clippers I use on my beard just kinda trying to get the hair off of this huge bovine. Truth be told, I'm not even sure I've seen a yak outside of a documentary, so I'm just imagining a big cow that has beautiful long flowing hair (cow Rapunzel?).

The origin of the phrase is someone named Carlin Vieri who was at MIT at the time. An email from Jeremy H. Brown explains:

You see, yak shaving is what you are doing when you're doing some stupid, fiddly little task that bears no obvious relationship to what you're supposed to be working on, but yet a chain of twelve causal relations links what you're doing to the original meta-task.

Some people, like Seth Godin, argue against it on his blog post "Don't Shave That Yak". Others, like Florian Gilcher, point out that good things come from shaving yaks.

My blog itself is a yakshave. I decided that I wanted to write a blog post. Then I decided that I didn't want to use Wordpress or Drupal and everything went downhill from there. I decided that it had to be an application of my own design. I wrote some blog software a long time ago in Ruby after I watched DHH's famous build a blog in 15 minutes demo. But I didn't want to write a blog in Ruby, I'd done that before. At work, we were using this cool language called Go.

I really enjoyed Go. I kept making little toy apps that would connect to a database, or validate JWTs, or muck around with general internet stuff. I built a small web crawler to do SEO audits, I built a little thing to handle OAuth to Google with and then persist it with client-side JWT's. None of it was really coming together into blogging software though.

Every time I tried to get started writing the blog software, I would look around for learning resources on routing in Go. I constantly saw people say things like "You just need the standard library". So I started a few times and came up with a few small routers using the standard library, and they were at best just ok. I didn't like how I ended up using them, so I kept falling back to using Gorilla or Echo.

Anyway, I decided to start writing the software at the bottom. Since I was already using Go, I looked at AppEngine - both are made by Google, so they had to go together. I knew that I wanted a way to write the posts, and that meant I needed a way to persist the data, a database. I hadn't used a NoSQL DB before, so I thought I'd try NoSQL. Google Cloud has a NoSQL DB called Datastore, but it was getting replaced by one called Firestore as I started this project, so I figured, let's just use Firestore.

I read the docs and got up and running pretty fast, but I knew that I wanted a way to post data to Firestore. So I started writing a small CMS to just be able to log in and post stuff. Firestore led me to Firebase, and I ended up with Firebase Auth kinda out of nowhere because I had decided to use Firestore. It worked great! I even thought about just making the whole application that SPA and displaying posts through the same program.

Back to Go. I wrote a server-side application that interfaces with Firestore. It uses basic IAM management to secure it. It doesn't include Firebase's JS because that bundle is large and I didn't want to. I also wrote a router in Go using the standard library. It's not great and I wouldn't recommend other people use it. On the other hand, I'm really happy that I dove in and wrote enough server code to get something working.

Writing your own router in a language you don't really know is very different than writing a blog post. There's a lot of cool stuff in these seemingly unrelated tasks that I learned because I was willing to follow the yakshave until I got what I wanted. I wrote a CMS, I wrote a router, I wrote a bunch of templates, all so I can shout into the ether.

Yak Shaving happens in real life too, not just in programming worlds. Sometimes you end up following a chain of tasks that don't seem related at all to what you should be doing. But if you're learning from those tasks and can build back up to what you should be doing, then those tasks are worth doing.

Instead of just saying "Don't Shave That Yak!", figure out if there's something worthwhile to learn from following the chain until you do have to shave the yak. Because sometimes shaving the yak is the right thing to do.

How To Secure Your AWS S3 Bucket With Cloudflare

Joshua Gilless — Fri, 09 Aug 2019 16:32:45 +0000

I've noticed a lot of people are hosting their website on AWS S3, with Cloudflare in front of it. It's a great option because it's cheap and easy to maintain for a single developer. A static site is fairly secure by default due to the small attack surface.

S3 lets you host websites from the bucket, with the condition that your bucket name is the same as the URL of the site you're hosting. For example, my website is https://www.joshuagilless.com, so to host a website at that location, the s3 bucket would be www.joshuagilless.com. This means that everything you put in the bucket has a predictable URL pattern. They have a fantastic guide for setting up static website properties.

In that guide, you're instructed to give your bucket public read access. That means that with public read access and a predictable URL pattern, any old joker off the street can go around your website and directly to your s3 bucket. The thing is, if you go on to the next guide in the series, setting your bucket up with a custom domain, it still instructs you to put a public read access bucket policy.

Next, in the Amazon guide series, you have instructions to add AWS Cloudfront. Cloudfront is a CDN and an excellent one at that. All of the tips in this post would apply to Cloudfront as well as Cloudflare. The thing about the instructions, they don't mention removing the public read access. So even the official guide is missing this piece of the security puzzle.

Security

Locking down the ACL (Access Control List) gives us all of the protections that Cloudflare promises.

Imagine a scenario where someone decides they want to cause you a large hosting bill and just start requesting objects directly from your S3 bucket. They could find the largest file you host and just request it over and over and over until you started getting a hefty bandwidth bill from AWS. Cloudflare has DDOS protection that would mitigate this.

So how do you do it?

Upload things without allowing public read access.
Remove the overall bucket rule allowing everything public read access.
Create a bucket policy

The default uploading strategy is to not allow the public-read-access, but a lot of guides have mentioned overriding it so that you can access the files from a handy dandy URL. It's important to not override this so that when you try to access it, you get a 403 Forbidden error. If you're making a new bucket, just go with the defaults.

To create a bucket policy, AWS Gives you a walkthrough on adding bucket policies.

So what we need to do is give it a statement for PublicReadGetObject:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<your-domain>/*",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": [
                        list,
                        of,
                        allowed,
                        ip,
                        addresses
                    ]
                }
            }
        }
    ]
}

There are two things you need to replace here:

What's between the "aws:SourceIp" brackets.
<your-domain> with your actual domain name. For example, I would replace <your-domain> with www.joshuagilless.com

For figuring out the updated list for Cloudflare, you can visit https://www.cloudflare.com/ips-v4 and https://www.cloudflare.com/ips-v6 and add those lists.

I learned that we can mix IPv4 and IPv6 addresses from the amazon docs on bucket policies in the section titled: "Allowing IPv4 and IPv6 Addresses". It doesn't matter which order you put them, just as long as they're formatted correctly.

Just Give Me Something to Copy-Paste

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<your-domain>/*",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": [
                        "173.245.48.0/20",
                        "103.21.244.0/22",
                        "103.22.200.0/22",
                        "103.31.4.0/22",
                        "141.101.64.0/18",
                        "108.162.192.0/18",
                        "190.93.240.0/20",
                        "188.114.96.0/20",
                        "197.234.240.0/22",
                        "198.41.128.0/17",
                        "162.158.0.0/15",
                        "104.16.0.0/12",
                        "172.64.0.0/13",
                        "131.0.72.0/22",
                        "2400:cb00::/32",
                        "2606:4700::/32",
                        "2803:f800::/32",
                        "2405:b500::/32",
                        "2405:8100::/32",
                        "2a06:98c0::/29",
                        "2c0f:f248::/32"
                    ]
                }
            }
        }
    ]
}

And again, just replace <your-domain> with your actual bucket name and website URL.

Any time you're using a CDN in production, you can do something similar to this to limit access to your locations that you don't want exposed. The same concept of only allowing a CDN's IP addresses applies to all resources, not just the ones in an S3 bucket.

I hope you find this useful, it took me a while to realize that I should be doing this, so I left my bucket unsecured for way too long. Nobody likes an unsecured bucket.

Forem: Joshua Gilless

Generating Responsive Images for Your Small Website

Motivation

Devices

The Web Is Accessible

Starting Questions

Example: Portfolio Website

Fin

Getting Started With Web Workers

Simple Web Server

Web Workers!

Tidying Up

The Browser Check

Use Cases

Conclusion

Fighting FUD

FUD - Fear, Uncertainty, and Doubt

Types of FUD

Combatting Your Own FUD

See Also

Skills For The Programming Job Seeker

Have "Soft Skills"

Be Able to Learn New Things Quickly

Know Your Tools

Business Skills

Data Structures and Algorithms

Conclusion

Shave That Yak!

How To Secure Your AWS S3 Bucket With Cloudflare

Security

Just Give Me Something to Copy-Paste