Forem: costica

Continuous Delivery: HTML to Kubernetes - the why

costica — Sat, 18 Feb 2023 08:15:00 +0000

I think Continuous Delivery is the magic sauce that allows web to be the go-to platform for all software nowadays.

Web development is about evolving software: from idea to actual requirements, from an MVP tested by your friends & family to a system used by millions of users.

Why

The internet is full of articles on “How to create a CRUD API with Node.js”, “How to configure Nginx as a reverse proxy” or “How to do X for Y”.

However, I think there’s a lack of details about why some patterns and ideologies emerged in the last few years.

What I think is missing (or maybe is only accessible via paid courses) from the general “internet archive” is how parts of a system work together in the whole lifecycle of a project, and how a CI/CD pipeline serves so many uses and it allows you to deliver fast, bit-sized iterations of your software so that users can benefit from it as soon as you are done coding it.

That’s not an easy task.

There are a lot of trade-offs that should be considered for the long-term success of a project: from small decisions like what library to use to validate the input to architectural decisions about how to make a system elastic but cost-efficient at the same time, all the way to being able to iterate fast and don’t pay 300 devs when 30 could suffice.

How I see things

Web dev is not about Javascript or Kubernetes - not only. And it certainly is not about “PHP is dying” vs “use Node!”.

It is about delivering software. It has evolved and nowadays it also means delivering all kinds of software in a browser.

My hot-takes about software engineering, with a naturally biased mentality coming from web development:

Translating the product requirements into technical ones is not enough; one should take into account actual deliverable milestones.
It is better to throw away 2 months of work that was tested by real users than spend 6 months to “do it right”.
If you know how to test a feature or project, the actual coding is a breeze.
Dev experience has a huge impact on time-to-market. Devs should have an easy way to code and to prove that it works.

Yes, I’m the kind of person that wants to get the software out the door as soon as possible. I’m sold on the idea that software not delivered is worth 0$.

Yes, this translates into the iterative / agile approach.

I also think that moving too fast and delivering bad software is just as bad.

The problem

So… how? How can one move faster, and get actual software out the door (in users' browsers) when web dev nowadays means frontend*

Backend
CDNs
Cloud
3000 javascript libraries
Microservices
Kubernetes
S3 (or S3-compatible)
Serverless
Kafka
Go
Rust
Java
Async processing
MapReduce
Nginx
Haproxy
L4 LB
L7 LB
Git
DevOps
etc

The list can go on. The list does go on.

My contribution

The bad news : if you want to create an environment where web dev is actually productive you need each web developer to be an entire IT department.

Yes, it used to be a joke running on LinkedIn a couple of years ago.

IMO, that joke has, unfortunately, become true: it doesn’t matter if you want to work as a Frontend, Backend, full-stack, DevOps, Cloud Engineer, etc.

As a “web developer”, you now need to understand the big picture of what delivering software in a browser means.

The good news : I don’t think it is impossible.

You don’t have to be a master in all areas in order to navigate through what web means.

Quite the opposite - having a general idea about how things work together is far more important than mastering only one topic.

That’s why I am writing down my mental model - my cheatsheet, if you will. This is what this whole “Continuous Delivery: HTML to Kubernetes” series is about.

How

Let this be the start of a “how to web in modern ages” series. My rough idea of what’s important and how to tackle it:

First, set up the basics of how the internet works:

Delivering software in a browser - Frontend apps
Then we’re going to look at how important it is to understand how your app is going to be deployed… by running it locally
Then we’re going to set up share-able dev setups using docker-compose (work-in-progress, sorry!)

The next step is to see how one can integrate them and how to test them. How to be efficient when coding. How to deliver fast.

After we have a working PoC, we are going to scale it up using a local Kubernetes cluster and deep dive into what delivering software at scale means, while still being able to be efficient when coding.

Necessary disclaimer

It is a biased tutorial of what I think one should focus on if one wants to deliver good, scalable software, in a timely manner.

That means there will be a lot of focus on the CI/CD, testing, and understanding how & why things work in the modern world.

Closing notes

Fair warning: most likely, the plan will change while I write different articles and realize I want to write about things in a different order - or about other things altogether.

And that’s ok because that’s what web dev is about: delivering software before requirements or priorities change.

I would also like to get input from the community. I am open to changing my plan (see what I did there?) if I find out certain topics or areas would be of greater interest than what I initially considered. So don’t be shy, say Hi 👋…

Ok now, go on, we can’t go into Kubernetes if we can’t display anything in a browser. Go get familiar with web apps, because understanding the front-end is the stepping stone to understanding how everything works. Just to be clear, that statement comes from a backend dev 😉

By-bye!

Backend Delivery - Hands-On Node & Docker 1/2

costica — Sat, 18 Feb 2023 08:15:00 +0000

We’re going to explore how to set yourself up for success - when it comes to exploring Kubernetes and running Dockerized applications.

It all starts with understanding whys and hows to deliver your application while being productive in your local dev environment.

Why

Every time I want to start a new project, I feel like reinventing the wheel. There are so many tutorials that cover Node, Docker, and debugging node… but not how to combine those.

While individually each of those is a 5-minute setup, the internet runs short when it comes to an explanation of how to set up the whole thing.

So let us follow The twelve factors and start thinking “cloud-first”, and have a project setup that makes sense for both delivery and development.

What’s covered

In this article: building & running your container. In-depth examples and explanations of how I reason about setting things up.

In the follow-up article: in-depth explanation about using containers for “one-command, ready-to-run” projects. Hot-reload, debugger, tests.

Delivery-first mentality

Say you finished your project and want to deploy it in the wild, so other people can access it. All you have to do is run node src/app.js, right? Right?!

Not quite.

What were the steps you took on your local environment to run your application?
Any prerequisites?
Any OS dependencies?
etc, etc…

The same goes for any server. It needs to have some things installed so that it can run your code.

This applies to your friend’s PC and to your mom’s laptop, too. You need to actually install node (and ideally, the same version) so that it runs the same way as it does for you.

The problem, if you only do this as a “final” step is that, most likely, your app won’t work anywhere else than on your local machine.

Or you are a remember-it-all kind of person, that knows exactly all the steps needed to run the application. And you’re willing and have the time to do that setup manually on multiple machines!

The project

Just in case you want to follow along, this is the app that we’re going to consider: a simple express application, with cors enabled, and nodemon for local development & hot-reload.

// src/app.js
import * as dotenv from 'dotenv'
dotenv.config()

import cors from 'cors'
import express from 'express'

const app = express()
app.use(cors())

app.get('/', (req, res) => {
    const testDebug = 4;
    res.send('Hello World!')
})

const port = process.env.APP_PORT
app.listen(port, () => {
    console.log(`App listening on port ${port}`)
})


// package.json
{
    "scripts": {
        "dev": "nodemon --inspect=9500 ./src/app.js"
    },
    "dependencies": {
        "cors": "^2.8.5",
        "dotenv": "^16.0.3",
        "express": "^4.18.2"
    },
    "devDependencies": {
        "nodemon": "^2.0.20"
    },
    "type": "module"
}

While there are many other things a real project would need, this should suffice for this demo. This is the commit used for this explanation, btw: node-quickstart-local-dev.

After cloning/downloading the files, all you have to do is to run a simple npm install; it will fetch all the dependencies defined in the package.json.

Running npm run dev will start the nodemon process simulating node --watch and exposing port 9500 for debugging.

Or we can run this in “prod” mode by calling node src/app.js. This is all that’s needed to run this project on our local machine.

🤦 No, it’s not, you silly! You also need node to run the app and npm to install the dependencies. To add to that, probably some specific versions of those.

Containers

While we’re discussing the simple case of just having node & npm here, in time, projects and their runtime dependencies grow.

While keeping in sync your laptop configuration with the server’s config is probably possible, wouldn’t it be easier if you… don’t have to do it?

How about a way of deploying your application on any server _(some conditions may apply*)_ or laptop without having to install node? Or any other dependencies our app might have?

Say ‘Hello’ to containerization, the buzzword for the past decade.

Instead of giving a wishlist of prerequisites to your mom and then emailing her the app.js file, we’re going to wrap everything inside a… let’s call it bundle, for simplicity.

What bundle means

This bundle is actually a… “bundle”. It has both application code (the app.js that we wrote) and all the dependencies needed to run our app: node engine and app’s dependencies, found in the node_modules folder.

At a first glance, this is nice. We have everything we need to run the app in the bundle.

However, I am still in the same uncomfortable position as earlier, the only difference being that instead of “you need to install node” I can now say to my mom “you need to install a container runtime so you can run the image I sent you”.

It is, however, an improvement. While not all laptops or servers will have the exact same version of node, most “dev-compatible” machines will have a container runtime: something that knows how to run the bundle. There are more of them out there, but, for brevity, we’re just going to call the runtime Docker.

So we have changed our run instructions from “install node” to “install Docker” and run the file I gave you like so: docker run bla-bla.

Why it is an improvement: I can now send any file to my mom and she will be able to run it. Let’s say we install Docker on her laptop beforehand - then all she has to do is to run them.

That’s a nice improvement, I’d say.

Let’s create a bundle

Creating a bundle starts from a template - a Dockerfile. It looks something like this:

# Dockerfile

FROM node:18.2.0

ARG WEB_SERVER_NODE_APP_PORT=7007
ENV WEB_SERVER_NODE_APP_PORT=$WEB_SERVER_NODE_APP_PORT

# Create app directory
WORKDIR /node-app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./

RUN npm install --production

COPY src/* ./src/

EXPOSE $WEB_SERVER_NODE_APP_PORT

CMD ["npm", "run", "server"]

I know it doesn’t make sense. And all tutorials that I’ve seen so far mix multiple concepts (like building, running, exposing ports, binding volumes, etc) without any explanation and everyone gets confused.

Let’s take it step by step.

Build & run your application - Files only

As we’ve seen so far, just running the app on our local machine is simple. Things get complicated only when we add different things into the mix. So let’s keep it simple:

## Dockerfile

# Set up the base: a Linux environment with node.js version 18.2 installed 
FROM node:18.2.0

# Create a new folder inside the "node:18.2.0" Linux OS"
# This command both creates the folder & `cd`'s into it (pwd output after running this command is `/node-app`) 
WORKDIR /node-app

# Copy our package.json & package-lock.json to `/node-app` 
# Because we're already in `/node-app`, we can just use the current directory as the destination (`./`)
# Copy HOST_FILES -> Bundle Files 
COPY package*.json ./

# Now we want to install the dependencies **inside** the bundle
# While you might be tempted to copy the `node_modules` 
# just because you already have it on your local machine, don't give in to the temptation! 
# The advantage of running this inside the bundle-container is that you get reproducibility; 
# it will run inside the context of `node:18.2` 
# and it doesn't even need your machine to have `node` or `npm` installed!
RUN npm install --omit=dev

# And finally, make sure we also copy our actual application files to the bundle 
COPY src/* ./src/

Behind the scenes, this is a rough estimation of what’s going on:

That is the template - the instructions of how the bundle should be built. The actual building is done by running docker build:

# Make sure the terminal has its current folder `cd`ed into the root project, same level as the `Dockerfile`.
docker build --no-cache --tag test_bundle .

docker build : I’m sure you got that part - we’re at the build step

–no-cache : Optional, but I like to add it because Docker cache is something I don’t master; why get frustrated things don’t get changed when we can make sure, with this simple flag, that we get a clean slate every time?

–tag TAG_NAME : set how the bundle is going to be called

"." : set the context of the command; in our case, since we are in the same folder as the Dockerfile, we can just mark it as “.” - which means the current folder

Running what we created

Now, let’s run our bundle:

docker run test_bundle

Aaand it might seem that nothing happened. But, it did actually run - even for just a moment.

But since we didn’t specify what command it should run, it defaulted to run a simple node command. It defaulted to that because that’s defined in the node:18.2.0 template we’re using as a base.

And because node is supposed to run and then exit, our container also exited instantly. Containers are supposed to be “short-lived”.

Let’s make it run our app, instead. And since our app happens to be a web-server-long-running-process, the container won’t exit immediately.

docker run test_bundle node src/app.js

Now we get the output:

App listening on port undefined

Environment values

Our app port is “dynamic”, listening on whatever port is specified in the environment variable called APP_PORT.

Let’s hard-code that value for a bit to something like 3003, just to test our bundle:

// const port = process.env.APP_PORT
const port = 3003

❗ We can’t test this simple change again by just re-running the docker run test_bundle node src/app.js again because the bundle was already built with the version of the app.js file where it reads the PORT value from an environment variable.

We will now have to rebuild the template & run the application again:

docker build --no-cache --tag test_bundle . && docker run test_bundle node src/app.js

Success!

App listening on port 3003

Let’s see it in action. In any browser, go to localhost:3003. And it won’t work…

Checking the app inside the container

Everything’s alright, I promise. The app really does work on the 3003 PORT, but that’s inside the running container.

Let’s run a GET request from inside the container, similar to what the browser does when we access it from the host machine:

First, we need to be inside the container. For that, we need to know what the container ID is:

docker ps 


docker exec -it CONTAINER_NAME /bin/bash

A more user-dev-friendly way to get a terminal into a container running in your local machine is via Docker’s Dashboard:

Now that we have a shell terminal inside the container, let’s test our application:

curl localhost:3003

Success! We get a “Hello world!” back.

Exposing ports so we can use our app from outside the container

That’s not helpful, is it?

A web server is supposed to be accessed from a browser, not from inside a container through a terminal. What we need to do next, is to set up some port forwarding.

-p HOST_PORT:CONTAINER_PORT

docker build --no-cache --tag test_bundle . && docker run -p 7007:3003 test_bundle node src/app.js

Finally, accessing localhost:7007 inside a browser works.

Ok, we tested our bundle. It works just fine, even though all we did was make sure we have some files inside of it.

A glance into the future - How containers are actually run

Let’s go back to the “hardcoded” app port. It isn’t an issue to hardcode your app port like this.

We can rely on the fact that the app will always listen on the 3003 port and then just set up the container to do the forwarding between CUSTOM_PORT<->3003.

I’m not a big fan of this solution, though. Simply because if I want to understand why a bundle behaves as it does I have to check its build command, its run command, and also the application code.

This is not a real issue now, but it will become one if we try to run this container inside Kubernetes, for example. Without going into too many details, in the future we will want to configure something like this:

    spec:
      containers:
        - image: localhost:55000/project1
          name: project1
          ports:
            - containerPort: CONTAINER_PORT

Preparing for the real world

It will be a lot easier if we just set our container and app port binding right from the beginning.

The trick here is to make sure we keep the two values in sync:

on one hand, we care about the PORT that the container exposes to the outer world
on the other hand, we care about the PORT that the apps listen to

So revert our app code to read the env variable instead of being hardcoded to 3003:

// const port = 3003
const port = process.env.APP_PORT

The end goal is to have complete control over:

what APP_PORT gets defined at build time (when the bundle is created);
keep APP_PORT in sync with the port exposed by the container
what HOST_PORT is bound to the APP_PORT

If the two (APP_PORT, CONTAINER_EXPOSED_PORT) are the same, the exposing part of the run command will be -p HOST_PORT:APP_PORT.

The advantage of doing that is that it allows our run command to be as slim as:

docker run -p HOST_PORT:APP_PORT_EXPOSED_PORT test_bundle node src/app.js

And the good news is we can find the APP_PORT_EXPOSED_PORT without looking into the app.js file - we can just check the Dockerfile!

Build arguments

For that, we’re going to use a combination of build arguments and env variables.

# Dockerfile

# Set the BUILD_ARG variable with the default value of 7007; 
# Can be overriden with the `--build-arg`; Note!: if overriden, the run command must also change...
# We rely on this build argument to be the same in the `ENV` and the `EXPOSE` commands
ARG ARG_WEB_SERVER_NODE_APP_PORT=7007

# If no env variable is provided, default it to the ARG we already set
ENV WEB_SERVER_NODE_APP_PORT=$ARG_WEB_SERVER_NODE_APP_PORT

# Expose the port we define when we build the container or the default setting
EXPOSE $ARG_WEB_SERVER_NODE_APP_PORT

The complete Dockerfile looks like so:

# Dockerfile 
FROM node:18.2.0

ARG ARG_WEB_SERVER_NODE_APP_PORT=7007
ENV APP_PORT=$ARG_WEB_SERVER_NODE_APP_PORT

WORKDIR /node-app

COPY package*.json ./

RUN npm install --omit=dev

COPY src/* ./src/

EXPOSE $ARG_WEB_SERVER_NODE_APP_PORT

Building the bundle command now becomes:

docker build --no-cache -t test_bundle .

And we can run it using:

docker run -p 5555:7007 test_bundle node src/app.js

Since we didn’t specify any build arg, both the APP_PORT and the exposed port defaulted to the defined 7007. That means all we have to do is to choose what host port to forward to the container exposed port - in the example above, we forwarded 5555 to the default 7007.

What happens if we want to use something else for the APP_PORT?

docker build --no-cache --build-arg ARG_WEB_SERVER_NODE_APP_PORT=8008 -t test_bundle .

And then run:

docker run -p 5555:8008 test_bundle node src/app.js

Short recap

So we know how to build our app and how to run it (using docker run ... node src/app.js).

However, that’s not a standard. That’s how I chose to run it. This is not ideal for multiple reasons:

figuring out how to run the container requires some application-code investigation
if I want to change the command, I should notify everyone using the container to run it using the new command; there’s also the chance that they will find out (surprise!) that their way of running it no longer works

So let’s default the command that is run inside the Dockerfile:

CMD ["node", "src/app.js"]

We can now rebuild & run the container in a much simpler way:

docker build --no-cache --tag test_bundle . && docker run -p 5555:7007 test_bundle

This doesn’t reduce our flexibility in any way, as the cmd can still be overridden at runtime:

docker run -p 5555:7007 test_bundle

Build & run - aka Deliver

A final overview of what we achieved so far is this:

From “install node” and run “npm install” and “node src/app.js” to docker run.

It’s not much, but it’s honest work.

However, this only solves the delivery part.

Coming up - shareable dev setups with Docker

While this article covers the delivery part and some basic concepts of Docker, environment variables, and port binding… we’re just scratching the surface.

The real fun begins in the 2nd part of the article, where we start creating Docker configurations for the dev environment.

The next article covers:

how to run the code and hot-reload it inside a container each time you hit “save”
how to debug your code when it is run inside a container
how to not rely on long, hard-to-remember docker run ... commands

And the good news is… it’s going to be easy since we’re going to piggy-bank on some of the concepts learned here.

Until the next one, by-bye!

Web apps - my mental cheat-sheet

costica — Sat, 21 Jan 2023 07:30:00 +0000

Intro

This article is part of the “Continuous Delivery: HTML to Kubernetes”.

While I can’t wait to get into the nitty gritty details of distributed systems, I found myself in an unpleasant position: I think its best if I start by writing about frontend.

That’s because web apps are the standard nowadays. And it doesn’t matter how many 3000 microservices you have in your ArgoCD-Kubernetes clusters deployed in multiple clouds; those are all irrelevant if users can’t or won’t use your product.

Why in a browser

To say delivering software in the browser has grown in popularity would not serve it justice. Heck, it is nowadays more or less the default for everything. Adobe’s moving frigging Photoshop in the web.

Ok, that’s an extreme example. Nonetheless, web apps have grown in popularity even if we still have the joke about JS' world changing weekly. Despite Javascript’s ecosystem state of the art, we could say.

So the default is now to deliver software over the internet, piggy-banking on the cross-platform-runs-everything-silver-bullet aka browser.

With great power comes great complexity

But what does that mean, exactly?

What’s the difference between a “simple site” like Wikipedia and a full-blown web app that can be “installed” on your phone and even send you notifications?
Or apps that you can interact with while you don’t have an active internet connection?
How does it differ when it is server-side-rendered or client-side-rendered?
Is it plain, static html? Or was it generated by a server and just returned?
Does the browser flicker when you change the page inside the same website?
Does it behave the same when you access the same URL from inside the web app versus when you hit enter in the browser’s URL bar?
We can do PWA with SSR… just do it, right? Right?!

A lot of questions, a lot of options. Here’s my mental model - it could serve as a “WebApps - associated complexity & when-to-use cheatsheet”.

The “overall complexity” that you saw in the table is the sum of how complex I think it is to build, maintain, evolve, test and deliver such an app. Not only for the first 3 or 6 months - but in time:

when requirements change
when evergreen browsers get updates
when libraries stop working with the evergreen browsers
when you need to update said libraries
when security patches
when X
when Y
when…

The list can go on.

I think there’s no reason (except for CV inflation, maybe) to create a PWA with offline capabilities for an admin interface that needs real-time data anyway. And that’s just one example.

While there are different pros and cons for each of the types of apps, I think it is worth looking into how complex delivering an app is versus what it can do. Versus what it should do.

Please note that my list builds upon itself: the simplest “web app” is a static HTML file, while the most complex is a full-fledged PWA.

WASM is a beast on its own - not detailed here.

Static HTML

The first one on my list is the classic static HTML. It is how the web started: a simple html file that is served by a server when a user hits a URL.

The process is quite simple:

User enters site.com in his/hers URL bar inside a browser. The browser (after some DNS fun) will make a request to the server.
Server receives the request, handles it, and returns a response: the html file.
Browser receives the html file, maybe even requests others (css, js) that were referenced in the initial file. It reads the whole file(s) and displays the contents.
User sees the information in the browser. Simple, elegant, beautiful.

Behind the scenes, developers working to write & deliver the files to a server looks like this:

The dev process is simple too: just write your html, css, js files, “link” them together and then deploy to your server. No fancy IDEs are needed, and no compiling is necessary.

✅ Pros : easy to develop, no tooling needed, loads fast, compatible with all browsers

⚠️ I’m considering js to be light or non-existent. For js-intensive sites, please see my definition of SPA.

❌ Cons : no code re-usability (for things like header, footer, etc), user agnostic (displays the same data for everyone).

❗ Not that cross-browser-compatible if you want the js & css to work the same everywhere :)

Static site generators

For things like code re-usability where the plain static HTML runs short, the static site generators come in and save the day.

❗The caveat: an extra build step is necessary for the developers before being able to deliver the site.

While request handling looks the same as for the static HTML:

The dev process requires an extra step:

While this is not the most complex scenario, it is still worth noting that now a simple refresh of the browser would not allow the developer to see the final result. Not without a local server with hot-reload or by running the build manually.

✅ Pros : loads fast, and is scalable in terms of re-usability for static content workloads.

❌ Cons : user agnostic, requires an extra build step (compared with the static html) before the site can be deployed.

Server-generated HTML

What about sites that need authentication? And to display the logged-in user’s information?

Pure static html or static site generators are not going to cut it anymore.

We are now moving to the “real-world” web functionalities, where server is no longer just a dumb content delivery mechanism.Server is now going to refer to a full-fledged machine, that is also capable of processing requests.

(Click here for higher-quality visuals)

From a user perspective, it is exactly the same process as for static websites. However, a more complete description of what’s actually happening is this:

User enters site.com in his/hers URL bar inside a browser. The browser (after some DNS fun) will make a request to the server.
The server, listening for connections, receives a new request.
[Optional] Depending on the application’s logic, it might call other services: database, emails, authentication, etc.
The server passes the required information to the templating engine, which transforms the templates (defined by the devs) + variables (“passed in” by the server) => into an html file.
Browser receives the html file, maybe even requests others (css, js) that were referenced in the initial file. It reads the whole file(s) and displays the contents.
User sees the information in the browser, not knowing anything of what happened under the hood.

A lot more powerful than the static websites - but also more costly. It requires way more resources than a simple content-delivery-only server. It is also more expensive in terms of dev costs: while static html and even static site generators’s only requirement is to understand html, the “knowledge bar” is now higher.

(Click here for higher-quality visuals)

You now need a local dev server so you can have a feedback loop and test out your changes. Ideally, it is kept in sync and works just as the production one. This adds a ton of complexity.

The build step might seem similar to the one from the static site generators, as you still need to copy a bunch of files in your deployment process; for some server-side languages, though, there’s also an additional compiling step.

Add configuration for the daemon listening for requests - congratz, you now have a complex setup to maintain.

✅ Pros : re-usability (due to the templating engine, similar to static site generators), can return user personalized responses.

❌ Cons : significantly more complex than static responses in terms of dev - deployment - maintenance, requires additional resources for processing requests each time.

Single Page Apps

For the server-side generated html, for each request, the server needs to do extra work to process it and return the entire html that will be rendered by the browser. That means extra work on the server plus a huge response (the entire html markup) being sent over the (possibly unreliable/slow) network. To top it off, the users see a blank page while the browser loads the new page.

(Click here for higher-quality visuals)

Wouldn’t it be great if we could tackle all of these problems and reduce:

the time users need to wait for a page to load (not applicable to the first-page load, that might actually be slower)
the load on the server - by removing the templating and just returning the dynamic part of the data
size of the payload that is sent over the network
the size of a deployment quanta

That’s where SPA come in and solve all those issues mentioned above:

(Click here for higher-quality visuals)

However, that’s not for free. We get extra complexity: in dev time, of course, but, we now have to start considering the implications for the request handling as well.

Let’s look at how a usual SPA gets loaded and is able to process the requests and display data:

(Click here for higher-quality visuals)

Alice enters site.com in her URL bar inside a browser. The browser (after some DNS fun) will make a request to the server.
The server receives the request, handles it, and returns a response: the html file.
In this case, it is safe to consider the initial html file returned by the server as an empty shell. It is usually a blank page with an empty div element that will be used by the actual app.
The html file references the application code, so the browser will make a new request for the app.js file.
The browser receives and starts processing the js file. Depending on its size, it might take a while…
Only now do we have a SPA loaded and run by the browser. It might give a clue to Alice that it started to do some work for her (in this case, gather the information needed to display her name).
Client-side hydration: the app will then make an asynchronous call to the API server (might be the same one that served the static files too, for all we care).
When it gets back the information (“Alice”), the browser can hide the loading indicator and render the page.
Alice can now interact with the loaded app.

A few notes:

Steps 1 to 6 need to be completed only when Alice first enters the app.

I made a clear distinction between the dumb content-delivery server and the API server for clarity - nothing stopping you from setting up a server to serve both static contents and respond to API requests.

❗ Not all SPAs work the same - while this example covers a lot of the work that is being done, it is by no means a complete & accurate example of what happens in a SPA: simply because it depends on how the SPA is implemented. Not to mention the browser cache, the fact that the app.js could be split, etc.

Now, for the development & deployment process, things get tricky too:

(Click here for higher-quality visuals)

For the local dev process, the frontend & content delivery server is most likely handled, along with the build system by the framework & setup of choice.

Just in case I wasn’t clear enough with the extra complexity: there’s already a build step in place for the simple process of having a feedback loop between the files edited by the developer and what the browser displays.

On top of that, more often than not, an additional API server is required so that the developer can get data for the SPA.

May be still on the local setup, or it may not. The fact remains that it is kinda necessary and another thing to maintain.

Oh, there’s more! Since you are working from the local development server, additional configuration might be needed only so that they work together. Fun, right?

💡 There are some nice things about it too, though:

If you do separate the two of them (SPA versus API), you can deploy them independently - thus, in theory, you should be able to deploy smaller chunks of functionality.

⚠️ There’s a caveat here, too: if you work on a vertical slice (touching both the SPA and the API) you will need to deploy both of them. The good news is that the advantage still applies: first, deploy the API functionality, validate everything is ok, then proceed with the SPA update. This should allow for easier rollbacks, more granular deployments, etc, etc.

To sum it up, for SPA s:

✅ Pros : code re-usability (due to the templating engine, similar to static site generators), can serve user-personalized content, reduces the load on the server compared to server-generated html, small deployment size, good UX after the app loads.

❌ Cons : multiple build systems to maintain, a lot of configuration on both local dev env and in prod, content not index-able by the web crawlers (crawlers only see an empty page)

Single Page Apps with Server Side Rendering

SPA s are awesome - as is static html. That’s what a SPA is, actually, static html on steroids. And just as static html have the static site generators to solve their shortcomings, welcome to the Single page apps on steroids - Server Side Rendered SPA.

What does that mean? Do we move the whole SPA on the server?

Not quite - the SSR-SPA still leverages the browser just like the usual SPA, with one “minor” change: at the time of the first request, when for simple SPAs we would have an empty div - we now get back from the server the whole HTML.

The immediate benefits are that web crawlers can now index the pages - and we still keep the same application code as before. On top of that, the user’s browser’s capability of interpreting and running js is no longer a bottleneck for displaying information - the browser can just display the html the server-rendered as soon as it receives it.

Sounds awesome, let’s make all our SPA to be server-side-rendered, right? Right?!

Ofc, do as you wish. But please look into the cost as well: for the first paint, the server needs to do some extra work - and handle the generation of the html that the browser displays right-away.

(Click here for higher-quality visuals)

Alice enters site.com in her URL bar inside a browser. The browser (after some DNS fun) will make a request to the server.
The server receives the request, handles it, and returns a response: the html file. Since it was the first paint request, the server will, under the hood, call the SPA methods and render the full web page that should be displayed for the user.
Server sends back the html for the browser to interpret & run, but also adds a reference to the app.js - our SPA.
The browser renders the html contents - Alice can see her page. For the moment, we can consider the contents of Alice’s browser as pure static html.
The browser also starts the request for the app.js file. The server will notice that now a static file is being requested, so it will simply return it.
The browser receives and starts processing the js file. When done, we can start considering that the switch was made from static html to SPA.
Alice can interact with the site as if it was a usual SPA. Let’s say she performs an action by clicking a button.

8 & 9. The app makes an asynchronous request to the server; the server knows that the SPA is fully loaded (information sent via the request headers) and that it can take the short path - returning only the “raw” response, without wrapping it in html.

Alice can see (almost instantly, maybe?!) that her operation was successful.

For the build and deployment of the SPA-SSR, things are even darker: we have to say goodbye to the option of choosing our own server-side language. There’s a hard, non-negotiable requirement of “run javascript” on the server side as well. Say “Hi” to node.js, bun, and deno.

We also lose the flexibility of separating the Static Bundle of the SPA from the Dynamic Server API server, because we need to bundle together the server and the SPA.

(Click here for higher-quality visuals)

To sum it up:

✅ Pros : SPA on steroids (see SPA above) in terms of functionality, responsiveness, and speed.

❌ Cons : locks in the server language of choice; bigger, riskier deployments; considerably harder to maintain long-term.

Progressive Web Apps

We’ve seen how strong SPA & SSR-SPA are; we’ve also looked at how expensive they are.

What about Progressive Web Apps - PWA? What purpose do they serve?

While they have a lot of power and are quite flexible, I tend to think of PWAs as the “offline web engine”. With the help of a service worker, we can now intercept the network requests.

Consider how, no matter the type of app we discussed so far, there’s only one common need: to fetch some files from the internet. What if we remove this constraint?

Well, we can now access web apps without the internet.

Wait, without internet?

There’s a catch - you need to have that application installed.

Basically, if there’s a PWA you’re interacting with, the first time you load that app, you will be asked (ideally) if you want to install it.

(Click here for higher-quality visuals)

While it looks 99% the same as for the SPA, take a look at the final pieces: 10 & 11. The app notices that it is, in fact, a PWA, not a simple SPA. It asks Alice if she’s ok with installing it. 12 - Alice gives her consent - the service worker is now registered, and the app is installed.

With an installed app & no internet connection , if Alice tries to use the same app, this is a rough look at what will happen under the hood: the service worker will pretend he is both the API & and Content Delivery Server.

(Click here for higher-quality visuals)

For the deployment & delivery, we have the usual build system where we build some static files into a bundle:

But we also have an extra step, an interesting detail: the deployment is no longer a dev-only process. It is noticeable by the user. It can even be allowed & delayed by the user if the app is set to take into account the user’s choice on the matter:

I know most people would disagree with my complexity score, especially for the SSR vs PWA. Hear me out, though:

I am reasoning about PWAs as a non-SSR-SPA with offline capabilities. While I am aware that you can do an SSR-PWA, I think it is out of the question how many needs that covers: basically all. But, the complexity score goes to the roof.
The SSR locks you in and couples the frontend with the backend very tightly. We know how that goes in the long term.
While PWA is complex to set up & understand… once you get the hang of it and have a working setup, it tends to just work. A lot of things are handled automagically by the service workers.
For the PWA you can opt-in & opt-out of various settings pretty much any time you want. The only risky thing is to mess up the service worker’s self-cache/update mechanism. Then you’re kinda screwed. SSR, on the other hand, is an all-or-nothing kind of setup.

✅ Pros : has the same capabilities as a normal SPA + it works when there is no internet connection or an intermittent one, and a lot of extra functionalities that can be configured on a per-need basis.

❌ Cons : complicated to set up initially, requires a more thoughtful approach - your web app is now susceptible to “the fallacies of distributed systems”.

WASM

Delivered as static html, with a bunch of js for integrations… and 100.000 other things behind the scenes.

WASM apps deserve a lot more in-depth dive than what I can do in a high-level overview here. Sorry, but I will skip this one for now - I need more knowledge on the matter before I can summarise it as I did for the other types of apps.

Necessary disclaimer

This is my mental model that helps me reason about things. It is in no way a comprehensive guide to all things possible in the web apps world.

However, in my experience, understanding these gets me through my workday quite easily, as I can expand on my knowledge of these basic concepts and then build upon them:

maybe some parts of an SSR SPA would have client-side hydration - for some reason.
or maybe we deploy a static blog with comments from a database embedded at build time: the new comments won’t show up until we rebuild & redeploy the website altogether.

The possibilities are endless.

Closing notes

❗For web dev to really make sense, look at the whole lifecycle of an app, from how long it takes to create a new <div>, all the way to how users (or web crawlers) will interact with your app.

❗ Understanding all the trade-offs between the type of software that can be run in a browser brings tenfold rewards: you can now do almost anything, on all platforms.

❗I was often able to reduce the challenge into a frontend/backend problem once I was able to understand the requirements: how do users expect to interact with the app and what features must the app support.

❗Understanding trade-offs doesn’t mean only “SPA or PWA with React or Angular - which one is best for my needs”. It means understanding how your app will scale in time: how complex & expensive it will be to develop and, most importantly… deliver it.

Dev.to community, if you've made it this far: please let me know if you have any feedback on the topic, if the pictures & visuals links are visible enough; also, if there's interest, I can go in-depth on certain topics - all you need to to is ask :)

Hacking a zero costs blogging system. With emails!

costica — Sat, 07 Jan 2023 12:09:00 +0000

I managed to spin up a blogging system, publicly available over the internet, with email subscription capabilities for basically 0$. This is a rough image of how it looks like:

This post will describe my thought process and how I decided what pieces to glue together.

In case you don't care about my twisty mind process, here's a step-by-step tutorial, just in case anyone wants to copy this setup for their own website.

Writing a blog...

So I've decided to start this blog. I've already explained why here.

As usual, I started to overthink it:

What platform to use?
How much is it going to cost?
GDPR?
Where do I deploy?
How much time will I spend on it maintaining it versus writing actual articles?
Am I willing to incur an additional monthly cost just to keep this alive?
Is it worth it?
Maybe it's better if I don't do it...

Ok, enough!
</panic-mode>

<engineering-mode>

With that out of the way, let us use some logical questions to figure out both the problem, and, hopefully, the solution should follow suit:

What do I need / want exactly?
How can I accomplish that?

Let's define the requirements

What do I know for sure?

So, I will be creating a blog - and host it somewhere. I know for sure that I want to finally use my .dev domain, which I own for a couple of years now and it just lays unused because I couldn't bring myself to start publishing my experiments and thoughts.

As I already pay a yearly fee for the Domain Name, I don't want to add to that. Ideally, I will run be able to run the blog with 0 extra costs.

For the rest of the story, we will consider the Domain Name cost as a necessary evil, but already accounted for, so we can ignore it and focus only on the "new" costs of running a blog.

List of features that I want

I need it to be as cheap as possible

I need to enjoy using it. If posting a 200 words article with 1 picture means that I need to gather mental strength to do it, it won't work in the long run. I also like writing Markdown, as it just makes sense for me.

I would like to see if people access it (analytics?)

I would like to have an RSS feed so people can subscribe and see when I post something new

I think it would be cool if I could give people the option to "subscribe" using their email and get notified via email when I post something new (yes, on top of RSS)

I think it would be cool if I could have create a sense of community and to allow people to review, leave feedback, start a conversation, etc

Please note that these are already sorted out: 'need's are must-haves, 'would like' are priority #2, while the "would be cool" thingies won't really matter in the overall decision process.

To sum it up, I want an easy-and-enjoyable-to-use-system (standards: :nerd: nerd), that is cheap to host and serve. If I can add some sort of analytics and an RSS feed, all the better.

How can I accomplish that?

The Hosting problem

It all boils down to how I'm going to serve the content.

Options that immediately crossed my mind, not an exhaustive list, of course :)

Option 1: AWS Ecosystem

The first thing that comes to mind is to use a static site generator, have the HTML files of a functional blog, and upload everything to s3. Then some CloudFront configuration magic to make it serve the static website (which I have to upload to s3 "manually") when the URL costica.dev is hit.

✅ s3 is cheap afaik, so it shouldn't be a problem. Being in an AWS ecosystem, I could then use a combination of HTML forms and lambdas integrated with some sort of storage behind a Gateway to store the emails. Store where? In a database? Another s3 bucket?
It seems a bit complex, but totally doable.

❗ The big "no" for me, in this case, is that I expose myself to potentially huge costs. Yes, there are budgets and alerts and different combinations available. But would I want to risk it? Hell no.

Still, might be a valuable lesson and would make me go really in-depth into cost analysis and budgeting, thus learning something new. Worth considering if nothing better comes up.

Option 2: Blogging platforms

❌ I could use an already existing platform, like Wix, Medium, Weebly, etc. Although it might work, and have some things working out of the box, I don't think I will have the control I want over the layout and overall feel of the site.

❌ WYSIWYG editors are cool, but they usually end up disappointing me and making me hate my life.

❌ Then there's the behemoth, WordPress, with enough plugins to support everything I want - but it also requires an up & running server all the time, with an actual database. Seems overkill for my needs, and, more importantly, goes against the main objective of keeping costs low.

Option 3: Static hosting

I've had a scratch inside my head for a long time about fastly, as it is something I hear about a lot in my day-to-day job, but that doesn't affect me at all.
Between that and my bias towards next, nuxt, nest.js, I end up googling for netlify instead. 🤦

💡 Surprise, surprise! Two minutes later, after a quick comb through their offerings, it looks like a no-brainer.

✅ Relying on netlify to serve static content which I somehow upload (but it can also integrate with github) is the same as the S3 option which I first considered - WITHOUT the costs or the risks of running it in "the cloud".

Option 4: Fastly

Realizing my mistake, I looked a bit into fastly as well.

❌ But the conclusion was that it looks more like a caching PaaS that relies on your site being hosted somewhere else... sounds pricey. No, thanks, back to netlify, as I am now biased and want to explore what it can do.

Choosing a Static site generator

Now I have my hosting & networking "architecture" figured out: I will rely on netlify to serve my website for free.

The next step is to choose a Static Site Generator that supports the other things. Let's explore the options: https://jamstack.org/generators.

I'm kinda tired of js, so I'll skip Next.js just because I don't think I can put up with javascript bundling errors and configurations in my free time. I'll save that "fun" for work.

The next ones on the https://jamstack.org/generators/ are Hugo and Jekyll (ignoring Gatsby, Nuxt, and Docusaurus for the same reason as Next.js).

Wow, this guy's an idiot. Not considering all the options et al..."

❗ It might seem like I'm being superficial and that I don't consider all the choices, some of them obviously more popular (by the number of github stars). However, before you try to prove me wrong, I am only adhering to the requirements (more specifically, the second requirement with priority #1: "I need to enjoy using it").

Yes, it is a subjective choice. Yes, I don't have arguments for it. Yes, it is my personal project, and yes, it is going to reflect my mood. Moving on.

Both Hugo and Jekyll seem pretty similar: customizable enough and pretty straightforward: install, write, generate, done!

As I scroll through the themes of both Hugo and Jekyll, the Hugo - Maverick theme by Tran Duy Canh was eye-popping due to its simplicity. I'm sold.

A quick playground with the Hugo installer and their "getting started" tutorial gave me the confidence that Hugo feels right for the moment. RSS feed capabilities, check.
I don't want to spend too much time on this, as I risk losing myself in the technical pieces and never getting to actually publishing a thing.
Hugo it is, then!

Analytics

I got help here: a friend recommended http://goatcounter.com/ for the sole reason that it doesn't require a GDPR notice. It is also free and "self-hosted", I can get a dashboard and see how many times each page has been accessed. Check, check, check. Moving on.

[Nice to have] Email subscriptions - part 1: "subscribe" and "store"

Now, to the interesting part. I got most of my requirements covered by my Hugo - Netlify - GoatCounter choice.

Up until this point, everything is completely free. This is a refreshing win, as it gives me the energy to go forward with the project. No extra stress from another monthly payment, yay!

When I think of "subscribing" to a blog, there are two options:

Use a (paid 😨) "platform" that offers a form and then use their dashboard to send emails to the people already subscribed
Implement it myself: create an HTML form, have it submit a request, save the email in some storage (database, file, etc). Later, I should be able to iterate over the emails saved in my system and email each of the saved addresses.

I'm ignoring the first option because it seems like a costly one. At this point, I'm decided that either I'm going to find a way to do this for exactly 0$ and create a blog post about it, or going to postpone this requirement and reconsider it in the future.

Compute (backend)

The thing that worries me the most is the backend that would link the HTML form with the call to write to the storage, whatever that might be. Also, how do I protect myself from spammers?
Costs wise this sounds like a serverless solution. Also, a serverless solution can be used for the "sending" of emails, too.

And while looking into various serverless offerings, I found https://www.scaleway.com/en/pricing/?tags=available, which offers 1M function calls per month. That looks amazing, btw.

Storage

Now that we have "solved" the compute part, let's look into storage. I remembered reading about https://www.ilovefreesoftware.com/01/windows/internet/plugins/how-to-store-files-as-bookmarks-in-browser-and-sync-across-devices.html, and then it struck me.
Yes, there are other free storage offerings in the cloud - found by a simple google search.

😎 But I'm not going to use them.

Compute & Storage - `Firebase`

Firebase is a Google SDK used mainly for real-time database, authentication, analytics, etc and it was designed initially to help Android developers. But, naturally, it has grown and it has support for other platforms as well.

Now, why do I think this is the magic bullet for my "email subscription" issue? It offers, along others, free "auth-as-a-service", using an email & password mechanism. I still don't have spam protection (wow, google!), but I can just ignore the unverified emails when I go through the list.

So, when a user registers (completes the "FirebaseForm"), it will be stored directly in the Firebase's storage - no compute / backend needed.

[Nice to have] Email subscriptions - part 2: "send" emails

The next thing I'm going to need is to get a list of all users "signed up" in my Firebase application and send them an email.
Email servers (server, that sounds pricey already, no thanks) can be tricky to install and maintain. AWS' SES (Simple Email Service) doesn't have any free quota.
A few google searches later, I find https://www.mailersend.com/ supporting sending up to 12k emails per month for the free tier.

I don't expect to have anywhere near a 10th of that subscribers, and I'm not going to send an email more than once a month anyway. Good enough.

❗ Yes, it is not a complete solution and it requires a script that would crawl the users registered in Firebase and then call the MailerSend API for each of them. But it solves the issue at hand and I am quite happy with it. Moving on.

[Nice to have] Conversation options

By the time I made the choices for the hosting, static site generator, and "subscription" system I realized I don't really want a "comments" section - or at least not for the MVP of this blog, anyways.

Maybe if enough people hit "subscribe" using their email, at some point I will reconsider it.

But, let's be honest... most likely this blog will only serve as another line in my already longer-than-should-be CV.

Later update: I got lucky here, as this comes out-of-the-box with the Maverick theme that I chose 🤦. It also uses the same mechanism I wanted to try out myself: embedding github comments.

Final solution overview

Putting it all together:

We have a guy (me) that writes something on his laptop, pushes the article to github
netlify is integrated with github, learns about the update and deploys the new version
New site version, containing the new article is now up. RSS feed is also updated
Users come in and read. GoatCounter knows about that
[Optional] User subscribes to the emailing list. Firebase takes care of that
[Sometimes] I decide to email the subscribers using Firebase and MailerSend

The one major sensible choice that is not ideal is using Firebase Auth as the "subscription" mechanism:

✅ It is free
✅ I think it is a cool solution and allows me to write a nice article about it
❌ The "subscribe" process needs to be documented - users might find it strange. Since this is a tech blog, and because I don't really care about marketing and conversion rates, not a real blocker
❌ The "unsubscribe" mechanism will only be done manually, requiring my intervention. Again, I should make sure I let users know how they can reach me so I can remove them from the list
❌ No sign-up spam protection
⚠️ it is a Google product, I know from experience that docs are gonna suck big time

Ideal? No. Good enough for starters? Yes, as it allows me to move on with this without being stuck in an "I am not doing it because of costs or fear of..." kind-of situation.

The most important part, however: it does work. Proof to that, you're reading this blog right now, with 0 hosting fees, and it is implemented exactly as described.

The next step would be to reduce the amount of work needed by my laptop when I want to send emails. I could leverage scaleway into running some functions for that. We'll see.

Takeaways

While this is not a pleasant task, rather a stressful one (see <panic mode>), I managed to get it done by following the rule of thumb of "deliver an MVP and build on top of it".
Defining what's important and what's not usually solves half of the problem: you get to tackle the must-haves first, and figure out the ones that are optional later.
Break the problem into small pieces and tackle them one by one (hosting, generator site, subscription).
Don't take anything for granted. I was really surprised to find out about the Firebase's lack of sign-up protection against spammers.

That's it. That's how I decided how to set this up. By-bye!

Debugging queries - lazy vs eager loading in Laravel

costica — Thu, 14 Mar 2019 21:05:48 +0000

Nowadays, frameworks bring a lot to the table. They bring so much, we often neglect what's actually happening under the hood and just take it for granted.
Don't get me wrong, I am not advocating against frameworks. I love them and for a long time I thought that using frameworks without giving a **** about their internals is fine.

I relied on magic. But, as someone old and bald keeps repeating:

There is no such thing as magic

So I started to pay more attention to what my code does ( or does not , hihi ).

Enough intro, let's cut to the chase. We want to list some posts and for each post its comments. One way to go at this is :

@foreach($posts as $post)
    <h3>{{$post->name}}</h3>
    @foreach($post->comments as $comment)
        <p>
            {{$comment->text}}
        </p>
    @endforeach
@endforeach

The relationships defined in my models are :

class Post extends Model
{
    protected $table = "posts";
    public function comments()
    {
        return $this->hasMany(Comment::Class);
    }
}

class Comment extends Model
{
    protected $table = "comments";
    public function post()
    {
        return $this->belongsTo(Post::class);
    }
}

Simple eloquent relationships. A lot of magic under the hood, so we can write this

    // Eager loading
    $posts = Post::with("comments")->get();

Or even simpler, this

    // Lazy loading ( the actual query which brings the comments for a post is called later ) 
     $posts = Post::get();

Remember, it is the same code that displays them:

     @foreach($posts as $post)
         @foreach($post->comments as $comment)
         ...
         @endforeach
     @endforeach

So, what is actually happening ?

Well, in the first case ( eager loading ), the following 2 ( TWO ) queries get executed :

    DB::enableQueryLog();
    $posts = Post::with("comments")->get();
    $query_dump = DB::getQueryLog();

    // dump value : 
    $query_dump === array:2 [▼
     0 => array:3 [▼
       "query" => "select * from `posts`"
       "bindings" => []
       "time" => 1.47
     ]
     1 => array:3 [▼
       "query" => "select * from `comments` where `comments`.`post_id` in (1, 2, 3)"
       "bindings" => []
       "time" => 0.2
     ]
   ]

These are the queries that Laravel does under the hood.

Now, for the lazy-loading :

    DB::enableQueryLog();
    $posts = Post::get();
    $query_dump = DB::getQueryLog();

    // dump value : 
    $query_dump === array:1 [▼
      0 => array:3 [▼
        "query" => "select * from `posts`"
        "bindings" => []
        "time" => 1.71
      ]
    ]

Only 1 query. But in view, for every post we get this query logged:

    @foreach($posts as $post)
        @php
            \DB::enableQueryLog();
        @endphp

        <h3>{{$post->name}}</h3>

        @foreach($post->comments as $comment)
            <p>
                {{$comment->text}}
            </p>
        @endforeach
        @php
            $query_dump = \DB::getQueryLog();
            dump($query_dump);
        @endphp
    @endforeach

    // dump value : 
    $query_dump === array:1 [▼
      0 => array:3 [▼
        "query" => "select * from `comments` where `comments`.`post_id` = ? and `comments`.`post_id` is not null"
        "bindings" => array:1 [▼
          0 => 1
        ]
        "time" => 0.2
      ]
    ]

Skipping the talk about design patterns and WHERE some code should be. Tl;dr is that queries SHOULD NOT be in the view.

So we have 1 initial query and 3 inside the view. For this simple example, we already have doubled the number of queries.

Now imagine having 100 posts, and calling things like $post->image->path and $post->users_who_like_it aaaaaaaaaaaaaaand awful nested $post->users_who_like_it->image->path.

$post->image->path is not nested, it's a simple join between post and its image -> path is the property of the image

$post->users_who_like_it->image->path is nested : post is "joined" with users , and then each user with its image

Yeah, nesting on this would make it even worse.

Now, you can do the math and if you think this is not an issue, go ahead and lazy load everything. It is, in the end, your decision. But now, hopefully, an informed one.

Penny for your thoughts || suggestions || feedback !

Cheers : )

Importance of data structuring with VueJs example

costica — Sat, 09 Mar 2019 08:34:05 +0000

Most of us focus on solving little problem after little problem, forgetting about the architecture of your application and neglecting how important it is to structure the data right for our needs - be it in a database or in a view/page.

Don't get me wrong, breaking your big, impossible mission into little ones IS the way to go. All I'm trying to say is: do not lose sight of the bigger picture: you deliver code that, in the end, solves a problem for somebody ( let's call that somebody our "client" ). As time passes, the client will have different needs - there is no "maybe" in there, trust me! - so you must anticipate as much of it as possible.

As for the part you can not anticipate, you must have the possibility of extending / modifying the app as the requirements change in a way that does not break it and in a reasonable time frame. "Testing" sounds cool in this situation? Spare the unicorns and rainbows, most likely your app does not have the needed tests. If you are part of the happy ones, keep reading anyway, this applies for apps that are test-covered as well.

Enough with the theory, time for real-life example.

Let's say we have to display more or less the same data in different ways, based on some flags / conditions - whatever you may want to call them.

Where these flags come from is irrelevant. They might be a filter a user applies in the page, or it could be an admin wanting to change the way a page looks every time he feels like it. In the end, all that matters, is the flag - true or false, vertical or horizontal, black or white.

VueJs is beautiful and awesome. "Magic", some might say. But it is like this because it takes care of the rendering and updates. It is your responsibility to give it data in the right format.

Let's say we have to display a list of products by category. Of course, that means we are going to need an object that looks more or less like this:

    const displayProducts = {
        "Category1": [ PRODUCTS ],
        "Category2": [ PRODUCTS ],
    }

This is fine, but what happens if we want to display the products by their color, or by some rank, or in a random order? Sure, we can parse the displayProducts and get a products array containing all the products.

But that's exactly the problem. You already have to parse your own data before being able to actually use it. When that happens, I think you should take a step back and reconsider what you're doing.

Take a second and think about the actual problem : you have to display a list of products . PRODUCTS, not categories. So our main data source and point of truth should be... well, yeah, products.

Now, being in Vue, a data-centered framework, we're good to go. We have an array of products that contain their infos: tags, category, rank, if they are featured or not, etc. Having all the data in one place, it is actually quite easy to display it in different ways to the end user.

Displaying the products markup is pretty much the same in all cases, but it can be customized depending on your needs - not going into details as article would get waaay too long. Each different listing could be a standalone component getting a list of products as props, and this component could render a list of product components. A basic listing would look like this:

    <template v-if="groupType === 2">
        <template v-for="tag of orderedTags">
            <br>
            <b>{{tag.label}}</b>
            <br>
            <template v-for="(item, index) in productsTagged(tag.id)">
                {{ item.name }}
                <br>
            </template>
        </template>
    </template>

Let's say we want to display the products by tag. Our computed that returns the products is as simple as this:

    productsTagged(tagId) {
        let itemsWithTag = [];
        for (let item of this.products) {
            if (item.tags.indexOf(tagId) !== -1) {
                itemsWithTag.push(item);
            }
        }
        return itemsWithTag;
    },

Still not convinced? Here's how easy it is to change the ordering of the whole page, when your client decides in the middle of the night that he wants the green products to be displayed first and the red ones third :

    orderedTags() {
        let collectedTags = coll(this.tags);
        return collectedTags.sortBy("position").all();
    },

Ok, now it's 2 o'clock in the morning and he doesn't give a single **** about the red and green products. Now he has some favorite tomatoes and some good tasty bacon that he wants displayed first. What do we do ? Change the front-end page to display 2 "different" types of products, that perhaps come from 2 requests like getFeaturedProduts and getOtherProducts?
Now someone - be it you or somebody else - needs to go into the backend code of your application and expose the same data in 2 different ways.

Or.... we could have the data and app structured the right, scalable way.

<template v-if="groupType === 3">
    <b>Our most bought products</b>
    <br>
    <template v-for="item in byFeaturedOption(true)">
        {{ item.name }}
        <br>
    </template>
    <hr>
    <template v-for="item in byFeaturedOption(false)">
        {{ item.name }}
        <br>
    </template>
</template>

And our filtering function:

byFeaturedOption(option) {
    let collectedProducts = coll(this.products);
    let itemsByCat = collectedProducts.where("featured", option);
    return itemsByCat.all();
}

What is that coll, you might wander ? It is nothing else than collect.js, a very nice and dependency free js library for lazy people, like myself. It provides convenient query functions over collections of data - 99% similar with the eloquent collections from laravel, as they are inspired from those.

Now you could write your own filtering and sorting functions, or you could use helpers like collect.js. It is entirely up to you. If you structure your data right and you have it in a "raw" form, you then have the liberty of manipulating it as you may wish, without having to modify half of your application's structure for every single change.

The snippet is beginner-oriented, while the purpose of the article is to make everyone realize how important it is to architecture your app right. This applies to so much more than a simple js-rendered list of products, but I think the VueJs examples are the easiest to understand and follow.

Full snippet can be found here:

https://codepen.io/costicaaa/pen/bZWGoB

As always, feedback and questions are more than welcome.

Cheers : )

VueJs type-hint imports support

costica — Sat, 02 Mar 2019 21:55:05 +0000

In applications bigger than a side / fun project, structuring your project right can be a life-saver in the long run, for both old-developers knowing the project's ins & outs and for new ones trying to get on-board.

But structuring your project right and using all kinds of different imports / syntax is confusing for your IDE. We had this problem with PhpStorm - but it exists on all JetBrains products, as their core is more or less the same. Other "lighter" IDEs have the same problem, but JetBrains and VS code support improves with the below fix, so I guess this would work on many more that respects the "specs".

Consider a nice, modern SPA written in VueJS, a library I like quite a lot, despite its caveats ( looking at you, array and object changes ).
We are also using VueRouter and VueX, hence we need some components for pages, some components for the actual rendering in the pages,
some mixins for shared variables / functionality, some modules for the Vuex Store, etc. Things are starting to get messy.

Now, when importing a component all over the place, we want the IDE to be able to type-hint its location, no matter how we structure our application. By default, PhpStorm / WebStorm supports imports that look like this :

import MyComp from "../../components/shopping-cart/myComp.vue";

This works fine, but only if you know where you are right now ( directory of the file you are trying to modify ).
My guess is you just smiled right now: you never do, you just rely on your cool shortcuts and chosen IDE to jump around files and folders in your project.

So what if we could do something like this ?

import MyComp from "COMPONENTS_ROOT/shopping-cart/myComp.vue";

Not only you can define your aliases for folders if you are using Webpack, but also you get the IDE's help jumping around folders and type-hint when writing the import manually.

So how can we achieve this? Quite simple actually :

config = {
    ............
    resolve: {
        alias: {
            "COMPONENTS_ROOT": path.resolve(__dirname, "/components"),
            "MIXINS_ROOT": path.resolve(__dirname, "/mixins"),
        },
    };

The only problem is… it does not work.

I tested on PhpStorm and VS code. These IDEs simply ignore the alias field from the webpack.config.js.

So... what can we do? Looking in the webpack docs, I found that according to some specification ( the classic hidden docs of something you did not know exists type-of-thing ), if we want the IDEs to actually take into account what we wrote in the alias, we must also define an array of aliasFields, like this

    aliasFields: [ "COMPONENTS_ROOT", "MIXINS_ROOT" ]

The aliasFields array lets the IDE know that it should parse the aliases declared and take them into account when parsing and type-hinting your files.
In my opinion, this is useless and uncalled for, I can not find a reason we would not want all of our aliases parsed. I'm curious if anyone knows the mindset behind this and why is this a thing.

For laravel-mix users: the laravel.mix.config.js file is different in structure and it is not supported atm.
Solution? Create a dumb new file containing only the alias and aliasFields and set it as webpack root config file.
The setback here is that you will have to update both files when adding / removing an alias.

TL;DR the required config ( by IDE ) must have both alias and aliasFields in the resolve key :

config = {
    ............
    resolve: {
        alias: {
            "COMPONENTS_ROOT": path.resolve(__dirname, "/components"),
            "MIXINS_ROOT": path.resolve(__dirname, "/mixins"),
        },
        aliasFields: [ "COMPONENTS_ROOT", "MIXINS_ROOT" ]
    };

Would be glad to hear what other tools you guys use for large VueJs projects!

Cheers : )

The "debounce"

costica — Sat, 23 Feb 2019 07:24:13 +0000

The why

a. Real life problem: user goes in and starts typing in a simple input form. We want to show him accurate results ( more often than not, requesting new data from a server ).
b. Real life example: We want to search in a list of states and show only the ones that match ( partially ) the user's input.

The problem

Take 1.b example and do that on "keypress", "change", [whatever_event_you_want_here]. Now you have a request on each keystroke. So on every key pressed by a user that means a new request on the server and/or some data processing using our mighty and all powerful JS.

This is in most cases pointless, the most relevant example being pressing space and then backspace as the user instantly changes his mind.

Why should I care, you might ask?

Also,

less js processing means less power consumption ( a lot of traffic is from mobile, don't ignore it anymore )
most of the pages are already js heavy, always adding extra dom changes and js processing without remorse will kill the page eventually

Another good example is when the person searching knows exactly what he or she is looking for : "Alabama". Most people write fast enough these days ( some random statistics found on the internet about this average about 150ms between 2 consecutive keystrokes, when excluding punctuation - which is the case for us ! ) . So consider this : a request is made for each of the following : "a", "al", "ala", "alab", "alaba" "alabam", "alabama". Taking a look at XHR requests, your network tab would look like this :

And that is for a single search from a single user.

The concept

Ignoring the same repetitive event on the same input element if it gets triggered multiple times in a given amount of time is known in the javascript ecosystem as "debouncing". A clasic debounce function takes 2 arguments: the function to be executed, and the delay.

It is implemented in jQuery. It exists as a support function in both lodash and underscore. If you already have one of this libraries included in your project, you are good to go.

But what happens if you don't? Should you add a new dependency to the project just because you did not know how easy it is to write it yourself ?

Bare with me and I will give you both a working example and an eli5 of how things work.

The how

Let's say we have a simple input that must do something at "keyup":

    <input type="text" @keyup="setFilteredStates()">

Now let's wrap the setFilteredStates inside our debounce function. It will look something like this:

    <input type="text" @keyup="debounce($event.target.value, 250)">

And the debounce function :

debounce(search, debounceDuration = 300){
    if(this.timeoutId !== null)
    {
        clearTimeout(this.timeoutId);
    }
    this.timeoutId = setTimeout( _ => {
        this.setFilteredStates(search);
    }, debounceDuration);
}

What happens is quite simple, actually. We let the javascript know that we want the function to get called after an arbitrary number of milliseconds, in our case 250. We can do that using setTimeout. All nice and easy so far, yes?

Next thing we want to do is to cancel the execution of the previous function, if it was not yet called. We can do that quite easily, as setTimeout returns an ID. So we call clearTimeout(ID) and voila, only the last key event will actually trigger the setFilteredStates, and only after 300ms have passed after last user's input.

Outro
The example is written using VueJs, but the same concept would work in vanilla or combined with any other library.
The complete snippet can be found here :
https://codepen.io/costicaaa/pen/xMorgO

For simplicity, the setFilteredStates does not actually make network calls, only filters a hard-coded array of states in JS.
I did not get into details about the binding of "this" inside or the arrow functions as this would get too long and there are already a lot of other great tutorials out there.

Open to suggestions / feedback . Also, if I got anything wrong, please let me know .

Cheers :)

Forem: costica

Continuous Delivery: HTML to Kubernetes - the why

Why

How I see things

The problem

My contribution

How

Necessary disclaimer

Closing notes

Backend Delivery - Hands-On Node & Docker 1/2

Why

What’s covered

Delivery-first mentality

The project

Containers

What bundle means

Let’s create a bundle

Build & run your application - Files only

Running what we created

Environment values

Checking the app inside the container

Exposing ports so we can use our app from outside the container

A glance into the future - How containers are actually run

Preparing for the real world

Build arguments

Short recap

Build & run - aka Deliver

Coming up - shareable dev setups with Docker

Web apps - my mental cheat-sheet

Intro

Why in a browser

With great power comes great complexity

Static HTML

Static site generators

Server-generated HTML

Single Page Apps

Single Page Apps with Server Side Rendering

Progressive Web Apps

WASM

Necessary disclaimer

Closing notes

Hacking a zero costs blogging system. With emails!

Writing a blog...

Let's define the requirements

What do I know for sure?

List of features that I want

How can I accomplish that?

The Hosting problem

Option 1: AWS Ecosystem

Option 2: Blogging platforms

Option 3: Static hosting

Option 4: Fastly

Choosing a Static site generator

Analytics

[Nice to have] Email subscriptions - part 1: "subscribe" and "store"

Compute (backend)

Storage

Compute & Storage - Firebase

[Nice to have] Email subscriptions - part 2: "send" emails

[Nice to have] Conversation options

Final solution overview

Takeaways

Debugging queries - lazy vs eager loading in Laravel

Importance of data structuring with VueJs example

VueJs type-hint imports support

The "debounce"

The why

The problem

The concept

The how

Compute & Storage - `Firebase`