Forem: Alejandro Oviedo

Node.js, TypeScript and ESM: it doesn't have to be painful

Alejandro Oviedo — Mon, 11 Dec 2023 13:40:54 +0000

I was thinking into starting a Node.js project from scratch and had an unsettling choice to make: to use ESM or CommonJS.

The decision of a few popular package authors to adopt ESM, coupled with the ongoing maturity of the tooling, strongly suggests that it is the direction most of the ecosystem will converge to eventually.

In this article, my goal is to present a solution that is not only fast but it also leverages widely adopted and trusted tools.

🙅🏻‍♂️🛑 Disclaimer
This approach is thought to comply with applications running in Node.js, not on a browser. The module resolution will change drastically for other scenarios.

Make it ESM

The first step is to make the project an ES Module. To do that you only need to include "type": "module" in your package.json.
An annoying detail for me (transitioning from CommonJS) is that ES Modules require the file extension on your imports, like this:

import { hey } from './module-b.js'

We will see in the next section how it could be improved.

tsconfig.json

This is a minimal version of a tsconfig that worked for my use case.

{
  "compilerOptions": {
    "skipLibCheck": true,
    "module": "ESNext",
    "moduleResolution": "bundler",
    "target": "ESNext",
    "isolatedModules": true,
    "esModuleInterop": true,
    "noEmit": true,
    "allowImportingTsExtensions": true,
    "outDir": "dist",
    "lib": [ "esnext" ],
    "types": [ "node" ],
    "baseUrl": "./",
  },
  "exclude": [ "node_modules" ],
  "include": [
    "src/**/*.ts",
    "bin/*.ts"
  ]
}

Notice allowImportingTsExtensions defined in there. This will allow to use your imports like the following:

import { hey } from './module-b.ts'

Compilation

I decided to use SWC to compile the project into JavaScript, primarily because of its speed. Bear in mind SWC will not run type-checking. For scenarios where type-checking is necessary, you can still use tsc to achieve this:

{
  "scripts": {
    "build": "swc src --out-dir dist/src",
    "build:ci": "tsc && npm run build"
  }
}

In order for the imports to work as expected this is the .swcrc config file that you will need:

{
  "exclude": "node_modules/",
  "jsc": {
    "parser": {
      "syntax": "typescript",
      "topLevelAwait": true
    },
    "target": "esnext",
    "baseUrl": ".",
    "experimental": {
      "plugins": [
        [
          "@swc/plugin-transform-imports",
          {
            "^(.*?)(\\.ts)$": {
              "skipDefaultConversion": true,
              "transform": "{{matches.[1]}}.js"
            }
          }
        ]
      ]
    }
  },
  "module": {
    "type": "nodenext"
  }
}

Live-reload

Also often times called hot-reloading, this functionality allows to watch your application files for changes and restart it when changes occur. I'm using SWC and nodemon to perform this, with exceedingly fast results. While using two different tools seems somewhat inconvenient I like it a lot because it leads to a single-responsibility goal: SWC only takes care of compiling our code and nodemon is in charge of watching JavaScript files and restarting the Node.js process.

Here are results showcasing the average time it took the server to restart, while watching for changes and running 10 times each using the example repository:

	time(avg)
tsx	260ms
SWC+nodemon	124ms

These numbers only measure from the time a change has been acknowledged to the new process has started successfully.

Why not include ts-node-dev in these benchmarks?
Sadly ts-node-dev is not currently compatible with the ts-node loader.

Running TypeScript files

There will be times when you need to run TypeScript files on its own. On Node v20 I couldn't make swc-node work for that (if you do, please let me know!), so I decided to use tsx. It works and is very simple to use.

{
  "scripts": {
    ...
    "migrate": "node --import tsx bin/run-migrations.ts"
    ...
  }
}

Final thoughts

Using TypeScript and compiling to ESM on Node.js involves a greater degree of complexity than is often recognized. There are a lot of tweaks and knobs to turn, and tools that are supposed to work but don't. I created an example repository with all the things I mention on this article in case you want to check it out: a0viedo/node-ts-esm-example

A decade of dotenv

Alejandro Oviedo — Wed, 06 Dec 2023 14:57:44 +0000

Node.js v20 has been released and quite a few new features were included. Among those and included in version 20.6.0, there is the --env-file flag, allowing you to import files for configuring environment variables natively. This is a popular feature among applications because dotenv has amassed the staggering number of 36 millions weekly downloads.

As an avid dotenv user I wanted to thank their maintainers for keeping the project alive for 10 years (wow). A perfect exemplary of dedication to Open Source.

Going back to the --env-file feature in hand, it can also be used to move NODE_OPTIONS environment variable to your .env file and avoid clutter on your Node.js commands. Last but not least, have in mind that you will still be able to pass environment variables on the Node.js command and that they will take precedence over your .env file:

$ SECRET=123 node --env-file=.env -e "console.log(process.env.SECRET)"

The previous command will always print "123", no matter the value for the variable SECRET you define in your .env.

Instrumenting Node.js internals

Alejandro Oviedo — Wed, 30 Sep 2020 15:31:17 +0000

Back with one of my clients, I remember a specific process that every engineer was too afraid to change because it was prone to failures and interacted with a few different parts of the infrastructure. It was a queue worker and I was told that whenever the worker just froze someone would had to restart it manually. I initially thought it made no sense but after a few failed attempts trying to reproduce the scenario I started to think different. I didn't had too much time to spend on fixing it, I was hired to focus on a newer architecture but it always resonated with me the lack of tooling to quickly get an overview of what a process is doing.

Fast-forward to today, I still think there's a lot of room to cover in the developer tooling space. With this in mind I've worked on an experiment called instrument that can collect information from your Node.js process and aggregate it in different ways.

Choosing your instrumentation spot

There are a few places where I can imagine you could intercept internal calls and collect data:

Intercepting system calls is doable and would also work for other runtimes/VMs not just Node, but you would have to target a specific operating system and it's generally more complex. With C++ (through node-gyp), you won't have to target a specific operating system but you will add some additional requirements (appmetrics falls in this space).
Intercepting calls from the javascript realm it's not only doable but also works great for portability: you don't have to install a different runtime or need additional requirements, you monkey-patch on top of it.

Configurability

I wanted to support different ways of running the tool for your existing application, much like dotenv does: include it programmatically or add an -r instrument/config flag for your command.
On top of these two alternatives, I've also added support for a standalone configuration file ("instrument.config.js") or you can also pass your own configuration as a parameter:

// at your entrypoint file
require('instrument')({
  summary: true,
  output: 'instrument-logs.txt'
})

My server is not a one-off

For cases when you're running a server and your process doesn't runs to completion you could still get the instrumentation running and separate it's output from your original process.

Reducing external noise

In most Node.js applications it's not uncommon to have a large list of dependencies, and sometimes you may not be interested in instrumenting the calls originated by your dependencies. Based on what you're looking for, you can switch on/off these logs by using the dependencies property from the configuration.

You can also choose the modules you want to instrument instead of having them enabled by default:

require('instrument')({
  modules: ['http', 'https']
})

The configuration above will only instrument the modules http and https.

Require-tree

I figured another useful feature would be to instrument the required modules. A great tool for this is madge, which focus on modules of your own and not node_modules dependencies. For my case, I choose a slightly different approach: you can choose to include the dependencies required by your dependencies or only to include one level of dependencies.

As an example, I exported the tree for running npm ls and graphed it using D3.js to end up with this SVG.

Measuring the overhead

Even if this experiment is a thin layer on top of some APIs it could have unexpected outcomes in terms of performance. I took fastify-benchmark and ran a few of the most common frameworks for HTTP servers with and without instrument enabled:

Library/framework	Throughput difference
built-in http	-11.32%
connect	-4%
express	-0.37%
fastify	-8.8%
hapi	-0.72%

Another benchmark I tinkered with was benchmarks-of-javascript-package-managers but for some reason I couldn't get consistent results out of the instrumented processes.

I see this mostly as a thought exercise, since my goal with this experiment is mostly for local development environments on which the performance overhead shouldn't matter that much.

Room for improvement

There were a lot of built-in modules or globals I didn't intercepted because I failed to see any added value (like process, path, util and the list goes on). It doesn't mean that those couldn't be instrumented, it just would take more time.
An interesting feature would be to measure the time for each of the calls that are instrumented but would require some extra work on figuring out how to graph the data to make sense of it.
The code is Open Source in case you want to take a look or if you found a bug:

a0viedo / instrument

Tool to collect information about Node.js native module calls

Another relevant question would be: could you monkey-patch other objects or functions that are inherent to JavaScript? The answer is yes!

const originalReference = Promise.resolve;
Promise.resolve = (...params) => {
  console.log('calling Promise.resolve', params);
  return originalReference.call(Promise, ...params);
}

It doesn't mean you should, though. Not only I can't think of a good reason for anyone to do it but it could also clutter your stack traces severely.
In the following weeks I want to explore exporting hooks from instrument to be able to build a warning mechanism for filesystems with strict permissions.

Shout-out to Marco Buono for donating the "instrument" package name!

Troubleshooting binaries in AWS Lambda

Alejandro Oviedo — Mon, 24 Aug 2020 13:10:02 +0000

I promised to write an entire article on this topic after struggling for a couple hours to get the graphviz library working in AWS Lambda. In this article I wanted to cover the tools I used to create a working version of the binary.

Build target

If you're looking to compile a binary for your Lambda you could either run an EC2 instance with the Amazon Linux 2 AMI or use the amazonlinux:2 docker image. Here's how you can get an interactive shell running:



$ docker run --rm -it amazonlinux:2 bash

In either case, have in mind these won't ship compilation tools by default so you will have to install them. sudo yum groupinstall "Development Tools" would get you a decent list to start with. To find out about the shared dependencies for a binary you could use ldd.
Once you compiled your binary you should copy it to the development environment for triaging.

Development environment

When working on an AWS Lambda function with a custom binary it's definitely useful to have a fast feedback loop. You can run a container with docker-lambda for this and it will replicate filesystem permissions and binaries bundled by default (among other things).
If you spawn commonly used binaries, like /usr/bin/find, make sure these are bundled by default (if they're not, you could include them yourself):



$ docker run --rm --entrypoint /bin/bash lambci/lambda:provided -c 'ls -lha /usr/bin'

It works on my machine

Once you have the binaries and libraries necessary you can include them in your Lambda deployment or in case you want to re-use them you can upload it as a layer. It won't really matter the flavor of serverless of your choice.

Layers are mounted in the /opt directory so some mangling is required to run these custom binaries. Supposing your layer included the bin and lib directories you will need to append these values to the $PATH and $LD_LIBRARY environment variables respectively. Since this is needed at runtime, it will be only language-specific step required. Here's how you could do it in Node.js:



process.env.PATH = `${process.env.PATH}:/opt/bin`;
process.env.LD_LIBRARY_PATH = `${process.env.LD_LIBRARY_PATH}:/opt/lib`;
module.exports.handler= async () => {
  /* code */
}

For a list of environment variables, including runtime-specific ones, see here.

Layer tooling

No matter which serverless flavour you're using if you want to download a layer from an ARN you will need to use AWS CLI. Even in some cases, when you're still browsing alternatives for a layer, you want to check for its content first before downloading it. It could be to verify the binary it's adding or to check the size of the files it contains.
I've built a small tool that can help to check binaries included in a layer, it's size and permissions: describeawslayer.com.

Convenience kills reproducibility

This is kinda the motto behind DockerHub's registry success, right? I know AWS SAR exists but it's not necessarily targeted for dependencies. It feels like there's still room for a layer manager (or layer registry?) to fill in the gap of re-using existing layers. I'm aware that there's also awesome-lambda-layers but it falls short to display if there's active usage on those layers or to even mention an ARN for all listed items. I trust that a tool like this could speed up the development lifecycle and would expand the notion of what we think it's suited for serverless even further.

If you used a different tool set for troubleshooting your binary within AWS Lambda or if you encountered different problems I would love to hear from you in the comments section.

Other resources:

Creating an SVG graph from your SQLite database with serverless

Alejandro Oviedo — Fri, 14 Aug 2020 17:55:33 +0000

1. Sending a binary file to an AWS Lambda function
Table of contents
2. SVG generation
3. Sending SVG as response
4. Putting it all together
5. Next steps

While working on my previous post I stumbled into great tools for visualizing SQLite databases into SVG graphs, but some of their drawbacks was that I needed to install something first. I started to wonder if I could make it easier, with no dependencies required at all.

I will split the work required for this project into three parts:

sending a binary file to an AWS Lambda function
SVG generation from a SQLite file stored in the filesystem
given an existing SVG on the filesystem, getting the same SVG as a request response

Thinking and coding it separately will help by having smaller tasks that can be independently verified.

Sending a binary file to AWS Lambda

For this step we want to allow multipart uploads into our serverless function. First, make sure you have enabled binary support in your API Gateway (see step-by-step here).

Screenshot of my API Gateway configuration for this project

Second, install lambda-multipart-parser and use it in the function's handler:

const parser = require('lambda-multipart-parser');
module.exports.handler = async (event, context) => {
  const result = await parser.parse(event);
  console.log('multipart files:', result.files);
  return {
    statusCode: 200
  };
};

Here's a minimal serverless.yml file for our function:

service: sls-sqleton

provider:
  name: aws
  runtime: nodejs12.x

functions:
  binary-upload:
    handler: index.handler
    events:
      - http:
          path: /binary-upload
          method: POST

Deploy the function. If everything went ok, when executing the function you should see something like the following:

SVG generation

For this step we will assume the SQLite file is safely stored in our filesystem (or AWS's filesystem but you get what I mean). The tool I will be using for the SVG generation is sqleton.

There is one requirement that sqleton lists on its README and that is the graphviz library. I searched the web to see if there was an existing Lambda layer or a statically linked binary that I could just import in my function to get sqleton working but the ones I found were compiled with really old versions of graphviz, like graphviz-lambda.
The amount of effort and trial and error to get graphviz working on AWS Lambda was totally unexpected. For that reason I think it deserves its own article and for the purpose of this post I'll just assume it's working.
For an initial test sqleton with some hardcoded parameters:

module.exports.svgGeneration = async (event, context) => {
  console.log(execSync(`ls -lha`, { encoding: 'utf8' }));

  await generateSVGFromSqliteFile2('test.sqlite', '/tmp/result.svg');
  console.log(execSync(`ls -lha /tmp`, { encoding: 'utf8' }));
  return {
    statusCode: 200
  };
}

function generateSVGFromSqliteFile(sqliteFilePath, svgFilename) {
  return new Promise(async (resolve, reject) => {
    const db = new sqlite.Database(sqliteFilePath, sqlite.OPEN_READONLY, async error => {
      if (error) {
        console.log('error', error);
        return;
      }

      let format = extname(svgFilename).slice(1);
      let stream, proc;
      if (format !== 'dot') {
        proc = spawn('dot_static', [`-T${format}`, `-o${svgFilename}`]);
        stream = proc.stdin;
      }

      await sqleton(db, stream, {});
      db.close();
      stream.end();
      proc.on('close', async (code) => {
        resolve();
      });
      proc.on('error', err => {
        reject(err);
      });
    });
  });
}

After deploying and invoking the function we should be able to confirm on the ls output that the SVG file was generated.

Sending SVG as response

Since we mentioned being able to verify each part individually we will not use the generated SVG but will upload one specifically for this purpose.

In the root of the project I created the test.svg file which displays a simple red dot:

<svg height="100" width="100" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
  <circle cx="50" cy="50" r="40" stroke="black" stroke-width="3" fill="red"/>
</svg>

In order to return it as a Buffer response in the body we will need to load it into memory first:

const fs = require('fs');
const readFile = util.promisify(fs.readFile);
module.exports.handler = async (event, context) => {
  const svgFile = await readFile('test.svg', { encoding: 'utf8' });
  return {
    headers: {
      'Content-Type': 'image/svg+xml',
    },
    statusCode: 200,
    body: svgFile
  };
};

After deploying this part of the function you could use curl or even Postman (which should render the red dot) to verify that the response is working as expected.

Putting it all together

After we've verified all three parts are working as expected we can start glueing them together.
To avoid filename collisions we will be using context.awsRequestId as the name for the SVG files being stored:

module.exports.generateSVG = async (event, context) => {
  const svgFilename = `/tmp/${context.awsRequestId}.svg`;
  const result = await parser.parse(event);

  const sqliteFilePath = `/tmp/${context.awsRequestId}.sqlite`;
  await writeFile(
    sqliteFilePath,
    result.files[0].content,
    { encoding: 'base64' }
  );
  const svgContent = await generateSVGFromSqliteFile(
  sqliteFilePath,
  svgFilename,
  qsParams);
  return {
    statusCode: 200,
    headers: {
      'Access-Control-Allow-Origin': '*',
      'Access-Control-Allow-Credentials': true,
      'Content-Type': 'image/svg+xml',
    },
    body: svgContent
  }
};

I built a beautiful frontend that uses the serverless function here.

As best-practice, every time you store files in the /tmp directory within an AWS Lambda execution remember to clean up before exiting.
Also, have in mind that the payload limit for AWS Lambda is 6mb.

Next steps

Given that sqleton offers some flexibility on the parameters you can use for the SVG generation (like the font or to enable edge labels) it would be nice to add those inputs in the frontend.
After I started working on this project I learnt that graphviz was being used for lots of cool projects like GraphvizOnline, so I might end up wrapping the library in its own serverless function that receives commands and returns images as the result.
For now, feel free to copy the layers/graphviz directory of the repo or you could also use this public ARN arn:aws:lambda:us-east-1:332013964457:layer:graphviz:11.

a0viedo / sls-sqleton

Handling sensitive fields with sequelize.js

Alejandro Oviedo — Thu, 30 Jul 2020 15:56:18 +0000

I've started working with sequelize again for the last few months and at first I struggled with a few things like TypeScript support and associations (so many different ways to configure them!) among other features. One of the things that I encountered recently was keeping specific fields (e.g passwords) out of default queries.
I want to walk you through my first solution to have a rationale, elaborate on hooks and then move into why it didn't worked as expected.

Setting up the models

We will go with a good ol' fashioned e-commerce for simplicity.

UML diagram of the models describing their associations

The UML diagram above tries to describe the following:

a company can have many products
a product can have many purchase orders
a user can have many purchase orders
a product must have one company
a purchase order must have one user
a purchase order can have many products

We will define the models altogether but keep in mind that in a real-word scenario I would suggest creating a different file for each model:



const User = sequelize.define('User', {
  firstName: types.string,
  lastName: types.string,
  email: types.string,
  password: types.string
});

const Product = sequelize.define('Product', {
  description: types.string,
  price: types.string,
});

const Company = sequelize.define('Company', {
  name: types.string,
  url: types.string,
});

const PurchaseOrder = sequelize.define('PurchaseOrder', {
  totalCost: {
    type: DataTypes.DOUBLE,
    allowNull: false
  },
});

// one-to-many
Company.hasMany(Product, {
  as: 'products',
  foreignKey: {
    name: 'companyId',
    allowNull: false
  }
});
Product.belongsTo(Company, {
  as: 'company',
  foreignKey: {
    name: 'companyId',
    allowNull: false
  }
});

// one-to-many
User.hasMany(PurchaseOrder, {
  as: 'purchaseOrders',
  foreignKey: {
    name: 'userId',
    allowNull: false
  }
});
PurchaseOrder.belongsTo(User, {
  as: 'user',
  foreignKey: {
    name: 'userId',
    allowNull: false
  }
});

// many-to-many
Product.belongsToMany(PurchaseOrder, {
  through: 'ProductPurchaseOrder',
  as: 'purchaseOrders'
});
PurchaseOrder.belongsToMany(Product, {
  through: 'ProductPurchaseOrder',
  as: 'products'
});

The problem

I've seen in many parts of the codebase that I'm working currently something similar to:



const user = User.findByPk(1);
console.log('User retrieved:', user.toJSON())

Why is that a problem? Just use the exclude property

One might be keen to suggest. Excluding the property for that specific line of code could work but it wouldn't be advocating for security by default. If, by human error, someone forgets to exclude the password for the next query it will still be a problem.

(Not) A Solution

While searching for different alternatives I found myself looking at the list of hooks that Sequelize has available: afterCreate, afterFind, and many others. I was already using those for something unrelated to the problem in hand and thought it could be a good approach too. After a few tests I added this to my model definition:



const User = sequelize.define('User', {
  /* props */
}, {
  hooks: {
    afterFind: result => {
      if(Array.isArray(result)) {
        for(const elem of result) {
          delete elem.dataValues.password;
        }
      } else {
        delete result.dataValues.password;
      }

      return result;
    }
  },
});

Pushed the code my model repository, updated my services and voilá, I was dealing with better query results by default.

A better solution

After starting to work on a different feature for another model I saw a response from one of my APIs:



"po":{
  "user": {
    "password": "howdy"
  }

Needless to say I grabbed my face for a slow but piercing facepalm. It turns out afterFind hooks will not run through queries with associations linking to that model.

Visual explanation of when afterFind hooks are executed

Finally, I found what I think is the best way for dealing with these type of properties: scopes. It's a powerful feature and allowed me to define the following:



const User = sequelize.define('User', {
  /* props */
}, {
  defaultScope: {
    attributes: {
      exclude: ['password']
    }
  }
});

All the associations of the User model will be able to include users without including the password field by default.

Feel free to play with the result on the Replit included here or checkout the repository on Github:

a0viedo / sequelize-sqlite3-example

Note to the reader: do not store your passwords in plain text. The examples here are especially crafted to increase the OMG! reaction.

Writing to a Google Sheet using serverless

Alejandro Oviedo — Fri, 24 Jul 2020 18:09:58 +0000

I've been working on personal projects that leverages Google Sheets for some time now and for multiple reasons: spreadsheets are super easy to create, view and edit for anyone. You can share them to the public and anyone would be able to see what you built or what you're working on.

My experience browsing the internet for resources on how to use Google Sheets programmatically was not what I expected though. Most of the examples I could find are focused on reading from spreadsheets, not writing onto them.

In this article I'll explain how you can create an AWS Lambda function using serverless from scratch that writes into a Google Sheet and will cover some of the pain points I've found.

Create a new project
Set up authentication
- Creating your service account
- Share the document
- Set up your environment variables
Implement the write function
Add a read function
Deploy it
Utils and room for improvements

Prerequisites

You will need a Node.js version installed (v12.x is recommended)
Serverless framework installed (npm i serverless -g)

1. Create a new project

We will use the CLI from serverless to bootstrap a new project. Run the following command to create a spreadsheet-example directory in the current folder:



serverless create --template aws-nodejs --path spreadsheet-example

Next we will add a new function called write with a POST endpoint as trigger in the serverless.yml that the CLI just created for us:



functions:
-  
+   write:
+    handler: handler.write
+    events:
+      - http:
+          path: /write
+          method: POST

Also add to the handler.js file:



module.exports.write = async event => {
  console.log('Starting write function');
}

A practical way to check we're on the good path is to execute sls invoke local -f write and verify the log is printing the correct message.

2. Set up authentication

I've read the documentation around Google Sheets and their google-apis library but still got frustrated from the lack of clarity on authentication mechanisms.
After some troubleshooting, I've found that using a Service Account is enough to get write permissions. For the sake of simplicity in these code examples I'll switch into using google-spreadsheet which abstracts a lot of the nitty-gritty details from Google's library.

2.a Creating your service account

Go to the Service Accounts page. You can choose an existing project or create a new one. Click on "Create Service Account" and enter a name for your new account.

You won't need to change anything for steps 2 and 3. Once it's created you should be able to see your new account on the list. Click on it and select "ADD KEY > Create new key".

After doing it a JSON file should be automatically downloaded, we will use it in the following steps.

2.b Share the document

After creating the service account you will also need to share the spreadsheet with said account.

You will have to do this for every spreadsheet you want to have access from the service account (an ideal world would have one service account per spreadsheet, but who am I to judge you?).

2.c Set up your environment variables

Now that your service account is fully configured you will want to use two fields from the credentials file: client_email and private_key. Create a new .env.yml replacing these two fields from the JSON file that was downloaded in the step 2.a:



GOOGLE_SERVICE_ACCOUNT_EMAIL: spreadsheet-example@yourexample.iam.gserviceaccount.com
GOOGLE_PRIVATE_KEY: "-----BEGIN PRIVATE KEY-----top-secret-do-not-share-----END PRIVATE KEY-----\n"

and include it in your serverless.yml like this:



provider:
  name: aws
  runtime: nodejs12.x
-
+  environment: ${file(.env.yml):}

We will abstract the usage of these variables from our handlers with the spreadsheetAuth function:



function spreadsheetAuth(document) {
  return document.useServiceAccountAuth({
    client_email: process.env.GOOGLE_SERVICE_ACCOUNT_EMAIL,
    private_key: process.env.GOOGLE_PRIVATE_KEY.replace(/\\n/gm, '\n'),
  });
}

3. Implement the write function

At this point we should be able to run our serverless function and also have authentication sorted out. In the next handler we're adding a few checks for the input of the function and we will always write to sheet with the index 0, but it could also be sent as parameter.



module.exports.write = async event => {
  console.log('Starting write function');

  if(!event.body) {
    return formatResponse(400, { message: 'body is missing' });
  }
  const body = JSON.parse(event.body);

  if(!body.cells || !Array.isArray(body.cells)) {
    return formatResponse(400, { message: '"cells" should be an array' })
  }

  // load up everything that's necessary to work with cells
  await spreadsheetAuth(doc);
  await doc.loadInfo();
  const sheet = doc.sheetsByIndex[0];
  await sheet.loadCells();
  for(const { identifier, content } of body.cells) {
    const cell = sheet.getCellByA1(identifier);
    cell.value = content;
  }
  await sheet.saveUpdatedCells();
  return formatResponse(200, { message: 'Cells saved successfully'});
};

4. Add a read function

Since we're already writing, why not offer the ability to read cells from the spreadsheet too? Modify the serverless.yml file and append at the end of the file the following:



  read:
    handler: handler.read
    events:
      - http:
          path: /read
          method: GET

and add to our handler.js file the next method:



module.exports.read = async event => {
  console.log('Starting read function');
  if(!event || !event.queryStringParameters || !event.queryStringParameters.cells) {
    return formatResponse(400, { message: 'Invalid parameters' });
  }

  const cells = event.queryStringParameters.cells;

  await spreadsheetAuth(doc);
  await doc.loadInfo();
  const sheet = doc.sheetsByIndex[0];
  await sheet.loadCells();
  const contents = cells.split(',').map(cell => sheet.getCellByA1(cell).value);
  return formatResponse(200, { contents });
};

5. Deploy it

In order to test it you can deploy it with sls deploy. If the deployment was successful you should be getting two endpoints within the terminal output. You can use curl to test it out:



$ curl --location --request POST 'https://your.own.endpoint.com/write' \
--header 'Content-Type: application/json' \
--data-raw '{
    "cells": [{"identifier": "A1", "content": "hello"}, {"identifier": "A2", "content": "world"}]
}'

If you want to both test the read function and verify the write went as expected you could run:



$ curl --location --request GET 'https://your.own.endpoint.com/dev/read?cells=A1,A2'

6. Utils and room for improvements

To avoid duplicating a few lines of code in the functions we discussed here I abstracted the response parsing into the formatResponse function. Here's how it looks like:



function formatResponse(statusCode, payload) {
  return {
    statusCode: statusCode,
    body: JSON.stringify(
      payload,
      null,
      2
    ),
  };
}

There's still parts that could be abstracted too into reusable components and for a real-world scenario you could also consider moving the read and write handlers into different files.

I've covered a very small example and there are many ways on how it could be improved. There are a few cases for error handling I've intentionally left out the code examples for simplicity like JSON parsing errors. I pushed these files to a repository if you want to check it out. For more complicated examples on how to edit the formatting of the cells, adding borders and removing columns you could check out these other projects I've pushed on GitHub:

a0viedo / dolar-historico

Finally, I'm convinced a lot of the documentation on this topic makes it seem harder than it actually is. If you're using spreadsheets as data-store or you worked on a side project that integrated with spreadsheets on your own I would love to hear about it in the comments.

JavaScript Interview Questions

Alejandro Oviedo — Wed, 12 Feb 2020 19:32:15 +0000

After a few interviews this month I wanted to share a few nice exercises that would come up in technical interviews, and how I originally approached them.

Asynchronous for-loop

This exercise is related to how asynchronous programming work. If you asked me, I would rather ask about how the event loop works instead of asking this particular question.

You are given the following code:

for(var i = 0; i < 10; i++){
    setTimeout(() => {
        console.log(i);
    }, 1000);
}

and you're asked two questions

what's the output?
what would you add to print the numbers from 0 to 9

I will leave the first answer as an exercise for the reader but question n°2 should already give you a hint.
To answer the second question what you want is to store for every iteration the value of the variable i so that they can be accessed later on by the setTimeout. One of the many alternatives is to create an IIFE (maybe better called IIAF in this case?):

for(var i = 0; i < 10; i++){
    (n => {
        setTimeout(() => {
            console.log(n);
        }, 1000);
    })(i);
}

Bonus

Would you be able to explain in the comments why this little change works as expected?

for(let i = 0; i < 10; i++){
    setTimeout(() => {
        console.log(i);
    }, 1000);
}

Reduce

The goal of this exercise is to define a function that does exactly the same than Array.prototype.reduce but receiving the array as first parameter (much like lodash's counterpart).

Iterative

This is the most intuitive solution for me, using an accumulator to carry on the result of each function execution.

function reduce(arr, fn, def) {
    let accum = def;
    for(let i = 0; i < arr.length; i++) {
        accum = fn(accum, arr[i]);
    }
    return accum;
}

Recursive

The recursive version has the requirement of knowing about an specific array operation: removing the first element of an array and get an array with the remaining elements. Of course I didn't knew which function did exactly that from the top of my head and the interviewer was kind enough to mention Array.prototype.slice. It is also important to clarify that Array.prototype.slice doesn't modifies the array itself but returns a copy of it.

function reduce(arr, fn, def) {
    // base case: there are no more elements in the array and the function should finish
    if(arr.length === 0) {
        return def;
    }
    return reduce(arr.slice(1), fn, fn(def, arr[0]));
}

Minimalistic

There's a growing interest for tweet-sized solutions, minimizing the characters used for a solution and thought it would be a nice exercise to add it here. I wouldn't recommend to include this type of solutions in your codebase though.

function reduce(arr, fn, def) {
    return arr.length === 0 ? def : reduce(arr.slice(1), fn, fn(def, arr[0]));
}

Map

For this exercise you're asked to write a map function that would replicate the behavior of Array.prototype.map.

Iterative

function map(arr, fn) {
    const result = [];
    for(const elem of arr) {
        result.push(fn(elem));
    }
    return result;
}

Using reduce

This is when things started to get interesting. I was told to only use the reduce function previously implemented to write the map function and my first thought was doing something like:

function map(arr, fn) {
    const result = [];
    reduce(arr, (curr, next) => {
        result.push(fn(next));
        return curr;
    });
    return result;
}

which is not incorrect, but was later asked "what if we don't want to use a intermediate variable?" which got me thinking...

function map(arr, fn) {
    return reduce(arr, (curr, next) => {
        curr.push(fn(next));
        return curr;
    }, []);
}

Minimalistic

function map(arr, fn) {
    return reduce(arr, (curr, next) => curr.push(fn(next)) && curr, []);
}

I've prepared a few playgrounds in CodeSandbox for you to try and solve them if you like:

Reduce
Map

There are many sites that you can use to practice with similar (or more complex) exercises like Hackerrank or Codility.

Are there any other interesting exercises that you were asked to do? Let me know in the comments!

X1 Carbon: 5th Gen (2017) versus 7th Gen (2019)

Alejandro Oviedo — Fri, 20 Dec 2019 20:14:22 +0000

I've been using an X1 Carbon for almost 3 years and recently switched to the latest model. I figured a comparison between these two could be useful for someone out there trying to decide whether to update or not. The model I got is the one with 14 inches and the following specs:

Intel i7-8565U
16GB RAM
512GB SSD M.2

Keyboard

Nothing really changed and that is certainly good news. Remains, in my opinion, one of the bests in the market.

Battery life

With almost all of the power savings enabled on Linux (see tools like powertop) I get around 6-7 hours on the 7th Gen while I would get 4 hours max on the 5th Gen.

Performance

This is one of the aspects that impressed me the most due to the nature of what I do for work. One of the build steps for a Node.js project that used to take 2 minutes on the 5th Gen model now runs on 50s on the 7th Gen!
If you will be only browsing the web I'm not very confident that you will notice any difference from that.

Disk

Without running specific benchmarks on what is shipped on the new model I can only say that the boot times and for my daily use the SSD seems to be pretty much unchanged.

Screen

I choose the 300nit configuration (WQHD, 2560 x 1440) for the new and it's about the same quality than the previous model I had.

Build quality

I was very happy with the quality of the X1 5th Gen, with only a bit of paint wearing off on one of the edges (if you squinch you can notice it on the photo above). I will say that the 7th Gen looks a bit more sturdy and somewhat more durable. YMMV.

Miscellaneous

With the 7th Gen version I noticed a problem with the touchpad I didn't had before on Linux which would make the touchpad unresponsive only when initiating the OS after suspension. After googling for solutions I finally found something that would work for my case which was creating a file /lib/systemd/system-sleep/touchpad-fix.sh with the following:

if [ "$1" = "post" ]; then
    echo 'fixing touchpad...'
        rmmod i2c_hid ; modprobe i2c_hid
fi

Did you experienced other problems? Let me know in the comments!

Decoupling code and secrets

Alejandro Oviedo — Fri, 05 Jul 2019 18:16:35 +0000

I was talking with a colleague on switching from Vault to Kubernetes secrets the other day. To my surprise a lot of our application code was dependent on our Vault implementation. Our migration would need opening a PR for every service we maintain. While trying hard not to grimace I started wondering on ways how to avoid this from the start. Two alternatives came to my mind which I'll describe here.

Solution A

An alternative is to wrap the logic of dealing with Vault's specifics and bundle that into a library. As result, once migrating to K8s secrets we could change the library's code once and it will impact the services using it. But it wouldn't be a cross-language solution i.e. you would have to maintain one library for Go and another library for Node.js.

Solution B

Another solution is to inject the secrets into the application process as environment variables. This approach decouples your applications from your secret management approach: the changes happen once for all your applications. Since it allows for more flexibility and portability than the solution A, this solution would be a better choice for a long-term scenario. This solution also follows the 12 Factor App approach for configuring applications.

In conclusion

Handling secrets for applications is a complex subject. Even if you use a solution that facilitates the concerns I covered here - being language-agnostic and having it configured as part of the provisioning of your images/instances - you're not exempt of application logic logging secrets in the wild.
You will spend more time thinking, designing and implementing decoupled components. It will also make your architecture easier to maintain and easier to apply changes.

How to bootstrap your NodeSchool event

Alejandro Oviedo — Thu, 13 Jun 2019 16:25:47 +0000

For years the team at NodeSchool Buenos Aires has been experimenting with different setups trying out tools like C9, runnable or glitch. On all those attempts we intended to minimize the setup time for the participant and avoid any kind of installations from their part. We were looking for an alternative that supported all Operating Systems, that doesn't requires a GitHub account and ideally that requires no sign-in/registration.

This is a tutorial on how to configure a modified Docker image of code-server to use a web-based VSCode editor with every workshop pre-installed on a self-hosted instance. To get a better idea I tried to put together the components in the following diagram:

1. Configure your instance

Use the cloud provider of your choice and any OS of your preference as long it's *NIX based (I haven't tried running it on Windows' WSL but should work too). It cal also run locally on a computer.
Requirements: docker, Node, a Zeit account and now installed.

2. Create your containers

SSH into your instance and define a few environment variables that will be used for the next steps, make sure you modify the environment variable CONTAINERS_COUNT to have the value you need:

export PUBLIC_IP=`curl https://ipinfo.io/ip`
export CONTAINERS_COUNT=thenumberyouwant

Create a template folder for your containers. In my case I created an introduction.js file so that the project is not empty:

$ mkdir -p nodeschool/template
$ echo "console.log('hello');" > nodeschool/template/introduction.js
$ ls nodeschool/template/
introduction.js

Each container should have a unique directory so that participants are able to save and edit files within their projects without affecting other running projects. Create the directories from your template running:

$ for i in $(seq 1 $CONTAINERS_COUNT); do cp -a nodeschool/template. nodeschool/$i; done

After that you can create the containers with the command:

for i in $(seq 1 2 $(($CONTAINERS_COUNT*2))); do docker run -e "SERVE_PORT=$((8442+$i+1))" -e "URL=$PUBLIC_IP" -d -p 0.0.0.0:$((8442+$i)):8443 -p 0.0.0.0:$((8442+$i+1)):5000 --entrypoint "/bin/bash" -v "${PWD}/nodeschool-servers/$i:/root/project" a0viedo/code-server initialize; done

Once all the containers are up you will need to extract the configuration from them running the following commands:

docker ps -q | xargs -n1 -I{} sh -c 'docker logs {} | tail -10' | grep Password | cut -d' ' -f4 | START_PORT=8443 npx https://gist.github.com/a0viedo/6707a836b16621263a31e7bd149bb6d8

The generated config.json should look like the following:

{
  "list": [
    {
      "port": 8443,
      "password": "arandompassword"
    },
    {
      "port": 8444,
      "password": "anotherrandompassword"
    }
  ]
}

3. Setup your event landing page

On this step we will configure a website that will assign each participant a unique URL and a password. You will need a Zeit account and have now installed (npm i now -g).

First, clone the minimal-nodeschool repository:

git clone https://github.com/a0viedo/minimal-nodeschool

Grab the config.json file that was generated for step 2 and copy it into the directory for "minimal-nodeschool". Finally run now in the directory and you should see something like the following:

$ now
> Deploying ~/Documents/dev/minimal-nodeschool under a0viedo
> Using project nodejs
> Synced 6 files (4.28KB) [2s]
> https://yourdeployment-hash.now.sh [v2] [2s]
┌ index.js           Ready               [17s]
└── λ index.js (2.86KB) [gru1]
┌ index.html         Ready               [1s]
└── index.html
┌ config.json        Ready               [1s]
└── config.json
> Ready! Aliased to https://yourdeployment.a0viedo.now.sh [in clipboard] [23s]

You can go a step further and alias your deployment URL to be something like nodeschoolcity.now.sh so that's easier for participants to type. Check their docs for more info.

4. Configure access to your server

Arguably one of the harder steps to automate since each cloud provider will provide different ways to enable ports on your instance (or enable those in your firewall in case you're using a local computer).
In case you're using AWS's EC2, you could open the ports on a security group with this command (you can either define CONTAINERS_COUNT or replace it manually with your value):

for i in $(seq 1 $((CONTAINERS_COUNT*2))); do aws ec2 authorize-security-group-ingress --group-name your-security-group-name --protocol tcp --port $((8442+$i)) --cidr 0.0.0.0/0; done

You are good to go! Participants can download their current projects by running download in the terminal, opening the URL that's displayed and clicking the project.zip file.

Final thoughts

Running 30 containers consumed around 17gb of memory but with a very low CPU consumption (which is not that bad!).

Forem: Alejandro Oviedo

Node.js, TypeScript and ESM: it doesn't have to be painful

Make it ESM

tsconfig.json

Compilation

Live-reload

Running TypeScript files

Final thoughts

A decade of dotenv

Instrumenting Node.js internals

Choosing your instrumentation spot

Configurability

My server is not a one-off

Reducing external noise

Require-tree

Measuring the overhead

Room for improvement

a0viedo / instrument

Tool to collect information about Node.js native module calls

Troubleshooting binaries in AWS Lambda

Build target

Development environment

It works on my machine

Layer tooling

Convenience kills reproducibility

Creating an SVG graph from your SQLite database with serverless

Sending a binary file to AWS Lambda

SVG generation

Sending SVG as response

Putting it all together

Next steps

a0viedo / sls-sqleton

Handling sensitive fields with sequelize.js

Setting up the models

The problem

(Not) A Solution

A better solution

a0viedo / sequelize-sqlite3-example

Writing to a Google Sheet using serverless

Table of contents

Prerequisites

1. Create a new project

2. Set up authentication

2.a Creating your service account

2.b Share the document

2.c Set up your environment variables

3. Implement the write function

4. Add a read function

5. Deploy it

6. Utils and room for improvements

a0viedo / dolar-historico

JavaScript Interview Questions

Asynchronous for-loop

Bonus

Reduce

Iterative

Recursive

Minimalistic

Map

Iterative

Using reduce

Minimalistic

X1 Carbon: 5th Gen (2017) versus 7th Gen (2019)

Keyboard

Battery life

Performance

Disk

Screen

Build quality

Miscellaneous

Decoupling code and secrets

Solution A

Solution B

In conclusion

Further reading

How to bootstrap your NodeSchool event

1. Configure your instance

2. Create your containers

3. Setup your event landing page

4. Configure access to your server