Forem: kontrollanten

DuckduckBang productivity

kontrollanten — Sat, 19 Sep 2020 18:28:40 +0000

I've been using DuckDuckGo for privacy reasons, but now I've actually found a real productivity upside with it.

As a developer I spend a lot of time read docs from different libraries and softwares. bang shortcuts makes that smooth and simple. If I want to search for a npm package I just write !npm algolia-places-react and DuckDuckGo will redirect that search query directly to npm:

There's over 13 000 different bangs so you can be sure that your favorite sites can be found, just go ahead and check the list at https://duckduckgo.com/bang

How are you doing to quickly find docs for the software you're using?

Bashing inspiring quotes

kontrollanten — Fri, 18 Sep 2020 21:51:33 +0000

I like history and I like quotes. I think it's an easy way to widen my perspectives and get more humble to the world. But I also like Linux. So I decided to combine those two in a golang session.

It resulted in bash-quotes, a CLI for printing quotes from http://quotes.cat-v.org/.

It's a pretty simple app that's divided into two parts:

download-quotes
As the name says this part downloads the webpage, scrapes it and uses the <hr /> to identify when a quote ends. When that's done all the quotes are saved into a filepath that's given as an argument.

read-quote
When calling this binary a randomly chosen quote is printed to the screen with some nice colors. This is the binary that I'm calling at shell login to keep me inspired with thoughtful quotes.

What's next?

The web page has a lot of sections with quotes, I've only scraped the first page so I guess I'll scrape further next time I dig into this.

You're welcome to download and use it by visiting https://github.com/kontrollanten/bash-quotes and follow the install instructions. What do you think? Would you've done it another way?

What Netlify got wrong

kontrollanten — Thu, 17 Sep 2020 20:14:18 +0000

I love Netlify. Netlify have helped me save thousands of minutes during my career. Previously I'd been using huge virtual machines and then complicated AWS S3 to host my frontend sites. But when I met Netlify it was intense love. We saw each other every day and every minute. Netlify's only focus was my prosperity.

But then came a day. She was coming home with a big bloat of stuff; AWS Lambda, Forms, Identity, plugins, etc. A bunch of features that I didn't need at all. And then I started to run into different bugs/problems in the build environment. I took a few steps back and saw that Netlify never was the perfect one.

With a bit perspective it seems clear that Netlify's main purpose isn't frontend hosting, but instead to be a service minded problem solver for website builders. As a software company you can choose to either do one thing well or everything okay, and I think Netlify is the latter. And that's fine, many people want it that way, just look at AWS.

But my requirements are quite different, I want it simple and easy:

Be able to run jobs before and after deployment.
Build environment should only be configured once.
Possibility to easy rollback the release if the e2e test failed.
The whole CD process should be monitored from the CI environment.

And to find that I have to search further. Thanks for everything, Netlify.

e2e testing Netlify deploy previews

kontrollanten — Sun, 17 Mar 2019 11:14:45 +0000

When working on my side projects I find it very important to get a straight testing process; since the projects can be untouched for months I want to be sure everything is ok when I push new code. Usually I use Netlify to deploy my sites; it's free (open source plan), it's easy to integrate with GitHub and it gives me deploy previews for each pull request.

Since e2e testing is all about being as close as possible to "real" environments I want my tests to run against Netlifys deploy previews instead of serve or some other local server setup (actually I've found bugs which was only produced in Netlifys production).

Netlify is creating deploy previews by listening for new PR's. I thought it was possible to deploy previews from CLI instead, and then wait for the response to run my e2e tests. Something like:

yarn netlify deploy && yarn e2e

But it seems there's no way to deploy previews from Netlifys REST API neither is it possible with their incoming webhooks, so I had to figure out an alternative way.

Bridge between Netlify and CI environment with AWS Lambda

To solve my problem I added an outgoing webhook who calls an URL upon deploy succeeded. I pointed the webhook to a custom AWS Lambda function I called netlify-travis-integration. netlify-travis-integration reads the deploy preview URL from the payload and triggers a Travis job to run e2e testing against.

Travis configuration

Since I couldn't find a way to run a specific Travis job I had to distinguish my non-Netlify required jobs from my Netlify required jobs by checking if the build was triggered by API or not:

jobs:
  include:
    - stage: lint
      if: type != api
      script: yarn lint
    - stage: test
      if: type != api
      script: yarn test:ci && codecov
    - stage: packtracker
      if: type != api
      script: yarn build
    - stage: lighthouse
      if: type = api
      script: yarn lh -- --perf=96 --bp=96 --no-comment $SITE_URL
    - stage: e2e
      if: type = api
      script: yarn e2e

This is a bit ugly, but I couldn't find a better way to solve it when using Travis. I think this is easier to solve with CircleCI, so I'll prefer to choose that for my next project.

GitHub statuses

To keep track of my e2e tests I added a webhook to my Travis config to let it call another AWS Lambda function. This function checks if the Travis job was triggered by an API and then creates a pull request status.

Conclusion

After one year and 100 pull requests I'm satisfied with the solution and it has caught a lot of errors I've missed. Especially it's comfortable when using Renovate who creates PR's for dependency updates, then it's easy to find out whether something breaks or not.

Way forward

I created netlify-travis-integration just to solve a problem I had, but in case there's others who'd like to solve the same problem it's not super easy to do the setup; you need to create an AWS account, setup AWS permissions, configure Travis and Netlify and also create a GitHub API token. It should be considered as a proof of concept. Maybe I'll take the next step and make it easier to integrate with new projects.

How are you solving this problem? Are you just running your e2e tests against a local web server?

netlify-travis-integration

Combine coverage reports from Electron tests

kontrollanten — Sun, 10 Mar 2019 09:35:42 +0000

We're using electron-mocha to test main process and jest to test renderer process for the ProtonMail unofficial Desktop application. To keep track of the test coverage we'd like to create coverage reports and send to codecov, but that was more troublesome than I first thought.

There's a good thread in the electron-mocha GitHub repo that helped us along the way. But to collect all information in one place I'll describe how we solved it.

Coverage report with jest

Since jest has coverage support out of the box the only thing needed is yarn jest --coverage. Simple.

Coverage report with electron-mocha

This wasn't just as easy as jest. To begin with I added babel-plugin-istanbul to instrument the code, which make it possible to trace the coverage.

.babelrc.js

    "env": {
      "test_main": {
        "plugins": ["istanbul"]
      }
    },

I didn't get babel-plugin-istanbul to work with nyc so I took this solution from bennyn which just saves the coverage data into coverage/coverage-main.json:

src/main/tests/coverage-after.spec.js

import path from 'path';
import fs from 'fs-extra';

const writeCoverageReport = coverage => {
  const outputFile = path.resolve(process.cwd(), 'coverage/coverage-main.json');
  fs.outputJsonSync(outputFile, coverage);
};


after(() => {
  if (process.env.NODE_ENV === 'test_main') {
    console.info('--- Writing coverage report');
    writeCoverageReport((global).__coverage__);
  }
});

To create the coverage report we then run yarn cross-env NODE_ENV=test_main yarn electron-mocha --require @babel/register src/main/**/*.spec.js src/main/*.spec.js.

This solution has one problem. It'll only show coverage for files included by tests. If you don't have any test you'll get 100% coverage. Almighty bennyn had a solution for that also; require all JS files before creating the coverage report. In our case it ended up with this:

src/main/tests/coverage-after.spec.js

import glob from 'glob';
import path from 'path';
import fs from 'fs-extra';

const writeCoverageReport = coverage => {
  const outputFile = path.resolve(process.cwd(), 'coverage/coverage-main.json');
  fs.outputJsonSync(outputFile, coverage);
};

// Load all source files to get accurate coverage data
const loadSourceCode = () => {
  const intrumentedCode = path.join(__dirname, '..');

  glob(`${intrumentedCode}/**/*.js`, {
    sync: true,
  }).forEach(file => require(path.resolve(file)));
};

after(() => {
  if (process.env.NODE_ENV === 'test_main') {
    console.info('--- Load all files to collect coverage');
    loadSourceCode();
    console.info('--- Writing coverage report');
    writeCoverageReport((global).__coverage__);
  }
});

Tie the bag together

Now we have to separate scripts to create two separate coverage reports. Let tie it together to one combined report.

jest will create a coverage report in ./coverage and we configured our main test to also put the report in ./coverage. By installing nyc we can combine the reports inside ./coverage and output it to ./coverage.json. If your test files isn't excluded by nyc's default config, you need to add the following to your config in package.json:

  "nyc": {
    "exclude": [
      "src/main/**/*.spec.js"
    ]
  },

Now you can run yarn nyc merge coverage combine the reports. That's it. To print the report in terminal we can run yarn nyc report -t ./coverage, which creates a report from the coverage folder, and it should look like this:

You can see the whole implementation commit here and the follow up cleanup here.

Super fast Electron builds with CircleCI 🚀

kontrollanten — Sun, 03 Mar 2019 18:01:54 +0000

Recently I switched from Travis to CircleCI, since I had some problems with getting my Travis stages working accurate. The results was overwhelming.

Simple Docker configuration

Since CircleCI jobs can run in Docker containers, it's get really simple to configure electron-builder multi build:

# Travis
script:
  - |
    if [ "$TRAVIS_OS_NAME" == "linux" ]; then
      docker run --rm \
        --env-file <(env | grep -iE 'DEBUG|NODE_|ELECTRON_|YARN_|NPM_|CI|CIRCLE|TRAVIS|APPVEYOR_|CSC_|_TOKEN|_KEY|AWS_|STRIP|BUILD_') \
        -v ${PWD}:/project \
        -v ~/.cache/electron:/root/.cache/electron \
        -v ~/.cache/electron-builder:/root/.cache/electron-builder \
        electronuserland/builder:wine \
        /bin/bash -c "yarn --link-duplicates --pure-lockfile && yarn release --linux --win"
     fi

# CircleCI
deploy-linux:
  docker:
    - image: electronuserland/builder:wine
  steps:
    - run: yarn --link-duplicates --pure-lockfile
    - run: yarn release --linux --win

The Circle config is much easier to read and maintain, since everything is based on Docker you don't need to pass environment variables you want to use in the container.

Almighty debug support

With Docker you can always run the jobs with the same environment locally, which makes everything easier to debug. If that's not enough, you can always SSH into the build container.

Super fast

For 4 minutes after a push commit I've the response from

Linting
React unit testing
Electron unit testing
Spectron e2e testing

Ten minutes later I'll also have Linux and MacOS builds deployed and ready to download. No more queues (at least yet) and no more waiting. Since a CI environment is supposed to support me as a developer I think it's vital that the feedback is quick and that I never have to wait for over 5-10 minutes until I have the response. A big plus is that I don't have to see the pride flag in my builds any more. 🥳

To see my whole Circle config, you can check here.

REST API e2e testing with cURL

kontrollanten — Fri, 15 Feb 2019 23:04:34 +0000

A while ago I was looking for a simple way to e2e test my REST API, written in Flask. Since I was only interested in make HTTP requests and parse the JSON response I didn't see any point of using any assertion library or test framework. To improve my bash skills i decided to write it with cURL and bash functions.

Readable output

Testing is to give us control and possibility to with ease find troubles. Therefore I made some colors and an icon to make it easy to parse the results for a human eye.

Green='\033[0;32m'
Red='\033[0;31m'
Color_Off='\033[0m'

Check_Mark='\xE2\x9C\x94'

Assertions

To keep the tests simple I tested everything with "assert equals", which was ran by this function.

assert_equals () {
  if [ "$1" = "$2" ]; then
    echo -e "$Green $Check_Mark Success $Color_Off"
  else
    echo -e "$Red Failed $Color_Off"
    echo -e "$Red Expected $1 to equal $2 $Color_Off"
    Errors=$((Errors  + 1))
  fi
}

Parsing the JSON

To parse the JSON I used jq, and created a helper function. The first argument is the JSON string and the second is the path, for example ".[0].title".

get_json_value () {
  echo $1 | jq -r $2
}

The tests

So finally came the tests that I run. They looked like this:

echo "### START /api/v2/shelters"
shelters_url="localhost:5000/api/v2/shelters/"

echo "When retrieving a shelter accurate properties should be given"
response=$(curl -s "${shelters_url}142784-8")
assert_equals "$(get_json_value "$response" ".shelterId")" "142784-8"
assert_equals "$(get_json_value "$response" ".address")" "Sockerbruksgatan 3"
assert_equals "$(get_json_value "$response" ".slots")" "80"

Am I happier?

Yes. I guess it'd been easier to write it with Python, but this was more fun. I also think that we developers may tend to use bloated frameworks that moves us away from the core and the control. This was a good lesson for me to think outside of my small box, which I think is necessary to keep learning.

You can see the full test suite here