Forem: Modern Web

Introducing check html links - no more bad links

Thomas Allmer — Wed, 27 Jan 2021 11:59:33 +0000

TL;DR : I created a standalone tool that can help you fix all the broken links in your websites/documentation. You can check it out on npm as check-html-links

In my developer career, I have put live multiple websites and honestly often within a few days, there was always this one issue raised. "This link on xxx is broken". 🤦‍♂️

Often these things happen as somewhere a page got moved or renamed and not every location got updated.
It's really hard to catch especially if you have a dynamic page like with WordPress or an SPA. And for users, there is nothing worse than landing on your documentation only to find a 404 staring back at them.

Luckily, with the rise of SSG (Static Site Generators), this problem becomes easier to tackle and can be solved in large part. The reason for that is that with all HTML rendered upfront as static files we can read all of them and check every link.

Evaluation and the decision for a new tool

Of course, I am not the first one to come up with that idea and there are multiple tools available on the market already.
However, when checking existing solutions I found out that most of them didn't satisfy me in at least on way 😅. Things I noticed: slow to execute, deprecated, large dependency tree, confusing output for the user, ...

Reviewing these tools I decided to create my own, with the following requirements :

Blazing fast
User-focused output
Few dependencies, to keep it lean
Preferably in the NodeJS ecosystem

Focusing on Useful Output

Most tools evaluated check files individually and report on their findings individually. That means if you have a broken link in your header or footer, you will get one line (or even multiple lines) of an error message(s) for EVERY page.

I tested this on the 11ty-website and there are currently 516 broken links in 501 files. However, the source of those 516 broken links is just 13 missing pages/resources.

In my implementation, I decided to switch from an "Error in File Focused" method to a "Missing File Focused". Let's see this with examples

Error in File Focused

This is what a lot of current existing solutions implement. Here is part of the output that is being produced:

[...]
authors/ryzokuken/index.html
  target does not exist --- authors/ryzokuken/index.html --> /speedlify/
authors/alex_kaul/index.html
  target does not exist --- authors/alex_kaul/index.html --> /speedlify/
docs/config/index.html
  target does not exist --- docs/config/index.html --> /speedlify/
  hash does not exist --- docs/config/index.html --> /docs/copy/#disabling-passthrough-file-copy
authors/cramforce/index.html
  target does not exist --- authors/cramforce/index.html --> /speedlify/
authors/accudio/index.html
  target does not exist --- authors/accudio/index.html --> /speedlify/
[...]

We get ~2000 lines of errors for /speedlify/ as it's not found ~500 times. In the middle of those errors, we also see some other broken links.
Because the reporting is focusing first on the files, and then on the actual error it is difficult to know where most errors originate from.

Missing File Focused

Let us turn that around and focus on missing references indeed. Here is the output for the same input website :

[...]
  1. missing reference target _site/speedlify/index.html
    from _site/404.html:1942:13 via href="/speedlify/"
    from _site/authors/_amorgunov/index.html:2031:13 via href="/speedlify/"
    from _site/authors/_coolcut/index.html:2031:13 via href="/speedlify/"
    ... 495 more references to this target

  2. missing id="disabling-passthrough-file-copy" in _site/docs/copy/index.html
    from _site/docs/config/index.html:2527:267 via href="/docs/copy/#disabling-passthrough-file-copy"

  3. missing reference target _site/authors/dkruythoff/github.com/dkruythoff/darius-codes
    from _site/authors/dkruythoff/index.html:2102:234 via href="github.com/dkruythoff/darius-codes"
[...]

We get one 5 line error for /speedlify/ and it tells us it's missing 495 times + 3 examples usages.
Afterward, we find very clearly more missing references and where they occurred.

A clear winner

Comparing those two outputs makes it pretty clear to me that Missing File Focused will make more sense if there is a chance that some links will be broken everywhere. My implementation focuses on missing links in its output. This is crucial because it allows developers to know where to focus their efforts first to get the biggest wins.

Focusing on Speed

Speed is always nice to have but in this case, it's probably vital. I need this to be fast so that I can run it potentially on every save. Speed is also very important in case the tool runs in a CI for example. For projects with extensive documentation, we don't want to hog the CI only to check for documentation.

Luckily HTML is an awesome language to analyze as it's declarative, which means you can read and analyze it at the same time. This may even mean that the HTML is already processed by the time the file is done reading.

With this knowledge I was hopeful - but reality didn't deliver 😅. The only tool that could keep up with the speed I needed was implemented in Go.

It seems that most tools use sophisticated parsers meant to create full syntax trees of your HTML.
In reality for link checking all you need to know are the id and the href attributes.

I have been using sax-wasm in a few situations before and I knew it supported streaming. I knew that way it could be FAST 🤞!

How fast are we talking about though?

As a rule of thumb, I decided that the analysis should be finished within 1s for a small site (up to 200 pages).
The main reason is already listed above: To not disturb during writing/development as it will run on every save.
For medium sites (200 - 1000 pages), it's reasonable if it takes a little longer - let's aim for less than 5 seconds. This will probably be a breaking point where you execute it only on-demand and in the CI instead of executing it on every save.

Results are gatherd on January 26, 2021:

Website	Pages	Duration
open-wc.org	90	~0.4s
11ty.dev	501	~2.5s
web.dev	830	~3.7s
eslint.org	3475	~12.4s

Being part of the NodeJS ecosystem

My daily workflow is hugely dominated by JavaScript, so it was only natural to want to stay in the same environment if I could reach my earlier requirements with it.
On top of this, the end goal is to integrate it within a bigger WIP system called Rocket which is node-based so therefore it will need to at least support NodeJS. Having it standalone (usable via npx) also makes it more versatile and easier to maintain/test.

Focusing on a small Dependency Tree

The JavaScript and NodeJs ecosystem is very active and constantly shifting. Lots of changes/improvements happen all the time. It's often hard to keep up. Therefore having a small dependency tree is something to always thrive for because it will reduce the maintenance burden down the line. And as an added benefit, it makes it smaller and easily embeddable as less stuff has to go down the wire. Lean is king 👑.

Solution

As already mentioned I went on and implement a link checker myself 😅. So far it fits all my requirements so I call it a success 🎉! You can find it on npm.

I called it check-html-links and its slogan is "no more broken links or assets".

The features so far are:

extracts every attribute value of id, href, src, srset
use a wasm parser (sax-wasm)
streams the html for performance
check if file or id within file exist
focus on missing references/sources

Usage

It does check your final html output so you need to execute it after your Static Site Generator.

npx check-html-links _site

Usage Github Action

Julien created a Github action available for the tool, so you can easily plug it in your existing CI. You can find it on the GitHub Marketplace.

Here is a complete example workflow that will check the result of the folder _site in the root of your repository on each push:

on: [push]

jobs:
  check_html_links_job:
    runs-on: ubuntu-latest
    name: A job to test check-html-links-action
    steps:
      - uses: actions/checkout@v2
      - name: check-html-links-action step
        id: check-links
        uses: modernweb-dev/check-html-links-action@v1
        with:
          doc-folder: '_site_'

Comparison

Checking the output of 11ty-website with 13 missing reference targets (used by 516 links) while checking 501 files. (on January 17, 2021)

Tool	Lines printed	Duration	Lang	Dependency Tree
check-html-links	38	~2.5s	node	19
link-checker	3000+	~11s	node	106
hyperlink	68	4m 20s	node	481
htmltest	1000+	~0.7s	GO	-

Future

The basic functionality is finished and it's reasonabley fast.

Topic to work on:

Allow to ignore folders (potentially via a cli parameter)
Support for <base href="/">
Big Sites Speed improvements (potentially running multiple parsers in parallel for 1000+ pages)
Speed improvements by introducing a "permanent cache" for the parse result (if file did not change, parse result will not change - we still check all links)
Memory consumption check (see if there is room for improvements)
Improve node api
Check external links

Acknowledgements

Thank you for following along on my journey on creating check-html-links. You can find the code on Github.

Thanks to Julien for feedback and helping turn my scribbles to a followable story.

If you think my open source work is valuable then I would like you to check out my personal Github Sponsor Page. Or you can support our whole group via the Modern Web Open Collective.

Photo by Teslariu Mihai on Unsplash

Introducing: Modern Web

Modern Web — Thu, 27 Aug 2020 00:00:00 +0000

We are excited to finally introduce our brand new project: Modern Web.

What is Modern Web?

A few years ago we started the Open Web Components project. Our goal was to help people develop web components, and we created guides and tools to help people do this. While working on this project, we realized that a lot of the things we were making were not necessarily specific to web components.

To maintain focus within the Open Web Components project, and to share our work with the larger developer community, we decided to split up the project and create Modern Web. Don't worry, Open Web Components is not going away! It will gain a renewed focus for web component specific topics, while in Modern Web we will work on generic tools and guides for web development.

The goal for Modern Web

Our goal is to provide developers with the guides and tools they need to build for the modern web. We aim to work closely with the browser and avoid complex abstractions.

Modern browsers are a powerful platform for building websites and applications. We try to work with what's available in the browser first before reaching for custom solutions.

When you're working with the browser rather than against it, code, skills, and knowledge remain relevant for a longer time. Development becomes faster and debugging is easier because there are fewer layers of abstraction involved.

At the same time, we are aware of the fact that not all problems can be solved elegantly by the browser today. We support developers making informed decisions about introducing tools and customizations to their projects, in such a way that developers can upgrade later as browser support improves.

Our plan for the future

This announcement marks the official release of Modern Web. Our website is live at modern-web.dev, and our packages are available on NPM under the @web namespace. Our code is open-source and publicly available at github.com/modernweb-dev/web.

For updates, you can follow us on Twitter, and if you like what you see please consider sponsoring the project on Open Collective.

We have been working on a lot of different projects in the last couple of years. In this post, we will walk you through some of our projects and how we are planning to fit them into the Modern Web project.

Guides

On our all-new website, we have a dedicated Guides section. It is meant to help you become confident in building for the modern web. It features step by step guides to work with our tools, and we document common issues developers run into when doing buildless development.

This section is a work in progress, we're looking to add more over time and would love your feedback and improvements. Feel free to hit "Edit this page on GitHub!", open issues or join the discussions.

Web Test Runner

We are very excited to announce web test runner, one of the major projects we have been working on for the past months.

There are already a lot of testing solutions out there today. Unfortunately, all of them either run tests in Node.js and mock browser APIs using something like JSDom, or they don't support native es modules out of the box.

We think that making browser code compatible for testing in node is unnecessarily complex. Running tests in real browsers give greater confidence in (cross-browser) compatibility and makes writing and debugging tests more approachable.

That's why we created web test runner.

Highlights

Some of the highlights of our test runner:

Headless testing using Puppeteer, Playwright, or Selenium.
Reports logs, 404s, and errors from the browser.
Debug opens a real browser window with devtools.
Mock ES modules via Import Maps.
Exposes browser properties like viewport size and dark mode.
Runs tests in parallel and isolation.
Interactive watch mode.
Fast development by rerunning only changed tests.
Powered by esbuild and rollup plugins.

Getting started

There is a beta version of the test runner available today on NPM as @web/test-runner. It is almost feature-complete, we will have a v1 release soon!

If you want get started now take a look at our Web Test Runner Getting Started Guide.

Web Dev Server

es-dev-server is the most popular package at Open Web Components, but it is not specific to web components alone. That's why we're working on its spiritual successor in the modern web project. We will call it web dev server, and it will be published as @web/dev-server.

If you're doing buildless development, you can use any web server for development. Our dev server helps out by providing developer productivity features and making your code compatible with older browsers.

Highlights

Acts like a real web server, without any flags it serves code untransformed to the browser.
Efficient caching of unchanged files between reloads.
Resolve bare module imports using --node-resolve.
Auto reload on file changes with --watch.
Compatibility with older browsers using --esbuild-target.
Extensive plugin system.
Integration with esbuild for fast transformation of JS, TS and JSX.
Reuse most rollup plugins in the dev server.
Plugin for polyfilling Import maps during development.

Our web dev server is not quite finished yet, but we've already built the basic parts to power our web test runner. This means that many of the listed features and plugins apply to our test runner as well.

We are working hard on finalizing the open tasks on web dev server so stay tuned for further updates.

Building for production

While we try to avoid complex builds during development, they are still a requirement for production optimizations. Here too things can get pretty complex. Through plugins and guides, we will make it easier to integrate production builds with buildless development workflows.

A good example here is @open-wc/rollup-plugin-html which we will move into the @web namespace. This plugin enables running rollup on an existing HTML page. Rollup will bundle and optimize any modules scripts found in the HTML. It works with single pages, but also with multiple pages, code-splitting and sharing common code between pages.

We plan to expand on this plugin further, adding support for optimizing assets such as images and CSS.

Progressive web apps

When we talk about modern web apps, we also talk about progressive web apps (PWA). PWAs are a great way to provide an engaging and user-friendly experience for your user, by allowing your app to work offline, and being able to install your web app on the user's home screen, among many other benefits.

Unfortunately, service workers are close to rocket science, and implementing PWA features isn't always as straightforward as it could be. That's why we'll provide technical guides and tools to make your life as a developer easier.

Not only do we ship rollup-plugin-workbox to help you generate your service worker at build time, in the future, we will also have a set of zero dependency pwa-helpers as web components and vanilla javascript functions, as well as codelabs to help you get started building your modern, progressive web apps.

Documentation (rocket)

In the past years, we've used different frameworks and tools to build our websites. Recently we've become big fans of 11ty. It's simple to use, works with markdown, and generates just plain HTML. There is no runtime javascript involved to display page content, making it super fast.

To add interactivity to our page we started using web components and applied rollup with our plugins for HTML and workbox.

This way our javascript is optimized and common code is shared between pages. Workbox adds a service worker, making our website available offline and precaches pages resulting is super fast navigation.

For a great content authoring experience, we integrated 11ty with our dev server. This adds features like resolving bare imports, and reloading the browser when files change.

By combining existing tools in this way, we think we've made something very powerful that will be useful for other people as well. That's why we started a child project in the modern web family, which we codenamed Rocket.

It's still in its early stages, but we're already using a prototype for our own website. We're still missing some features, but we'll continue to evolve it, and an announcement will follow as soon as it's finished. So keep an eye out for it!

Modern Web Family

Welp, that was a lot of information for a first announcement post — we realize, but many of these projects have been years in the making, and are finally finding their right home. As we mentioned before, Modern Web is all about making the life of developers easier, reducing the complexity of tools, and staying close to the browser.

While we have now spread out over multiple repositories like Open Web Components, Modern Web, and Rocket, we'd like to assure you that all of these projects fall under the same Modern Web Family, and aim to help make your life as developers easier.

Thanks for reading

We are incredibly proud of what we have achieved so far and the direction we are heading and we invite you to join us on our path forward.

There is much, much more to come so follow us on Twitter and if you like what you see please consider sponsoring the project on Open Collective.

Written with ♥️ by the Modern Web Core Team

Photo by Vladislav Klapin on Unsplash