Forem: Hexydec

A Better User-Agent Parser for PHP

Hexydec — Sun, 20 Aug 2023 16:11:42 +0000

I know, for years the real nerds have been telling us not to rely on user agents to detect capabilities, but the ubiquitous user agent string still has it's uses.

For one thing it is still present in most web server logs, one of only 8 pieces of information stored there (Common Log Format), and can provide a treasure trove of information about the devices that are accessing your website.

Even as the user agent string is sort of being phased out and replaced with (in my opinion) a not too well thought out replacement, the user agent is still going to hang around for a while, certainly bots and crawlers will still use it to announce themselves.

Server Logs

I am currently working on some software that provides analytics based on web server logs, it will be different to the sort of analytics you get from front-end javascript trackers, in that it will analyse all your website traffic, crawlers, assets and all.

Here the user agent string is critical in segmenting the data into users and robots, desktop and mobile, search engine or commercial crawler.

The software is written in PHP, and has a scheduler to process the server logs at regular intervals into a MySQL database. For each row in the logs, the user agent must be processed to determine information about the session.

Initially I was using the built in PHP function get_browser() which it turns out is fraught with problems.

The Problem

The first issue is that it requires a file called browscap.ini, that doesn't come with PHP, and must be regularly updated.

The second is that the browscap project has slowed it's updates in recent years, and with vendors now knocking out a new version of their browsers every month, often it fails to detect current browsers.

And thirdly is speed, due to the liberal use of regular expressions and the tens of thousands of patterns stored in the browscap.ini file, the performance of the get_browser() function isn't great, fine for one or two lookups, but when you need to do hundreds, you start to notice.

Indeed when reading the server logs in my software project, processing logs can chug along at as little as 50 rows per second, whereas when this function is commented out, it powers along at well over 1,000 rows per second.

Building a Solution

As a software engineer, my brain kicked into gear on how to solve these problems in a simple yet future proofed way.

The core problem in my mind is trying to match the whole string, it would be simpler and more flexible to extract smaller patterns of data out of the string to build up the contained information such as browser, platform, and device name.

In this way, if it happened upon a new combination of features contained within the UA string, it could still make best effort to interpret it, giving it a level of resilience, and meaning the config doesn't have to be updated everytime a new device or browser version is released.

Introducing AgentZero

It was actually quite quick to get a really basic version of my UA parser working, by looking at some common UA strings, and matching the patterns within it, the software was able to extract quite detailed information.

Then the sprawl happened, it turns out there are quite a lot of browsers and platforms and devices and architectures! But this did not deter me, I got more organised with my config files that contained the patterns and started writing some unit tests to make sure the output was consistent.

I devised a first come first serve system, where the order of the captured patterns is significant. For example, the Edge browser uses Chrome's rendering engine, and always presents the Chrome string with version number. If it was the Chrome browser, this would be listed in the returned features, but because the Edge pattern appears first, this information is filled with the Edge information, when the Chrome string is matched, does not overwrite it.

This allows each matched pattern to fill in the blanks as features are matched, even if there is cross over, providing the most significant information first, and falling back into more common patterns.

AgentZero Performance

There are currently around 360 strings that can be matched in order to extract the information from a UA string, and this will likely increase in the future as more devices, apps, and browsers are supported.

Currently on my laptop the information can be extracted in around 0.03s on average, which is a big leap in performance compared to the get_browser() function, whilst also providing more granular information.

This difference in performance is in the design of the program architecture, by looking for simple string features, regular expressions are avoided, and less strings need to be compared as each feature is matched, not every possible full UA string.

Indeed the only regular expression used is to tokenise the UA string, everything else is matched with string functions.

Ensuring Consistency and Correctness

Firstly, UA strings are tricky, some give lots of information, and others hardly any. My library is designed to extract a core set of information, not every possible detail that is presented in the string, even so it is quite comprehensive.

In order to prove it works consistently over a broad range of UA strings, I created a test suite for it with PHPUnit. The test suite is broken down into feature categories and then there are test methods for each feature with a couple of example UA strings to test against.

All the result data is tested against, providing a lot of cross-over with each suite. All in all there are nearly 400 user agent strings in the test suite providing good coverage across all the target platforms, browsers, devices, crawlers etc.

Next Steps

There is still a bit more work to do to test the performance of AgentZero in the wild, I have already started integrating it into my server logs software, and upgraded the user agent database to take the extra information, whilst also using it to identify UA strings that didn't get matched fully.

The results are already very promising with the server log software able to import at over 1,000 rows per second, and there are more metrics to segment the data with.

AgentZero will also enable me to develop features in the software that would not have been possible withget_browser(), such as showing device vendors and models, rendering engines and applications, or segmenting the robots into categories.

Availability

I have made AgentZero a free and open-source user-agent parser for PHP, available for you to download and try in your own projects. It is MIT licenced so you can pretty much do what you want with it.

I built the software to with my server logs project in mind, but I wanted to make it comprehensive, robust, and well documented so that others could use it to.

If you download AgentZero and give it a try, or you just found this article interesting, please leave some feedback in the comments below 👇.

My Portfolio: Vanilla PHP, for Fun, Promotion, and Performance

Hexydec — Fri, 11 Aug 2023 23:51:11 +0000

Its hard to believe that I have been developing website professionally for 25 years. I built my first website using Microsoft Frontpage in 1996, and got my first job building websites in the summer of 1998. Since then I have worked on countless web systems, and have been very lucky that most of them I developed from the ground up using vanilla code.

I have spent my career building systems, frameworks, and products and throughout all those project, have developed a passion for website performance.

I also consider myself a full stack developer: I did a degree in multimedia, and like working all areas of systems development such as planning, branding and identity, UX design, HTML, CSS, Javascript, databases, and server config.

TL;DR

Check out my new portfolio at Hexydec.com.

Access My Portfolio

Professional Handle

After university I ran my own freelancing business for 10 years, under the moniker Hexydec. I then took on my current role as Digital Experience Manager at SWGfL, a charity in the UK that works to help people reap the benefits of technology, free from harm.

Since my switch from freelancer to employee, the name Hexydec has become my professional handle.

My Portfolio

This year will be my 9th year at the charity, most of my freelancing is long behind me, but I am still a hungry developer and work on my open source projects in my spare time.

The time just feels right to create a portfolio of my work to flex my skillz and improve my developer profile. It is a chance to demonstrate the software I have developed, and publish some tools that I need when I am working such as my Password Generator.

Also it is an opportunity to construct something with a bit of flair, have some fun, and push the boundaries of what I can do.

Philosophy

During my career I have focused on building well optimised websites that present data in a structured and dynamic way. It seemed fitting that I should do the same with this site, I am not sure I thought about how big my portfolio should be, considering most portfolio's are only one or two pages, currently this one is running at over 60 pages.

The basic premise is to categorise all my skills and projects, write a short page about each, and then link them together with a tagging system.

Logo & Navigation

I worked hard on a few elements that are reused throughout the website, in the template the logo is animated along with the burger menu. I wanted the menu itself to be a little different, and so I came up with the fan menu which opens out sequentially when opened. Each section has an icon and a colour to represent it.

Content Navigation

Cards are a good mechanism for presenting links to more content, for this site I wanted to use 3D flip cards, with a simple title and a logo or icon, which when hovered over, presents more information.

The animation sequencing took a while to get the feel right, but the effect is pleasing. Each card also has a colour to represent the section it links to.

Project Gallery

The main content pages of the site are to present the projects that I have created. Images of the project are important, but displaying screenshots can be a little dry, so I created an animated polaroid gallery to make the interaction more interesting.

The side menu displays links to all the skills, languages, software, and projects used to build the each project and link to their specific skill page from where you can find other projects that use that thing.

Apps

Most of my open source projects are developed in PHP, so whilst I can publish the code on GitHub, I cannot run the code for users to try out there.

The apps section of my portfolio enables anyone to try out my code, such as my minification tools, and my User-Agent parser. I also created some some other programs specifically for my portfolio, such as an encoder/decoder, anagram solver, and a capital cities quiz.

Solari Board

One of the tasks that gives me enjoyment is performance optimisation. This starts from the ground up in writing efficient code that doesn't waste resources, all the way to compressing output code and images, and optimising transport.

For my portfolio as well as offering a testing platform for some of the tools I have developed, I wanted to show some of the inner workings of how this website has been optimised.

I also like trains, so I developed a Solari board, similar to the old mechanical split-flap departure boards you used to see at airports and railway stations to display metrics about the website, such as load times, memory usage, asset sizes, and compression stats. Just click the stats button floating in the bottom right hand corner of each page.

What do you think?

It has taken a year and a half, and 114 commits to get my portfolio up and running, and I now present it to the world. You can view my portfolio at hexydec.com.

Access My Portfolio

If you decide to have a look around, I would appreciate your comments and feedback, which you can leave below. Thanks!

Wordpress vs Optimised vs Custom

Hexydec — Tue, 03 Jan 2023 08:05:21 +0000

I happened to inherit a Wordpress site for a project I was working on. When I received it, it was in an ok state, it wasn’t that well optimised but it was ok. To get it working a bit faster I did some work on the site to optimise the network usage and reduce the asset sizes.

The site is multilingual, and one of the jobs I had to do was add the translations for a new language. Is used a plugin called WPML, and this is really where all the problems started. Entering the new language and updating some content in all languages was so problematic, I was left with no choice to rebuild the whole site using our custom framework, I had planned to do this at some point in the future, so this just moved it up the timetable.

This left an interesting situation, here I have 3 versions of the same website:

The original Wordpress site
An optimised version of the Wordpress site
A custom built version

For this article, I thought it would be interesting to compare the performance of them, to document some of the optimisations I performed and the issues I encountered.

Optimising Wordpress

The site as it arrived wasn’t too bad, it loaded in just under a second, but the homepage weighed in at 2MB. It scored 59 on lighthouse for the performance test.

The first optimisation I performed was to add my Wordpress plugin Torque to the plugins list. This combined and minified the CSS and Javascript, and minified the HTML, reducing the size and number of requests. It also sets some headers to optimise subsequent page loads.

There was a rather large block of inline CSS that was loading into every page, so I moved this into an external stylesheet. The theme had a custom interface to set some other settings, which I used to reduce the size of the icon bundle. The fonts that were used were included externally from Google fonts, but it had the option to upload them locally so I swapped those out too.

Deciding to Rebuild

A little more detail on the issue that led me to rebuilt the site from scratch, this website now contains over 20 languages, I am not sure how the original developers managed to enter all the content in the languages it had when I inherited it, but I am guessing that the one I entered just tipped it over the edge.

The simple fact is that the language plugin beats Wordpress into doing something it was simply never designed to do (and quite frankly something it should do out of the box). This plugin invades almost every interface of Wordpress to enable you to do the same thing once for every language.

Having looked in the database, for a page it creates one post for every language, and then hides the pages except the one for the language it is currently showing and has some obfuscated JSON blob within each page that enables it to be tied all together.

Where it fell down was that I was unable to make an edit to a page in existing languages, I was only able to edit it in English. Also the menu synchroniser was so buggy I couldn’t link my menu in the new language, I ended up assigning 2GB RAM and 10 minutes of time to it, but it just sat there spinning until it timed out.

Building the Custom Site

The new site was built with our custom PHP framework, the base setup has a login and CMS, the pages are constructed with content blocks that can contain applications. The applications for this site were a prebuild FAQ application, and a custom-built partners list, as well as a number of different content layouts, and a carousel.

CSS is compiled from SCSS files, using some framework files, and some custom files to implement the design and site layout. Javascript the same using ES6, my jQuery clone library Dabby.js, and Rollup.

Copying the content over was really what took the time, even though the site only has 7 pages, with 20 languages, some of them right to left, I had to make some upgrades to our language system to support them, and then there was a LOT of copying and pasting to get all the content over.

Wordpress vs Optimised Wordpress vs Custom Build

So let’s compare the performance of all three sites:

Client-Side Assets

	Wordpress	Optimised	Custom
HTML Size	211kb	120kb	47kb
HTML Size Brotli	90kb	20kb	11kb
CSS Size	520kb	462kb	179kb
CSS Size Brotli	88kb	76kb	20kb
Javascript Size	845kb	492kb	27kb
Javascript Size Brotli	229kb	134kb	10kb
Assets Size	473kb	571kb	247kb
Assets Size Brotli	473kb	478kb	185kb
Total	2.05mb	1.62mb	499kb
Total Brotli	888kb	710kb	227kb

Whilst the optimised version showed an overall reduction in download size of 21%, the custom version smashed both with a reduction of 76%.

The HTML size reduced significantly from the base to the optimised site (-43%) as there was 3 huge blocks of Base64 encoded Figma code commented out in the body. Also I moved some CSS from inline to the external stylesheet. But the custom site enabled the HTML to be much leaner (-78%) as it was designed specifically for purpose, and didn’t have extra bloat of the generalised block builder.

The reduction in CSS and Javascript sizes between the base and optimised version showed that the base site was not that well optimised, the removal of unneeded plugins and combine/minify of the assets enabled a 30% reduction in size.

The CSS on the custom site was 65% smaller than the base site, even though it contained all the styles for the admin system which the other two sites have in a separate stylesheet. Including Javascript the reduction was 85%. Developing the styles and Javascript of the website specifically for this site meant that there wasn’t much bloat, leading to a drastically smaller size, and less work for the browser to do to consume it.

The asset size of the optimised version went up due to the external fonts being counted as Javascript in the base site, but overall the size was reduced. Whilst the same assets were used in the base and optimised sites, I ran the PNG’s through a PNG optimiser, and minified the SVG’s.

On the custom site the assets were 48% smaller than the base site, this is because I remade some of the bitmap images as SVG’s, and I trimmed a load of whitespace out of the other PNG images.

Performance Metrics

	Wordpress	Optimised	Custom
Lighthouse Performance	59	66	100
First Contentful Paint	2.5s	2.0s	1.1s
Time to Interactive	5.5s	5.6s	1.1s
Largest Contentful Paint	5.9s	4.6s	1.7s
Total Blocking Time	390ms	320ms	0ms
Cumulative Layout Shift	0.084s	0.084s	0s

Optimising the Wordpress site improved most metrics across the board, the performance score was 7 points higher, and page draw times reduced. The time to interactive was slightly higher, this may be to do with moving the fonts to local, meaning whilst it had one less domain to lookup and connect with, it couldn’t download as many assets in parallel.

Due to the size of the HTML/CSS/Javascript on both Wordpress sites, the performance score was always going to have a ceiling, the size of those assets not only increasing the download time, but more importantly making the browser have to work much harder to process them. The custom site with its leaner assets gained a perfect score of 100, a 69% increase.

The render metrics were also significantly better across the board, a result of the reduction in work the browser has to do to render the page.

Server-side Assets

Whilst this doesn’t matter so much to users, I thought it would be interesting to compare some metrics from the server-side.

	Wordpress	Optimised	Custom
Website File Size	746mb	258mb	9.45mb
Page Generation	0.5062s	0.7251s	0.0318s
Peak Memory Usage	21.7mb	25.5mb	1,014kb

Here we can see the size of the website code and assets, there was a fair amount of leftover junk from some plugins that weren’t being used that caused the reduction of 488MB (65%) from the base site to the optimised. The custom version is way smaller at 9.45MB, a reduction in size of 736MB (98.7%).

When running the sites, the optimised version takes longer and uses more memory than the base version, I think this is down to the fact that I tried to add another language to the optimised version, and this seems to increase the generation time and memory usage.

The custom version absolutely annihilated both Wordpress versions using 95% less RAM, and generating the page 1,592% faster.

Database

The database is a critical component to the functioning of a dynamic website, and here both Wordpress sites and the custom site use MySQL (Specifically MariaDB).

	Wordpress	Optimised	Custom
Database Size	155mb	181mb	12.5mb
Database Tables	71	45	25
Database Rows	139,589	70,825	700
Queries	151 - 221	175 - 243	6
Query Time	0.0453s	0.0526s	0.0058s

I managed to cut down the number of tables used in the optimised site as many were used in plugins that were no longer active or needed. Interestingly whilst the number of records in the optimised site was half that of the base site, the size actually went up, as did the number of queries to generate the homepage. Again I attribute this to the extra language I added to the optimised site.

The custom site’s database is 92% smaller, with 99.5% less rows. In terms of queries, the homepage generates around 96% - 97% less queries. I have expressed the number of queries for the Wordpress site as a range as it changes on every page load, something to do with caches or plugins writing back to the database.

In terms of query time the optimised site was 16% slower due to the larger number of queries, whereas the custom site completed its queries 87% faster.

Conclusion

First to reiterate, I have dealt with a fair few Wordpress sites in my career, and this one was one of the better ones. It’s problems were not down to what the website developers did, it was a product of Wordpress, the language plugin, and the demands of the site. For example, the language plugin generated 136 database queries (55%), whilst the theme’s block builder generated 76 queries (31%).

The point of this article is to show how much more a website can be optimised by building to exact specifications, and also to show how problematic Wordpress can be when you want to get everything working really well.

Clearly the custom built website was better on all metrics, the homepage was 89% smaller than the original website, the server-side assets were 98% smaller, generating the page 1,592% faster, with 95% less RAM. It’s performance metrics were also significantly better. The result is that for users the site feels more responsive and requires less of their resources.

On the server-side, the custom built site had a footprint that was 95.5% smaller than the optimised site (including database), and should be able to handle ~16x the number of page requests on the same hardware, and with RAM to spare (based on the numbers here and without stress testing the sites).

The database size/load/queries were all over 90% less than the Wordpress sites, as database resources can be the bottleneck to a growing website, this allows a lot more room to scale in the future.

Wordpress is so popular because it is easy to get going, there are a lot of plugins, it is cheap, and there are a lot of developers out there who develop for it. The penalty for this is without the support of good plugins or suitable plugins, you don’t always get exactly what you want, and the base code and plugin architecture are very generalised, meaning it won’t be very well optimised for your specific website.

My experience is that plugins tend to add bloat quite readily to your pages, don’t offload CSS/Javascript to external files, or they try to do too much, and aren’t optimised at all well for delivery of the page.

I should also mention hidden costs, apart from your hosting bill, which if this example is anything to go by, will likely be much higher, but unoptimised back-end processes cost you money every time they are used if they are inefficient, taking longer to complete than they would if the process has been designed specifically for your needs, often annoying staff who find the laborious processes frustrating.

On the flip-side, with custom development you get exactly what you want, but it takes longer, will cost more and any architectural changes you want to make will likely need a developer. It also relies on having a good developer who knows what they are doing, and has a bank of trusted software, such as a CMS that they can tailor to your needs, and finding good developers these days can be challenging.

But when you have the right people, you will get a lean mean website that will trounce most off the shelf platforms on just about every metric, enabling you to do things that may not be possible to do or do well with something like Wordpress. It will oil your entire business with efficient and automated processes that are a pleasure to use, ultimately pushing it forward rather than holding it back.

The State of Minification in PHP – How 1 Project Grew into 6 (Part 2)

Hexydec — Sun, 30 Jan 2022 22:06:16 +0000

In part 1 of this post, I presented to you my journey of writing a set of compilers to minify HTML/CSS/JS using PHP, along with the software I wrote to see how they stacked up against the competition.

This is the results.

The Contenders

I used google, packagist, and GitHub to find some projects to go up against, I picked projects that were most used. Where the original project had been abandoned, I picked forks that maintained them, although some of those forks themselves hadn’t been updated in a while.

I also picked a script on GitHub Gist that seemed to have some traction, it combined an HTML, CSS, and Javascript minifier.

So here they are (Note stats may not be completely up to date):

HTML Minifiers

HTML Minifier	Type	Commits	Stars	Dependents	Maintained
hexydec/htmldoc	Compiler	212	11	1	Yes
mrclay/minify	RegExp	764	2900	3731	Yes
taufik-nurrohman	RegExp	48	199	Unnknown	Yes
voku/html-min	Compiler	203	106	151	No
pfaciana/tiny-html-minifier	Linear Consumer	46	123	0	No
deruli/html-minifier	Compiler	39	2	0	No

CSS Minifiers

CSS Minifier	Type	Commits	Stars	Dependents	Maintained
hexydec/cssdoc	Compiler	169	1	1	Yes
mrclay/minify	RegExp	764	2900	3731	Yes
taufik-nurrohman	RegExp	48	199	Unknown	No
matthiasmullie/minify	RegExp	439	1700	4544	Yes
wikimedia/minify	RegExp	220	0	12	Yes
tubalmartin/cssmin	RegExp	123	223	5586	No
natxet/cssmin	Compiler	38	79	1308	No
websharks/css-minifier	RegExp	10	17	159	No

Javascript Minifiers

JS Minifier	Type	Commits	Stars	Dependents	Maintained
hexydec/jslite	Compiler	64	2	1	Yes
mrclay/jsmin	Linear Consumer	764	2900	3731	Yes
taufik-nurrohman	RegExp	48	199	Unnknown	No
matthiasmullie/minify	RegExp	439	1700	4544	Yes
wikimedia/minify	RegExp	220	0	12	Yes
tedivm/jshrink	Linear Consumer	110	25	0	No

Software

The tests were run on a VPS running Ubuntu and PHP 8, with 2 Intel Xeon CPU's and 2GB RAM.

Metrics

To understand the results, there are 3 metrics here that we need to look at:

Compression Ratio – How much compression was achieved by each package (Gzipped and non-gzipped)
Speed – How fast does each package perform the compression
Reliability – Is the output valid

The first two were fairly straight forward to calculate, but the third needed some extra programming to plugin some packages or external services to validate the code, and there are some caveats:

Not all the input is valid, so where the number of errors in the input equals the number of errors in the output, the result is considered valid
The HTML and CSS minifiers both use the W3 validator service, on certain errors it doesn’t present the errors in that tree, if the minifier fixes the error, it may show more errors than the input if the tree in question has more errors underneath it, causing a false positive

Because of the false positives on the output, the software allows you to see the validation errors, and to view the output.

Scoring System

For each metric, a score will be calculated like this:

(100/(Max-Min)*(Value-Min)

This means that the slowest / lowest compression / least reliable will score 0 and the fastest / highest compression / most reliable will score 100, and other values will be graded between the highest and lowest.

To get the totals, the scores will be added up, but since there are two metrics for the compression (gzipped and non-gzipped), the reliability and speed metrics will be multiplied by 2, allowing the average score of the two compression metrics to be calculated in.

A Note

Before looking at the results I want to note that all the developers including me have worked very hard on their creations, and many of them are well tested and are used by millions of websites everyday.

Currently my software is only used by a handful of websites.

The Results

HTML Minification Test

The scores for the HTML test are as follows:

	Reliability	Speed	Compression	Gzip	Total
voku/html-min	93	23	60	45	56%
mrclay/minify	80	30	41	32	49%
taufik-nurrohman	0	100	0	0	33%
deruli/html-minifier	100	0	44	36	47%
hexydec/htmldoc	94	44	100	100	79%

taufik-nurrohman’s minifier was the fastest followed by mrclay/minify, both beat the next fastest which was mine, by about 8x. Unsurprisingly these were both RegExp based ones. All the compilers were slower, the slowest of which was deruli/html-minifier, which is a PHP port of the Blink Tokeniser. It was 4x slower than the fastest compiler (mine), but this is not surprising since the original code was written in C++, which is much faster than PHP.

My software gave the highest compression at 11.00%, 39% better than the next highest voku/html-min at 6.71%. Just to point out here that my HTML minifier also minified inline CSS and Javascript, and I don't think any of the others did, which is perhaps why the compression was so much higher than the others. Without compressing the inline CSS and JS, the compression ratio for my software would have been 8.2%.

All the compilers performed better than their RegExp counterparts on the compression metric.

The worst performer was taufik-nurrohman at 0.20%, note that only the results considered valid were taken into account, and because of the amount of errors in the results, the compression was always going to be lower, although strangely most of the results that were valid seemed to produce bigger output than the input.

All the minifiers tested showed more errors in the output than the input on at least one test, although after a brief analysis of the input and output, it looks like most of them are false positives. Certainly in my software, the errors captured by the validator were all present in the input code, but for some reason were not reported in the input.

There was once exception to this, and this was the Gist script by taufik-nurrohman, it showed output errors in over 90% of the test websites.

Even though there were false positives, I took the results as the came out for scoring.

Overall I am pretty happy with the performance of my software, discounting the errors which I think are all false positives, it was the fastest compiler and also produced the best compression, along with the highest overall score.

The script from taufik-nurrohman did not perform well, whilst it was the fastest, not only did it have the most errors, it also had by far the lowest compression.

The most popular project mrclay/minify performed well, a few more errors were reported than on my software, but again I think these are all false positives.

CSS Minification Test

The scores for the CSS test are as follows:

	Reliability	Speed	Compression	Gzip	Total
matthiasmullie/minify	100	0	46	43	48%
tubalmartin/cssmin	100	79	68	73	83%
mrclay/minify	100	63	36	45	68%
natxet/cssmin	100	89	0	0	63%
cerdic/css-tidy	100	73	29	60	73%
websharks/css-minifier	100	100	12	67	80%
wikimedia/minify	100	100	37	55	82%
taufik-nurrohman	100	99	28	7	72%
hexydec/cssdoc	100	72	100	100	91%

All the contenders performed well here, there were no minifiers that produced more validation errors that there were in the input.

The lack of errors is here is likely an output of the fact that CSS is easier to parse than other languages due to its fixed and predictable format.

The fastest minifier here was wikimedia/minify, along with websharks/css-minifier it blitzed the rest of the field, minifying all 12 test files in 0.047 seconds. The slowest was matthiasmullie/minify which completed the tests in 8.4 seconds. Again the RegExp based minifiers were all faster than the compilers with the exception of the above, the fastest compiler was natxet/cssmin at 0.94 seconds.

All the contenders were very close in their compression ratio, again this is likely due to the predictable format of CSS. The lowest compression was natxet/cssmin at 21.17%, 16.70% gzipped. The highest compression ratio was achieved by my software at 24.63% and 20.14% gzipped.

One anomaly here was matthiasmullie/minify, it mostly performed well in compression and would have completed the tests in good time, but it for some reason had trouble processing a couple of the test files, one has significantly lower compression than the others, and another took 6 seconds out of its 8 seconds overall to minify it.

Here my software was not the fastest compiler, but it did achieve the highest compression. I ran the results here on Ubuntu, but on my laptop which is Windows, it was by far the fastest compiler, I guess the OS and platform implementation of PHP makes a difference in the performance of the code that is running.

Javascript Minification Test

The scores for the Javascript test are as follows:

	Reliability	Speed	Compression	Gzip	Total
matthiasmullie/minify	100	0	0	0	33%
mrclay/jsmin	100	33	96	96	77%
tedivm/jshrink	100	76	96	96	91%
wikimedia/minify	100	86	97	94	94%
taufik-nurrohman	0	100	7	0	34%
hexydec/jslite	100	27	100	100	76%

For the Javascript tests all the contenders produced valid results except for taufik-nurrohman which had 2/10 invalid results. It was also the fastest, completing the tests in 0.18 second, wikimedia/minify was the next fastest at 1.75 seconds.

With regards to speed, the RegExp based minifiers were out front again (except one). But there are more linear consumers and my compiler in the mix, showing how much more difficult it is to parse Javascript than HTML and CSS. Indeed when I started my project it was RegExp based, but I rewrote it as a compiler because I found that I couldn’t achieve what I wanted to do using regular expressions alone.

After the RegExp based minifiers, the Linear Consumers came next in the speed ranking, with my compiler and then matthiasmullie/minify bringing up the rear. My software had the highest compression ratio in both non-gzip and gzip with 33.97% and 26.67% respectively.

The slowest and lowest compression was matthiasmullie/minify, again there was an anomaly where it was unable to compress one of the tests files to anywhere close to the ratio achieved by all the others. This had a big impact on its overall compression ratio which ended up being about half of the others (17.55%), it would have been on par without this problem. It also struggled in the speed department, clocking in a total of 11.4 seconds, interestingly on Windows it was one of the fastest, whilst the most popular mrclay/jsmin was the slowest.

Happy here to be leading the pack across the board on compression ratio, my goal was to produce software that was fast enough to use on the fly with inline javascript, so I didn’t want to get too crazy with the optimisations, I only picked the ones that were safe to perform and didn’t need too much context to know whether it was safe.

Hats off to the devs who write the javascript based minfiers where their AST tracks all of the variables and scopes, and thus are able to perform much more heavy-handed optimisations in a safe manor, something that all the software packages here opted not to do, probably for both processing speed and to reduce the complexity of their software.

Conclusion

It is interesting how the structure of the input language determined the spread of speed and ratio, with the more predictably structured CSS showing the contenders much closer on all metrics. Also the difference in speed/ratio/reliability between the RegExp based minifiers and the compilers, there was definitely a clear gap in speed with the RegExp software being faster, whilst also being slightly less reliable and lower compression.

The results here are just an indication of the performance and quality of the software packages presented here. With different inputs, running on other OS's the results will be different. Also the scoring system levels each metric evenly against the others, whereas real world requirements may be different.

On quality, I think this is harder to measure than the benchmark presented here, which didn't take into account metrics like code coverage for tests, or whether the project is maintained.

What do you think? Please tell me your thoughts in the comments.

How My Software Performed

To the performance of my own software I am definitely happy, whilst none were the fastest overall which I expected, they were also not the slowest, and proved to be reliable in all tests (bar the false positives which all the competitors seemed to suffer from in the HTML test).

Proud also that my software achieved the highest compression across the board, helped by the software I wrote to compare it to the competition.

In the scoring my HTML and CSS minifiers came out in 1st place, with the Javascript minifier coming in 4th. Speed was the issue here, and being the only compiler in the list it wasn't a suprise that it didn't score that well on the speed metric, although it wasn't the slowest and also was only 1% behind the most popular package. Perhaps I will rework it into a linear consumer at some point.

For my own process, writing and testing this software has been extremely challenging, but has enabled me to learn new concepts, implementation techniques, and overall improve my process. Before starting these projects, I hadn’t:

Done any serious testing with PHPUnit
Published anything on packagist
Used composer or published anything that required it
Written a Wordpress plugin
Done much with promises in Javascript
Produced a code coverage report
Used GitHub’s build system
Published any articles on dev.to

To all developers who are starting out or who haven’t published their own project yet, I would highly recommend it. You will be pushed out of your comfort zone, and be required to learn new techniques, processes, platforms and systems to be able to get your software to a point where it is stable, tested, documented, and basically good enough for other developers to pull into their production projects.

If you want to run these tests yourself, or want to try out my software, all my projects are available on GitHub and other platforms with the following links:

Me (Hexydec) on Twitter
HTMLdoc on GitHub and Packagist
CSSdoc on GitHub and Packagist
JSlite on GitHub and Packagist
Tokenise on GitHub and Packagist
Torque on GitHub and WordPress
Minify-Compare on GitHub

The State of Minification in PHP – How 1 Project Grew into 6

Hexydec — Sun, 30 Jan 2022 22:05:38 +0000

My name is Will, alias Hexydec, and I am a developer. I have been developing websites professionally for 23 years, and I have always been obsessed with performance, how you can optimise your web application to give users the best possible experience.

In this article I want to talk about minification, the process of removing extraneous bytes from the code you deliver to the client, to make the payload as small as possible. I want to talk about all the open source PHP minifiers there are out there, and try to show how they stack up against each other.

First let me fess up here, I have written a number of PHP minification programs in my career, and 3 years ago I started my own GitHub project to write an HTML minifier in PHP. Partly this article is to present that software to you, but it is also to show you the research I did myself to see how my project stacked up against the competition and improve my software, hopefully you will find the results interesting, as I did.

TL;DR

If you are not interested in how all this came about and just want to skip straight to the results, read part 2.

How does a minifier work?

I just went back and looked at the first commit for this project, it looks like I started with the previous minfiers I wrote for HTML and CSS, and I was planning to use a program called JSMin to minify any inline javascript.

The HTML minifier I committed used some loops of Regular Expressions to find and replace patterns within the code to remove the unwanted whitespace etc, but the CSS minifier was a much more interesting beast – it broke the CSS down into an internal array that represented the document, changed some properties in the array, and then compiled it back to CSS – it is a compiler.

So why did I write a pattern based replacement mechanism for HTML and a compiler for CSS (I wrote it years ago)? I am guessing this is because the structure of CSS is much flatter, and much more predicable. You will see later on how this plays out with the other minifiers I tested.

So how does a minifier work? Well it either:

takes the input code, uses regular expressions to find patterns within the code, and replaces them with the same code minus all the bits you don’t need, or;
it splits the input code into tokens, loops through them, consuming each one to generate an internal representation of the document, performs optimisations on that representation, and then compiles it back to code

Regexp or Compiler, which is better?

It really depends on what you are trying to do, regular expressions alone are faster, but they have their limits. The issue with them is that they just look for patterns, what is matched doesn’t have any context.

If you have a language like CSS, it is quite predictably structured, this would be easier to use regular expressions to minify, but even so you will need some callback code to process the values to do individual optimisations.

Compare this to a language like Javascript, where the code can be infinitely nested, and is full of pitfalls such as determining whether a line ending is the end of a command, or matching regular expressions in the Javascript code, suddenly you will find a lot of edge cases which need more and more code to handle – it ends up feeling like whack-a-mole.

A compiler on the other hand has a number of stages to parse the code into a more understandable structure, therefore it is slower, but the result is that each bit of code has a relationship to the other bits of code, and once you have it in this format it is then much easier to work with.

Edge cases are now much easier to handle because your code has an understanding of what each piece of code is, it is then simpler to work out what you can do to it. You can also handle errors in the input code, the context enables you to know what to expect next, and if the token that appears is not what is expected, then you can discard it or throw an error, whatever is appropriate for what you are trying to achieve.

Because of the context, you can also perform more optimisations than you would be able to with regular expressions. Take HTML minification for example, an optimisation such as removing closing tags is much easier when you know what tags are next to or ancestors of the current tag.

So with regular expressions you get speed and ease of development, with compilers you get reliability, the ability to handle more complex input, handle errors, and better optimisation.

There is also a third option, as sort of halfway house, I am not sure what you would call it, so I am going to call it a “Linear Consumer”. It processes the input in a linear fashion, using a tokeniser, it then processes the code and spits out the optimised code as the tokens are generated.

By doing all the processing as you go, you don’t need to create an intermediate representation, thus improving performance, but the tradeoff is that you can only perform optimisations within close range of the current tokeniser position.

How does my HTML minifier work?

My GitHub project is called HTMLdoc, and as you might have guessed it is a compiler. It does use regular expressions, but only to tokenise the input. This keeps the regular expressions fairly simple; it just splits the input into bits ready for consumption by the parser.

Note that I have tried writing a string based tokeniser, but regular expressions are about 3 times faster. This is probably because of the amount of userland code required to chop the input up, whereas with RegExp all that work is done in optimised C code.

The output tokens are then parsed into objects, each representing a part of the code, such as a tag, textnode, or comment. A minify() method then distributes the minify command to each object in the tree, which performs any optimisation on each bit of the code.

Finally the compile() command orders each object to render itself and its children as HTML. There is a fair amount of complexity in doing all of this, hence the reason for using objects which splits the functionality into smaller more modular bits, I have also tried to make it all as configurable as possible, so you can control things like which tags are pre-formatted, and what minification optimisations to perform.

You can also query the object using CSS expressions to extract text and attributes. If you have ever used a program called simple_dom_html then this functionality is akin to that, but with a built in minifier (and actually maintained by someone!).

CSSdoc: Project Number 2

So I already had an old CSS compiler, and in my pursuit of the highest compression, it was clear that I needed to rewrite this to compile any inline CSS in the HTML, enter another Github project - CSSdoc.

It uses the same sort of code layout and tokeniser as HTMLdoc, but due to CSS’s predictable structure, I elected not to retain the input whitespace, this was for better memory and performance. Instead it has two compiler modes, minify and beautify.

JSlite: Project Number 3

I mentioned earlier that I was planning on using JSMin to minify any inline Javascript, if you have used any other minifier in PHP, you have probably used JSMin. It is a port of Douglas Crockfords Javascript minifier, and was (and kinda still is) the defacto PHP minifier for Javascript. Most Wordpress plugins use this behind the scenes.

But the port is not actively maintained (Last commit was Dec 2018), and was created before ES6, so I started another project called JSlite. This time I didn’t want to be too heavy handed with the compression, the goal was to minify inline Javascript in a webpage on the fly, if I could just remove the unneeded whitespace, I figured that would be enough.

I started out with a big regular expression to do this, I wanted the code to be as fast as possible. But it soon became clear that this was not going to cut it. So I redeveloped it as a compiler.

Tokenise: Project Number 4

All 3 projects now used similar code layouts and a similar tokeniser, so I unified the tokeniser across the projects by moving the tokeniser into its own GitHub project. I also had to put all the projects on packagist and bring the projects together using composer.

Unit Tests, Bugs, and Production

I always wanted to make sure my code was reliable, therefore along the way I made sure to write unit tests for all these projects, it is not something I normally do on a regular basis, so it was a good learning curve and has proved itself incredibly useful as the codebases got more complex and features were added.

I had written a load of code, now I wanted to make sure it was production ready, my plan here was to download the code from a load of popular websites, minify them all, and log the processing time and compression achieved. So I wrote a little script to achieve this, which is included in the tests folder.

Running all these sites through my code, threw up loads of issues, which took me a while to work through, but each issue added more unit tests and discovered more bugs until I could minify the sites reliably, and see what the output compression / minify time was.

Torque: Project Number 5

Now I have some good code, I want it to be used. My first thought here was Wordpress, it is the most popular CMS on the Internet, writing a Wordpress plugin seemed like a good way to help my projects gain traction.

So I wrote a Wordpress plugin called Torque, using my minification software and a load of other security and performance options, the Wordpress sites I have tested it on show a good speed improvement in Lighthouse.

There are of course already lots of plugins that minify your Wordpress website, and during development I was intrigued to find out how they were minifying their code, and I discovered that under the hood of most of them, the same projects kept appearing. It made me want to test my software against them to see how it stacked up, and to improve my code.

Minify-Compare: Project Number 6

So I started another project to compare minifiers, the idea here was to expand on the code that downloaded all the website code to throw through my minifier, and compress the code with each of the minifiers I found, logging the time each one took, and what the compression ratio was.

It is this software that I will be using to pit all the different minifiers against each other.

Read The State of Minification in PHP: How 1 Project Grew into 6 (Part 2) to find out how they stacked up.