Forem: Paul Calvano

What can the HTTP Archive tell us about Largest Contentful Paint?

Paul Calvano — Tue, 08 Jun 2021 03:59:31 +0000

Largest Contentful Paint (LCP) is an important metric that measures when the largest element in the browser’s viewport becomes visible. This could be an image, a background image, a poster image for a video, or even a block of text. The metric is measured with the Largest Contentful Paint API, which is supported in Chromium browsers. Optimizing for this metric is critical to end user experience, since it affects their ability to visualize your content.

Google has promoted this metric as one of the three "Core Web Vitals" that affect user experience on the web. It is also slated to become a search ranking signal over the next few weeks, which has created a lot of awareness about it. The suggested target for a good Largest Contentful Paint is less than 2.5 seconds for at least 75% of page loads.

Source: https://web.dev/lcp/

Some of the recent posts on WPOStats feature interesting case studies about this metric. For example,

Google's research found that when Core Web Vitals are met, users are 24% less likely to abandon a page before it finishes loading.
Vodafone improved LCP by 31% and saw an 8% increase in sales.
NDTV improved their LCP by 55% and saw a 50% reduction in bounce rate.
Tokopedia improved their LCP by 55% and saw a 23% increase in session duration.

Identifying the Largest Contentful Paint Element

The name of this metric implies that size is used as a proxy for importance. Because of this, you may be wondering specifically which image or text triggered it as well as the percentage of the viewport it consumed. There are a few ways to examine this:

One way to visualize the Largest Contentful Paint is to look at a WebPageTest filmstrip. You’ll be able to see when visual changes occurred (yellow outline) as well as when the Largest Contentful Paint event occurred (red outline).

In Chrome DevTools, you can also click on the LCP indicator in the “Performance” tab to examine the Largest Contentful Paint element in your browser. Using this method you can see and inspect the exact element (image, text, etc) that triggered it.

Lighthouse also has an audit that identifies the Largest Contentful Paint element. If you examine the screenshot below you’ll notice that there is a yellow box around the largest element, as well as an HTML snippet.

How Large is the Largest Contentful Paint?

The HTTP Archive runs Lighthouse audits for approximately 7.2 million websites every month. In the May 2021 dataset, Lighthouse was able to identify an LCP element in 97.35% of the tests. Since we have the ability to query all of these Lighthouse test results, we can analyze the result of the LCP audits and get more insight into what drives this metric across the web.

Using the same boundaries that Lighthouse uses to draw the rectangle around the LCP element, it’s possible to calculate the area of it. In the above example, the product of the LCP image’s height (191) and width (340) was 64,940 pixels. Since the Lighthouse test was run with an emulated Moto G4 user agent with a screen size of 640x360, we can also calculate that this particular LCP image took up 28% of the viewport.

The graph below shows the cumulative distribution of the LCP element as a percentage of screen size. The median LCP element takes up 31% of the screen size! At the 75th percentile the LCP element is nearly twice as large, taking up 59% of the screen size. Additionally 10.6% of sites actually had an LCP element that exceeded the viewport (which is why the y axis doesn’t reach 100%).

The graph below illustrates the same data in a histogram. From this we can see that 4.03% of sites (285,751) had a LCP element that took up 0 pixels. Upon further inspection, the 0 pixel elements appear to have been used in carousels, so by the time the audit completed the LCP element slid out of the viewport.

Node Paths of LCP Elements

Another interesting aspect of the Largest Contentful Paint audit is the nodePath of the element, which shows you where in the DOM this element was. In the example we looked at earlier, the nodePath was:

1,HTML,1,BODY,8,DIV,2,SECTION,1,DIV,0,DIV,0,DIV,0,UL,0,LI,0,ARTICLE,1,DIV,0,DIV,0,A,0,IMG

If we look at the last element in the node path, we can get some insight into the type of element that triggered the Largest Contentful Paint. The most common node that triggered the Largest Contentful Paint was <IMG>, which accounted for 42% of all sites. Next was <DIV> at 27% (which could include text or images). The <H1> through <H5> header elements accounted for 7.18% of all Largest Contentful Paints.

LCP Node (last element in path)	Number of Sites	Percent of Sites
IMG	3067354	42.12%
DIV	1981416	27.21%
P	766977	10.53%
H1	291091	4.00%
	192498	2.64%
SECTION	182267	2.50%
H2	144534	1.98%
A	107501	1.48%
SPAN	85245	1.17%
HEADER	67762	0.93%
LI	64212	0.88%
H3	60679	0.83%
RS-SBG	51623	0.71%
TD	48470	0.67%
H4	19039	0.26%
VIDEO	15649	0.21%
ARTICLE	12860	0.18%
FIGURE	9208	0.13%
BODY	8859	0.12%
image	8077	0.11%
CENTER	7960	0.11%

The <VIDEO> element only accounted for 0.21% of sites. According to the Web Almanac, the <video> element was used on 0.49% of mobile websites - so from this we can estimate that half of sites loading videos are triggering LCP with video poster images.

Image Weight for the LCP

One of the Lighthouse audits looks for opportunities to preload the Largest Contentful Paint element, and estimates the potential savings in performance. This audit also identifies the URL for the LCP element - which can give us some insights into what type of images are being loaded as a LCP element. In the HTTP Archive data, only 67% of the Lighthouse tests were able to identify a URL for an LCP element. Based on this, we can infer that text nodes are used for the LCP on approximately 33% of sites.

The graph below shows the distribution of sizes for the image element that was associated with the Largest Contentful Paint. The median LCP element size was 80KB. At the 90th percentile, the LCP element size was 512KB. If you have a large LCP image then you should consider optimizing it before you attempt to follow the Lighthouse preload recommendation.

Additionally, 70% of the LCP element images were JPEG and 25% were PNG. Only 3% of sites served a webp as their LCP element.

format	sites	% of Sites
jpg	3161991	69.37%
png	1122585	24.63%
webp	141441	3.10%
gif	84829	1.86%
svg	34123	0.75%
Other	13272	0.29%

When we look at the LCP element as a percentage of page weight, we can see that the median LCP element is 4.17% of the total page weight. At the higher percentiles, the LCP elements are larger and also a larger percentage of page weight.

percentile	ImageRequests	ImageKB	TotalKB	LCP as a % of Page Weight
p25	15	422	1,138	3.01%
p50	26	1,142	2,185	4.17%
p75	45	2,692	4,108	5.58%
p95	103	8,008	10,036	8.42%

Since images account for 52% of the median page weight (for the sites that have a LCP image element), we can infer that at the median 8% of page weight is used to render content to 31% of the screen.

How does this change based on Site Popularity?

The HTTP Archive now contains rank groupings, obtained from the Chrome User Experience Report. This can enable us to segment this analysis based on the popularity of sites. The rank grouping indicator buckets sites into the top 1K, 10K, 100K, 1 million and 10 million.

When we look at the Largest Contentful Paint image size based on popularity, it’s interesting to note that the most popular sites tend to be serving smaller images for the LCP element. While there may be numerous reasons for this, I suspect that the more popular sites are investing in image optimization solutions.

Page weight follows the same pattern, with the least popular websites having some of the largest page weights. If we look at the LCP element based on the percentage of page weight, you can see that within the top 100K sites the ratios are very close. In the less popular sites, the LCP element tends to be a much greater percentage of page weight.

Largest Contentful Paint Image as a Percentage of Page Weight
rank	p25	p50	p75	p95
Top 1k	1.61%	2.12%	2.85%	5.67%
Top 10k	1.76%	2.27%	3.00%	4.96%
Top 100k	2.07%	2.87%	3.77%	5.78%
Top 1 million	2.53%	3.49%	4.60%	6.95%
Top 10 million	3.11%	4.30%	5.75%	8.65%

We can also make some interesting observations about how popular sites are optimizing their LCP assets. Looking at the various image formats, JPG images are the most common LCP element. Some other formats such as PNG, WebP, GIF and SVG are used more frequently in the more popular sites.

Conclusion

Largest Contentful Paint is an important metric that helps illustrate when a page’s most significant content is rendered to the screen. In reviewing the HTTP Archive data, we can see that this area represents between 30% and 60% of a mobile viewport for a majority of sites.

There are a shocking number of sites that have a LCP element that consumes a large percentage of the viewport and are delivered as large unoptimized images. Site owners should evaluate both what is triggering the Largest Contentful Paint as well as how it is loaded. Optimizing for the Largest Contentful Paint will ensure that the browser has the opportunity to load and render this content as quickly as possible.

If you are interested in seeing some of the SQL queries and raw data used in this analysis, I’ve created a post with all the details in the HTTP Archive discussion forums. You can also see all the data used for these graphs in this Google Sheet.

Originally posted at https://paulcalvano.com/2021-06-07-lcp-httparchive/

Growth of the Web in 2020

Paul Calvano — Tue, 29 Sep 2020 19:28:48 +0000

For the past 10 years, the HTTP Archive has tracked the evolution of the web by archiving the technical details of desktop and mobile homepages. During its early years, the Alexa top million dataset (which was publicly available until 2017) was used to source the list of URLs included in the archive and the number of sites tracked increased from 16K to almost 500K as testing capacity increased. To keep the archive current and include new sites, towards the end of 2018 we started using the Chrome User Experience Report as a source of the URLs to track.

Throughout 2019 the size of the HTTP Archive dataset was mostly constant. However, the sample size has grown quite a bit in 2020 as you can see in the graph below! Additionally, if we combine both desktop and mobile URLs, there was a recent peak of 7.5 million sites!

How are sites Included in the Chrome User Experience Report

The Chrome User Experience Report (CrUX) is sourced from performance data collected from real Chrome users that have opted into syncing their browser history and sharing anonymized usage statistics reporting.It’s essentially real user measurement (RUM) data for Chrome users.

You can read more about CrUX on Google’s Developer website, as well as this informative blog post from Rick Viscomi. I’ve also written about it previously here.

While Google doesn’t publish a definitive list on what it takes to be included in the Chrome User Experience Report dataset, they have indicated that:

Origins are automatically curated based on real-user Chrome usage
Websites must meet a traffic threshold to be included.
Websites must be publicly accessible

Essentially, a website’s inclusion in the Chrome User Experience Report indicates that they’ve reached a certain threshold of activity. According to the CrUX changelog, there have been no changes in the methodology. So it can be inferred that analyzing the number of websites included in this dataset should provide some interesting insights into the month on month growth of the number of websites being visited by real users.

Note: The Chrome User Experience Report does not contain traffic details, and as such this analysis should not be interpreted as growth of traffic on the internet. This analysis is specifically about the growth in the number of websites that people are visiting.

Global Growth of Origins Accessed in 2020

The graph below illustrates the total number of websites that the Chrome User Experience reported across all form factors during the previous 12 months. There are a few interesting observations we can make:

In both 2019 and 2020 there were increases in the number of websites at the start of the year.
There was a linear increase in the number of websites through the first half of 2020.
The drop in March and April 2020 is interesting, since that coincides with the start of the global COVID-19 pandemic.

There is a similar pattern with the total number of registered domains. This indicates that much of the growth is new domains and not necessarily subdomains of existing domains.

When we look at the month over month rate of change, you can see that the max change in 2019 was +/- 5%. The number of sites tends to fluctuate month to month. For example, between August and December 2019 there was an 8.6% decrease in sites. However at the start of 2020 there was a 7.5% increase.

When comparing the number origins between December 2019 and August 2020, the total number of origins increased by 28.9% this year alone! That’s huge!

Mobile vs Desktop Growth

Looking at this by device type, we can see that there are consistently more mobile websites compared to desktop websites. And over the past year the fluctuations between them have been fairly consistent. The one exception is between May 2020 and June 2020, where desktop increased by 0.7% and mobile increased by 6.2%.

Overall, there are 22.9% more mobile websites in the CrUX dataset compared to desktop. We know from sources like statcounter that mobile usage has grown significantly over the years, and consistently surpasses desktop. But why are mobile users navigating to so many more websites compared to desktop users?

Is there something about the mobile experience (such as social media links, email marketing, etc) that increases the change a user may navigate to an unfamiliar website?

Or could it be growth in regions where mobile is more dominant?

How Has this Varied by Region?

At the start of 2020, most regions of the world saw an increase in the domain of sites. The exception to this was western Asia. The regions that had the most substantial increase at the start of the year were Northern Europe, North America and South America.

Between May and June there was another large uptick in the number of sites. This appeared to be mostly South-East Asian and Western European countries

The tables below detail the number of sites included in the CrUX dataset during December 2019 as well as January, May and June 2020. This first table contains the top 10 regions, most of which saw an increase of 15% to 25% during the previous six months!

	Number of Sites Included in CrUX dataset
Sub Region	Dec 2019	Jan 2020	May 2020	June 2020	August 2020
Northern America	1,257,159	1,406,284	1,676,120	1,681,454	1,730,260
Western Europe	713,202	768,644	874,164	908,560	943,891
Eastern Europe	640,145	694,632	821,024	913,037	926,202
Eastern Asia	720,926	740,767	871,854	882,322	901,008
South America	486,894	540,685	668,604	726,410	784,854
Southern Europe	506,054	541,526	661,416	710,543	724,323
Northern Europe	453,591	516,527	601,459	638,744	661,790
South-Eastern Asia	473,962	485,143	524,249	584,214	629,815
Southern Asia	403,325	419,118	441,328	462,282	500,600
Western Asia	274,339	273,610	327,425	351,186	362,340

	Percent Change of Sites Included in CrUX dataset
Sub Region	Dec 2019 - Jan 2020	May - Jun 2020	Jun - Aug 2020	Dec - Aug 2020
Northern America	10.60%	0.32%	2.82%	27.34%
Western Europe	7.21%	3.79%	3.74%	24.44%
Eastern Europe	7.84%	10.08%	1.42%	30.88%
Eastern Asia	2.68%	1.19%	2.07%	19.99%
South America	9.95%	7.96%	7.45%	37.96%
Southern Europe	6.55%	6.91%	1.90%	30.13%
Northern Europe	12.18%	5.84%	3.48%	31.46%
South-Eastern Asia	2.30%	10.26%	7.24%	24.75%
Southern Asia	3.77%	4.53%	7.65%	19.43%
Western Asia	-0.27%	6.77%	3.08%	24.29%

Looking at the next 10 in the list, we can see significant growth in Central America, Australia as well as West, Southern and South Africa. Overall the regions with the most growth during the 7 month period was Australia and New Zealand, South America, and Central America.

	Number of Sites Included in CrUX dataset
Sub Region	Dec 2019	Jan 2020	May 2020	June 2020	August 2020
Central America	155,057	179,295	236,255	242,132	257,043
Australia and New Zealand	124,763	141,523	194,212	196,841	214,757
Northern Africa	68,754	69,497	83,312	88,672	88,606
Southern Africa	50,618	59,139	64,978	66,392	70,218
Central Asia	45,932	49,192	57,098	57,508	62,112
Western Africa	44,692	49,868	47,834	51,257	50,853
Caribbean	33,840	37,445	44,090	45,910	45,395
Eastern Africa	31,010	34,822	36,073	37,388	38,609
Middle Africa	8,873	9,149	9,121	10,057	10,032
Melanesia	2,733	2,991	2,580	2,779	2,818

	Percent Change of Sites Included in CrUX dataset
Sub Region	Dec 2019 - Jan 2020	May - Jun 2020	Jun - Aug 2020	Dec - Aug 2020
Central America	13.52%	2.43%	5.80%	39.68%
Australia and New Zealand	11.84%	1.34%	8.34%	41.91%
Northern Africa	1.07%	6.04%	-0.07%	22.40%
Southern Africa	14.41%	2.13%	5.45%	27.91%
Central Asia	6.63%	0.71%	7.41%	26.05%
Western Africa	10.38%	6.68%	-0.79%	12.12%
Caribbean	9.63%	3.96%	-1.13%	25.45%
Eastern Africa	10.95%	3.52%	3.16%	19.68%
Middle Africa	3.02%	9.31%	-0.25%	11.55%
Melanesia	8.63%	7.16%	1.38%	3.02%

Many of the regions that had an increase in sites visited (based on CrUX data), also have a high percentage of mobile visitors compared to the global population (based on statcounter). So while it’s difficult to say for certain, it’s entirely possible that location is a large factor in the gap between Desktop and Mobile.

Analyzing by Top Level Domain

The .com top level domain accounts for 43.7% of all websites tracked in the Chrome User Experience report. The next largest top level domain is .org, which consists of 3.7% of all sites. Overall there were 4111 TLDs in the dataset, and the top 20 of them represented 75% of all websites.

Most of these top level domains experienced a > 20% growth in active websites since December 2019, with the exception of .info and .net. The domains with the largest percentage growth were co.uk, com.au and de.

If we look at the month to month growth trends for these TLDs, we can make a few interesting observations:

There was a significant drop across all TLDs in March 2020.
The largest percentage drop was for .it domains in March 2020, although that rebounded with increases in April, May and June.
In February 2020, there was a 23.9% increase in .edu domains receiving traffic.
In May 2020, more than a dozen popular TLDs saw a double-digit increase in the number of sites.
In August 2020 there was a 10.4% increase in edu domains

Conclusion

The web is constantly growing and evolving, and clearly it’s rate of growth can vary quite a bit. During this analysis we explored a public dataset that Google provides to show how the web has grown during 2020, and which regions are growing the most. While this doesn’t speak to the traffic levels experienced in these locations, the number of websites can be used as a proxy for understanding usage of the web. As this analysis shows, 2020 has been a year of substantial global growth for the web.

Originally posted at paulcalvano.com

Browser Back/Forward Caches and their Benefit to Web Performance

Paul Calvano — Mon, 03 Aug 2020 18:52:58 +0000

Overview

There are a few different types of top-level navigation events that browsers handle. The most common one is a simple “navigation”. You enter a URL in the address bar and hit enter, or you click on a hyperlink to visit another webpage. Another is a “reload”, which causes the browser to revalidate it’s cached entries. There’s also a “hard reload”, where the browser ignores cache and reloads the page. And another type of navigation, which we will discuss in this post, is “Back/Forward” navigation. These occur when you click the back or forward buttons in your browser to return to a previously visited page.

If you are using a real user measurement (RUM) service like mPulse then you are likely collecting the stats for every navigation type that results in a page load. The navigation timing API used by RUM services also includes the navigation type which makes it possible to determine which one was used. You can see the value in your browser console by checking the value of performance.navigation.type. The values defined by the specification are below:

Navigation Event	Type Attribute	Description
TYPE_NAVIGATE	0	Navigation started by clicking a link, entering a URL in the address bar, submitting a form, or triggered by a script (except the ones used by reload or back_forward)
TYPE_RELOAD	1	Navigation through the reload operation or the `location.reload()` method.
TYPE_BACK_FORWARD	2	Navigation through a history traversal operation.
TYPE_RESERVED	255	Any navigation types not defined above.

mPulse collects the navigation types for every real user measurement across thousands of websites. Analyzing this data can provide an interesting perspective on the performance of back/forward navigations as well as how their usage varies by device and browser.

Back/Forward Caches in Popular Browsers

The general idea behind a back/forward cache is to preserve the state of the page so that the DOM does not need to be rebuilt when a user returns to a previously visited page. With a back/forward cache the render tree and layout remains intact. The page is not technically loaded again because it was already loaded.

Chrome, Firefox and Safari all have implementations of a back/forward cache, although Chrome’s is currently hidden behind an experimental flag (chrome://flags/#back-forward-cache).

The Chrome team has announced its intention to enable the back/forward cache feature for Chrome Mobile users starting with Chrome 86 (October 2020). In this post we will examine how frequently back/forward navigations are used across different devices and browsers. We’ll also look at the performance opportunities that the back/forward cache implementation may be able to solve.

_Note: Because of how back/forward caches work, we do not have real user measurements for back/forward cache hits. In this analysis we’ll explore the opportunity presented by back/forward caches, and not the performance gained by any specific browser’s implementation. _

Summary of Navigation Types

The graph below illustrates the breakdown of different navigation types by device type. While the content of this graph is very high level and seems to be simple, it’s important to note that this is an aggregation of tens of billions of real user measurements from thousands of websites, measured during the month of June 2020.

There are a few interesting observations we can make from this graph:

Across all device types, ~85% of navigations are type=navigation.
There is a greater percentage of back/forward navigations on mobile and tablet devices compared to desktop
Desktop browsers had 60% more reloads compared to mobile/tablet.

Desktop Browsers

The most popular desktop browsers are Chrome, Safari, Edge and Firefox. In fact, combined they account for 91.4% of all Desktop pages!

_Note: When looking at this data it’s also important to note that in Firefox and Safari we only have the back/forward navigation events for navigations that did not result in a back/forward cache hit. _

When we look at this by desktop browser, we can see that

Chrome and Opera have a large % of reloads compared to other browsers
Almost 17% of Opera navigations are back/forward!
All browsers except for Safari Desktop and legacy IE have at least 10% of back/forward navigations.
There were 50% less back/forward navigations measured for Safari compared to Chrome. This seems to indicate that approximately half the time users are getting a cache hit from this feature.
Firefox had a similar rate of back/forward navigations to Chrome. This seems to indicate that either Back/Forward navigation are much more prevalent in Firefox, or that their implementation is not resulting in many cache hits.

The graph below illustrates the median load times by navigation types for these browsers. For each browser, the reload events are significantly slower. On Chrome, reloads took 30% longer than navigations. Firefox saw the biggest degradations with reload events, taking almost twice as long as navigations.

Comparing the navigation and back/forward load times, we can see that legacy IE and Opera back/forward load times were only slightly faster than a navigation. With Chrome, the back/forward navigations are 16% faster than navigations. Safari back/forward navigations are 27% faster. That’s without any back/forward cache, but merely relying on the efficiency of the browser cache.

Mobile Browsers

The most popular mobile browsers are Mobile Safari and Chrome Mobile, which together accounts for 67.6% of mobile page views. The graph below also includes other popular mobile browsers such as Samsung Internet, UC Browser, Crosswalk, and Firefox Mobile. Additionally included are webviews and in-app browsers are included - such as Facebook, Instagram, LINE, Flipboard, Snapchat, Pinterest, etc.

Some observations:

Back/Forward usage is much more prominent in mobile browsers compared to desktop browsers.
Mobile browsers have a much higher back/forward usage compared to WebViews and in-app browsers.
Considering the difference between Chrome Mobile iOS vs Mobile Safari (both Wekbit based), as well as Chrome Mobile vs Samsung Internet vs Crosswalk (all Chromium based), the amount of back/forward navigation events may be influenced by the browser UI.

Is this a Blind Spot for Synthetic?

When measuring the performance of a synthetic transaction, you are usually starting a new session - which means that the page loads will be navigations. Since most synthetic services identify themselves in their User-Agent, we’re able to compare navigation types across popular synthetic solutions using mPulse data. In the graph below you can see that the vast majority of requests from synthetic services are navigation events. There are a small percentage of reloads, likely due to multi-step scripts that repeat a page view. The only synthetic measurement service that I was able to see back/forward navigations from was PingdomBot.

Regardless of what we see today, on the measurement platform you use it may be possible to script these types of navigations. For example, in WebPageTest I can trigger a back/forward navigation by using a script such the one below:

// Turn off logging of results
logData 0   
// Navigate to the first page.  Then navigate to another page.
navigate    https://www.google.com/
navigate    https://www.google.com/search?q=back+forward+cache
// Wait for 2 seconds
sleep   2 
// Turn on logging of results
logData 1 
// Use JavaScript to trigger the back button
execAndWait window.history.back();

What does this mean for RUM?

Many RUM implementations, such as mPulse, will aggregate and report on all user navigations. So the results that you are looking at will include a mix of navigation, reload, back/forward and reserved navigation. Based on the data here, ~ 10% of Desktop and ~20% of Mobile page loads utilize Back/Forward navigation.

If a navigation is served from the Back/Forward cache, then it will not trigger a beacon since the browser is unfreezing the browser state. In this case there will be no load event to trigger a beacon.

We may look at ways to support timing back/forward navigation from cache in the future - but for now, I would expect the following:

Once Back/Forward cache rolls out for a browser, you might see a drop in page loads from that browser (since those navigation types will no longer be reported)
Additionally, load times may appear to be slightly higher because some of the faster experiences (a back/forward navigation utilizing the browser cache) will no longer be reported. Essentially, these measurements will be affected by survivorship bias.

I would expect this to apply to other analytics tools as well.

Conclusion

While we often think of navigating the web as the simple operation of loading a web page, there are different types of navigation events which can be measured. Reloads are slower, and many of us are accustomed to that in normal use. However we expect the Back/Forward operations to be fast - and they are faster. The implementations of Back/Forward caches in popular browsers are helping to improve this experience even further - which has the benefit of significantly speeding up the web for up to 20% of navigations!

Originally posted at paulcalvano.com

Many thanks to Nic Jansma and Utkarsh Goel for reviewing this.

An Analysis of Cookie Sizes on the Web

Paul Calvano — Mon, 13 Jul 2020 14:28:47 +0000

Cookies are used on a lot of websites - 83.9% of the 5.7 million home pages tracked in the HTTP Archive to be specific. They are essentially a name/value pair set by a server and stored in a client’s browser. Sites can store these cookies by using the Set-Cookie HTTP response header, or via JavaScript (document.cookie). On subsequent requests, these cookies are sent to the server in a Cookie HTTP request header.

In this article we’ll be looking at the size of cookies across the web, and discuss some of the web performance implications of them.

First vs Third Party cookies?

When a cookie is set by the domain you see in your address bar, it is considered a first party cookie. These can be used for session management, authentication, personalization, etc. When a cookie is set by a different domain, then it’s considered a third party cookie.

Based on an analysis of over 109 million cookies, third parties account for 79% of all cookies.

Note: I wrote another blog post exploring the use of the SameSite attribute in cookie files, and how third party cookies are affected. You can read it here.

Cookie Sizes

An example of a cookie set by a server would be:

Set-Cookie: loggedIn=true; Domain=.example.com; Path=/; Max-Age=14400; HttpOnly

Directives such as Domain, Path, max-age, and HttpOnly affect how the cookie is stored, and which hostname a browser should share them with. In this example, loggedIn=true is the name/value portion of the cookie, and that is what we’ll be exploring in this post.

The median length of all cookies in the HTTP Archive is 36 bytes as of June 2020. That statistic is consistent across both first and third party cookies. The minimum is just a single byte, usually set by empty Set-Cookie headers (which is likely an error).

Cookie Size	First Party	Third Party
Min	1	1
Median	36	37
95th Percentile	181	135
99th Percentile	287	248
Max	29,735	8,500

The largest cookie size was 29,735 bytes, which is quite large! This is so large in fact that it is rejected by all modern browsers. I was curious to see what the limits are, and decided to dig into the source. Both Chrome and Firefox will reject cookies greater than 4KB. This is likely due to the implementation limits defined in RFC 6265.

So who is setting these large cookies?

The largest first party cookie was set by https://www.ridewill.it/, and it is named menu. It’s value was a long urlencoded string that contained multiple <div> layers and links. All in all, there were 240 links in a single cookie!
Many large first party cookies included large session cookies.
The largest third party cookie was set by web.taggbox.com, and consisted of a large JSON array named “liveWall”.
Most of the largest third party cookies were set from web.taggbox.com as well as a small number of advertising third parties.

If we look at the entire distribution of cookie sizes, it gets even more interesting. 88% of the cookies being set are less than 100 bytes. The 99th percentile is 372 bytes. So really large individual cookies are not common.

Cookies Sent to First Party Domain

What about cookies that clients send back to servers? A client can send multiple cookies in a single Cookie request header. Since the HTTP Archive only collects information on homepages, there is a limit to the insight we can collect here. If we look at just the request headers for favicon.ico, we can get an idea of how large the Cookie request header might be for a subsequent request. However this does not include any additional cookies set later in a session (ie, such as after logging in)

The median size of cookies sent on the favicon request was 161 bytes and the 95th percentile was 681. The largest was 7,795 bytes, and you can see the distribution below.

It’s important to note that the cookies set by the time favicon was requested may under represent the size of the cookies users would send later in a browsing session. For example, when logging into an application, a few additional cookies might be set. Some third parties that use a first party subdomain (ie, if www.example.com loaded a resource from metrics.example.com) also set a first party cookie.

Performance Implications of Large Cookies

When a browser sends an HTTP request, the HTTP request headers are usually 400 - 500 bytes. In the example below the request headers total 407 bytes.

GET / HTTP/1.1
Host: www.example.com
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

Adding a cookie to that will increase the size of the request header. If we add more than 1KB of cookies to that request, then we exceed 1500 bytes, which is the standard maximum transmission unit (MTU) used by TCP. This means that the HTTP request would span multiple TCP packets, which may result in multiple round trips and increase the risk of retransmission. This can potentially increase the TTFB of the response since it would take longer to make the request.

With HTTP/2, we have HPACK compression - which was designed to help reduce the size of HTTP request headers by utilizing a dynamic index table. Receiving endpoints advertise the maximum size of this table in bytes (default is 4096), the sender can insert headers up to this limit in one request or response and subsequently reference it in another. In theory HPACK seems like it could help reduce the overhead of these large cookies. However, it’s not as easy as it sounds. Consider the following:

If a request is made on a new HTTP/2 connection (ie, for a return visitor, or a navigation that happened after the previous connection times out), then the entire cookie string would be sent. This would impact TTFB for that first request (usually the HTML).
Some servers intentionally exclude cookies from HPACK compression, since cookies are a very valuable target for an attacker.

Because of the practical constraints on compressing cookies, and the impact of cookie size on the first flight of requests and responses, it is beneficial to use smaller cookies. Based on the observations here, 900 bytes seems like a good budget for a total cookie size, which leaves room for other headers such as user-agent (which can benefit from HPACK compression).

Conclusion

Cookies are everywhere, and they are set by both first and third party requests. Most browsers will limit the size of cookies stored to 4KB - which is still quite large.

At a minimum, setting a cookie larger than 4KB will simply not work. However, 4KB is still too large and can negatively impact your TTFB by increasing the number of round trips on the HTTP response.

Even moreso, it’s important to keep track of how many cookies are being sent, as bloated cookies can also impact your TTFB by increasing the time it takes to make an HTTP request.

Many thanks to Lucas Pardue and Matt Ringel for reviewing this.

Investigating Duplicate HTML Requests on a Page Load

Paul Calvano — Fri, 10 Jul 2020 21:44:30 +0000

Why it occurs, and what is the impact on web performance?

Overview
Ever see a web page load an embedded object that is really the same HTML as the main document? I’ve seen this show up on a handful of websites recently - including a few large ecommerce sites, a news site, a financial services site and a utility company’s website. Every time I've run into this, I've said I need to write a blog post about it. So, here's that post!

Two of the most recent ones I’ve seen made the same mistake. Can you spot the error?

The body was styled with the CSS background property, which is shorthand for background-color, background-url, and others (this is explained better here). In this example, the developer probably wanted to use background: none to give a blank background color, but used an empty URL instead. The url() function attempts to load a background image, and it takes a string as an argument. Since an empty string is a valid string this doesn’t trigger an error in the console. It simply results in a relative request to the base URL.

I created a simple test page to demonstrate this. In the WebPageTest waterfall below you can clearly see the extra page request.

In Chrome DevTools, you would see the duplicate request show up with an initiator of the main document. In the console there is no indication or an error. This may not be a technical error, but it certainly is unintentional and comes with a performance penalty.

In this article we are going to explore why this is a performance issue, how duplicate requests for HTML can be inadvertently triggered, and how you can avoid them.

What is the Performance Impact?

For many sites, HTML pages are dynamically generated - which means that there is a backend cost associated with delivering the content. We often measure this by using the Time to First Byte metric, which tells us the time from when a request was made until the browser starts to receive a response. Often these pages are not cacheable either - so each request gets handled dynamically.

The overall capacity of the application backend is usually driven by the its ability to process the workload volume it receives. And if you are accidentally sending twice the number of requests for a dynamically generated HTML page, your infrastructure could be working twice as hard for nothing.

For example:

If the backend is struggling to keep up with the load, then all pages will be affected. This has the potential to degrade time-to-first-byte for all pages.
There are costs involved with scaling out capacity at the origin. Unnecessary HTML requests can skew capacity planning in this case.

Increased page weight is an additional reason. Many of the examples below were used to generate placeholders or empty content. In these cases, the additional HTML could be unnecessarily increasing page weight.

What Causes This?

Here are a few reasons I’ve seen for duplicate HTML requests

<body style="background: url('')">

This is the example I started off this post with. When you style an element with a background URL using url(), any string is considered a value argument. Since url(‘ ‘) is passed a valid string, it gets translated to a relative URL. This triggers a duplicate request in Chrome as well as Firefox. Safari ignores the empty url string

This also applies to other elements that can be styled with a background URL. For example, <div style="background: url()">

And it also happens with url(), url('') and url(' ')!

Element src’s with ? or #
For some elements, a src attribute of either “?” or “#” would get interpreted that as a blank relative URL. For example <img src=”#” /> looks like an empty placeholder image, but it will cause the base HTML to get fetched again.

The same applies for <script>, <iframe>, and <video> - although browsers vary a bit. For example:

src=” “ returns a duplicate request for Firefox, but not the other browsers.
<script src=”#” /> also affects only Firefox
<script src=”?” /> affects only Chrome browsers
Chrome does not send duplicate requests for relative video src attributes, but Firefox and Safari do.

The table below lists the full summary of the tests I ran across Chrome, Safari and Firefox to see how they handle these errors. A warning icon indicates that they result in multiple HTML requests -

	Chrome	Safari.	Firefox
`<img src="#" />`
`<img src=" " />`
`<img src="?" />`
`<script src="#" />`
`<script src=" " />`
`<script src="?" />`
`<iframe src="#" />`
`<iframe src=" " />`
`<iframe src="?" />`
`<video src="#" />`
`<video src=" " />`
`<video src="?" />`
`<body style="background: url()">`
`<body style="background: url('')">`
`<body style="background: url(' ')">`
`<div style="background: url()">`
`<div style="background: url('')">`
`<div style="background: url(' ')">`

* iFrames with src=”?” and “#” resulted in 3-4 HTML requests for the same page.

Service worker Cache
Creating a separate cache layer via a service worker is a great way to preload your browser cache for resources that it will need later. However, what happens when you include an HTML page as one of those resources to fetch?

Here's an example of a site doing exactly that. Even worse, the HTML was not cacheable...

If you are going to use a service worker to cache a resource, make sure that resource is cacheable!

display:none
While this example is not specifically about HTML, it occurs enough that it's worth mentioning. As @dougsillars says "display:none does not mean download:none".

// Detect dark theme var iframe = document.getElementById('tweet-1261888005364756480-648'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1261888005364756480&theme=dark" }

In this example, the developer referenced video content that used a media query to hide the video for certain screen sizes. The end result was downloading both the desktop and mobile videos, while hiding one of them. That's a lot of wasted bytes!

Why do I bring this up in an article about duplicate HTML requests? I've seen some cases of this technique used alongside <img src="#" /> - resulting in duplicate HTML!

How to Detect?
There aren't any tools or audits I know of that look for this, as it does tend to be a bit of a niche error.

Probably the easiest way to spot this is by looking at the CPU processing overlay in WebPageTest. If you see JS execution for a request before the request is loaded, then it was likely a duplicate.

In DevTools, you can also filter by your domain name using domain:www.example.com. Filtering and sorting by name, makes these easier to spot. If you see the same URL with a type of both document and text/html, then you are loading a duplicate HTML page.

Clicking on the initiator can help find what caused the duplicate HTML to load.

Conclusion
If you are analyzing a page and see what looks like a duplicate request for the base HTML page, don’t ignore it! This could be a serious performance and capacity issue. Instead use DevTools to attempt to locate what initiated that request. You may find that you have accidentally included a placeholder or set an incorrect CSS property or element src attribute.

SameSite Cookies - Are you Ready?

Paul Calvano — Tue, 07 Jul 2020 20:43:39 +0000

Last year Google announced updates to Chrome that provide a way for developers to control how cross site cookies should work on their sites. This is a good change - as it ultimately improves end user security and privacy by limiting which third parties can read cookies that were set while visiting a different site. It also defeats cross site request forgery attacks. The implementation is fairly simple, and only requires developers to add the SameSite attribute to their cookies.

The SameSite attribute is supported by all modern browsers, and most have historically defaulted to a permissive use of cookies if the attribute isn’t present.

Google changed the default behavior of SameSite attribute to secure cookies by default when Chrome 80 was released in February 2020. However it was rolled back in April 2020 to ensure stability during the initial stage of the COVID-19 response. Now they are planning to resume SameSite cookie enforcement with Chrome 84, which will be released on July 14th.

Despite almost a year of notice and warnings in the browser console, this seemed to catch many by surprise in February. How ready are third parties for this now?

What is a Cross Site Third Party Cookie?

If a request on www.example.com sets the following cookie from its domain, then the browser will store this cookie and send it back on subsequent requests to the same domain. This is an example of a first party cookie. Essentially a cookie whose domain matches the domain that appears in the address bar...

Set-Cookie: session=abc; path=/; Secure; HttpOnly;

Now let’s assume that www.example.com includes a third party analytics provider, metrics.analyticsexample.com. When the third party request is made, that third party can also set a cookie in the end users browser. And that third party will be able to read the cookie. This is an example of a third party cookie.

If that same user then navigated to www.example2.com, which uses the same third party analytics provider, then their third party cookies would be readable by them across both sites. The third party is then able to track the user across multiple websites.

SameSite Cookies

The SameSite cookie attribute was introduced in a 2016 IETF draft, but had not been widely adopted initially. This attribute provided developers with the ability to control when a browser would send a cookie to a third party. Using them is simply a matter of adding the SameSite attribute to a cookie declaration, with one of the three supported values: “None”, “Lax”, and “Strict”.

This provides the following controls:

SameSite=None
- The browser will send cookies with both cross-site requests and same-site requests.
SameSite=Lax
- Same-site cookies are withheld on cross-site sub-requests, such as calls to load images or frames, but will be sent when a user navigates to the URL from an external site; for example, by following a link.
SameSite=Strict
- The browser will only send cookies for same-site requests (requests originating from the site that set the cookie). If the request originated from a different URL than the URL of the current location, none of the cookies tagged with the Strict attribute will be included.

An example of how this is configured is:

Set-Cookie: key=value; SameSite=Strict

If the SameSite attribute is not included, then most browsers have historically defaulted to the most permissive behavior: SameSite=None.

Google Chrome’s Update

Google has been planning to update the behavior of SameSite within the Chrome browser to default to the more secure SameSite=Lax. Additionally, if a SameSite=None attribute is present, then they would require that the cookie have the “Secure” attribute. There was some concern that this change would cause breakage for some third parties, so a warning message was included in Chrome since version 77 (September 2019).

A cookie associated with a cross-site resource at was set without the SameSite attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with SameSite=None and Secure. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.

SameSite Usage Across the Web

The HTTP Archive stores a tremendous amount of detail for every HTTP request and response for approximately 5.8 million homepages. In the June 2020 data, there were approximately 108 million third party cookies set across 3.79 million homepages. Of these cookies 35,721,768 (32.9%) included the SameSite attribute. Comparatively in August 2019, 21.4% of cookies had the SameSite attribute.

Note: Due to a collection issue described here, ~18.6% of third party cookies were unreadable in the June 2020 HTTP Archive data. The remainder of this analysis is on the cookies we could read.

The “Secure” flag of a cookie ensures that the browser only sends the cookie over HTTPS. Chrome made this a requirement to use SameSite=None. Out of the 35 million cookies, nearly 75% of them use the Secure flag.

When we look at this graphically, there are a few interesting observations we can make:

SameSite=Lax, which will be the new default, is in use by only 10.82% of secure cookies, but 97% of insecure cookies.
SameSite=None is present on 89.10% of Secure cookies.
2.65% (238,810) insecure cookies are set with SameSite=None, but not including the Secure flag. These will default to SameSite=Lax.
Only 0.06% (16,000) of secure cookies are using SameSite=Strict!
There are less than 1000 SameSite attributes set to an erroneous value (ie, not Lax, Strict or None).

Who is using SameSite=None incorrectly?

There were 238,810 third party cookies set with SameSite=None, but missing the Secure flag. These will default to SameSite=Lax in Chrome 84 unless the Secure flag is added. Overall, there were 1749 third party domains that made this error. The top 5 account for 48% of the erroneous SameSite cookies. This includes Spotxchange, ETargetNet, SmartAdServer, BazaarVoice and EntityTag.

Even more concerning than the number of cookies, is the number of websites that are affected. For example spotxchange.com is setting SameSite=None with insecure cookies on 26,174 websites. EntityTag.co.uk is doing the same for 14,358 websites.

Who is using SameSite Strict?

I thought it was odd that SameSite=Strict was used so infrequently. The table below shows some of the third parties that are using it. The top 10 account for 67% of all SameSite=Strict usage.

What Erroneous SameSite Attribute Values are in Use?

I was surprised to see that there was such a low percentage of erroneous usage of the SameSite attribute. The top 10 errors listed below account for 80% of the erroneous uses. Most were SameSite=Secure, SameSite with no value, and SameSite: Lax.

Many of these errors were from a small number of third parties. For example:

SameSite=secure was set by wildbeardbg.com:
- dg_perm_sessid=95951591788161; secure; SameSite=Secure
SameSite: Lax was set by nytimes.com.
- nyt-purr=cfshcfhssc; Expires=Fri, 18 Jun 2021 01:02:51 GMT; Path=/; Domain=.nytimes.com;SameSite: Lax;Secure
SameSite=; was set by cloudengage.com
- CEUID=E80y%2BgcVMl0o6Skdol5X9FRCrdrbqBNssQeOYrNUnQxC42U4gpGNV6VX; expires=Wed, 01-Jul-2020 16:16:13 GMT; Max-Age=2592000; path=/; samesite=; domain=.cloudengage.com; secure; HttpOnly

SameSite Usage Across Popular Third Parties

So far we’ve looked at how SameSite is being used across third parties. But which third parties are not setting SameSite attributes? By not setting it, they will default to SameSite=Lax. Whether or not this is intentional or will cause breakage will depend on each third party's usage of the cookies. But not setting an explicit SameSite attribute could be indicative of whether a third party has prepared for this.

The graph below shows SameSite usage across some of the most popular third parties. Very few sites third parties are setting SameSite=Lax, which is about to become the new default. Some of these third parties are used by hundreds of thousands of sites.

Conclusion

SameSite cookies are a huge win for privacy and security, but there is a risk that Chrome’s new default settings will cause problems. For many cases, this will likely render some cross site tracking techniques ineffective with little change to end user experience. However, with any changes like this there are risks of breakage and serious site issues. With SameSite adoption at less than 33% of third party cookies, this raises the question of how prepared third parties are for this.

It’s important to note that the absence of a SameSite attribute does not necessarily mean that there will be breakage. However depending on how the cookie is used, it has the potential of becoming problematic. It may be worth checking the JS console in Chrome DevTools to see if you see the SameSite warning for any of your third parties. You can also set a flag in Chrome to test how this will affect your site ahead of the Chrome 84 release. The Chrome team has published a useful guide for debugging SameSite cookies.

Certificate Validity Dates

Paul Calvano — Thu, 20 Feb 2020 20:55:52 +0000

Back in 2017 the maximum validity lifetime for an HTTPS certificate was set to 825 days, a decision that was widely supported by both browsers and certificate authorities. However, since then there have been multiple unsuccessful attempts at reducing the maximum lifetime to one year. Scott Helme has written about this previously, and his blog post noted that browser vendors unanimously supported this while some certificate authorities objected to it.

Information is still trickling in, but it seems that Safari is planning to enforce a max validity lifetime of 398 days effective September 1st, 2020.

Dean Coclin

@chosensecurity

Today's big news: One year max public TLS certs are coming, starting 1 Sept 2020, if you want to be trusted in Safari.

22:10 PM - 19 Feb 2020

103 149

Let’s take a look at the state of certificate validity ranges today, so we can track how this evolves over the next few months. The data for this comes from the HTTP Archive, which is an open source project that tracks how the web is built.

Certificate Validity Dates in the Wild

The HTTP Archive requests table contains certificate details for every HTTPS request that was served to the 5.3 million websites tracked. The details are in the $._securityDetails and the data contains information on 4,397,690 unique hosts. Out of all of these, 136,007 hostnames served a validity date greater than 825 days! That’s 3.09% of all requests!

Certificate validity dates are widely distributed, but there seems to be a few popular ranges. THe most common validity date is 90 days, likely to the popularity of LetsEncrypt Overall 55% of certificates have a validity date of less than 364 days, which I’ve highlighted in green below. An additional 20% of certificates have a validity between 365 and 398 days, which will meet Safari’s requirements. The remaining 25% have a validity range of more than 398 days.

Looking at this by certificate authority is also quite interesting. The graph below shows the top 15 certificate authorities. LetsEncrypt accounts for 38.4% of all certificates -

When we look at certificate validity date ranges for these certificate authorities, we can see that there’s a mix. Certificates issued by LetsEncrypt, Cloudflare, cPanel and Amazon meet the 398 day requirement already. However, Sectigo, GoDaddy, DigiCert, Comodo and RapidSSL have a very large percentage of certificates that exceed 398 days.

DigiCert released a public statement yesterday, which confirms that existing certificates with a validity range >398 days will continue to be trusted by Safari, but that certificates issued after August 30th won’t be able to exceed 398 days.

For your website to be trusted by Safari, you will no longer be able to issue publicly trusted TLS certificates with validities longer than 398 days after Aug. 30, 2020. Any certificates issued before Sept. 1, 2020 will still be valid, regardless of the validity period (up to 825 days). Certificates that are not publicly trusted can still be recognized, up to a maximum validity of 825 days.

I’m interested in seeing how this will evolve in the coming months, especially once there are more formal announcements about this. If you would like to see the queries used for this analysis, I've detailed them in this HTTP Archive discussion forum post.

Many thanks to Scott Helme for reviewing this.