Forem: MylifeforAiur

React SEO: manage sitemaps

MylifeforAiur — Thu, 22 Dec 2022 09:25:00 +0000

background

This is another step of React SEO improvement series with Gatsby. Sitemap serves as report of site's content and help crawlers to grab the nuts and bolts of site.

According to google:

A sitemap is a file where you provide information about the pages, videos, and other files on your site, and the relationships between them.

Sitemap are specially helpful if:

Your site is large and have a big number of pages with low traffic
Your site is new and eagerly wants to get noticed
Your site is rich in media content like video, images etc

Problem

Web site needs sitemap automation tool to generate sitemap with your production build so it is maintenance free.
Some of the urls like generated by client side rendering are not recognised by sitemap tool. You need to attach it manually, meanwhile keep the automated part running.

Solution

Again, base on my guinea pig gatsby blog starter site, picked Gatsby sitemap plugin as a tool to generate sitemap on build. Simply install it: npm install gatsby-plugin-sitemap and add it to gatsby-config.js

It will check all pages and generate sitemap like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
  xmlns:news="http://www.google.com/schemas/sitemap-news/0.9"
  xmlns:xhtml="http://www.w3.org/1999/xhtml"
  xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
  xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
  <url>
    <loc>https://gatsbystarterblogsource.gatsbyjs.io/gatsby-starter-blog/hello-world/</loc>
    <changefreq>daily</changefreq>
    <priority>0.7</priority>
  </url>
  <url>
    <loc>https://gatsbystarterblogsource.gatsbyjs.io/gatsby-starter-blog/my-second-post/</loc>
    <changefreq>daily</changefreq>
    <priority>0.7</priority>
  </url>
</urlset>

⚠️ changefreq and priority will be ignored by google bots

That's the moving part, but how to append fixed urls not auto-generated? Answer is to use multi-location sitemap. This map will be uploaded to google search console of your site and set in robot.txt. Sample like:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://gatsbystarterblogsource.gatsbyjs.io/gatsby-starter-blog/sitemap-0.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://gatsbystarterblogsource.gatsbyjs.io/gatsby-starter-blog/sitemap-manual.xml</loc>
  </sitemap>
</sitemapindex>

With above, sitemap-0.xml is generated by sitemap plugin so it keeps the book of all gatsby site urls, and site-manual.xml is the extra you want to declare.

Last but not least, don't forget to submit the url in google search console:

Here is the teleport to (commit)[https://github.com/gatsbyjs/gatsby-starter-blog/commit/b0d1f7c7dd0b0362925c3998090f77889ca63eca] for devs too busy to read:

Champagne🍾 time now, with sitemap and robot.txt improvement, your site has a good start for SEO and you should be able to see some improvement of impressions and clicks in google search console.

Next topic will be tips on site metadata. SEO on!

ref links:

Gatsby SEO: Manage robot text file in different environments

MylifeforAiur — Wed, 14 Dec 2022 23:22:34 +0000

Background

When we are talking about SEO, the very first thing to handle is robot.txt that tells search engine crawlers which URLs they can access on your site. Within dev environment you want none of your pages on cook to be visible in google search results. And in prod you want the opposite. Robot txt is the gate keeper for so. Robot txt file has four directives:

User-agent - specify which bots like Googlebot, AdsBot-Google, bingbot, slurp
Disallow - specify files, directories to block bots
Allow - by default it is allow. This is to whitelist urls like you can disallow a directory and allow just some files in it.
Sitemap - reference the location of site maps. Let's leave it for another post for sitemaps.

Problem

Site starter/template doesn't have robot.txt or the file doesn't change with development/production environments. Even accidentally leaves disallow setting to production env.

site with mis-configured disallow can block search engine badly

Solution

Give an example for Gatsby starter site but principle applies to all starters:

install plugin/node modules to manage robot txt file - This is overkill in my opinion. Update static files according to different build stages is simple enough. Maintaining a library is always an overhead.
Save two versions of text files and use them with build stages. I vote this solution for the mere simplicity. List steps here:

Step 1. Change default file used by site to disallow. Sample is gatsby site so the default one is in static folder.

Step 2. Add robot-prod.txt to directory like SEO, only lists urls disallow here. Urls are relative and regular expression ready.

Step 3. Update your build file to copy it to root folder

Step 4. Verify search results change, notice it will work after search engine cache refresh. For Google it will take 24 hours.

For devs reference, here is the sample commit to make it working on my gatsby blog starter

https://github.com/gatsbyjs/gatsby-starter-blog/commit/075e61748c8e90eb09621ae6b812225d7607da07

Call out

Robot.txt must be placed under root folder of site.
Rules applies to relative urls only.
Disallow does not guarantee your pages will fly under the radar. They will still get indexed if other pages refer to them. Use the noindex robots meta tag or X-Robots-Tag HTTP header to completely block bots.

Hola, one commit and 3 steps to bring your robot text completely under control. 🐢 Fruit can't be lower, chuck the code in your repo now. Next post I will talk about site menu of crawl bots - sitemap.

Ref links

Gatsby site SEO action list

MylifeforAiur — Wed, 07 Dec 2022 05:07:04 +0000

Background

Search Engine Optimisation is critical for new start up business site to rank themselves up on search engine(or google) search result list. Even for well established business brand, it is still handy for their help/FAQ pages to be top listed on relevant search so people won't call support centre to increase operation cost. And usually, SEO work is more of development practice improvement. After process is set up, unlike accessibility or security features, those SEO improvements are easy to maintain. Development team can also set up a coding SEO guide wiki page or even custom eslint rules to share knowledge and maintain standard for new joiners.

Gatsby js, as a static site generating tool, is known for its friendly SEO, scalability, security and Content Management System (CMS) support through Multipage Page Application (MPA) architecture. Gatsby gradually gains popularity among react frameworks that usually build app into Single Page Application(SPA). For google bots, executing javascript to get html content is always second choice. More about javascript SEO

Problem

Gatsby js framework has extensive support for SEO. However serious set up is needed to ensure the SEO optimisation is baked in your site starter.

Action list

Based on Gatsby site, attached a chart with effort and impact. Gatsby's starter gives you plenty of options, but none of them are fully or even partially SEO optimised. Rated with impact and effort, actions are classified with three layers:

Golden(low effort high impact) 🥇 :
- Robot.txt control
- Page metadata
- Sitemap generation
- URL structure
- Navigation links
Silver 🥈 :
- SPA/MPA optimisation
- SEO Monitoring
- Site Moves
- Canonical urls
Bronze 🥉
- Site speed

This is an overview of the action list, I will write a serious of articles to elaborate tech details of items on the chart. I will also be using most starred Gatsby starter - forked blog starter to demo the SEO credits you can claim with Gatsby site. Happy SEO!

Forem: MylifeforAiur

React SEO: manage sitemaps

background

Problem

Solution

Next

ref links:

Gatsby SEO: Manage robot text file in different environments

Background

Problem

Solution

Call out

Next

Other articles of this series

Ref links

Gatsby site SEO action list

Background

Problem

Action list

Next

Ref links