Forem: Sanchit Dikshit

The Two UNDERRATED Files Every Website Needs

Sanchit Dikshit — Wed, 15 Apr 2026 08:42:17 +0000

How robots.txt and sitemap.xml quietly control your site's relationship with the internet.

While going down an SEO rabbit hole one evening, I came across something I had completely overlooked — two small, plain-text files that sit quietly at the root of nearly every website on the internet. Old or new. Big or small. Built on WordPress or hand-written in vanilla HTML.

robots.txt and sitemap.xml.

The story of these files goes back further than most people realize. In the early days of the web — mid-1990s — search engine bots had no rules. They crawled everything, followed every link, hammered servers, and indexed pages that site owners never intended to be public. It was chaos. So in 1994, a developer named Martijn Koster proposed a simple, voluntary agreement: websites would place a plain text file at their root, and crawlers would read it before doing anything. That file was robots.txt — and it became an informal standard almost overnight.

sitemap.xml came later, in the early 2000s, when the web exploded in complexity. Crawlers were getting better, but they were still missing deeply nested pages and dynamically generated URLs. Google introduced the sitemap format to let site owners tell crawlers exactly what existed — rather than waiting for them to discover it.

Two files. Decades of web history. And we might set them up wrong, or skip them entirely.

This post is everything I wish I had found in one place.

First, a Mental Model

Think of your website as an office building.

robots.txt is the security guard at the front desk — it tells visiting bots (Googlebot, Bingbot, GPTBot, you name it) what they're allowed to access and what's off-limits.

sitemap.xml is the floor directory on the wall — it tells those same bots exactly what rooms (pages) exist, where they are, and how important each one is.

Neither file is magic. But without them, you're expecting search engines to figure out a maze blindfolded.

`robots.txt` — The Bouncer Your Site Deserves

What it is

A plain text file that lives at the root of your domain: https://yourdomain.com/robots.txt

It follows a simple format:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Breaking it down

Directive	What it does
`User-agent: *`	Applies the rules to ALL bots
`User-agent: Googlebot`	Applies rules only to Google's crawler
`Disallow: /path/`	Tells bots: stay out of here
`Allow: /path/`	Explicitly permits access (useful to override a broader Disallow)
`Sitemap:`	Points bots directly to your sitemap — gold.

Real-world patterns

Block everything (staging/dev environments):

User-agent: *
Disallow: /

Use this on staging.yoursite.com. You do NOT want Google indexing your half-finished pages.

Block AI scrapers specifically:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

Yes, AI companies have their own crawlers. Yes, you can block them. Whether you should is your call.

Protect the back-end, expose the front:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /cart/
Disallow: /checkout/
Allow: /

Classic WordPress setup. No reason for Google to index your login page.

The golden rule people forget

robots.txt is a request, not a lock.

Ethical crawlers (Google, Bing, etc.) will respect it. Malicious scrapers won't. If you have genuinely sensitive data, don't rely on robots.txt — use authentication.

Also: listing a URL in Disallow does NOT stop Google from knowing the URL exists if another page links to it. It just stops Googlebot from crawling it. Subtly different.

`sitemap.xml` — Your Site's CV for Search Engines

What it is

An XML file that lists every page you want indexed, along with optional metadata: when it was last modified, how often it changes, and its priority relative to other pages.

Bare minimum sitemap:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yourdomain.com/</loc>
  </url>
  <url>
    <loc>https://yourdomain.com/about/</loc>
  </url>
  <url>
    <loc>https://yourdomain.com/blog/my-first-post/</loc>
  </url>
</urlset>

Full-featured entry:

<url>
  <loc>https://yourdomain.com/blog/robots-and-sitemaps/</loc>
  <lastmod>2026-04-15</lastmod>
  <changefreq>monthly</changefreq>
  <priority>0.8</priority>
</url>

The optional fields, honestly rated

Field	Honest usefulness
`<lastmod>`	✅ Actually useful. Tells Google when you updated content. Use it.
`<changefreq>`	⚠️ Mostly ignored by Google in practice. Include it anyway for other crawlers.
`<priority>`	⚠️ Google largely ignores it too. But it's a signal, however faint.

Sitemap index files

Got a large site? You can't cram 50,000 URLs into one file (50,000 is actually the limit per sitemap). Use a sitemap index:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://yourdomain.com/sitemap-posts.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://yourdomain.com/sitemap-pages.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://yourdomain.com/sitemap-products.xml</loc>
  </sitemap>
</sitemapindex>

Point Google Search Console to the index file — it handles the rest.

Specialized sitemaps

Don't sleep on these:

Image sitemaps — Helps Google index your images in search
Video sitemaps — Required for rich video results
News sitemaps — For publishers wanting Google News inclusion

The Connection Between the Two

Here's the part most tutorials skip:

Your robots.txt should reference your sitemap.

User-agent: *
Disallow: /admin/

Sitemap: https://yourdomain.com/sitemap.xml

That last line? It means any crawler that reads your robots.txt — even if they weren't specifically looking for a sitemap — now knows exactly where to find all your content. It's a two-for-one.

And the inverse: don't list URLs in your sitemap that you've disallowed in robots.txt. You're sending conflicting signals — "here's a page I want indexed" + "don't crawl this page." Google will be confused. Confused Google = bad rankings.

Framework Quickstart

Most frameworks handle this with plugins or built-in support:

Framework / Tool	Sitemap	robots.txt
Next.js 13+	`app/sitemap.ts` (built-in)	`app/robots.ts` (built-in)
Nuxt 3	`@nuxtjs/sitemap` module	`nuxt-simple-robots` module
Astro	`@astrojs/sitemap` integration	Manual or `astro-robots-txt`
WordPress	Yoast SEO / Rank Math	Yoast SEO / Rank Math
Hugo	Built-in	Manual in `static/robots.txt`
Gatsby	`gatsby-plugin-sitemap`	`gatsby-plugin-robots-txt`

If you're rolling a custom backend, generate the sitemap dynamically from your database/CMS and serve it at /sitemap.xml. Most frameworks make this straightforward.

Submitting to Search Console (Don't Skip This)

Waiting for Google to find your sitemap organically can take weeks. Do this instead:

Go to Google Search Console
Select your property
Left sidebar → Sitemaps
Enter sitemap.xml → Submit

You'll see index status, errors, and how many pages Google's actually processed. Check it once a month. Fix errors promptly.

Common Mistakes (I've Made Most of These)

🚫 Blocking CSS/JS files in robots.txt — Google needs to render your pages. If you block your stylesheets, Googlebot sees a broken page.

🚫 Including noindex pages in your sitemap — If a page has <meta name="robots" content="noindex">, don't put it in the sitemap. Contradictory signals.

🚫 Hardcoding www vs non-www inconsistently — Your sitemap URLs should match your canonical domain. All https://domain.com or all https://www.domain.com. Not a mix.

🚫 Forgetting to update the sitemap — If you add new pages but never regenerate the sitemap, those pages wait in the dark. Automate this.

🚫 Using relative URLs in sitemap — Every <loc> must be an absolute URL with the full scheme: https://yourdomain.com/page/ not /page/.

Things Most People Don't Know (The Interesting Bits)

This is the part I find genuinely fascinating — the stuff that doesn't show up in beginner SEO tutorials.

robots.txt has its own crawl budget implications.
Google has a limited "crawl budget" per site — a rough cap on how many pages it will crawl in a given period. For small sites this rarely matters. For large sites (thousands of pages), a poorly written robots.txt that doesn't block useless URLs (faceted navigation, session IDs, duplicate filtered pages) can eat your crawl budget on junk pages, leaving your important content under-crawled.

Google publicly logs its robots.txt parsing rules — and they're stricter than the spec.
The original robots.txt spec from 1994 is informal and not an official standard. Google published their own formal specification and even open-sourced their parser. One non-obvious rule: if your robots.txt file returns a 5xx server error, Googlebot treats the entire site as disallowed and stops crawling until it recovers. A broken server = Google treating your site as fully blocked.

noindex in HTTP headers works too — no HTML needed.
Most people know <meta name="robots" content="noindex"> in HTML. Fewer know you can send the same instruction as an HTTP response header: X-Robots-Tag: noindex. This is the only way to noindex non-HTML files like PDFs, since they have no <head> tag. Very useful for documentation or internal files served publicly.

Sitemaps are how Google first learned about many dark web discoveries.
This one's more trivia than actionable — but search engines have found accidentally public sitemaps on misconfigured servers that listed URLs the site owner never intended to be public. A sitemap is essentially a complete map of your site handed directly to any crawler that asks. Be intentional about what goes in it.

The robots.txt file is publicly readable by anyone — always.
It cannot be password protected (that would defeat its purpose). This means anyone can visit yourdomain.com/robots.txt and see exactly which paths you're trying to hide. Security researchers and curious people do this routinely. Don't rely on Disallow to obscure sensitive directory names — it's an announcement, not a curtain.

Google can index a page it has never crawled — if enough other sites link to it.
This means Disallow in robots.txt doesn't guarantee a page won't appear in search results. It just prevents Googlebot from reading the content. The URL can still show up as a "known but uncrawled" result. To truly remove a page from Google, you need noindex on the page itself (or use the URL removal tool in Search Console).

TL;DR

robots.txt → tells crawlers what to skip
sitemap.xml → tells crawlers what to index
Always link your sitemap from robots.txt
Submit to Google Search Console manually — don't wait
Keep them consistent with each other

Two files. One at the root of your domain. They take 20 minutes to set up properly and can meaningfully change how search engines see your site.

Every Website Has One. Go Check.

Here's something I want you to do right now.

Open a new tab and visit: youtube.com/robots.txt

You'll see hundreds of lines — specific bots being blocked, crawl-delay rules, dozens of sitemap references for different content categories. YouTube is one of the most visited sites on the planet, and they have a meticulously maintained robots.txt.

Now try it on any website you use regularly. Just append /robots.txt to any domain.

chatgpt.com/robots.txt. github.com/robots.txt. reddit.com/robots.txt. amazon.com/robots.txt.

Every single one has it. A startup's landing page. A government website. A 1999-era forum that hasn't been updated in a decade. This is what a 30-year-old voluntary internet standard looks like when it works — so universally adopted that it's basically invisible.

robots.txt isn't just a config file. It's a piece of living web history.

Tried it on a site and found something interesting in their robots.txt? Drop it in the comments.

I Stuck a ₹10 Sticker on My Wall and Now I Control My Light Like a Wizard

Sanchit Dikshit — Sat, 28 Mar 2026 09:19:49 +0000

No Alexa. No hub. No voice commands. Just tap and done.

The Problem That Started It All

It's 11:45 PM. I'm in bed, half asleep, and the light is still on.

My options:

Get up and hit the physical wall switch (feels like climbing Everest at midnight)
Unlock my phone, swipe past notifications, find the Wipro app, wait for it to load, tap the bulb tile, confirm the action

Neither of these should be this hard in 2026.

I already had a Wipro Wi-Fi smart bulb installed. Great bulb — RGB, tunable white, music sync, the works. But the app-based control was killing the "smart" experience. And I wasn't going to spend ₹3,000-5,000 on an Amazon Echo or Google Nest just to say "Alexa, lights off" before bed.

Then I remembered something I'd seen online: NFC tags.

Small stickers. Programmable. Cost almost nothing. And iPhones can read them without even unlocking.

What if I just... stuck one next to my bed and tapped my phone on it?

Spoiler: It worked. Here's exactly how I built it, what I learned, and how you can replicate it in an afternoon.

What Even Is NFC? (Bear With Me, This Is the Fun Part)

Before we get into the setup, let me geek out for a moment — because understanding why this works makes the whole thing more satisfying.

NFC stands for Near Field Communication. It's a short-range wireless technology that operates at 13.56 MHz and works over a maximum distance of about 4 centimetres. That ridiculously short range is actually a feature — it means you have to deliberately tap your device to trigger it, so false triggers are essentially impossible.

NFC is a subset of the broader RFID (Radio Frequency Identification) family of technologies. You've been using RFID/NFC your whole life without knowing it:

Your credit/debit card for tap-to-pay? NFC.
Your Mumbai Metro card? RFID.
That office access card you wave at the door? Also RFID.
Apple Pay, Google Pay on your phone? NFC.

The difference between passive NFC tags (like the stickers we're using) and your phone is power. Your phone has a battery and an active NFC controller. The tag has neither — it's completely passive. When your phone comes close, it generates a tiny electromagnetic field, which inductively powers the tag just long enough for it to transmit its stored data back. No battery. No charging. Ever.

The tag I used: NTAG213

This is the most widely deployed consumer NFC chip in the world. Here's what makes it special:

Memory: 144 bytes of user-accessible storage — enough for a URL, a command, or a short text string
Standard: ISO 14443 Type A compliant — meaning it works with virtually every NFC-enabled device
Rewritable: You can reprogram it hundreds of times, so if your setup changes, just overwrite the tag
iPhone compatibility: iPhone 7 and above can read NFC tags. iPhone 13 (what I use) supports background NFC tag reading — meaning the screen just needs to be on and unlocked, no app needs to be open

One fun fact: the tag has no moving parts, no battery, and theoretically no expiration date. That ₹10 sticker could outlast your phone by a decade.

The Shopping List

I only bought two products for this entire project.

Product	Price	Link
Wipro 12.5W B22D Wi-Fi Smart LED Bulb (16M Colours, Music Sync, Voice Control)	₹704	Amazon India
NTAG213 NFC Tag Stickers, Rewritable — Pack of 10	₹169	Amazon India
Total	₹873

Everything else I used was free:

Wipro Smart Home app (iOS/Android — free)
Google Home app (iOS/Android — free)
Google Assistant app (iOS — free)
Apple Shortcuts (built into every iPhone — free)

💡 Pro tip: The pack of 10 tags is the right call even if you only need 3. You'll find uses for the rest faster than you think — business card tag, desk work mode tag, silent mode trigger. More on that at the end.

The Architecture — How It All Connects

This is the part most blogs skip, and then readers get lost halfway through setup. Let me show you the full picture first.

Why so many layers? Great question. Ideally, the NFC tag would talk directly to the bulb. But Wipro/Tuya doesn't expose native iOS Shortcuts actions, and neither does Google Home. So we build a bridge: iPhone Shortcuts → Google Assistant → Google Home → Bulb. Each layer does one specific job, and together they create a seamless tap-to-control experience.

The whole chain from tap to bulb response takes about 2-3 seconds. Not instant, but completely hands-free.

Physical Setup — Where to Put the Tags

I placed 3 tags in total:

Tag 1 — Near the door (ON only)
Stuck on the wall right next to the light switch. When I enter the room, I tap this to turn the light on — without touching the physical switch.

Tag 2 — Bedside (ON)
Stuck on the headboard or nearby wall. When I'm in bed and want to turn the light on without getting up.

Tag 3 — Bedside (OFF)
Right next to Tag 2. Tap this to turn off before sleeping.

Why two separate tags instead of one toggle?

This is important to understand. NFC tags can only store and send a fixed command. They have no way of knowing the current state of your bulb (is it on or off right now?). A single "toggle" tag would be unpredictable — tap once and it turns off, tap again quickly and it turns on again. Two dedicated tags — one for ON, one for OFF — is far more reliable and predictable.

⚠️ Critical warning: The physical wall switch is the Achilles heel of this entire system. If someone flips the physical rocker switch to OFF, the bulb loses mains power completely. No power = no Wi-Fi = no smart control. The NFC system stops working until the physical switch is turned back ON. I recommend putting a small label on the switch that says "Don't switch off — smart bulb".

Step-by-Step Setup Guide

Step 1 — Set Up the Wipro Bulb

Screw in the Wipro bulb and turn the physical switch ON
Download the Wipro Smart Home app (iOS / Android)
Create an account — note your email and password carefully, you'll need them again
In the app, tap + → Add Device → Lighting → follow the Wi-Fi pairing instructions
Make sure your phone is on 2.4 GHz Wi-Fi during setup (the bulb doesn't support 5 GHz)
Once paired, test on/off from the app to confirm it's working

Step 2 — Link Wipro to Google Home

Install Google Home app and sign in with your Google account
Tap + (top right) → Link app or service
Tap Works with Google
Search for "Wipro Next Smart Home"
Tap it → tap Continue → tap Agree and continue
A Wipro login page opens — change the country from "United States" to "India" first (critical step many miss)
Enter your Wipro app email and password → tap Next Step
Your bulb will appear — tap Next to add it to your home

You should now see your bulb in Google Home, showing its live status.

Step 3 — Create Google Home Routines

This is the engine room. We're creating two voice-triggered routines that Google Home will execute when Google Assistant hears a specific word.

Routine 1 — Lights ON:

Open Google Home → tap Automations (bottom tab)
Tap + → New Routine
Under Starters → When I say to my Assistant → type: Lights
Under Actions → Add action → Control home devices → select your bulb → Turn on
Tap Save — name it "Lights"

Routine 2 — Lights OFF:

Repeat the above process
Starter phrase: Nights
Action: Select your bulb → Turn off
Save — name it "Nights"

💡 Why "Lights" and "Nights"? These words are phonetically distinct enough that Google Assistant won't confuse them with each other or with random speech. Avoid words like "on" and "off" as standalone triggers — they're too common and might fire accidentally.

Step 4 — Set Up iPhone Shortcuts Automations

This is where NFC comes in. We'll create one automation per tag.

For the Lights ON tag (Door or Bedside):

Open Shortcuts app → tap Automation tab → + → New Automation
Scroll down → tap NFC
Tap Scan → hold your iPhone's top edge close to the NFC tag
Name the tag "Lights" → tap OK
Tap Add Action → search "Hey Google" → select it
In the Hey Google action, type: Lights
Tap + below → search "Wait" → set to 3 seconds
Tap + → search "Go to Home Screen" → add it
At the top, set Automation to Run Immediately (not "Run After Confirmation")
Toggle Notify When Run to ON (optional but useful — you get a lock screen confirmation)
Tap Done

For the Lights OFF tag (Bedside):

Repeat the exact same steps above, but:

When you scan the tag, name it "Nights"
In the Hey Google action, type: Nights

Step 5 — Stick the Tags

Clean the surface with a dry cloth before sticking
Press firmly for 5 seconds
For high-traffic areas, consider an epoxy NFC tag instead of the sticker variant — more durable

The Result

Here's what happens now when I want to sleep:

Phone is locked on my nightstand
I pick it up — screen wakes (Face ID or tap)
I tap the top edge of my phone to the small sticker on my headboard
Lock screen shows: "Nights detected — Running your automation"
Google Assistant quietly fires in the background
Light turns off in ~2-3 seconds
Phone returns to lock screen

No app. No voice command. No getting up. Just tap.

It genuinely feels like a superpower the first time it works.

Honest Limitations

I want to be upfront about what this setup doesn't do:

iPhone must be unlocked (screen on) for NFC background reading to work — it doesn't fire from a completely dark, locked screen
Internet required — the entire chain goes through cloud services (Wipro → Tuya → Google Home). No internet = no control
~2-3 second delay from tap to bulb response — not instant, but acceptable
Physical switch is the weak point — if it's off, nothing works
Google Assistant app must stay installed — if you delete it, the Hey Google Shortcut action breaks

What's Next — Ideas to Extend This

Now that you have 7 leftover NFC tags, here are ideas to use them:

NFC Business Card — Program a tag with your website URL. Hand someone a card, they tap their phone, your portfolio opens. Zero app install needed on their end.

Desk Work Mode Tag — Tap when sitting down: bulb switches to cool white 100% brightness automatically.

Bedtime Scene Tag — Instead of just OFF, trigger a scene: dim to 10% warm white for 10 minutes, then off. Great for winding down.

Phone Silent Mode — NFC tag on your desk at work that triggers a Shortcut to enable Do Not Disturb.

Spotify Playlist — Tag on your gym bag that opens a specific playlist when tapped.

Future upgrade: If you ever replace this bulb with a Matter-compatible smart bulb, it will work natively with Apple Home — no Google Assistant bridge needed. The NFC Shortcuts setup remains identical.

The Final Tally

Item	Cost
Wipro Smart Bulb	₹704
10x NFC Tags	₹169
Smart speaker	₹0 (didn't need one)
Hub/Bridge	₹0 (didn't need one)
Total	₹873
Setup time	~90 minutes (including troubleshooting)
Satisfaction level	Disproportionately high

Wrapping Up

This started as a lazy man's quest to avoid getting out of bed. It became a genuinely fun afternoon of exploring NFC technology, building a multi-layer automation chain, and learning why Wipro uses the Tuya platform under the hood.

The best part? It cost under ₹1000, uses zero extra hardware, and works every single time.

If you're on Android, the setup is even simpler — Tasker or the built-in Google routines can trigger directly without the Hey Google workaround.

What would you automate with your leftover NFC tags? Drop it in the comments — I'm genuinely curious what creative uses people come up with.

Found this useful? Follow me for more experiments where I build things I probably didn't need but definitely enjoyed making.

Products used:

Wipro 12.5W B22D Wi-Fi Smart LED Bulb — ₹704
NTAG213 NFC Tag Stickers Pack of 10 — ₹169

Apps used (all free):

Wipro Smart Home (iOS/Android)
Google Home (iOS/Android)
Google Assistant (iOS)
Apple Shortcuts (built-in on iPhone)

How I Built a Fully Private Portfolio Using GitHub and Cloudflare — For Free

Sanchit Dikshit — Thu, 03 Jul 2025 06:02:30 +0000

A complete guide to deploying a portfolio site from a private GitHub repo using Cloudflare Pages, DNS, HTTPS, redirects, and DNSSEC — with zero cost and maximum security.

Why I Built this

I wanted a clean, secure, and professional online presence:

Without exposing my GitHub code
Without paying for hosting or domain-linked mail servers
Without sacrificing speed, reliability, or security
This setup delivers on all of that.

Goals

✅ Deploy from a private GitHub repo
✅ Use a custom domain with full HTTPS
✅ Enforce DNSSEC and redirect rules
✅ Host everything 100% free

Step-by-Step Setup

Create & Push Your Portfolio Repo

Initialize a private GitHub repo named portfolio
Add your static site files (index.html, images, etc.)
Add a CNAME file with your custom domain (e.g., www.example.com)

Deploy via Cloudflare Pages

Go to pages.cloudflare.com
Click Create Project → Select your private repo
Choose "None" as the framework, set output directory as .
Deploy

✅ You now have a live site at https://your-project.pages.dev

Transfer DNS to Cloudflare

Add your domain to Cloudflare
Let it scan for existing records (optional)
Update your domain's nameservers in Namecheap to the ones Cloudflare gives you (e.g., dion.ns.cloudflare.com)

Add DNS Records

In Cloudflare DNS:

Add a CNAME:

Name: www
Value: your Pages URL (e.g., `your-project.pages.dev`)
Proxy status: DNS only (important)

Add an A record:

Name: @
Value: 192.0.2.1 (dummy IP to enable redirect)
Proxy status: Proxied

Add a Custom Domain to Pages

In Cloudflare Pages → Your Project → Custom Domains
Add www.example.com
Choose "My DNS provider"
Verify CNAME setup
Enable HTTPS (Cloudflare will issue a free cert)

Redirect example.com to www.example.com

Go to Rules → Page Rules
Add two rules:

http://example.com/* → Forward to https://www.example.com/$1

https://example.com/* → Forward to https://www.example.com/$1

Enable DNSSEC

In Cloudflare → DNS → DNSSEC → Enable
Copy the DS record Cloudflare gives you
Go to Namecheap → Domain → Advanced DNS → DNSSEC → Add DS Record
Paste the values from Cloudflare

✅ Now your domain is cryptographically signed

Why This Matters

In 2025, you shouldn’t have to:

Pay for basic static hosting
Expose your GitHub repo just to host a site
Settle for a default pages.dev domain
Compromise on security or speed

This setup checks all the boxes:

✅ Zero cost
✅ Zero code exposure
✅ 100% secure
✅ 100% under your control

Final Result

Accessible at: https://www.example.com

Private GitHub powered
HTTPS + DNSSEC secured
Lightning-fast via Cloudflare

Want Help Doing the Same?

Drop a comment or connect on X/LinkedIn — happy to help others set this up.
If this guide helped you, please react, save, or share to support more privacy-first builders.

Forem: Sanchit Dikshit

The Two UNDERRATED Files Every Website Needs

First, a Mental Model

robots.txt — The Bouncer Your Site Deserves

What it is

Breaking it down

Real-world patterns

The golden rule people forget

sitemap.xml — Your Site's CV for Search Engines

What it is

The optional fields, honestly rated

Sitemap index files

Specialized sitemaps

The Connection Between the Two

Framework Quickstart

Submitting to Search Console (Don't Skip This)

Common Mistakes (I've Made Most of These)

Things Most People Don't Know (The Interesting Bits)

TL;DR

Every Website Has One. Go Check.

I Stuck a ₹10 Sticker on My Wall and Now I Control My Light Like a Wizard

The Problem That Started It All

What Even Is NFC? (Bear With Me, This Is the Fun Part)

The Shopping List

The Architecture — How It All Connects

Physical Setup — Where to Put the Tags

Step-by-Step Setup Guide

Step 1 — Set Up the Wipro Bulb

Step 2 — Link Wipro to Google Home

Step 3 — Create Google Home Routines

Step 4 — Set Up iPhone Shortcuts Automations

Step 5 — Stick the Tags

The Result

Honest Limitations

What's Next — Ideas to Extend This

The Final Tally

Wrapping Up

How I Built a Fully Private Portfolio Using GitHub and Cloudflare — For Free

Create & Push Your Portfolio Repo

Deploy via Cloudflare Pages

Transfer DNS to Cloudflare

Add DNS Records

Add a Custom Domain to Pages

Redirect example.com to www.example.com

Enable DNSSEC

`robots.txt` — The Bouncer Your Site Deserves

`sitemap.xml` — Your Site's CV for Search Engines