Forem: Adam Tauber

How I Cut My Google Search Dependence in Half

Adam Tauber — Thu, 02 Apr 2026 15:05:43 +0000

TL;DR: I built Hister, a self-hosted web history search tool that indexes visited pages locally. In just 1.5 months, I reduced my reliance on Google Search by 50%.

The Problem: Online Search Isn't What It Used to Be

Like many developers and knowledge workers, I found myself constantly reaching for Google Search throughout my workday. It had become such an ingrained habit that I barely noticed how often I was context-switching away from my actual work to perform searches. But over time, something had changed about the experience. The search results that once felt reliable and helpful were increasingly problematic in several ways.

Too Many Advertisements

What used to be a clean list of relevant links now requires scrolling past multiple sponsored results, shopping suggestions, and promoted content just to reach the organic results. Often, the actual information I'm looking for doesn't appear until halfway down the page, after I've mentally filtered out all the commercial noise.

Manipulative SEO Tactics

Organic results themselves have been manipulated by SEO tactics rather than truly reflecting the most relevant and helpful content. Websites optimized for search engines rather than humans dominate the rankings, while genuinely useful resources from smaller sites or personal blogs get buried on page two or three. The signal-to-noise ratio has degraded significantly.

AI Suggestions

Google has recently added AI-generated summaries at the top of many search results. While sometimes helpful, these summaries often miss crucial nuance, provide oversimplified or occasionally incorrect information, and add yet another layer between me and the actual source material I'm trying to find. For technical queries where precision matters, these AI answers can be misleading or incomplete.

Lack of Privacy

Google tracks every query I make, building a detailed profile of my interests, work patterns, and information needs. This data is used for ad targeting and who knows what else. The convenience of search comes at the cost of giving away intimate details about my work and life.

The Insight

But the realization that pushed me to build a solution was that I was often searching for pages I'd already visited. That documentation page I read last week but forgot to bookmark. That GitHub issue I commented on yesterday but couldn't remember the exact project name. Those internal wiki pages with crucial information about our infrastructure. I was using Google as a personal memory aid, outsourcing my recall to an external service that was tracking my every query. And for content behind authentication (internal tools, documentation, private repositories) Google couldn't help at all, since it can't index pages it can't access.

Two Types of Search

Thinking on how to replace Google led me to a crucial realization about the nature of search itself. When we type queries into a search box, we're actually doing one of two fundamentally different things, even though the interface is identical:

Discovery Search: Finding New Information

Discovery search is what we typically think of when we imagine "searching the internet". It's about finding information we've never encountered before. This is true exploration, we're venturing into unknown territory, discovering new resources, learning about topics we're unfamiliar with, and finding answers to questions we've never asked before. For this type of search, we genuinely need the vast index of the internet that services like Google provide. We need to cast a wide net and see what the collective knowledge of the web has to offer.

Recall Search: Refinding Known Information

But then there's the other type of search what I call "recall search". This is when we're trying to find information we've already encountered. We're not discovering something new; we're trying to remember where we saw something. Examples of this include searches like "That authentication bug I fixed last month..." when you remember solving a problem but can't recall the exact solution, or "The Bleve docs page about result highlighters..." when you know you've read the documentation before but can't remember the specific URL or section title. Another common example: "That Stack Overflow answer about async/await..". when you remember reading a particularly clear explanation but didn't save the link.

A significant portion of my daily searches — probably more than half — were recall searches, not discovery searches.

The revelation that changed everything for me was this: A significant portion of my daily searches - probably more than half - were recall searches, not discovery searches. I was constantly using Google to search my own browsing history, to refind pages I'd already visited and information I'd already read. But Google's interface treats both types of search identically, and it has no special optimization for helping you refind your own content. Worse, for pages behind authentication or on private networks, Google can't help at all because it can't index content it can't access.

This insight suggested a solution: What if I had a dedicated tool optimized specifically for recall search for refinding my own browsing history, and only fall back to Google for true discovery search?

The potential benefits were enormous:

faster results
better privacy
access to authenticated content
results tailored specifically to my interests and work

The Solution: Index Everything Locally

The solution seemed obvious once I'd articulated the problem: what if I could search my entire browsing history - including the full page content, not just URLs and titles - locally and privately? This would give me a personal search engine optimized specifically for recall search, while still allowing me to fall back to Google for discovery search when needed.

I started looking for existing solutions. Surely someone had built this before? Browser history exists, but it only stores URLs and page titles, making it nearly useless for finding pages based on their content. Some note-taking apps like Evernote or Notion offer web clippers, but they require manual action for each page you want to save. Personal knowledge management tools like Omnom exist, but they're focused on curated notes rather than comprehensive browsing history, but they require conscious decisions about what to save.

None of the existing tools I found met all my requirements. I needed something that combined the comprehensive automatic capture of browser history, the full-text search capabilities of a search engine, the performance of local software, and the privacy of self-hosted solutions. Since nothing existed that checked all these boxes, I decided to build it myself.

What I Needed

The requirements for my ideal solution were clear:

Fast lookup If searching my local index took longer than just Googling, I'd never use it. I needed instant, sub-second search response times, keyboard shortcuts to make it faster to search locally than to context-switch to Google.

Automatic indexing I didn't want to manually save pages or make conscious decisions about what to index. It needed to capture pages as I browse with zero manual work on my part. The tool should disappear into the background and just work.

Authentication aware indexing So much of the content I reference daily is behind authentication: internal wikis, private documents, authenticated API documentation, internal dashboards. Any solution that couldn't handle authenticated content would miss a huge portion of my actual browsing.

Full-text search Meant searching the actual page content, not just URLs and titles. Browser history is useless when you remember reading something about "microservice authentication patterns" but can't remember which blog or doc site it was on. I needed to be able to search the words within the pages.

Powerful query capabilities Like Boolean operators (AND, OR, NOT), field-specific searches (search only URLs, or only titles), and wildcard matching would make it possible to narrow down results quickly.

Zero cognitive overhead The tool needed to work seamlessly in my workflow. It should integrate naturally with how I already browse and search.

Transparent fallback to online search engines If I searched locally and didn't find what I wanted, I should be able to immediately fall back to Google with the same query, making adoption gradual rather than requiring a complete workflow change.

Fine-tuning capabilities Let me customize the experience over time. I wanted to be able to blacklist irrelevant sites I never want to see again, prioritize important sources, and create keyword aliases for common searches.

Offline preview of saved content I could read indexed pages even if the original site went down or the page was deleted; a nice bonus that would occasionally save me from link rot.

Import existing history I wanted to start with years of browsing data already indexed, rather than building up an index from scratch over months.

Free software Self-Hosted, with no recurring costs or vendor lock-in. My browsing history is my personal data, it should not be owned by companies.

No existing tool checked all these boxes. So I decided to build Hister.

Introducing Hister

Hister is a self-hosted web history management tool that treats your browsing history as a personal search engine.

The Results: 50% Reduction in 1.5 Months

After using Hister for six weeks, I analyzed my search patterns:

~50% of my Google searches now answered locally
Found content Google couldn't (authenticated pages, deleted content)
Zero privacy concerns No tracking, no profiling
Better results for my specific needs (because it's MY history)

The more I use it, the better it gets. My local index is now:

More relevant than Google for my common queries
As fast as opening a new browser tab
Comprehensive across authenticated services
A personal knowledge base of everything I've read

Unexpected Benefits

Rediscovery: I'm finding valuable content I'd forgotten about. That article I bookmarked 2 years ago but never revisited? Now it shows up in relevant searches.

Learning patterns: Seeing what I search for reveals my knowledge gaps and interests.

Offline access: When documentation sites go down or pages get deleted, I still have the content.

Conclusion

We've accepted that search means "go to Google" for so long that we've forgotten there are alternatives. But for a huge portion of my daily searches - probably more than half - I don't need the entire internet. I need OUR internet: the pages I've read, the docs I've opened, the internal tools I use daily.

Hister isn't trying to replace Google for discovery. It's trying to replace Google for recall. And in that domain, it's already better than Google could ever be, because:

It knows about authenticated pages Google will never see
It searches YOUR history, not the entire web
It's instant, private, and ad-free
It gets better the more you use it

After 1.5 months, I've cut my Google dependence in half. I expect this number will increase as my index grows.

If you're a developer, researcher, or knowledge worker who constantly re-searches for information you've already found, give Hister a try. It might just change how you find information on the internet.

Before Hister:

Open Google
Search: "bleve query"
Click first result (probably wrong)
Click second result (looks familiar…)
Realize I've been here before
Finally find the specific page I wanted

Time: ~1-2 minutes, 5-10 clicks

With Hister:

Open Hister
Type: "bleve query", press enter
First result is opened with the EXACT page I visited last month

Time: ~5 seconds, few keystrokes

Take Back Your Search

To get started with Hister check out the following links:

Future Development

I'm actively developing Hister with these goals:

Improve usability
Add automatic indexing capabilities based on the index and opened results
Find a secure and privacy respecting way to connect local Hister's to a distributed search engine
Export search results
Advanced analytics and search insights

Hister is open source (AGPLv3) and welcomes contributions!

Ways to Contribute

🐛 Report bugs and suggest features on GitHub Issues
💻 Submit pull requests (check out good first issues)
📖 Improve documentation
🎨 Design better UI/UX
🌍 Translate to other languages
⭐ Star the repo and spread the word!

Have questions or feedback? Open an issue on GitHub or reach out to @asciimoo.

How to Scrape Instagram Profiles

Adam Tauber — Mon, 13 Nov 2017 00:00:00 +0000

Scraping can be tedious work especially if the target site isn't just a standard static HTML page. Plenty of modern sites have JavaScript only UIs where extracting content is not always trivial. Instagram is one of these websites, so I would like to show you how it is possible to write a scraper relatively fast to get images from Instagram. I'm using Colly, a scraping framework for Golang. The full working example can be found here.

Information gathering

First, if we view the source code of a profile page (e.g. https://instagram.com/instagram), we can see a bunch of JavaScript code inside the body tag instead of static HTML tags. Let's take a closer look at it. We can see that the first script is just a variable declaration where a huge JSON is assigned to a single variable (window._sharedData). This JSON can be easily extracted from the script tag by finding the first { character and getting the whole content after it:

jsonData := scriptContent[strings.Index(scriptContent, "{") : len(scriptContent)-1]

Note that because it is a JavaScript variable declaration it has a trailing semicolon what we have to cut off to get a valid JSON. That's why the example above ends with len(scriptContent)-1.

The formatted view of the extracted JSON reveals all the information we are looking for. The JSON contains information about a user's images and some metadata of the profile (e.g. the profile ID is 25025320). There is an interesting part of the metadata called page_info:

 "page_info": {
  "has_next_page": true,
  "end_cursor": "AQBiQhGRC6c6f-YOxdU0ApaAvotN4zI601ymkAtQ8SutdWz2n-bKFCkv51PMAoi9im3tNDTFLyhV969z8a6JnAkQMzHbYVwNI4Ke7jbk99nvFA"
 }

Probably, the value of end_cursor' is the attribute of the URL to get the next page when has_next_page is true.

Format JSONs with the handy jq command line tool

Paging

The next page of the user profile is retrieved by an AJAX call, so we have to use the browser's Network Inspector to find out what is required to fetch it. Network Inspector shows a long and cryptic URL which has two GET parameters query_id and variables:

https://www.instagram.com/graphql/query/?query_id=17888483320059182&variables=%7B%22id%22%3A%2225025320%22%2C%22first%22%3A12%2C%22after%22%3A%22AQBiQhGRC6c6f-YOxdU0ApaAvotN4zI601ymkAtQ8SutdWz2n-bKFCkv51PMAoi9im3tNDTFLyhV969z8a6JnAkQMzHbYVwNI4Ke7jbk99nvFA%22%7D

It seems like Instagram uses a GraphQL API and the value of variables GET parameter is an URL encoded value. We can decode it with a single line of Python code:

$ python -c 'import urlparse;print(urlparse.parse_qs("variables=%7B%22id%22%3A%2225025320%22%2C%22first%22%3A12%2C%22after%22%3A%22AQBiQhGRC6c6f-YOxdU0ApaAvotN4zI601ymkAtQ8SutdWz2n-bKFCkv51PMAoi9im3tNDTFLyhV969z8a6JnAkQMzHbYVwNI4Ke7jbk99nvFA%22%7D")["variables"][0])'
{"id":"25025320","first":12,"after":"AQBiQhGRC6c6f-YOxdU0ApaAvotN4zI601ymkAtQ8SutdWz2n-bKFCkv51PMAoi9im3tNDTFLyhV969z8a6JnAkQMzHbYVwNI4Ke7jbk99nvFA"}

As you can see it is a JSON object and the value of the after attribute is the same as the value of the end_cursor and id is the ID of the profile.

The only unknown information in the next page URL is the query_id GET parameter. The HTML source code does not contain it, nor the cookies or response headers. After a little bit of digging it can be found in a static JS file included in the main page and seems it is a constant value.

The format of the response is also JSON but the structure is different from what we've found on the main page. This JSON contains the same information as the previous one, however we cannot use the same method to extract data due to structural differences.

Building the scraper

The information gathering phase clearly shows that we need four building blocks to be able to fetch all images found on an Instagram profile. Let's do it using Colly.

Extract and parse JSON from the main page

To extract content from HTML we need a new Collector which has a HTML callback to extract the JSON data from the script element. Specifying this callback and when it must be called can be done in OnHTML function of Collector.
The JSON can be easily converted to native Go structure using json.Unmarshal from the standard library.

c := colly.NewCollector()

c.OnHTML("body > script:first-of-type", func(e *colly.HTMLElement) {
    // find JSON string
    jsonData := e.Text[strings.Index(e.Text, "{") : len(e.Text)-1]

    // parse JSON
    data := struct {
       EntryData struct {
           ProfilePage []struct {
               User struct {
                   Id    string `json:"id"`
                   Media struct {
                       Nodes []struct {
                           ImageURL     string `json:"display_src"`
                           ThumbnailURL string `json:"thumbnail_src"`
                           IsVideo      bool   `json:"is_video"`
                           Date         int    `json:"date"`
                           Dimensions   struct {
                               Width  int `json:"width"`
                               Height int `json:"height"`
                           }
                       }
                       PageInfo pageInfo `json:"page_info"`
                   } `json:"media"`
               } `json:"user"`
           } `json:"ProfilePage"`
       } `json:"entry_data"`
    }{}
    err := json.Unmarshal([]byte(jsonData), &data)
    if err != nil {
        log.Fatal(err)
    }

    // enumerate images
    page := data.EntryData.ProfilePage[0]
    actualUserId = page.User.Id
    for _, obj := range page.User.Media.Nodes {
        // skip videos
        if obj.IsVideo {
            continue
        }
        c.Visit(obj.ImageURL)
    }
    // ...
}

Create and visit next page URLs

The format of the next page URL is fixed, so a format string can be declared which accepts the changing id and after parameters.

const nextPageURLTemplate string = `https://www.instagram.com/graphql/query/?query_id=17888483320059182&variables={"id":"%s","first":12,"after":"%s"}`

Parse next page JSONs

This is pretty much the same as the conversion of the main page's JSON except these responses have some different attribute names (e.g. the image url is display_url instead of display_src).

Download and save images extracted from JSONs

After requesting images from Instagram using the Visit function, responses can be handled in OnResponse. It requires a callback as a parameter which is called after the response has arrived. To select responses which include images, we should filter based on Content-Type HTTP header. If it is image, it must be saved.

c.OnResponse(func(r *colly.Response) {
    if strings.Index(r.Headers.Get("Content-Type"), "image") > -1 {
        r.Save(outputDir + r.FileName())
        return
    }
    // handle further response types...
}

Epilogue

Scraping JS-only sites isn't always trivial, but can be handled without headless browsers and client side code execution to achieve great performance. This scraper example downloads approximately 1000 images a minute on a single thread over a regular home Internet connection.

It can be tweaked further to handle videos and extract meta information.