Forem: Goscrapy

Efficiently Extracting Business Data from Google Maps

Goscrapy — Fri, 17 Apr 2026 16:25:14 +0000

Hi everyone,

Extracting reliable business data from Google Maps is a common requirement for market research and local lead generation. However, the technical implementation is a bit difficult not because of anti-bot measures, but because of how Google structures its responses.

Instead of clean HTML or standard JSON, Google serves data in massive, obfuscated arrays that change based on your geographic center. While many developers default to heavy headless browsers to solve this, there is a much more efficient, "browserless" approach that relies on understanding the underlying API logic.

In this guide, we'll walk through how to solve these using GoScrapy in a step by step manner.

Let's get started.

💡 Tip: For this you will need to setup goscrapy. If you haven't checked out our last article on how to do it, you should—it's very simple and will get you up and running in minutes.

Step 1: Defining the Data Blueprint

The first step in to define a model for the scraped data. Google Maps provides a wealth of information, but without a clear schema, your extraction logic will quickly become unmanageable. In GoScrapy, data is modeled using a record

The Benefit:
Defining a typed structure upfront ensures that your data collection is consistent and that your downstream processing (like CSV or Database exports) remains stable even when dealing with varied business listings.

// google_maps_scraper/record.go

type Record struct {
    Query              string           `json:"query" csv:"query"`
    QueryLoc           location         `json:"-" csv:"-"`
    Status             string           `json:"status" csv:"status"`
    OpenHours          OpeningHours     `json:"opening_hours" csv:"-"`
    OpenHoursStr       string           `json:"opening_hours_str" csv:"opening_hours_str"`
    Phone              string           `json:"phone" csv:"phone"`
    WebResultsUrl      string           `json:"webresults_url" csv:"webresults_url"`
    ShortDescription   string           `json:"short_description" csv:"short_description"`
    Description        string           `json:"description" csv:"description"`
    TimeZone           string           `json:"timezone" csv:"timezone"`
    Categories         []string         `json:"category" csv:"category"`
    Title              string           `json:"title" csv:"title"`
    Website            string           `json:"website" csv:"website"`
    Review             float64          `json:"review" csv:"review"`
    ReviewDistribution string2Uint64Map `json:"review_distribution" csv:"review_distribution"`
    ReviewsUrl         string           `json:"reviews_url" csv:"reviews_url"`
    ReviewsCount       uint64           `json:"reviews_count" csv:"reviews_count"`
    Street             string           `json:"street" csv:"street"`
    City               string           `json:"city" csv:"city"`
    ZipCode            string           `json:"zipcode" csv:"zipcode"`
    State              string           `json:"state" csv:"state"`
    StateCode          string           `json:"state_code" csv:"state_code"`
    Country            string           `json:"country" csv:"country"`
    CountryCode        string           `json:"country_code" csv:"country_code"`
    Details            Details          `json:"details" csv:"details"`
    ReservationUrls    []string         `json:"reservation_urls" csv:"reservation_urls"`
    OrderUrls          []string         `json:"order_urls" csv:"order_urls"`
    OrderPlatforms     []string         `json:"order_platforms" csv:"order_platforms"`
    GoogleReserveUrl   string           `json:"google_reserve_url" csv:"google_reserve_url"`
}

Step 2: Overcoming the "Spatial Anchor" Requirement

Most developers start by searching for a keyword, only to find the results are inconsistent. This is because the Search API requires a spatial "anchor" (Latitude and Longitude) to provide high-density results.

The Technique:
You can resolve any text query (e.g., "Gyms in Austin, TX") by first hitting a geocoding endpoint. Once you have the coordinates, you can center your search exactly where the density is highest, ensuring you don't miss any data points. As you can see, for jobs, where don't have lat/lng in advanced, we try to grab the location for the query using the parseGeoding step.

// google_maps_scraper/spider.go

func (s *Spider) StartRequest(ctx context.Context, job *Job) {
    if job.loc == nil {
        // Resolve coordinates to ensure a high-density search area
        req := prepareRequest(s.Request(ctx), generateGeocodingUrl(job.query), *job)
        s.Parse(req, s.parseGeocoding)
        return
    }

    // If we have coordinates, the search results will be far more accurate
    req := prepareRequest(s.Request(ctx), generateSearchUrl(job), *job)
    s.Parse(req, s.parseMapListing)
}


func (s *Spider) parseGeocoding(ctx context.Context, resp core.IResponseReader) {
    job, ok := getJob(resp)
    if !ok {
        return
    }

    lat, lng, _, ok := extractGeocoding(resp.Bytes())
    if !ok {
        return
    }

    job.setLocation(lat, lng)
    s.Logger().Infof("Location found for %s: %f, %f", job.query, lat, lng)
        // check util.go for details on prepareRequest
    req := prepareRequest(s.Request(ctx), generateSearchUrl("https://www.google.com", job), job)
    s.Parse(req, s.parseMapListing)
}

Step 3: Handling Large Datasets via Cursor Offsets

Google Maps doesn't use simple "Next Page" links or page numbers. Instead, it relies on a cursor-based offset system embedded within a pb query parameter.

The Technique:
To crawl thousands of results, you need to track this offset state across your requests. Using GoScrapy's Meta context we pass this "tracking job" between asynchronous workers without the risk of data races or state corruption that global variables would introduce.

// google_maps_scraper/spider.go

func (s *Spider) parseMapListing(ctx context.Context, resp core.IResponseReader) {
    job, ok := getJob(resp)
    if !ok {
        return
    }

    if resp.StatusCode() != 200 {
        return
    }
        // Here we basically try to parse the data surgically
    records := extractMapResults(resp.Bytes())
    s.Logger().Infof("Found %d records for [%s] (At cursor %d)", len(records), job.query, job.cursor)
    if len(records) <= 0 {
        return
    }

    // update cursor for the next page
    job.SetCursor(job.cursor + 20)
    for _, record := range records {

                //  this is where we export the record to be saved by csv/json pipeline, which ever we have configured.
        s.Yield(&record)
    }

    req := prepareRequest(s.Request(ctx), generateSearchUrl("https://www.google.com", job), job)
    s.Parse(req, s.parseMapListing)
}

Step 4: "Surgical" Extraction from Deeply Nested Arrays

Google's internal JSON is not served in standard "key-value" pairs. It is a deeply nested array structure designed for internal engine processing, not external consumption.

The Benefit:
Instead of trying to map the entire massive response into memory, you can use GJSON to perform "surgical" extractions. By reaching directly into specific array indices (like index 11 for Title).

// google_maps_scraper/utils.go
// check repo examples for full code.

func extractMapResults(data []byte) []Record {
    // After removing the non JSON prefix, we dive directly into the results array
    records := gjson.GetBytes(data, "0.1.#.14")
    // This approach allows for extracting deep values with zero-allocation speed
   // Refer to full code in repo attached. Pasting it here would create clutter
}

Conclusion: Performance in Production

By moving away from browser engines and toward a raw HTTP approach, you can achieve significantly higher throughput with minimal resources.

A GoScrapy driven scraper compiles into a standalone 13MB binary, making it incredibly efficient to deploy in cloud environments compared to multi-hundred-megabyte headless browser images. This efficiency translates directly into lower infrastructure costs and faster data turnaround.

If you’re looking to scale your data extraction projects, you can find the full source code for this example and more in the Google map scraper.

Note: This scraper was created using **goscrapy* and for educational purposes only to showcase the capabilities of goscrapy and I am not liable for any misuse of this scraper.*

Scrapy 🕷️, but in Go: Building High-Performance Scrapers without the Boilerplate

Goscrapy — Wed, 15 Apr 2026 16:28:56 +0000

Hi everyone 👋

Web scraping can start out pretty basic. You just loop through some pages, grab the HTML, pull out what you need, and store it somewhere. But then when you try to make it bigger, like dealing with a ton of requests or figuring out retries and cookies for different sites, it turns into a real hassle quick.

I remember using Scrapy in Python, and it handled all that stuff without me having to think too hard about the details. It felt structured, you know. When I switched over to Go, there are good tools for the basics, but I missed that kind of ready-made setup. So thats why I ended up making GoScrapy.

What is GoScrapy? (A quick intro)

GoScrapy is basically this framework that tries to mimic the experience of Scrapy but in Go. It is not just for pulling HTML, it manages the whole process from start to finish for extracting data. And it uses Go's built in concurrency, so you get fast performance without messing around with goroutines yourself all the time. I think that part is what makes it stand out, especially if you are coming from other languages.

To get going with it

There is a CLI tool that feels similar to Scrapy's. You install it with go install and it needs Go 1.22 or later.

go install github.com/tech-engine/goscrapy/cmd/...@latest

Then verify the installation with gos -v.

Starting a project is simple too

gos startproject my_scraper

It sets up everything for you, like the module and files.

gos startproject books_to_scrape

🚀 GoScrapy generating project files. Please wait!

📦 Initializing Go module: books_to_scrape...
go: creating new go.mod: module books_to_scrape
go: to add module requirements and sums:
        go mod tidy
✔️  books_to_scrape\base.go
✔️  books_to_scrape\constants.go
✔️  books_to_scrape\errors.go
✔️  books_to_scrape\job.go
✔️  main.go
✔️  books_to_scrape\record.go
✔️  books_to_scrape\settings.go
✔️  books_to_scrape\spider.go

📦 Do you want to resolve dependencies now (go mod tidy)? [Y/n]: y
📦 Resolving dependencies...

The output shows it's scaffolding the project will all the necessary file, so you don't have to manually. It even asks if you want to run go mod tidy right then. Pretty handy, I guess, though sometimes it takes a second to finish.

Project Structure

Once you have the project, the structure looks like this for something like books_to_scrape

books_to_scrape/
├── main.go               # The starting point
└── books_to_scrape/
    ├── base.go           # Initializing the engine, middlewares etc
    ├── constants.go      # Shared stuff
    ├── errors.go         # Custom errors, if you need any
    ├── job.go            # Parameters for your spider
    ├── record.go         # The structure of data you scrape
    ├── settings.go       # Configs like pipelines
    └── spider.go         # Where the actual extraction happens

main.go sets up the context, instantiate the spider, starts the request, and waits for it to finish. It keeps things straightforward.

// main.go

// this template will be autogenerate as well like all the files
func main() { 
    ctx := context.Background() 
    spider := books_to_scrape.New(ctx) 
    // start the first request
    spider.StartRequest(ctx, nil) 

    // wait for the spider to finish, true mean, the spider would
    // exit as soon as it's done. But if you are running goscrapy
    // as a server, you can set it to false(default), and it will keep running, accepts more jobs via the StartRequest() method.

    if err := spider.Wait(true); err != nil { 
        log.Fatal(err) 
    } 
}

Writing Spiders

In spider.go, thats where you write how to handle requests and parse responses. There are two main things you use all the time: s.Request and s.Parse.

s.Request(ctx) is what you call each time for a new fetch. It gives you a request object where you can chain things like Url, Meta, or Headers. Each request has to be its own thing don't try to reuse them for different pages/requests.

s.Parse(req, callback) is what actually sends the request off to the engine. You also tell it which function should handle the response once it comes back. Usually, you start with a StartRequest method that kicks things off.

For example, with books.toscrape.com, in the parse function you grab product links, make a full URL, and parse each one with parseProduct. Then check for the next page, and if there is one, follow it back to parse.

// spider.go

func (s *Spider) parse(ctx context.Context, resp core.IResponseReader) {
    // grab product links
    for _, productUrl := range resp.Css("article.product_pod h3 a").Attr("href") {
        req := s.Request(ctx)
        req.Url(fmt.Sprintf("%s/%s", s.baseUrl, productUrl))
        s.Parse(req, s.parseProduct)
    }

    // check for next page
    if next := resp.Css("li.next a").Attr("href"); len(next) > 0 {
        req := s.Request(ctx)
        req.Url(fmt.Sprintf("%s/%s", s.baseUrl, next[0]))
        s.Parse(req, s.parse) 
    }
}

func (s *Spider) parseProduct(ctx context.Context, resp core.IResponseReader) {
    product := resp.Css("article.product_page")

    // yield a Record
    s.Yield(&Record{
        Title: product.Css(".product_main h1").Text()[0],
        Price: product.Css(".price_color").Text()[0],
        Stock: product.Css(".availability").Text()[0],
    })
}

It handles pagination kind of naturally, but you have to be careful with the base URL.

Middlewares and Pipelines

The framework has middlewares and pipelines like Scrapy. Middlewares for things like retry with backoff or dupe filter are set in settings.go as a slice:


// settings.go
var MIDDLEWARES = []middlewaremanager.Middleware{
    middlewares.Retry(), 
    middlewares.DupeFilter,
}

Pipelines take the yielded records and export them, say to CSV. Every yield goes right there. It seems efficient for output, though I am not totally sure how it scales with huge datasets yet.

var export2CSV = pipelines.Export2CSV[*Record](pipelines.Export2CSVOpts{ 
    Filename: "output.csv", 
})

TUI Dashboard

There is also this TUI dashboard for watching progress in the terminal. You can add it in base.go by making a tui.New(app.Logger()), a telemetry hub, and starting it in a goroutine.

//base.go

// Tweak base.go to wire up the TUI
dashboard := tui.New(app.Logger())
// create a telemetry hub
hub := ts.NewTelemetryHub()
// add the dashboard as an observer
hub.AddObserver(dashboard)
// set the telemetry hub to the app
app.WithTelemetry(hub)

go func() {
    _ = gos.StartWithTUI(ctx, app, dashboard)
}()

It shows stats visually, which is nice if you are fond of that.

Anyway, thats the gist of it

GoScrapy is still in early versions, v0.x, and under development. If you want to check it out or help, feel free.

GitHub: github.com/tech-engine/goscrapy
Discord: Join our community

I think it could be useful for Go folks doing scraping, but it might need more tweaks.

Thank you for reading this far 💚