Forem: Murrough Foley

How to Use rs-trafilatura with Scrapy

Murrough Foley — Fri, 03 Apr 2026 14:23:44 +0000

Scrapy is the standard Python framework for web scraping. It handles crawling, scheduling, and data pipelines. rs-trafilatura plugs into Scrapy as an item pipeline — your spider yields items with HTML, and the pipeline adds structured extraction results automatically.

Install

pip install rs-trafilatura scrapy

Setup

Add the pipeline to your Scrapy project's settings.py:

ITEM_PIPELINES = {
    "rs_trafilatura.scrapy.RsTrafilaturaPipeline": 300,
}

That's it. Every item that passes through the pipeline with a body (bytes) or html (string) field will get an extraction dict added to it.

Writing the Spider

Your spider yields items with the response body and URL:

import scrapy

class ContentSpider(scrapy.Spider):
    name = "content"
    start_urls = ["https://example.com"]

    def parse(self, response):
        yield {
            "url": response.url,
            "body": response.body,  # raw bytes — rs-trafilatura auto-detects encoding
        }

        # Follow links
        for href in response.css("a::attr(href)").getall():
            yield response.follow(href, self.parse)

The pipeline picks up body (bytes) or html (string). When it finds one, it runs extraction and adds the results under item["extraction"].

What the Pipeline Adds

Each processed item gets an extraction dict:

{
    "url": "https://example.com/blog/post",
    "body": b"<html>...",
    "extraction": {
        "title": "Blog Post Title",
        "author": "John Doe",
        "date": "2026-01-15T00:00:00+00:00",
        "main_content": "The full extracted text...",
        "content_markdown": "# Blog Post Title\n\nThe full extracted text...",
        "page_type": "article",
        "extraction_quality": 0.95,
        "language": "en",
        "sitename": "Example Blog",
        "description": "A blog post about...",
    }
}

Enabling Markdown Output

Add to settings.py:

RS_TRAFILATURA_MARKDOWN = True

This populates item["extraction"]["content_markdown"] with GitHub Flavored Markdown.

Filtering by Page Type

The page type classification lets you route items differently based on what kind of page they are:

class ContentSpider(scrapy.Spider):
    name = "content"
    start_urls = ["https://example.com"]

    custom_settings = {
        "ITEM_PIPELINES": {
            "rs_trafilatura.scrapy.RsTrafilaturaPipeline": 300,
            "myproject.pipelines.PageTypeRouter": 400,
        },
    }

    def parse(self, response):
        yield {"url": response.url, "body": response.body}
        for href in response.css("a::attr(href)").getall():
            yield response.follow(href, self.parse)

# myproject/pipelines.py
class PageTypeRouter:
    def process_item(self, item, spider):
        ext = item.get("extraction", {})
        page_type = ext.get("page_type", "article")

        if page_type == "product":
            # Save to products table
            save_product(item)
        elif page_type == "forum":
            # Save to discussions table
            save_forum_post(item)
        elif page_type == "article":
            # Save to articles table
            save_article(item)
        else:
            # Default handling
            save_generic(item)

        return item

Filtering by Extraction Quality

Drop items where extraction quality is low:

class QualityFilter:
    def process_item(self, item, spider):
        ext = item.get("extraction", {})
        quality = ext.get("extraction_quality", 0)

        if quality < 0.5:
            raise scrapy.exceptions.DropItem(
                f"Low extraction quality ({quality:.2f}): {item['url']}"
            )

        return item

Add it before the router:

ITEM_PIPELINES = {
    "rs_trafilatura.scrapy.RsTrafilaturaPipeline": 300,
    "myproject.pipelines.QualityFilter": 350,
    "myproject.pipelines.PageTypeRouter": 400,
}

Exporting to JSON Lines

Scrapy's built-in feed exports work out of the box:

scrapy crawl content -o output.jsonl

Each line in output.jsonl will contain the full item including the extraction dict. You can then process it with any tool that reads JSON Lines.

Performance

rs-trafilatura extracts in ~44ms per page via compiled Rust (PyO3, no subprocess). On a typical Scrapy crawl, extraction adds negligible overhead compared to network latency. The pipeline processes items synchronously in the Scrapy reactor thread, but since extraction is CPU-bound and fast, it doesn't block the download pipeline.

For very high-throughput crawls (1000+ pages/second), consider running extraction in a separate process and communicating via Scrapy's item pipeline.

Items Without HTML

If an item doesn't have body or html, the pipeline passes it through unchanged:

# This item has no HTML — pipeline ignores it
yield {"url": response.url, "custom_data": "something"}
# → No "extraction" key added, item passes through as-is

How to Use rs-trafilatura with Firecrawl

Murrough Foley — Fri, 03 Apr 2026 14:22:45 +0000

Firecrawl is an API service for scraping web pages. It handles JavaScript rendering, anti-bot bypass, and rate limiting — you send it a URL, it gives you back the page content. By default, Firecrawl returns Markdown. But if you request the raw HTML, you can run rs-trafilatura on it for page-type-aware extraction with quality scoring.

This is useful when you need structured metadata (title, author, date, page type) or when you want to know how confident the extraction is.

Install

pip install rs-trafilatura firecrawl

You also need a Firecrawl API key from firecrawl.dev.

Basic Usage

from firecrawl import FirecrawlApp
from rs_trafilatura.firecrawl import extract_firecrawl_result

app = FirecrawlApp(api_key="fc-your-api-key")

# Request HTML format (required for rs-trafilatura)
result = app.scrape("https://example.com/blog/post", formats=["html"])

# Extract with rs-trafilatura
extracted = extract_firecrawl_result(result)

print(f"Title: {extracted.title}")
print(f"Author: {extracted.author}")
print(f"Date: {extracted.date}")
print(f"Page type: {extracted.page_type}")
print(f"Quality: {extracted.extraction_quality:.2f}")
print(f"Content: {extracted.main_content[:200]}")

The key is formats=["html"] — this tells Firecrawl to return the raw HTML alongside whatever else it produces. Without it, you only get Markdown and there's nothing for rs-trafilatura to extract from.

Why Not Just Use Firecrawl's Markdown?

Firecrawl's built-in Markdown output is good for articles. The difference shows on non-article pages:

Product pages: Firecrawl may include navigation, filters, and "related products" sections in its Markdown. rs-trafilatura recognises the page type and extracts just the product description, falling back to JSON-LD structured data when needed.
Forums: Firecrawl treats the entire page as content. rs-trafilatura identifies user posts and excludes voting controls, user profile panels, and moderation UI.
Service pages: Firecrawl may over-extract or under-extract multi-section layouts. rs-trafilatura's multi-candidate merge handles hero + features + testimonials + pricing sections.

The other advantage is the quality score. Firecrawl doesn't tell you how confident it is. rs-trafilatura's extraction_quality field gives you a 0.0–1.0 score so you can flag unreliable extractions.

Getting Both Firecrawl Markdown and rs-trafilatura Extraction

You can request both and compare:

result = app.scrape("https://example.com", formats=["html", "markdown"])

# Firecrawl's own Markdown
firecrawl_markdown = result.markdown

# rs-trafilatura extraction
extracted = extract_firecrawl_result(result, output_markdown=True)
rs_markdown = extracted.content_markdown
rs_quality = extracted.extraction_quality

print(f"Firecrawl markdown: {len(firecrawl_markdown)} chars")
print(f"rs-trafilatura markdown: {len(rs_markdown)} chars")
print(f"Extraction quality: {rs_quality:.2f}")

Batch Scraping

Firecrawl supports batch scraping. Combine it with rs-trafilatura for structured extraction at scale:

from firecrawl import FirecrawlApp
from rs_trafilatura.firecrawl import extract_firecrawl_result

app = FirecrawlApp(api_key="fc-your-api-key")

urls = [
    "https://example.com/products/widget",
    "https://example.com/docs/getting-started",
    "https://example.com/blog/announcement",
    "https://forum.example.com/thread/help",
]

batch = app.batch_scrape(urls, formats=["html"])

for doc in batch.data:
    extracted = extract_firecrawl_result(doc)
    print(f"[{extracted.page_type}] {extracted.title} (quality: {extracted.extraction_quality:.2f})")

Note: the batch API returns a result object with a .data attribute containing a list of Document objects. The extract_firecrawl_result adapter handles both Document objects (v4) and legacy dicts (v1).

Options

# Stricter filtering — less noise
extracted = extract_firecrawl_result(result, favor_precision=True)

# More inclusive — captures more content
extracted = extract_firecrawl_result(result, favor_recall=True)

# Get Markdown output
extracted = extract_firecrawl_result(result, output_markdown=True)

What You Get

extract_firecrawl_result returns an ExtractResult with:

title, author, date — structured metadata
main_content — clean extracted text
content_markdown — GFM Markdown (when enabled)
page_type — article, forum, product, collection, listing, documentation, service
extraction_quality — 0.0–1.0 confidence score
language, sitename, description — additional metadata
images — extracted image data with src, alt, caption

How to Use rs-trafilatura with spider-rs

Murrough Foley — Fri, 03 Apr 2026 14:21:31 +0000

spider is a high-performance async web crawler written in Rust. It discovers, fetches, and queues URLs — but content extraction is left to you. rs-trafilatura slots in as the extraction layer, giving you page-type-aware content extraction with quality scoring on every crawled page.

Setup

Add both crates to your Cargo.toml:

[dependencies]
rs-trafilatura = { version = "0.2", features = ["spider"] }
spider = "2"
tokio = { version = "1", features = ["full"] }

The spider feature flag enables rs_trafilatura::spider_integration, which provides convenience functions that accept spider's Page type directly.

Basic: Crawl Then Extract

The simplest approach — crawl a site, then extract content from every page:

use spider::website::Website;
use rs_trafilatura::spider_integration::extract_page;

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://example.com");
    website.crawl().await;

    for page in website.get_pages().into_iter().flatten() {
        match extract_page(&page) {
            Ok(result) => {
                println!("[{}] {} (confidence: {:.2})",
                    result.metadata.page_type.unwrap_or_default(),
                    result.metadata.title.unwrap_or_default(),
                    result.extraction_quality,
                );
                println!("  Content: {} chars", result.content_text.len());
            }
            Err(e) => eprintln!("  Extraction failed: {e}"),
        }
    }
}

extract_page takes a &Page and returns Result<ExtractResult>. The page URL is automatically passed to the classifier for page type detection.

Streaming: Extract As Pages Arrive

For large crawls, you don't want to wait until everything is fetched. spider's subscribe channel lets you process pages as they arrive:

use spider::website::Website;
use rs_trafilatura::spider_integration::extract_page;

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://example.com");
    let mut rx = website.subscribe(0).unwrap();

    let handle = tokio::spawn(async move {
        let mut count = 0;
        while let Ok(page) = rx.recv().await {
            if let Ok(result) = extract_page(&page) {
                count += 1;
                println!("[{count}] {} → {} ({:.2})",
                    page.get_url(),
                    result.metadata.page_type.unwrap_or_default(),
                    result.extraction_quality,
                );
            }
        }
        println!("Extracted {count} pages");
    });

    website.crawl().await;
    website.unsubscribe();
    let _ = handle.await;
}

Each page is extracted in the spawned task as soon as spider fetches it. Extraction takes ~44ms per page, so it easily keeps up with typical crawl rates.

Custom Options

Use extract_page_with_options for fine-grained control:

use rs_trafilatura::{Options, spider_integration::extract_page_with_options};
use rs_trafilatura::page_type::PageType;

let options = Options {
    output_markdown: true,          // Get GFM Markdown output
    include_images: true,           // Extract image metadata
    favor_precision: true,          // Stricter filtering
    page_type: Some(PageType::Product),  // Force page type
    ..Options::default()
};

let result = extract_page_with_options(&page, &options)?;

if let Some(md) = &result.content_markdown {
    println!("Markdown:\n{}", md);
}

for img in &result.images {
    println!("Image: {} (hero: {})", img.src, img.is_hero);
}

If you provide url in the options, it takes precedence over the page URL for classification. If you don't, the page URL is used automatically.

Quality-Gated Processing

The extraction quality score lets you filter or flag low-confidence results:

for page in website.get_pages().into_iter().flatten() {
    let url = page.get_url().to_string();
    let result = extract_page(&page)?;

    if result.extraction_quality < 0.80 {
        eprintln!("⚠ Low confidence on {url}: {:.2}", result.extraction_quality);
        // Log for manual review, or route to a fallback extractor
        continue;
    }

    // Process high-confidence extractions
    save_to_database(&result);
}

On the WCXB benchmark, about 8% of pages score below 0.80. These are typically product pages with content in JSON-LD, forums with unusual markup, or service pages with highly distributed content.

What extract_page Returns

ExtractResult gives you:

Field	Type	Description
`content_text`	`String`	Main content as plain text
`content_markdown`	`Option<String>`	GFM Markdown (when enabled)
`content_html`	`Option<String>`	Extracted content as HTML
`metadata.title`	`Option<String>`	Page title
`metadata.author`	`Option<String>`	Author name
`metadata.date`	`Option<DateTime>`	Publication date
`metadata.page_type`	`Option<String>`	Detected page type
`extraction_quality`	`f64`	0.0–1.0 confidence score
`images`	`Vec<ImageData>`	Image URLs, alt text, captions

Why Not spider_transformations?

spider ships with its own spider_transformations crate that can convert pages to Markdown or plain text. It works, but it's a basic readability-style extractor without:

ML page type classification
Type-specific extraction profiles (forum comment handling, multi-section merge, JSON-LD fallback)
Extraction quality scoring
Structured metadata extraction from JSON-LD, Open Graph, and Dublin Core

rs-trafilatura gives you all of these. For article-heavy crawls, spider_transformations is fine. For crawls that hit diverse page types, rs-trafilatura produces substantially better results.

How to Use rs-trafilatura with crawl4ai

Murrough Foley — Fri, 03 Apr 2026 14:19:29 +0000

crawl4ai is an async web crawler built for producing LLM-friendly output. By default, it converts pages to Markdown using its own scraping pipeline. But if you want page-type-aware content extraction with quality scoring, you can swap in rs-trafilatura as the extraction strategy.

This tutorial shows how to set that up.

Install

pip install rs-trafilatura crawl4ai

If this is your first time with crawl4ai, you also need Playwright browsers:

python -m playwright install chromium

Basic Usage

rs-trafilatura provides RsTrafilaturaStrategy, a drop-in replacement for crawl4ai's built-in extraction strategies:

import json
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from rs_trafilatura.crawl4ai import RsTrafilaturaStrategy

async def main():
    strategy = RsTrafilaturaStrategy()
    config = CrawlerRunConfig(extraction_strategy=strategy)

    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com", config=config)

    data = json.loads(result.extracted_content)
    item = data[0]

    print(f"Title: {item['title']}")
    print(f"Page type: {item['page_type']}")
    print(f"Quality: {item['extraction_quality']}")
    print(f"Content: {item['main_content'][:200]}")

asyncio.run(main())

The extracted content is a JSON array with one item containing the extraction result. crawl4ai serialises it automatically — you just json.loads() the extracted_content field.

What You Get Back

Each extraction result is a dict with these fields:

Field	Description
`title`	Page title
`author`	Author name (if detected)
`date`	Publication date (ISO 8601)
`main_content`	Clean extracted text
`content_markdown`	Markdown output (if enabled)
`page_type`	article, forum, product, collection, listing, documentation, service
`extraction_quality`	0.0–1.0 confidence score
`language`	Detected language
`sitename`	Site name
`description`	Meta description

Enabling Markdown Output

Pass output_markdown=True to get Markdown alongside plain text:

strategy = RsTrafilaturaStrategy(output_markdown=True)
config = CrawlerRunConfig(extraction_strategy=strategy)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(url="https://example.com", config=config)

data = json.loads(result.extracted_content)
markdown = data[0]["content_markdown"]

This gives you GitHub Flavored Markdown with headings, lists, tables, bold/italic, code blocks, and links preserved.

Precision vs Recall

By default, rs-trafilatura balances precision and recall. You can tip the scale:

# Stricter filtering — less noise, may miss some content
strategy = RsTrafilaturaStrategy(favor_precision=True)

# More inclusive — captures more content, may include some boilerplate
strategy = RsTrafilaturaStrategy(favor_recall=True)

Crawling Multiple Pages

crawl4ai handles concurrency. rs-trafilatura runs extraction in a thread per page, so it doesn't block the async crawl loop:

async def main():
    strategy = RsTrafilaturaStrategy(output_markdown=True)
    config = CrawlerRunConfig(extraction_strategy=strategy)

    urls = [
        "https://example.com/blog/post-1",
        "https://example.com/products/widget",
        "https://example.com/docs/getting-started",
        "https://forum.example.com/thread/123",
    ]

    async with AsyncWebCrawler() as crawler:
        for url in urls:
            result = await crawler.arun(url=url, config=config)
            data = json.loads(result.extracted_content)
            item = data[0]
            print(f"[{item['page_type']}] {item['title']} (quality: {item['extraction_quality']:.2f})")

Each page gets classified into its type and extracted with the appropriate profile. A product page gets JSON-LD fallback. A forum thread gets comment-as-content handling. A docs page gets sidebar removal. All automatic.

Using the Quality Score for Hybrid Pipelines

The extraction_quality field tells you how confident rs-trafilatura is in its extraction. You can use this to build a hybrid pipeline — fast heuristic extraction for most pages, with LLM fallback for the hard cases:

from crawl4ai.extraction_strategy import LLMExtractionStrategy

async def extract_with_fallback(crawler, url, config):
    result = await crawler.arun(url=url, config=config)
    data = json.loads(result.extracted_content)
    item = data[0]

    if item["extraction_quality"] < 0.80:
        # Low confidence — use crawl4ai's built-in LLM extraction as fallback
        llm_config = CrawlerRunConfig(
            extraction_strategy=LLMExtractionStrategy(provider="openai/gpt-4o-mini")
        )
        result = await crawler.arun(url=url, config=llm_config)
        return result.extracted_content

    return item["main_content"]

On the WCXB benchmark, about 8% of pages score below 0.80. Routing just those pages to a neural fallback improves the overall F1 from 0.859 to 0.862 on the development set and from 0.893 to 0.910 on the held-out test set.

How It Works Under the Hood

RsTrafilaturaStrategy inherits from crawl4ai's ExtractionStrategy when crawl4ai is installed, so it passes the isinstance() check in CrawlerRunConfig. It sets input_format="html" which tells crawl4ai to pass raw HTML (not Markdown) and to skip chunking. The extraction runs in Rust via PyO3 — no subprocess, no binary to find.

rs-trafilatura: Page-Type-Aware Web Content Extraction in Rust

Murrough Foley — Fri, 03 Apr 2026 14:17:29 +0000

Web content extraction is the task of isolating the main content of a web page from its surrounding boilerplate — navigation menus, cookie banners, ads, sidebars, footers, and the other 80% of a page that isn't the actual content. If you process web pages at scale, you need it. Search engines use it for indexing. RAG pipelines use it to feed clean context to LLMs. SEO practitioners use it to approximate what Google sees when it evaluates a page.

The open-source ecosystem for this is strong. Trafilatura (Python), Readability (JavaScript), jusText, BoilerPy3 — all solid tools that work well on news articles and blog posts. On articles, the top systems all converge above F1 = 0.90. The problem is largely solved.

But the web is not just articles.

The Problem: Everything That Isn't an Article

When I was running SEO audits across thousands of competitor pages from search results, I kept hitting the same issue. The extraction tools worked on articles but fell apart on everything else:

Product pages encode descriptions in JSON-LD structured data rather than the visible DOM. Extractors that only parse what they see in the HTML miss the content entirely.
Forum pages wrap user posts in CSS classes like comment and reply — the exact patterns that article extractors have been trained to strip as boilerplate.
Service pages spread content across 5 to 15 independent <section> elements. An extractor that picks a single best node grabs the hero section and throws away the rest.
Documentation pages embed content alongside sidebar navigation, version pickers, and table-of-contents panels that extractors include as if they were content.

These aren't edge cases. In the WCXB dataset, 47% of pages are non-articles. And the failures are architectural — no amount of parameter tuning within an article-focused extractor can fix them. You need a different extraction strategy for each page type.

What rs-trafilatura Does Differently

rs-trafilatura is a Rust library that classifies pages into seven types (article, forum, product, collection, listing, documentation, service) and applies type-specific extraction profiles. The classifier is a three-stage pipeline:

URL heuristics — fast pattern matching on domain and path. URLs containing /forum/, /products/, or docs. subdomains resolve immediately. This handles ~63% of pages.
HTML signal analysis — JSON-LD @type values, Open Graph meta tags, DOM patterns like product grids or code blocks. This catches ~15% more.
XGBoost ML classifier — 200 trees over 181 features (DOM structure, vocabulary density, link ratios) for the remaining ambiguous pages. 86.6% accuracy.

Once the page type is known, extraction uses a type-specific profile:

Forums get comments_are_content = true and platform-specific selectors for XenForo, vBulletin, Discourse, phpBB, and others.
Service pages get multi-candidate content merging — selecting the top-scoring sections and concatenating them.
Products get JSON-LD structured data fallback when DOM extraction produces poor results.
Documentation gets framework-specific boilerplate removal for Sphinx, Rustdoc, MDN, and ReadTheDocs.

Extraction Quality Predictor

After extraction, an ML quality predictor estimates how reliable the result is. It's a 27-feature XGBoost regression model that predicts the expected F1 score. Pages scoring below 0.80 are candidates for LLM fallback — you can route them to MinerU-HTML or another neural extractor while keeping the fast heuristic path for the 92% of pages where it works well.

This is the basis for hybrid extraction pipelines: heuristic speed on most pages, neural quality on the hard cases.

Performance

Benchmarked on the Web Content eXtraction Benchmark WCXB — 1,497 pages from 1,295 domains across 7 page types:

System	F1	Speed
rs-trafilatura	0.859	44 ms/page
MinerU-HTML (0.6B)	0.827	1,570 ms/page
Trafilatura (Python)	0.791	94 ms/page
ReaderLM-v2 (1.5B)	0.741	10,410 ms/page

On a separate 511-page held-out test set (never used during development), rs-trafilatura achieves F1 = 0.893, confirming that results generalise. With a hybrid pipeline routing low-confidence pages to MinerU-HTML, the held-out F1 reaches 0.910.

On articles, every top system converges around F1 = 0.93 — the differences are marginal. The gap opens on non-article types:

Page Type	rs-trafilatura	Trafilatura
Article	0.932	0.926
Documentation	0.931	0.888
Service	0.843	0.763
Forum	0.792	0.585
Collection	0.713	0.553
Listing	0.704	0.589
Product	0.670	0.567

Forums: +0.207 over Trafilatura. Collections: +0.160. These aren't marginal improvements — they're the difference between getting most of the content and getting almost none.

Using rs-trafilatura

Rust

use rs_trafilatura::extract;

let result = extract(html)?;
println!("Title: {:?}", result.metadata.title);
println!("Content: {}", result.content_text);
println!("Page type: {:?}", result.metadata.page_type);
println!("Confidence: {:.2}", result.extraction_quality);

[dependencies]
rs-trafilatura = "0.2"

Python

import rs_trafilatura

result = rs_trafilatura.extract(html, url="https://example.com")
print(result.title, result.main_content, result.page_type, result.extraction_quality)

pip install rs-trafilatura

The Python package bundles four Rust crates into a single native extension: content extraction, page type classification, HTML cleaning, and Markdown conversion. No subprocess overhead — it's compiled Rust called directly from Python via PyO3.

Spider-rs Integration

use spider::website::Website;
use rs_trafilatura::spider_integration::extract_page;

let mut website = Website::new("https://example.com");
website.crawl().await;

for page in website.get_pages().into_iter().flatten() {
    if let Ok(result) = extract_page(&page) {
        println!("[{}] {}", result.metadata.page_type.unwrap_or_default(), result.metadata.title.unwrap_or_default());
    }
}

[dependencies]
rs-trafilatura = { version = "0.2", features = ["spider"] }

The Benchmark

WCXB is the dataset behind these numbers — 2,008 pages from 1,613 domains, with a dev/test split and ground truth annotations for all seven page types. It's the first benchmark that measures extraction quality across structurally different page types rather than just articles.

Website: webcontentextraction.org
Dataset: GitHub · Zenodo · HuggingFace
Rust crate: crates.io/crates/rs-trafilatura · GitHub
Python package: pypi.org/project/rs-trafilatura · GitHub

If you're extracting content at scale and need reliability beyond articles, give it a try. If you run it on your own data, I'd love to hear how it performs.

Murrough Foley — Technical SEO consultant and researcher

6 Things Dev.To Can Do To Improve Organic Traffic

Murrough Foley — Wed, 03 Aug 2022 08:45:00 +0000

Dev.to is a great site, with a great community. It's grown rapidly over the years from it's beginnings in 2016, amassing over 5.2 million backlinks from almost 65 thousand domains and cemented itself as one of the most authoritative community sites around coding on the net.

But why has it's organic traffic stalled and is it important?

Has Traffic Plateaued?
Is It Important?
So what's going on?
What Are The Issues & How To Fix Them?
#1 - Dev.to & The Bloated Index
#2 - Lonely Pages & Limited Internal Interlinking
#3 - Tag Pages Need Lovin Too
#4 - Spam Abuse & Content Moderation
#5 - Smaller Considerations That Don't Deserve A Number
#6 - Curated content to highlight the community posts

Notes, Exceptions and Caveats: If you find this beneficial and have an open-source or commercial project you want me to take a look at, connect with me, Murrough Foley, on Linkedin. I've moved notes and asides to the bottom. It's there for the curious and interested and linked via the numbers.

Dev.to is first and foremost a community site built with UX/UI in mind. Ben Halpern, Jess Lee, Peter Frank and the Forem team have done a fantastic job at building a safe platform where developers are encouraged to share their knowledge and discuss each other's content respectfully.

As a corporate enterprise, they must balance a host of aspects such as moderation, technical maintenance, features vs ease of management and the commercial side of the business too.

To be clear, I am an SEO guy and only looking at the Dev.to site in this article. I do not have the wider considerations of the open-source platform and it's commercial aspects in mind. For example, a change to dev.to for more visibility may not fit with the Forem commercial partners etc.

Has Traffic Plateaued?

I am only looking at organic traffic and have no inside baseball as to the direct or social traffic. Ahrefs traffic estimates aren't great but in general do a good job of showing the general trend and it does look like organic took a big hit during the January '22 roll back of the December '21 update.

Organic growth looks steady but it's not on that explosive path it's started on.

Why Is This Important?

I would say it's very important.

I know that Dev.to is an open-source project and that it's main focus is the community. But an essential part of driving the community is page views, interaction and discussion, it's the dopamine hit that encourages people to set aside time and share their endeavors. It's the ability to reach others with similar interests and share ideas.

For Dev.to it's also a force multiplier. Added visibility in the search engine results pages(SERPS) encourages more people to join and get involved.

Increased visibility for Dev.to puts it on the map(even more) for it's corporate partners and advertises the platform to new parties, all without a major marketing drive.

So what's going on?

The site is making good progress and has accrued a good number of quality links in recent months.

But there are some SEO fundamentals that are holding the site back.

Changing the first two and a half of these issues should see a great improvement and put the site back on to that original trajectory.

Okay get on with it

1 Dev.to & The Bloated Index.

Problem: A serious problem that Dev.to has is that there are too many low quality pages available for Google to crawl. These consist mainly of comment pages that contain little content and those same comments create a duplicate content issue as they appear on the article page too.

Wasting Google's Resources

Google's spiders are constantly roaming the net, visiting pages and making decisions on whether to index a pages content or not. Each time a Googlebot visits a site, there is a cost associated with that.

Now think of the exponential growth of content across the web and think of a bunch of engineers at Google, looking at the problem of resources and costs.

Google has long encouraged sites to have a slim and trim sitemap, with multiple statements about thin content ,low quality doorway pages and the need to no-index pages that aren't useful to the public.

So the idea here is pretty simple, be Google's friend and don't waste their resources and do your bit for the planet too.

Okay, so how bad is the problem?

Welllll, it's pretty bad.

Let's take a quick look at Dev.to in the Google Index using Google dorks/advanced operators.

The two operators we'll be using are site: and inurl:

What's in the index?

Using site:dev.to we can show the number of results in the index.

site:dev.to

This number is unreliable as mentioned in the notes, but it's useful nonetheless.

site:dev.to/t

Shows the number of tag pages in the index. It's important to be aware how many there are and I'll come back to this later.

But here is the kicker. Using the operator below, we are asking Google to show us the results from the site dev.to that include the word 'comment' in the url structure.

site:dev.to/ inurl:comment

Now the first 50 results are articles where the url contains the word "comment" but all the other results are user comment pages that follow the url structure below.

dev.to/user1/comment/1pp5k

Google estimates that there are 961,000 of these pages in it's index.

I've randomly picked a comment page out of the SERPS to give you an idea of what a low quality page that Google doesn't like looks like.

This page has about 400 words, no real topic, doesn't answer a question and holds no value without the article that in accompanies.

So if it's useless to the user(coming at it from the search results page), Google can't understand it, why is it in the index taking up space?

Which bring us neatly to the next issue these pages create.

Duplicate Content

All the comments on these discussion pages mirror the comments left under the articles.

Duplicate content has been a contentious issue in the SEO world for years and without rehashing the arguments, there are many(me) that believe it is disregarded or attributed to a single page and then ignored.

Perhaps these discussion pages are a great resource for the account holders to view all their discussions in one place but, there is no reason for them to be accessible via the search engine.

The Solution

The solution is easy to remedy, adding the noindex tag to the page or via the robots.txt file.

User-agent: *
Disallow: /*/actions_panel*
Disallow: /users/auth/twitter*
Disallow: /users/auth/github*
Disallow: /report-abuse?url=*
Disallow: /connect/@*
Disallow: /search?q=*
Disallow: /search/?q=*
Disallow: /search/feed_content?*
Disallow: /listings*?q=*
Disallow: /mod/*
Disallow: /mod?*
Disallow: /admin/*
Disallow: /reactions?*
Disallow: /async_info/base_data

Sitemap: https://dev.to/sitemap-index.xml

Would there be an immediate positive move in the SERPS?

It's difficult to say how long it would take for these pages to wash out of the index and to see the rewards. It depends on the crawl budget, how proactive the team are at getting the site recrawled and other factors. The Forem team might even have to wait for the next update(usually November or December) to see the benefit.

2 - Lonely Pages & Limited Internal Interlinking

Problem: Google rewards experts in a subject matter. Currently Dev.to has an enormous amount of high quality content around different technologies but the interlinking is extremely limited and inconsistent(the links generated to other articles are random to each user/session).

I can hear people saying, "What does that mean?".

Google rewards subject matter experts. When Google understands that a site or site section is about a singular topic and covers it to exhaustion, Google rewards that site with better positions and more traffic.

Let me take a mundane example from the SERPs to explain.

My choice of keyword is about as boring as you can get. The image above is taken from Ahrefs for the search term "best dehumidifier" using a UK proxy. I've highlighted the little affiliate site choosedehumidifier.co.uk in red. In green you can see how authoritative the sites are.

That site is ranking alongside huge national newspapers due to all it's content being about a single topic, and all the pages being interlinked in a coherent, hierarchical way.

So what's the situation with Dev.to?

Good question. Authors have the option to cross-link other relevant pages on Dev.to but there is no reminder(visual cue) and no incentive to do this.

The team have done a good job at prompting users to add "Alt text" to images as shown below.

Currently, cross-linking is reliant on the "Read Next" box/module at the end of each article.

And this has a serious issue too!

The "Read Next" module (probably) uses the tags and keywords of users to generate links to recent articles. Refresh the page in the same browser and by magic you'll get the same suggestions. Check the page in a private browser (with a fresh session) and you'll get a new set of suggestions.

All this is perfect for the user, but what about Google's crawler - Googlebot?

As it stands at the moment, every time Googlebot crawls the site, it will see different links to somewhat related topics but these change every time Googlebot comes back to take a look at the page.

How can Google understand these silos of inter-related content without permanent links telling it that these pages are related and share a similar topic?

Without permanent links between closely related articles, Google can never understand that dev.to has a huge amount of resources around Javascript tutorials or CSS tips and tricks or any of the incredible number of topics that are covered here.

It gains no bonus for all the specialised knowledge within each of the tags or topics.

The Solutions

Incentivise the user to cross link relevant articles on the site or other good resources.

Visually nudge them with a reminder. The team have done it with image alt-tags, why not prompt the user to reward other good content on the site with a link?

Well, that's all well and good pal, but what about the million great pieces already on the site?

Yeah, that is a conundrum. But there are a couple of ways you could tackle that.

I'm not going to go into them here as it would add a couple of thousand words and I'm already tired, but basically you are looking at categorising and subcatagorizing content in topic clusters and then adding permanent links in a second module/bot called related content.

After fixing this, how long would it take to see a benefit in search engine visibility?

Applying good interlinking is pretty fundamental to helping Google understand content, and it's relationship to other content. There should be an almost immediate improvement in rankings.

3 - Tag Pages Need Lovin Too

Problem: The tag pages have been built for UX purposes without consideration of how Google views them. They are great for getting new content crawled but due to the lack of content Google doesn't understand the context of these hub pages and doesn't rank them outside of "tag title +community".

The most used tag on Dev.to(I think) is javascript and the site has a huge amount of content around this topic and all it's subtopics with 62980 at the time of writing. So why does it perform so poorly in the SERPS?

Outside of the dynamic content, there are only about 250 words that are there consistently(each time Googlebot comes to visit) that give an indication as to the purpose of the page.

The Solutions

The solutions here really depend on the goal. A lot of written content that is not dynamic would broaden the pages visibility across 100s of keywords associated with Javascript.

Contextually cross linking of sub-topic hub pages would also be of great benefit.

So you might have an expandable box at the top of the page instead of the highlighted text, with an article linking to the front-end frameworks, Node, TypeScript and related tag pages.

These pages are the most linked pages internally (via tags)and could be huge drivers of organic traffic with a little love.

Seriously, Dev.to is on page 6. for the term "Javascript" Why?

It bounces from page 6 to page 3. But you are right, there must be more to it.

And there is.

Dev.to has thousands of posts about Javascript and each of these pages link to the hub page "https://dev.to/t/javascript
" via a tag just below the page title. So all that link equity from related pages should place it better than page 6 or 3.

When we link a page in html it's pretty simple stuff.

<a href="url">link text</a>

The url is the target page and the link text is the anchor. The link anchor gives Google context, it tells Google what the target page is about and this is one of the primary ranking factors.

**So what's the issue.

The issue is that Dev.to tag links do not contain anchors or any indication as to the topic of the target page. The text in our example is javascript, which is in a child span tag, so how can Google understand the link and it's topic?

Let's not forget that anchors are used for accessibility for those who are sight impaired too.

** If Dev.to changed how the tags are coded, would they see a jump in SERP position for these tag pages?**
Yes, it would be a change from Google's point of view as you are giving context to thousands of links. I would expect rankings to jump around for a few weeks before settling into a higher position.

** What about that thing about adding a hidden article in a dropdown box at the top of the tag pages, what would be the benefit of that?**
By adding more static content to the page and cross linking these tag pages, Google should start to rank the page for a whole host(several hundreds) of related keywords.

4 - Spam Abuse & Content Moderation

There are a lot of shady tactics in SEO. Some work for a while, then fall out of fashion as the search engines address the issue and sometimes, inexplicably start working again years later.

But one tactic that hasn't really gone away is webspam to a tier 1 property.

You are speaking gibberish again!

In the image below you can see a spam profile page that links to a Malaysian Lotto page.

The idea is to build low quality links to this spam profile page and "wash" the link equity through the authoritative domain. In this case Dev.to

In the image below, I've pulled some of the top linked pages as seen by Ahrefs.

Starting from the bottom in purple, we have the Lotto site who have built 1,106 do-follow links and 623 no-follow links from 390 different domains.

I'll not go through the others, and the issue is not as widespread as it is on other platforms.

Ideally, the solution would be to delete this accounts and redirect the links to a 410 server error and add the toxic links to a disavow file. There are ongoing arguments as to whether this is necessary though as Google is pretty good at ignoring spam links.

This tactic still works in some non English geos.

5 - Smaller Considerations That Don't Deserve A Number

There are a number of smaller areas that could be considered when looking to boost the site visibility on Google and the other search engines. I'll briefly touch on those here.

Different Languages

Due to an overzealous rate-limit by the dev.to server on my scans, I don't have hard figures on the level of multi-lingual or articles in other languages present on the site. It may not amount to much as a percentage but as a driver of future growth, dev.to could make a small change to help Google find these.

No hard figures but anecdotally I have come across quiet a bit of Spanish and Portuguese on the site with the Ahrefs traffic estimates showing about 5% from Brazil.

One way to help the search engines digest content on a multilingual site is to explicitly tell it the language and or region being targeted.

To my knowledge there is no way to add a hreflang tag to a Dev.to article in the markup and the html lang attribute is set to as the sitewide default.

*Would this be difficult to implement? Would it be worth it? *

This is one of those questions that I can't answer, every year the internet is a more diverse place with more and more users coming online across the globe. Perhaps different languages aren't a priority for the Forem team, or maybe they have a different solution. Smart people make smart decisions.

Other Considerations

I think this article is already long enough and I still have one other thing to cover.

6 - Curated content to highlight the community posts

Dev.to has some great stuff written each week by the community. It has a points system that allows users to highlight great articles to the admins.

And the community admins do a great job of picking out quality pieces every week. These are featured in roundups pulled out by @graciegregory and @michaeltharrington.

So if everything is great, what's your point?
These posts are put together for an internal audience. If you want to boost organic traffic you need to cater to an audience coming from search engines too.

I follow you, how would Dev.to do that
It's pretty simple really. You take a look at the great user generated content and create roundup posts around that tailored to search traffic.

Let me give you an example of two.

SVG & CSS Animation Tutorials [56 Cool Guides]

And you build an article that categorises and introduces the best dev.to has to offer on this topic.

Dev.to articles are often hyper focused on a problem or topic, but rounding up related articles into more generalised categories and serving them to Google would be targeting more exploratory search terms with much higher search volume.

Time for one more example.

Secure Node.js | 18 Tutorials To Harden Node Server

This title is going after the search term "secure node.js" and "harden node server"

I think you get the idea.

This is pretty time intensive and would require a lot of curation, how long would it take to see the benefits?

Yeah no pain, no gain. It would take time and some knowledge to curate the best content on dev.to and produce these types of round up posts. But you are broadening the types of search queries that dev.to ranks for, highlighting the great efforts of the community and creating a positive feedback loop of more fresh eyes on the site and content.

Conclusion

Okay that's it. If you read all the way to here, pat yourself on the back. If you understood everything give youreself a round of applause, SEO lingo can be difficult to get your head around.

Any corrections, comments or complaints reach out to M Foley on Linkedin. And if you have a web facing open-source project, a commercial project or question, please reach out. I'm always working.

Notes, Exceptions & Caveats

Dev.to server rate-limiting: There were a number of technical SEO checks I wanted to run through but the server kept throwing a 429 error every now and then. Things that would have been interesting to know include.

Amount of articles less than 400 words. (Thin content)
Number of articles in different languages.
Max crawl depth(maximum number of hops to a piece of content)
Number of canonicalised pages.
Average number of internal cross-linking per article(not in the "Read Next" module)
How series posts perform in the serps vs regular ole posts.

Ahrefs Traffic Estimates: It should be noted that all traffic estimate sites produce wildly inaccurate data at the best of times. I'd hazard to say that this is particularly true for dev.to due to the nature of the topic. I'd also say that a lot of organic traffic comes from highly specific long tail search queries which will never turn up on these tools.

Advanced Google Operators: The site operator has become less and less useful over the years and without access to the Search Console Data this proves the point but it does highlight the issue and so I've included it.

Different Languages: As a side note, cross linking topically relevant articles within the same language would probably be of more benefit then adding the hreflang tags. This of course would be it's own headache to implement.

Murrough Foley — Technical SEO consultant and researcher

How BBC News Uses This One Hack to Steal Hundreds of Thousands of Visitors Every Month

Murrough Foley — Fri, 31 Jul 2020 22:52:19 +0000

How BBC News Uses This One Hack to Steal Hundreds Thousands of Visitors Every Month from Competitors

An awful headline, I know but it’s true. I was curious how the Corona Virus had effected traffic for some of the large media publications in the UK and whilst taking a look at the top 10 news sites, I came across something interesting.

Is the BBC using shady tactics to grab more organic traffic or is it an SEO quick fix for something that has gone unaddressed for years?

Anyone who juggles clients or who works for a large company knows that there is always work to do. Much of a task schedule is a matter of prioritisation and triage. What’s the most important thing, that needs to be done yesterday? Are the BBC using a trick to generate more organic visitors and if so, should the UK national broadcaster be operating in such a manner?

So what are BBC News doing to grab hundreds of thousands of extra visitors per month?

Ranking The Same Page Twice For Numerous Terms

The BBC’s server set up, has all of its category pages duplicated and ranking for multiple terms.

https://www.bbc.com/news//world
https://www.bbc.com/news/world

https://www.bbc.com/news//england
https://www.bbc.com/news/england

https://www.bbc.com/news//uk
https://www.bbc.com/news/uk

And as far as I can see, it’s been like this for a long time. Judging from the number of 301s shown in the Wayback Machine, it looks like they have tested changing things at various points.

The duplicated link is canonicalised to the main URL as shown below but still tagged index.

The canonical tag doesn’t stop them from ranking for some very large keywords. Below Ahrefs shows that the canonicalised version is sitting at position 6 for the keyword today’s news (70,000 monthly) in the UK.

The same goes for the larger keyword “world news” (222,000 monthly) in position 6.

On closer inspection, the BBC are taking up both position #1 and a lower position 5/6 for thousands of keywords using this trick/bug/exploit/mistake.

If I was doing SEO for the BBC, would I want to fix an issue that could be generating an extra 8–12% of traffic from all these searches? Probably not.

The canonical is supposed to indicate to the search engines, that one page’s content is similar to another’s and indicate the primary source to be ranked. Maybe when you reach the levels of trust and authority that the BBC.com has, things work differently.

As a public service, should the BBC be limiting the choice of the public by taking up two positions via this hack?

And can you do the same for your site? The only way to know for sure, is to test.

Also by Murrough Foley

SEO, Data Science & Correlative Analysis For Google Organic Traffic

Murrough Foley — Technical SEO consultant and researcher

11 Sneaky Negative SEO Attacks & How To Protect Against Them

Murrough Foley — Tue, 28 Jul 2020 20:53:51 +0000

Negative SEO is nothing new. As long as there have been ways to improve your position in the organic results, there have been malicious actors who target their competition with negative search engine optimisation techniques.

This article will cover the common types of SEO attack, one or two less common types and a couple of sneaky ways bad actors can disrupt your business.

It is important to note at the outset that the best ways to protect your sites or your clients businesses, is to be aware of these various different strategies and closely monitor what is going on.

If you think your site has been the victim of an attack, a good SEO specialist should be able to diagnose the problem quickly and most times, remediate the situation. But like all things, prevention is better than cure and some of the attacks listed below require security measures put in place at the outset.

Spam Links / Link Farms / Manual Penalties
Anchor Text Ratio Unbalancing
Purchased Links / Cheap PBNs
Link Removal
Email Blacklisting Or Reducing Sender Score
Hot Linked Images
Scraped Content
DDOS
Slow Loris
Poisoned Canonical
Straight Up Hacking

If good search engine optimisation is knowing what and where the boundaries are, then negative SEO is knowing how to push the target into and over those boundaries.

Strictly speaking, the term negative SEO should only really be applied to attacks that aim to reduce your SERP visibility, but I’ve come to look at it more holistically as any digital attack meant to disrupt your online business operations. That can include brand sabotage, some variations of hacking or disruption of communications. This view may seem excessively broad, but as we go through the examples, you may start to agree with me.

Spam Links / Link Farms / Manual Penalties

Spam links are low quality links that can be built to negatively impact a site’s SERP visibility. Within this category of attack, there’s a lot of scope to do damage in multiple different ways. But to understand spam links, it’s best to understand a little bit of the history.

During the hay day of automated tools when Blackhat SEOs were able to rank poor quality sites for competitive keywords with ease, Google needed a way to improve the quality of organic results and filter these automatically generated links out of their link index.

It’s my view that Google’s web spam team were ready to throw a machine learning algorithm at the problem but needed a data source to separate the good links from the bad.

Cue the arrival of Google’s disavow tool.

Google introduced the disavow tool under the auspices of giving webmasters the ability to remove poor quality(purchased, automatically built) links from negatively effecting their site.

It was the proverbial confessional box for any nervous webmaster and promised a swift return to page 1 for those that had sinned.

Couple this technical “Mea Culpa” machine with a hearty dose of manual penalties applied to both the most egregious of offenders and Google had the perfect mood of fear, uncertainty and doubt spreading to fuel submissions to the disavow tool.

This fear drove countless webmasters to use the tool and feed Google’s Machine Learning system.

Using this constantly growing bank of submitted links, built by tools such as GSA SER, Xrumer and Senuke, Google was able to identify and devalue these links. And pretty quickly these automated tools fell out of widespread use to all but the most creative users.

How the Disavow tool works is open to debate. Is it a complicated machine learning algo that identifies common variables from the submitted pages? Or is it a far more simple, weighted list of bad domains with poor user registration controls and moderation?

Due to the sheer amount of computing power needed to analyse the common factors of a “bad site”, and then apply these identified signals to the entire web, my money is on a simpler type of solution.

So when John Muller ** says;**

FWIW looking at the site, I don’t see any negative seo effects there. Like the others in the forums posted, most of those links have no effect at all — sites collect links from all kinds of weird & spammy places over time, we’re pretty good at ignoring them .

He is not completely lying. But he is doing his usual thing of telling half truths.

Whilst a massive number of low quality article type links, showing up one day in your backlink analysis tool might look shocking, it’s probably not going to do too much damage. You ought to still disavow them though.

Google can filter out many automated links such as those that turn up for purchase on link lists. However, the more creative people out there will have built their own engines to post on more niche areas or done in a way that blends in with user generated content much better. This can’t be filtered.

Modern use of low quality links for negative SEO is more likely to be much slower and stretched out over several months so as not to arouse suspicion. The aim is to take down your site and not alert you to the process so that you will take action.

A large part of it is manipulating the link anchors that are pointed to your site.

Anchor Text Ratio Unbalancing

The Penguin update targeted webmasters who had over optimised anchor text, that is too say, webmasters who were gaming the system by using targeted key phrases in their anchor text and spawned the use of the full stop and coma as an anchor as SEOs tried to bring things within acceptable levels.

Unbalancing anchor text happens when a malicious actor builds ‘good links’ with over optimised anchors to one(or all) of your pages. Initially this can look like a positive signal, your page is generating natural links but if not monitored can push you over the edge into a penalty.

Things get more complicated when you understand that anchor text ratios flow through do-follow links.

Situation 1 — New tier 1 exact match do follow links are built to your site. A good SEO can easily monitor new inbound links on tier 1 and the effects to anchor text. Then react and disavow links when he/she feels things are getting critical.

Situation 2 — Exact match anchors are built on tier 2 to do follow tier 1 links. This is more difficult to identify and deal with, as most backlink tools are just not set up to easily provide you with the anchor used on tier 2. So without spending the time to dig into your website’s tier 2 anchors, you may never know why your rankings have dropped.

How To Mitigate Or Prevent

The only way to prevent anchor text unbalancing is to regularly do anchor text audits that look at Tier 2 and maybe even Tier 3 anchors. This is a proactive step that most businesses won’t be able to afford.

Obviously Purchased Links

Buying links is against Google’s terms of service. Most search engine optimisation professionals will keep a wide berth of any site which is clearly selling “guest posts” or indicates that a post was sponsored.

A bad actor can use this to his/her advantage seeking out domains that have been obviously penalised and dropped dramatically in rankings to buy a link and point it at your site.

The other variation of this would be buying links on the cheapest PBNs available on Blackhat SEO marketplaces.

Private/Public Blog Networks

A PBN, unless built very carefully, will leave a footprint that Google is well versed in spotting, the simplest of which would be all the sites on the same IP address but there are many more.

If a large number of your links are coming from a PBN with an easily discernible footprint, your site may be at risk of a penalty.

The problem you may face with this is, that many PBN operators cloak their sites from the Backlink analysis tools such as Majestic and Ahrefs, so you may have to trawl through your Search Console data to spot this type of attack.

How To Mitigate Or Prevent

There is no way to completely protect yourself against this type of attack but there are a number of steps to mitigate the risk of a malicious actor being successful.

#1. Monitor your backlink profile and investigate any new links to see if they have natural anchors and check the quality of the domain.

#2. Use the disavow tool for anything that you are uncertain about.

Link Removal

As you may have guessed, links are a major part of how Google identifies and rewards good sites. And large chunks of a companies marketing budget goes into raising brand awareness and the natural links that accompany that spend.

Those links are hard earned and what makes this type of attack really sneaky is that you may never be aware it’s going on.

An attacker might set up a branded domain with a different TLD or CCTLD for the purpose of impersonating someone from your company and requesting good links be removed.

All the pages linking to your site can be pulled from the various backlink databases and contact details of webmasters can be scrapped or manually retrieved.

It’s at this point the negative outreach campaign begins from an email that may look official;

Hey {first-name/}

It’s Jane Bloggs from YourCompany. I just have a quick request. We are facing some SEO issue that, to be honest, I don’t really understand. Anyway, I’ve been tasked with removing some of our links that we’ve accumulated over the years. Can I get you to remove the link on the page {target-for-link-removal}. I know it would only take a minute and it would really help me out. Thanks!

Regards

Jane Bloggs

The emails don’t have to come from a similarly branded domain, but obviously it’s the kind of thing that adds some authenticity to the email request.

These link removal requests can also be perpetrated from social media accounts that impersonate your brand.

How To Mitigate Or Prevent

There is no way to completely protect yourself against this type of attack but there are a number of steps to mitigate the risk of a malicious actor being successful.

#1. The first would be to own all the top level domains that match your main domain. So if your site is greatcompany.com , then buy the .net, .org and the main country tld where you operate whether it be .co.uk or .ru.

#2. The next step would be claiming all of your brand accounts on social media. Having a legitimate account with a little bit of activity on each of the major social media platforms should be enough raise alarm bells if someone tries link removal outreach via a different account.

#3. Finally, adding some security to your email service such as SPF, DKIM and DMARC records will stop anyone from spoofing your company email addresses.

Which brings us on to email. Most people wouldn’t classify this as a negative SEO attack, but at one time, spam reports could have your server IP pulled.

Email Blacklisting Or Reducing Sender Score

Email is the life-blood communication of most businesses, it’s used for everyday back and forth, in the sales process, in the generation of leads and it provides delivery of all sorts of essential information between systems.

Disrupting a business’s communications could lead to a massive decrease in revenue and might not be identified until it’s too late.

One very important aspect that affects email deliverability, is a domain’s ‘sender score’. It’s calculated by the ISP (or Email Service Provider) based on a number of factors including email open rates, spam reports and . If an attacker can lower this score, your company’s emails will take a hit.

This can be done by sending spammy emails purporting to be from your domain.

Imagine hundred’s of thousands of emails being sent from your spoofed domain with subject lines like;

Buy Cheap C*alis — International Delivery!
College E**ays | Best Prices | Quick TAT
Beautiful Slavic M*il O*der Br*des — From The Russian Steppes To Your Doorstep

A sustained attack of this nature could get your domain blacklisted and stop all of your email reaching it’s intended target.

How To Mitigate Or Prevent

#1. The main way to prevent this type attack is by installing SPF, DKIM and DMARC records for your email server. You might also consider moving the email to a subdomain of your main domain.

#2. Checking your domains Sender Score and whether your domain can be found on the email blacklists might be overkill unless you notice issues.

This is not a common type of attack, that I am aware of but one that could be devastating to many companies. It’s likely that the issue wouldn’t be diagnosed for weeks and in the mean time, all those missed sales would be gone.

Setting up the needed records in your DNS and with your email provider shouldn’t be overlooked.

Hot Linked Images

Image hot linking is when another site uses your image on their site. More often than not, they will not have permission and the low quality site will look something like this with thousands of pages covering various niches.

The category pages are tagged ‘index’, whilst the individual pages for each image are tagged ‘no-index’. Very often these sites are built in subdomains of sites with tlds like .tk or .xyz and also on blogspot sites.

I’ve seen sites ranking locally and nationally in Ireland, the UK and the US powered mainly by these type of links.

The problem is that once your images are being hot-linked by a couple of these sites, you will get caught up in more and more over time. When your site tips over some unknown ratio, you’ll see rankings and traffic plummet.

Is this a directed negative SEO attack? In most cases, probably not. But it is an attack vector that can be prevented.

How To Mitigate Or Prevent

#1. Preventing hot linking of images is relatively simple. It just requires adding a little bit of code in your htaccess file.

RewriteEngine on
RewriteCond %{HTTP\_REFERER} !^$
RewriteCond %{HTTP\_REFERER} !^http://(www\.)yoursite.com/.\*$ [NC]
RewriteRule \.(gif|jpg|jpeg|bmp|zip|rar|mp3|flv|swf|xml|php|png|css|pdf)$ - [F]

You will of course need to change ‘yoursite.com’ to your own domain name.

#2. This blocks the hot linking of the images, but a more preventative method would be to block all unwanted bots in both your website’s htaccess/nginx.conf file and the robots.txt.

Blocking all malicious bots has a couple of benefits including limiting the amount information available to your customers and is always a good idea.

But this brings us neatly along to the next topic

Scraped Content

Imperva / Incapsula reported earlier in 2020, that up to 37% of all internet traffic is some form of automation. Personally, I’d like to see exactly how they arrived at that figure, but as proxy providers have entered the corporate world servicing big data, content scraping is becoming more pervasive.

Whilst content scrapers and other bots pollute your analytics, from a marketing perspective, it’s not the major problem that it once was.

Google, and the other search engines to an extent too, have become much better at assigning ownership to the original content author.

The main problem, arises if your site(or page) goes offline for a time. This could be down to server error, a misconfigured index tag, or unpublishing a page by mistake.

Given enough time, Google will reassign the content’s authorship to the next most authoritative source. In this case, a low quality scraper site would now own your content.

To check how many sites are scraping your content, grab a random sentence and Google it surrounded by double quotes.

How To Mitigate Or Prevent

A determined adversary will be able to circumvent any controls you try and put in place, but most content scraping bots are simple and not targeted.

#1. Block bots. Stoping bots that declare their useragent, is the first and most obvious step. This just requires editing your .htaccess or nginx.conf files. CCarter has a regularly updated bot list on BuSo, with Mitchell Krog being another good option.

#2. Invest in a Web Application Firewall(WAF) — Whether you opt for a paid service such as CloudFlare and AWS, or go for an open source solution such as ModSecurity, a firewall will prevent a large number of bots, hitting your sites.

#3. Modify your RSS Feeds — RSS feeds have a range of different uses but can be exploited by scraper sites. Modify your RSS feeds to serve only a summary that doesn’t include any hyperlinks.

#4. DMCA Request — If scraper sites are causing you issues, your last resort may be a DMCA request.

#5. Google Copyright Infringement Tool — Google provide a tool for copyright infringement, if your report is successful, the infringing party is notified via Google Search Console and the infringing page is delisted. This tool is open to abuse by bad actors and negative SEO.

(Distributed) Denial Of Service Attack

Picture this, it’s Black Friday, the busiest day of the year for online sales and your e-commerce site is under constant attack. A DOS attack will effect your site’s responsiveness and a slow server means visitors and customers will drop off like flies.

A sustained attack could overwhelm your server under the load of connections, robbing you of sales for hours.

Depending on the traffic to your site, prevention is probably better than cure.

How To Mitigate Or Prevent

Like other automated attacks, the best way to mitigate a DDOS is by paying for enterprise level protection.

#1. Invest in a WAF — A Web Application Firewall will protect your site against all but the most leveraged types of Denial of Service Attack.

The next type of attack is related to a DDOS but with a twist.

Slow Loris

A slow loris attack’s main aim is to slow down your server by limiting the number of connections available to real visitors and GoogleBot. The attacker send partial HTTP requests that are never finished, and one by one, takes over all your servers sockets.

If Googlebot is unable to fully crawl your site several time, your rankings will drop site wide.

Most modern Apache servers are configured to prevent this type of attack out-of-the-box and whilst I’ve heard of a modified version called Gloris used against Nginx servers, I’m not familiar with it.

Poisoned Canonical

Anyone who has bought domains on the secondary market is aware of how diligent you have to be before buying a dropped domain. You have to check the existing and historical links and anchors for indications of spam.

You have to eyeball the domain’s archived history in the Wayback machine to see if it was used to sell knockoff Nike’s or Ugg Boots at any stage and then sleuth for a missing year, just in case the spammers were blocking the Internet Archive’s bot “ia_archiver”.

But it’s par for the course that after doing all the checks, you’ll pick up a domain, load it up on a server, register it in the Search Console only to see a penalty of some sort appear after a few days. Or worse still, it’ll just never perform.

The point is that penalised domains are not uncommon. They can be bought and used for negative SEO. That might be to redirect toward a target’s site but more recently bad actors have been exploiting the canonical tag.

The canonical tag tells the search engines which is the master copy of a page. It combines similar pages into one in the eyes of Google including the links associated with each canonicalised page.

Using the canonical tag, an attacker can tell Google that your domain is the master copy for the penalised domain and Google will share the penalty with your site.

Straight Up Hacking

If someone is determined and patient, they will be able to breach your security at some stage. Whether this is a simple password are a 0day vulnerability. The key is not to leave the key in the lock. Take basic precautions.

Once an attacker has control of your site, there is a range of things they can do, from defacing the site, to adding links to gambling domains, or changing your robots.txt file to disallow all.

Conclusion

Whilst negative SEO attacks are unscrupulous, I don’t believe they are against the law, with the clear exception of hacking. Add to this the fact that they can be difficult to identify and when done right, impossible to trace and you have a recipe for disaster.

Most SEOs are not in the business of destroying the competition out of spite and want to compete on a level playing field. Negative SEO is far less common than you’d think.

If you are considering negative SEO as a strategy for beating the competition, understand that it’s not an exact science and you may end up strengthening the domain that you were trying to impact.

Also consider the moral issues, if you negatively effect someone’s business, you are not just hurting the owner, but also the employees and their families. Plot these effects on a timeline and you might be casting a very long shadow.

Murrough Foley — Technical SEO consultant and researcher

Increase Email Outreach Campaign Effectiveness With These Tips

Murrough Foley — Wed, 15 Jul 2020 18:50:00 +0000

There's nothing worse than spending all that time setting up your email campaign only to discover that nobody is opening your carefully crafted mail. So, where did you go wrong? Here's a list of simple steps and things to bear in mind before pressing the send button.

Check Your Domain Against Email Blacklists
Authenticate Your Email With SPF, DKIM & DMARC
Sender Policy Framework
DomainKeys Identified Mail
Domain Message Authentication Reporting and Conformance
Add A Feedback Loop To Your Email Header
Avoid Spam Traps
Use Trusted Email Servers
Manage Your Email Lists
Warm Up A New Email Account
Avoid Spammy Subject Lines & Content
Creative Subject Lines

Most of the following steps can be broken up into two main categories, technical issues, and content mistakes. Let's deal with the technical aspect first.

Check Your Domain Against Email Blacklists

If you've purchased a domain with a history, it's a good idea to check to make sure that it wasn't used for email spam in the past. This is a relatively easy step and just requires you to plug your domain name into one of the many email blacklist checkers that can be found online.

My favorite tool for this is MXTools, which checks around 100 different blacklists within seconds.

Anytime you see your email deliverability drop, the first check should always be the blacklists so you can rule it out and zero in what the problem is.

Authenticate Your Email With SPF, DKIM & DMARC

Just like regular post, protecting the privacy and ensuring the integrity of email is essential. And as the threats to that mail system have become more sophisticated, so to have the protocols that protect us.

Sender Policy Framework

Sender Policy Framework, was probably one of the first protocols used to authenticate the sender. Simply put, the SPF record allows the receiving email server to check that the sender is on a list of authorised IP addresses. You can think of this like checking a return address on a regular letter.

To set up an SPF record you'll need to add a TXT record to your DNS. SPFWizard is a handy little tool to assist in creating the TXT record.

DomainKeys Identified Mail

DomainKeys Identified Mail or DKIM is a newer and more robust way to ensure the authenticity of the sender and the integrity of the email contents and uses a public / private key cryptographic system. You can think of DKIM authenticated email as akin to the registered post.

Setting up and configuring DKIM can be a more involved process, depending on your email server set up, but essential can be broken down into the following steps.

Generate your key pair
Add the public key as a txt record in DNS
Generate and save the DKIM-Signature

Zerobounce have a great little tool to help generate the needed keys. Ideally, for best security practice, you should be doing this on your local machine.

If you are using GSuite for email, there's a pretty concise tutorial below.

Domain Message Authentication Reporting and Conformance

Domain Message Authentication Reporting and Conformance or DMARC for short, is the latest protocol and it relies on SPF and DKIM. You can think of it as the final check to see whether the email is coming from the verified sender, if anything is amiss, the receiving Email server will drop the message and let the proposed sender know of the issue.

Dmarc.org, have a great list of tools and resources to help with the process, but you shouldn't have too much difficulty with the process. You might consider setting up a specific email address to store and monitor the DMARC daily emails.

By setting up email authentication, you should improve mail deliverability, particularly for new domains. But this won't assure that all your emails get to where they are going.

Check Your Email Sender Reputation Score

Your email sender reputation score is assigned by each of the ISPs and indicates how trusted your email is. Each ISP will have different weighted values for different attributes giving you a different score for each ISP.

The sender reputation score is an essential part of email deliverability. Here are some services to check your email sender score.

Add A Feedback Loop To Your Email Header

Am email Feedback Loop or FBL is a system that reports when an email is marked as spam in a user's inbox. It can be used to keep an email marketing list clean by removing users who indicate they are not interested in your email.

This type of service is provided by many of the large ISPs and some of the more specialized services such as SendGrid have done all the heavy lifting for you.

NB Google offers their FBL to high volume senders.

Avoid Spam Traps

Spam traps are the honeypots of unwanted email. They are specifically set up email accounts that are only available to spammers who scrape email addresses from sites or use some other questionable way of compiling their lists.

To avoid spam traps, don't buy email lists. Make sure you use an email validation service before sending emails and clean your list every few months.

Use Trusted Email Servers

Google aren't the only game in town, but they are the largest with over 1.8 billion users worldwide and one recent report stating that they are responsible for around 43% of email opens.

Google trusts its own systems and some argue that by using Google's own email servers, you'll see a higher level of deliverability.

Personally, I have only used Google and AWS SES, so can't compare.

Other email providers include;

There are many more, each with their own unique offering that will suit different types of use cases.

Manage Your Email Lists

Managing your email list is an essential part of ensuring high deliverability. The more recipients who don't open your email or worse, consign it to the spam folder, the worse your future campaigns will be effected.

Managing your email list once a quarter should be part of any digital marketing calendar. The cleaner your list the better, but there are lots of ways to go about this task.

Removing anyone who hasn't interacted in the last 90 days
Remove duplicate email addresses
Remove emails that have a history of a 'hard bounce'
Remove any clearly spammy email addresses.

Many marketers will advise on sending a reengagement email before cleaning your lists.

Validate Your Email List Before Sending

Many email apps will have a function to validate the list of email addresses before sending. For larger lists, this can get expensive but is always a good idea.

There are a range of different services that offer this feature, some use an API to work with your existing mail application, others are baked into the service such as Mail Gun.

Warm Up A New Email Account

Any sudden spike in email activity will trip spam alerts. It's always best to start slow and build up the number of emails you send per day. This is especially important on a new email account on a brand new domain.

Historically, spammers would set up and start mass emailing from an account within hours. If you mimick this type of activity, even by mistake, you'll be subject to the same low rates of email deliverability.

Warm up the account by progressively increasing the number of emails you send each day and don't be overly aggressive.

Avoid Spammy Subject Lines & Content

This could be a blog post in itself. There are any number of key phrases which if included in your subject line may result in your email being filtered to spam. Take a look at your own inbox for things like;

I need your urgent reply
Win Big!!!
Free Discounts!
High DA Domains

Creative Subject Lines

A well crafted subject line will result in more email opens. These opens improve your sender reputation so care should always be taken.

Also, be aware that the Google email client shows the first couple of words of the email body, so you can often elicit curiosity within this space.

That's it for improving email deliverability, if you think I've missed anything, I'm always looking to learn so be sure to drop a comment.

Murrough Foley — Technical SEO consultant and researcher

An Almost Complete List Of SEO Tools

Murrough Foley — Sun, 05 Jul 2020 20:16:00 +0000

A List Of SEO Tools (2020)

There are a huge number of great paid tools on the market, from suites that try and do everything like SEMrush, to the highly specialised ones that focus on a single task such as Copyscape. I’ve collected a few here that are worth investigating for junior search engine optimisation professionals.

Keyword Research
Browser Extensions
Wordpress Plugins
Conversion Rate Optimisation
Email Tools
Rank Trackers
Automation
Link Analysis Tools / Backlink Checkers
Crawlers
Backlink Monitoring Tools
Content Analysis Tools
Productivity & Project Management Tools
OSINT & Research Tools
Citation Resources & Services
Proxy Providers
Content Delivery Networks
Search Engine Provided Tools & Services
Search Engine Tools / Hacks
Quick & Dirty Tools
Python Libraries

Keyword Research

The most important aspect of search engine optimisation. Without the correct targets, how can you have the correct goals. Keyword research should lead the site architecture and give you an in-depth insight into how to group your content. It will also indicate how best to interlink that content and uncover the most profitable keywords and phrases.

Keyword volume is one of the key metrics you’ll be using to priortise or rank your keywords but care should be taken. Many of the keyword research tools below just pull the data straight from Google’s Keyword Planner, this data is not reliable particularly when looking at keywords with low volume. Ahrefs is slightly more reliable as they have an internal process that references data from Alexa/Clickstream. But the important takeaway is that keyword volume is guideline and not a reliable metric of the traffic you can expect to see.

Keyword Sheeter

Originally under a slightly different spelling, this great tool *sheets* out keywords from a variety of sources and is a great resource when doing the time intensive process of keyword research.

You will need to filter and prioritise these results as the amount of keywords you’ll get back can be intimidating.

SECockPit

Swiss Made Marketing came out with this expensive tool in 2014/2015 and simplified the arduous task of completing comprehensive keyword research. Whilst I don’t use it so much anymore, it’s still worth a look.

Browser Extensions

Depending on what task you are doing the will be a number of add-ons or extensions that will give you more insight. So probably the most important extension is your extension manager, which will allow you to active and deactivate these plugins as needed. This is not an exhaustive list but more a place to get started.

SEO Quake

A long time staple of search engine optimisation folks, the browser plugin provides a wealth of information at the page and domain level. There are literally too many features to list. It can be used to tighten up content for on page SEO, is a handy way to pull the top 100 SERP results into a CSV file, plugs into many backlink checkers and many more.

Detailed — SEO Extension

Not quite a replacement for SEO Quake but a much more streamlined plugin with many of the same features. Things like the robots.txt, schema and the archive.org are only a click a way.

Detailed — Remove Bredcrumbs

Another one from Viperchill’s big boy pants company. This removes the breadcrumbs that obscure the url path in Google’s SERPS.

Browseo V3

This is not a browser extension but a customised browser that allows you to maintain multiple personas each with an individualized proxy attached. There are a large number of reasons why this can come in handy and whilst this is a great tool, the recurring fee makes it a little too expensive unless you have a particular use-case. And there are other options.

What Runs Where

What Runs Where gives a basic overview of the web technologies used to build a site or web-app. If you are checking the competence of your competition, this is a good place to start.

Builtwith

Built With Site Profiler provides a huge amount of useful data including connected websites that are using the same tags, redirected domains, a much more in-depth look at the technologies being used on the site and meta information.

Mangools SEO Extension

A like for like replacement for SEO Quake, you may prefer the interface. It’s not as feature rich with things like a like to domain whois and archive.org missing.

Disable Javascript

There will be multiple occasions when you’ll need to see how a page is rendered before javascript gets involved. This is a handy little switch that gives you an immediate view of the vanilla HTML and CSS.

SEERobots

I highly recommend this extension as checking the X-Robots tags is needed on an ongoing basis, whether you need to check if a link is on an no-indexed page, or you have correctly switched a transferred domain back to the live version.

MagicCSS

A handy little extension that allows you to edit the CSS on a live page.

Grammarly

Poor spelling and grammar is something that can be easily avoided. There are privacy concerns with this tool due to their rumoured deal with ClickStream but it does one thing very well.

Ghosterly

Whilst marketed as a way to block tracking scripts around the web, it’s also very useful for checking that your own Facebook pixel / Google Ads / Remarketing cookies have been installed correctly.

Wordpress Plugins

There is a sea of Wordpress plugins that have value, but ideally we should all be avoiding using plugins as much as possible due to their inherent security risk. One increasing trend is the use of plugins as a backdoor to place links on your site, so take care and do some research before installing something.

RankMath

After Yoast has proved themselves completely unreliable in recent years with bug after bug that negatively effected rankings, a better alternative is RankMath who are adding functionality to their core plugin rather than reducing features.

Redirection

Includes a host of useful features including 404 error log files from within WP, which allows anybody to find and fix misconfigured inbound links, and lost link juice.

WP Tag Manager

There are a number of ways to integrate Google’s Tag Manager into a WP site, the best option is to include the code in a child theme but this is plugin is useful if that’s not an option.

Conversion Rate Optimisation

Conversion rate optimisation or what HR like to advertise as “post click analysis” when done right, can have a huge impact on your ROI per visitor.

VWO

Feature rich and at a great price point, VWO allows A/B testing segmentation, mutually exclusive groups all based on an intuitive platform. Highly recommended for sites that have the needed traffic to gain useful insights. Customer support is also excellent.

Effective Experiments

Not very familiar with this platform yet but a number of people have recommended it. It’s on my to do list.

Hotjar

Handy platform particularly the heat mapping function allows you to pinpoint any CRO errors on specific pages. It has a generous free tier and should be implemented on any landing pages.

Email Tools

Email is the bread and butter of any digital marketer and is probably one of the biggest time-sinks there is, if not done right. Here are a couple of tools that are useful at reducing the amount of time you’ll spend on email.

Hunter.io

There are a lot of tools that provide a similar service with more being brought to market all the time. Hunter.io has a pretty reliable database of domain and email addresses and has a handy little tool baked in for deducing the email structure of a company.

GMass

One of the first mass emailing services I tried that allows you to use GMail’s email servers. When I used it, it wasn’t a mature product and lacked a number of important features, the most important being error checks. It was possible to send several hundred email(or thousand) by mistake, and not have a fallback for the first name. I’m sure it’s been improved since I looked at it.

StreakCRM

Streak CRM allows for open and click tracking, automation of leads through the sales process, integrates with Google Sheets and is pretty intuitive to use.

Despite the advertised limitations of the free plan, there is no limit on the number of open trackers per month, or at least there wasn’t in April of 2019 when I last tested.

Mailshake

Send email using Google’s trusted email servers allowing for a higher level of deliverability. It has integrated automated email follow-up and helpful functions that stop you emailing the same person multiple times.

Rank Trackers

Rank Trackers are pretty boring, generally you need them to do one thing and one thing well. There has been some innovation in this space, but not a whole lot.

SERPWoo

Is a not really a rank tracker, at least not in the conventional sense. It tracks the SERP and not just your position within it. This allows for detailed analysis on changes after updates, and investigation of competitors who are making gains.

Ahrefs

When you are tracking keywords for a large project conventional rank trackers can get prohibitively expensive. Ahrefs can give you a top down view of your site with delayed results. It does have an integrated “Project Tracker”, but like the more conventional trackers out there, this can get very expensive.

SEMrush

SEMrush tries to do it all and in some areas fails, the rank tracking part of their set up is quite robust and feature rich though.

SERPRobot

The best bang for your buck rank tracker out there. A no frills service that does one thing really well.

Automation

UBot Studio

Ubot studio is not cheap and to be honest is pretty buggy. It does give you the ability to automate almost any task quickly from building a twitter bot to manipulating RSS feeds. Over the years, I find I use it less and less and many things can be done with a rudimentary understanding of Python.

Scrapebox

Scrapebox has a bad reputation due to its overuse by comment spammers but it has a 101 small features that always come in handy including the quick url dedupe feature.

GSA SER

Not particularly relevant anymore but like Scrapebox, can still have its uses.

GSA Email Finder

Due to changes in EU legislation, scraping information of any kind is now illegal. A quick look at any server logs will show that this is not adhered to by many and enforcing the new law appears to be impossible, but the fact remains that tools such as GSA’s are now illegal.

Link Analysis Tools / Backlink Checkers

Backlink profile analysis is one of the fundamental aspects of SEO and a true understanding of why some sites rank and others tank often comes down to careful investigation of the a domain’s link profile. Whilst proprietary metrics such as Moz’s DR and Ahrefs DA give an indication as to a site’s authority, the devil is in the detail and you have to look at the links on tier 1 and tier 2 to really understand what is happening.

Ahrefs

Considered by many to be the best tool in the game. Whilst Ahrefs don’t have the biggest link index, they seem to be able to filter the low quality links better than the competition. New links appear sooner than the competition and the company have steadily improved the feature set over the years. Plus the interface allows you to drill down to get at useful information without exporting the data to a spreadsheet.

Moz Open Site Explorer

The industry standard for many years, Moz’s metrics were too easily manipulated and they really stopped trying in 2015/2016 allowing others to move ahead. They have an impressive link index but for everyday use, just don’t seem to find new links for months and bad data makes for bad decisions.

SEMRush

Have recently boasted of the largest link index in the industry, but again like Moz, new links just don’t appear in their system quickly enough to be relied upon. SEMrush may be a good option for small shop digital marketing firms with a wide range of tools, but link analysis and understanding a domain’s link profile, I personally find it difficult to work with.

Majestic

Majestic’s two proprietary metrics, TrustFlow and CitationFlow were modelled closely after one of Google’s original patents and give a useful view of the power and trust of a link or domain. It’s still open to manipulation but is more useful than many other proprietary metrics. Majestic’s link index is comprehensive and their crawler surfaces links quicker than many of the competition, even turning up links that Ahrefs fails to.

Search Console

Google’s search console contains a selection of links that Google’s crawler has come across but like most data Google provides is purposefully incomplete.

Linkody

Linkody is a meta type service that allows you to combine backlink sources such as MOZ and Ahrefs.

Link Research Tools

I haven’t used LRT’s platform as it’s always been outside of my budget. They do pull data from a number of sources and deduplicate the results. They have an excellent reputation with many and a great blog.

Netpeak Checker

Very useful tool, you plug in your API keys for the services such as Ahrefs, Majestic, etc and pull only the specific data you need.

Crawlers

Technical SEO is an essential part of any search optimisation role and it usually begins with a crawl.

Screaming Frog

The long time industry standard continues to add features as technical SEO evolves with recent integration of page speed tests, schema mark up and an update to its server log file analysis. Some argue that the UI could do with an update but why change when everyone is used to it as it is?

SiteBulb

SiteBulb has a similar feature set to Screaming Frog but gives you that information in a way that is easier to act upon. It’s also more beginner friendly, in that issues are categorised and ranked in “perceived importance”.

Xenu Link Sleuth

A bit of a throwback, this Win32 crawler is free and could still be useful to some.

Deepcrawl

Deepcrawl is an enterprise level SASS targeting sites with a footprint of larger than 500,000 pages. I have worked on sites of a max 380,000 pages and never had the need to employ Deepcrawl. It has a good reputation though.

Backlink Monitoring Tools

When you’ve worked hard to earn your links, you don’t want to give them up without a fight, or at least an email. Like all things nowadays, the trend is toward recurring fee SASS applications. This of course, is far from ideal and whilst applications benefit from a SASS type set up, this is unnecessary for backlink monitoring.

Inspyder Backlink Monitor

A great little tool that can be run when needed that checks backlinks and notifies you when you loose a link. You can set it up for tiered links too.

Content Analysis Tools

The term ‘Scientific SEO’ has been thrown around in recent years but it’s really just a sales pitch for an industry that has to juggle so many ranking factors. That’s not to say that content analysis tools aren’t important.

Content analysis allow you to statistically investigate the factors that correlate with a high position for a given search term. Peeling back some aspects of your page that you may have overlooked, that need improvement. More tools are coming to market all the time offering these features, but here are the most popular.

SurferSEO

Reasonably priced, SurferSEO has the most streamlined user interface that allows you to focus improving the content. It’s easy for writers to understand and make the most of and you still have the data when you need to look under the hood.

CoraSEO

CoraSEO the greatest amount of raw data. The developer has gone to some lengths recently to present that data in an easily digestible way. But you are still working with a Spreadsheet. And whilst you can gain the most insight from this tool, it’s not something you can hand off to a writer or junior SEO. The data has to be interpreted and the SERP has to be analysed in context.

Page Optimiser Pro

I was an early adopter of Pop after seeing the results Kyle Roof got with his now infamous local lorem ipsum test. The UI passed through several iterations as features were added and after trying SurferSEO, I made the switch.

Siteliner

This is a great tool to help quickly diagnose the amount of duplicate content on your site. You won’t need a paid account to quickly get an estimate of how serious an issue you have. Remember the more unique your page, the higher the chance that Google will index it and keep it indexed.

Copyscape

One of the first checks you perform on any content you receive from writers is a plagiarism check. Whilst there are several new services out in the last number of years, many of them use Copyscape’s API, so you might as well get the service from the source and save in the process.

Productivity & Project Management Tools

OmniFocus

One of the first checks you perform on any content you recevie from writers is a plagiarism check. Whilst there are several new services out in the last number of years, many of them use Copyscape’s API, so you might as well get the service from the source and save in the process.

OmniPlan

Asana

OSINT & Research Tools

Open source intelligence tools may seem like overkill but are important if you need to find a competitor’s PBN or want to map a business’s various web properties to understand the full breadth of their business.

TinyEye

Excellent reverse image look up service. Numerous applications.

DomainIQ

Serves some data on domains, not very reliable, the data should be verfied using other services.

ipfingerprints

Standard reverse IP lookup service

DomainTools Whois

Comprehensive whois service with paid historical data.

Citation Resources & Services

You may think that citations aren’t relevant for international SEO, but have a variety of uses. There are some countries, where you need a minimum number of links from local ccTLDs. They can also be used as pillowing links to dilute your backlink profile.

Brightlocal

Managed service based in the UK with a nice portal for keeping an eye on progress and indexation. Worth the money if you prefer a more hands off approach.

WhiteSpark

Similar to bright spark but with a greater database of niche specific directory sites. Won’t suite every time of business but things like dentists and lawyers are covered.

Proxy Providers

There are a variety of situations where you may need proxies. Describing the different types of proxies and their uses is outside the scope of this article.

Infatica

Residential and mobile proxies with a global network. Pricing is per GB and of course, not cheap.

BuyProxies

Haven’t used them in a number of years, they were the go to provider for a long time. Unaware of the proxy quality or reliability in recent years.

Content Delivery Networks

CloudFlare

Free service with an integrated WAF to protect against DDOS attacks. CloudFlare greatly simplifies the process of updating or switching servers with immediate DNS record updates.

They provide a free SSL cert but the increasing number of sites that use CloudFlare and the way the free SSL cert is implemented mean you are trusting them with your data.

Amazon Web Services / CloudFront

Enterprise level content delivery network that can greatly reduce a server’s time to first byte. Integrates with S3 and the Amazon WAF meaning faster and more secure sites, at a price though.

Incapsula / Imperva

I have yet to use this Incapsula / Imperva but I’ve heard good things and would like to take a closer look when the opportunity presents itself.

E-Commerce Tools & Resources

Out of the box, most e-commerce platforms are horribly optimised for SEO. The worst offender being WooCommerce.

CartFlows

Cartflows doesn’t solve all of WooCommerce issues, but it does remedy a couple of the major shortfalls of the default cart.

Search Engine Provided Tools & Services

The various search engines provide some useful tools that can be used in various ways for digital marketing. Everything from a customised search engine to notify you of when your brand is mentioned, to help identifying global trends can be found.

There is one caveat though and that is that the data should be taken as an indication rather than the cold hard truth. Google’s keyword volume is the prime suspect here. Google has no responsibility to provide accurate data and nor would it be in their best interests to provide anything more than an approximation.

Google Trends

A good way to identify seasonal trends for a topic or product. It’s always best to compare your target with a topic or product that you have hard data on so that you can infer the general levels of interest.

Google AdWords Keyword Tool

The source data for most keyword volume tools, but is wildly inaccurate. Keyword groupings can provide some insight.

Google Alerts

There are a number of SASS based projects that sell Google Alerts for ridiculous prices with a nice front end. Google Alerts allows you to set up notifications for keywords. When Google’s crawler encounters your keyword, you get an alert. Very handy tool to keep an eye on your brand name and competitors.

Google Analytics

Industry standard analytics program, very powerful but with a couple of missing features, or features that require a deep dive into the config panels.

Google Custom Search Engine

Google allows you to “create” your own search engine and only include your competitors. This helps to identify new posts or major changes within your niche.

Google Search Console

Allows for mining a plethora of information on your progress in organic search. It often turns up keywords that you’ll miss via traditional keyword research.

Bing Webmaster Tools

With less than 3% of the population using Bing, it’s ignored by most SEOs and digital marketers as the results are too easily manipulated. The webmaster tools used to turn up a number of inbound links that did not appear in Google’s offering and this data can come n handy when you are disavowing links and want the largest dataset to work with.

Searx

Not useful for SEO particularly but it’s a really interesting project. It’s meta-search engine that you self host. They have numerous engines that allow for a range of customisations plus integration of proxy support.

Search Engine Tools / Hacks

USearchFrom.Com

The dynamic nature SERP results mean that the Google results you are seeing, is not always the same as what your clients or competitors are seeing. Whilst there are ways around this, one quick hack to see the results in a different country is usearchfrom.com. The creator built it out of frustration with a similar site who failed to implement some basic but essential upgrades.

Penguin Analysis Tool

A handy little tool for identifying the nature of a penalty. Less useful in the last number of years with the rolling core releases.

Quick & Dirty Tools

Text Mechanic

Whilst many of the features of this web app can be found in a good text editor, the ease of use of text mechanic makes it a big winner. Started as a small weekend project years ago, the creator checked his analytics one day to the shock that if was seeing high traffic and that SEOs and web masters had come to rely on it. He has since monetised it and the restrictions are a bit of an annoyance but hey, still a great tool in a pinch.

Mergewords

Handy tool when doing Adwords or keyword research.

Httpstatus.io

When performing a 301, installing an SSL cert or a million other small tasks, you may need to check that all versions of a URL have been correctly configured. This little tool comes in handy.

Wordcounter.io

Another quick single use tool if you need to check word count on the fly.

Python Libraries

Python is a great language and one I ’m still learning.

CloudFail

Python script to find ip addresses hidden behind CloudFlare name servers.

Beautiful Soup

Worth taking a look at but not worth spending much time on. A pretty limited framework for scraping data.

Scrapy

Far more robust and feature rich library capable of querying APIs.

You can find Murrough Foley on Twitter and Linkedin or at his personal blog mfoley.ie