Why a Crawler Purpose-Built for AI Is Different

Most site crawlers were designed to be used by humans — you run a crawl, get a dashboard full of issues, triage them manually, and work through a fix list. They’re powerful tools, but they’re built around the assumption that a person is sitting in front of a screen making decisions.

FreeCrawl is built around a different assumption: that an AI is going to be doing the analysis. Rather than producing a fixed set of reports, it exposes 96 individual MCP tools that Claude can call selectively, in any combination, based on what the analysis requires. It’s the difference between handing someone a pre-printed report and handing them direct access to the database.

This means Claude can do things that no conventional crawler UI supports — like combining crawl data with GA4 conversion rates and GSC rankings in a single analysis, or identifying the specific technical issue that’s causing a cluster of pages to underperform organically.

Why not just use Screaming Frog?

Screaming Frog has been the industry standard for as long as I can remember. However, in the article I decided to try something different and I’ve been really impressed with the results. Not only is FreeCrawl built with this purpose in mind, it’s also completely free.

If you’d rather be using Screaming Frog then you can enable the MCP server under Configuration > MCP Server in version 23 and above.

What FreeCrawl Can Do

The 96 available tools cover the full spectrum of technical SEO analysis:

• Crawl management — start, pause, stop and configure crawls; set crawl depth, include/exclude rules, crawl speed
• URL-level data — HTTP status codes, response times, redirect chains, canonical tags, robots directives, hreflang, structured data/schema markup
• Content analysis — title tags, meta descriptions, heading structure, word counts, duplicate content detection, spelling errors
• Link analysis — internal link structure, broken links, orphaned pages, link anchor text distribution, link depth
• Image analysis — missing alt text, oversized images, image formats
• Core Web Vitals — page speed scores via the PageSpeed API for crawled URLs
• Sitemaps — generate, validate, and compare sitemap contents against crawled URLs
• Analytics integration — pull GA4 and GSC data alongside crawl data for each URL
• Export tools — CSV, JSON, tabular and HTML reports

Setting Up FreeCrawl

FreeCrawl runs as a desktop application with an integrated MCP server. Download the latest version from freecrawl.net and install it on your machine. The installer handles the MCP server configuration automatically.

Once installed, open FreeCrawl and create a new project for your site. Set the start URL (your homepage or the root of the section you want to crawl), configure your crawl settings (I recommend starting with a crawl depth of 5 and following internal links only), and click Start Crawl. For a typical site of a few hundred pages, a crawl takes 5-15 minutes.

The MCP server runs in the background as long as FreeCrawl is open. To connect Claude to it, add the following to your claude_desktop_config.json:

{
  "mcpServers": {
    "google-analytics": { ... },
    "google-search-console": { ... },
    "freecrawl": {
      "command": "npx",
      "args": ["-y", "freecrawl-mcp"],
      "env": {
        "FREECRAWL_PORT": "3456"
      }
    }
  }
}

Restart the Claude desktop app and you’ll see FreeCrawl appear in your active MCP servers. Claude now has access to all 96 tools.

Running Your First AI Technical Audit

With FreeCrawl connected alongside GA4 and GSC, you can ask Claude to run a comprehensive technical audit in a single conversation. Here’s a prompt that demonstrates the power of having all three data sources available:

“Run a full technical SEO audit on the crawl data. Identify the top 10 issues by severity. For each issue, tell me how many pages are affected, give me three example URLs, explain why it matters for SEO, and suggest a specific fix. Cross-reference with Search Console to flag any issues that are affecting pages that currently rank in the top 20 for high-volume queries.”

Claude will work through this methodically, querying FreeCrawl for the technical data and GSC for the performance context, then synthesise a prioritised remediation plan specific to your site.

The Top Issues Report

One of the most useful starting points is the top_issues tool, which gives Claude a ranked summary of the most common problems found in your crawl. Ask Claude: “What are the top SEO issues on my site right now, and which pages are most affected?” It will pull this data from FreeCrawl, interpret it in the context of your GA4 and GSC data, and give you a prioritised action plan rather than a raw issue list.

Next up: Part 5 — Running Multiple Microsoft Clarity Projects, including the workaround for the single-project limitation that makes Clarity usable at scale.