Fix the website

The stray noindex. The blocked AI crawler. The twin city pages. Caught weekly, ranked by damage.

Site Health sweeps up to 60 of your money pages every week for the technical problems that actually move rankings: HTTPS and redirect sanity, robots.txt - including whether you're blocking the AI crawlers that now send customers - noindex accidents, canonical problems, broken pages, titles and metas. Issues rank by severity, brand-new criticals raise an alert, and a manual run takes about 2 minutes.

RankLens Site Health screen - health score gauge with trend arrow, AI crawler access card showing GPTBot blocked, and issues ranked critical first

☝ One red row in the crawler card - GPTBot blocked - and the criticals sorted above everything else. That's your week's technical to-do, already prioritized.

AI access

The robots.txt from 2021 is blocking the customers of 2026

Four AI crawlers now matter to a local business: GPTBot (ChatGPT training), OAI-SearchBot (the one behind ChatGPT search), ClaudeBot and PerplexityBot. A blanket "block all bots" line a past developer added stays invisible until somebody checks - so every sweep checks.

Your robots.txt is evaluated the way Google evaluates it - longest match wins, Allow beats Disallow on ties - for Googlebot, Bingbot and all four AI bots. What we report is what the crawlers actually do, not what the file looks like.

  • One green "allowed" or red "blocked" row per AI crawler - no robots.txt spelunking
  • No robots.txt at all? Reported honestly: everything is allowed, and that's fine
  • Sitemap listed and llms.txt present are checked on the same card - with a generator for the llms.txt
SEARCH & AI CRAWLER ACCESS · ROBOTS.TXT
BLOCKED  GPTBot (ChatGPT training) - your robots.txt shuts it out sitewide.
THE OTHER THREE
ALLOWED  OAI-SearchBot (ChatGPT search) · ClaudeBot · PerplexityBot
THE FIX
One robots.txt edit. Next sweep, the row turns green.

Illustrative example - sample business.

The score

A 0-100 score with a formula we publish - and a memory

The health score isn't a vibe. Start at 100: each critical costs 20 points (capped at 60), each warning 4 (capped at 30), info items up to 5 total. Green at 85+, amber from 65, red below - with the delta since your last sweep right next to the gauge, so "better or worse" takes one glance.

Under the hood it's plain, careful HTTP - four concurrent fetchers, a 12-second timeout per page, redirects followed hop by hop. No paid APIs, no AI calls, no credits.

  • Every critical and warning files into your Actions list automatically, deduped by fingerprint - brand-new criticals also raise an alert
  • Click "Fixed" and the score recomputes on the spot; closed issues show in a green "Fixed since last sweep" list
  • Mark an issue "Intentional" and it's fingerprinted and never flagged again in any future sweep - until you restore it
MONDAY'S SWEEP · 2 CRITICALS
CRITICAL  Noindex on /water-heater-repair - your best service page is invisible to Google.
YOU CLICK "FIXED"
RESOLVED  Score recomputes instantly. The matching Actions card closes itself.
NEXT SWEEP
Trend arrow up. "Fixed since last sweep" shows the win in green.

Illustrative example - sample business.

What gets checked

Fifteen categories, ranked by what they cost you

Host & redirect sanity

All four versions of your domain get probed - https and http, www and bare. Two live, non-redirecting versions raise a duplicate-site warning, and redirect chains of 3 or more hops get flagged, every hop counted (followed by hand, up to 6).

Weekly sweep

Robots.txt & AI access

Google-style longest-match parsing for Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot - each AI bot gets its own allowed or blocked verdict, plus sitemap and llms.txt checks.

Weekly sweep

Indexability & canonicals

Noindex accidents on money pages, canonicals pointing at the wrong host, broken pages, and title and meta problems - the quiet mistakes that unrank a page without showing an error anywhere.

Weekly sweep

Doorway-page detector

Location pages that are the same template with the city swapped get caught: pages 70%+ similar after every city name is stripped out are flagged as near-duplicates, with an on-demand AI rewrite to differentiate the weaker page (you confirm before it runs).

Weekly sweep · rewrite on demand

Internal link plan

Orphaned money pages - zero internal links but earning impressions - become warnings; pages with one link and real traffic become notes. The plan prescribes up to 10 target pages with up to 3 source pages each, and the suggested anchor text is that page's best real search query.

Refreshes with the weekly sweep

Do the pages agree with each other?

Every fetched page is read together, not one at a time: one business type in the schema or three, city pages whose structured data never names their own city, phone numbers and addresses that differ between pages, breadcrumb trails that disagree, and templates that have drifted apart (a skip link on 3 of 40 pages, no main landmark on 3). The copy-paste mistakes a per-page checker cannot see.

Weekly sweep

Security headers & broken assets

Checks the five security headers scanners look for, and catches scripts and stylesheets that 404 together with the pages that still reference them. The sweep only ever requests pages and files the site links to itself, so it never looks like an attack to the host's firewall.

Weekly sweep

Trust, conversion & performance hygiene

Accreditation badges rendered as images that link nowhere, the same three photos on every city page, analytics installed with no phone or form events, response-time promises that differ between pages, images without dimensions, and CSS cached for a week with no version in the filename.

Weekly sweep

Schema, FAQ & llms.txt generators

Copy-paste LocalBusiness JSON-LD built from your verified profile data, FAQ markup drafted from the question-shaped searches your site already gets, and an llms.txt listing your top services and service areas by real clicks - no AI needed for that last one.

On demand
Illustrative scenario A plumber's old developer added a blanket AI-bot block to robots.txt in 2023, "to be safe." The first sweep shows GPTBot and PerplexityBot in red, plus two city pages that are 84% identical once the city names are stripped - classic doorway twins. One robots.txt edit, one AI rewrite of the weaker page, and the next sweep moves both into the green "Fixed since last sweep" list. Total effort: one evening. Found by hand: probably never.
Honest limit: the sweep checks up to 60 pages per run - your money pages, prioritized home first, then service and location pages, ranked by real 28-day impressions. It is deliberately not a 50,000-URL enterprise crawler: no sitewide 404 dumps, no crawl-budget noise, no 400-row CSV nobody reads. If an issue won't move rankings for a local business, it doesn't make the list.

The numbers

Small sweep. Sharp teeth.

Up to 60money pages per sweep - home and service pages first, by real impressions
~2 minfrom "Run sweep" to a scored, severity-ranked report
Weeklyautomatic re-sweep - no credits, no separate audit tool

Find out what's quietly broken - this week, not next quarter

Site Health, the internal link plan and the schema generators are all in the $69 flat plan. No site-audit add-on, no credits.

Start free - 5 days on us or see pricing →