Go-Live Clearance

Indexability · robots.txt

robots.txt Checker

The #1 silent launch killer: a preview template that ships Disallow: /. Paste your production URL — we read /robots.txt and stamp CLEARED / HOLD / DENIED with a fix you can paste.

Accidents we flag

These look harmless in staging and become permanent SEO debt after launch day.

  • Disallow: / on production

    Every crawler is told to stay out. Product Hunt and ads send traffic that never becomes indexed pages.

  • Missing robots.txt (404)

    Crawlers fall back to defaults. You cannot point to a sitemap or block private paths on purpose.

  • No Sitemap: line

    Search engines may find pages slowly. A one-line Sitemap: URL speeds discovery after launch.

  • Allow rules that contradict Disallow

    Conflicting patterns confuse crawlers and waste crawl budget on the wrong paths.

Copy-paste robots fixes

Prefer app/robots.ts on Next.js App Router; plain robots.txt works everywhere.

Next.js App Router — app/robots.ts

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/' },
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

Static public/robots.txt

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Still check by hand

  • □Confirm you are scanning the apex / www you will promote
  • □Open /robots.txt in an incognito window after deploy
  • □Submit sitemap in Search Console once the domain is live

Related