Go-Live Clearance

Launch accident · robots.txt

Disallow: / in robots.txt

The second most common launch accident after noindex: Disallow: / in robots.txt tells every crawler to stay away from your entire site. It looks harmless in staging and becomes permanent SEO invisibility after launch. Paste your URL — we read robots.txt and stamp CLEARED / HOLD / DENIED.

How Disallow: / happens

These are the real patterns — not hypotheticals.

  • Starter template shipped with Disallow: /

    Many Next.js starters and CMS templates include Disallow: / for development. It gets committed and deployed to production unchanged.

  • app/robots.ts copied from a preview project

    The App Router robots.ts was written for a staging site. rules: { userAgent: '*', disallow: '/' } ships to production.

  • CMS or plugin override

    WordPress, Shopify, or a SEO plugin set 'Discourage search engines' during setup. The checkbox is easy to miss in production settings.

  • Manual edit to block AI crawlers gone wrong

    Adding a Disallow rule for GPTBot or CCbot but accidentally applying it to User-agent: * instead of a specific bot.

  • Conflicting Allow + Disallow rules

    Allow: / and Disallow: / for the same user-agent. Crawlers interpret conflicts differently — some block, some allow. Unpredictable.

Fix Disallow: / — two approaches

Use the Next.js App Router API when possible; static file as fallback.

app/robots.ts — App Router (recommended)

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/' },
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

public/robots.txt — static file

User-agent: *
Allow: /

# Block AI crawlers (optional, specific user-agents only)
User-agent: GPTBot
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Block specific bots without blocking all crawlers

// app/robots.ts — multiple rules
export default function robots(): MetadataRoute.Robots {
  return [
    { userAgent: '*', allow: '/' },
    { userAgent: 'GPTBot', disallow: '/' },
    { userAgent: 'CCBot', disallow: '/' },
    {
      userAgent: '*',
      disallow: ['/api/', '/admin/'],
    },
  ]
}

Still check by hand

  • □Open yourdomain.com/robots.txt in an incognito window after deploy
  • □Search site:yourdomain.com in Google a few days after fixing — confirm pages appear
  • □Submit sitemap in Search Console to accelerate re-crawling
  • □If you block AI crawlers, verify that Googlebot is NOT affected

Related