robots.txt vs Meta Robots vs X-Robots-Tag: Which to Use When

The three mechanisms look interchangeable and are not. robots.txt decides whether a crawler is allowed to fetch a URL. <meta name="robots"> and the X-Robots-Tag HTTP header decide what a search engine may do with the response once it has fetched it. That distinction — access versus disposition — explains almost every robots-related bug in production, including the famous one where a page you explicitly blocked ends up in Google's index anyway. This guide covers what each layer actually controls, a decision table for the goals people usually have, real header syntax for non-HTML files, the full set of directive values worth knowing, and how to verify any of it.

Two different machines

Think of it as two gates in sequence.

Gate one — the fetcher. Before requesting a URL, a compliant crawler checks robots.txt for the same scheme, host and port. https://example.com/robots.txt governs https://example.com/* and nothing else — not the http version, not cdn.example.com, not port 8443. If the file disallows the path, the crawler does not make the request. It never sees a status code, a body, a header, or a meta tag.

Gate two — the indexer. Given a fetched response, the indexer reads X-Robots-Tag headers and, for HTML, <meta name="robots"> in the head. These say: index this or not, follow the links or not, show a snippet or not, cache a copy or not.

Gate two can only run if gate one let the request through. That is the whole trick, and it is why the two most common instructions people give — "block it in robots.txt" and "noindex it" — are not substitutes.

The classic failure: Disallow plus noindex

Here is the bug, in full:

# /robots.txt
User-agent: *
Disallow: /internal-search

<!-- https://example.com/internal-search?q=widgets -->
<meta name="robots" content="noindex" />

Googlebot obeys the Disallow, never fetches the page, and therefore never reads the noindex. But indexing does not strictly require fetching. If enough external or internal links point at that URL with meaningful anchor text, Google can index the URL itself — no title from the page, often a description reading "No information is available for this page", listed in Search Console under Indexed, though blocked by robots.txt. You have achieved the exact opposite of the goal, and you have also removed your ability to fix it, because the fix instruction lives on a page nobody is allowed to read.

The resolution is counterintuitive but simple: to keep something out of the index, you must let it be crawled.

# /robots.txt — remove the Disallow
User-agent: *
Allow: /

<meta name="robots" content="noindex" />

Let Googlebot fetch the URL, read the noindex, and drop it. Once the URL has been reprocessed and removed, you may then add a Disallow to save crawl budget — but be aware that doing so freezes the state: if the page is later reintroduced, or Google re-evaluates, it can no longer see the noindex. In most cases the cleaner long-term configuration is noindex alone, crawlable. Google reduces crawl frequency of persistently noindexed URLs on its own.

A second, related trap: noindex as a robots.txt line. Google announced in July 2019 that it would stop honouring unsupported directives — noindex, nofollow and crawl-delay in robots.txt — from 1 September 2019. If you have inherited a file containing Noindex: /private, it does nothing.

Decision table

That last row deserves emphasis. None of these mechanisms are security. robots.txt is a public file that advertises the paths you consider interesting; plenty of people read it precisely for that reason. If a URL must not be seen, put it behind a login.

The staging row is the same argument. A Disallow: / on staging fails the moment someone links to a staging URL from a public page, and it fails completely for crawlers that do not honour the file. HTTP basic auth returns 401 to everyone, which is unambiguous and cannot be misconfigured into visibility. It also avoids the migration disaster where staging's robots.txt gets deployed to production.

robots.txt syntax that people get wrong

User-agent: Googlebot
Disallow: /search
Allow: /search/featured

User-agent: *
Disallow: /cart
Disallow: /*?sessionid=
Disallow: /*.json$

Sitemap: https://example.com/sitemap.xml

Points worth internalising:

• Paths are prefix matches. Disallow: /search blocks /search, /searchresults, /search/anything. If you meant only the directory, write /search/.
• * and $ are supported by Google and Bing: * matches any sequence, $ anchors the end of the URL.
• Most specific rule wins, measured by path length — not file order. Allow: /search/featured (16 chars) beats Disallow: /search (7 chars) regardless of which appears first. On a genuine tie, the least restrictive rule wins.
• A crawler obeys exactly one group. Googlebot reads the User-agent: Googlebot group and ignores User-agent: * entirely. If your specific group omits a rule from the wildcard group, that rule does not apply. This bites people who add a group for GPTBot or ClaudeBot and accidentally grant them broader access than the wildcard group allows — see llms.txt explained for how AI crawler access is evolving separately from this file.
• Paths are case-sensitive; user-agent names are not.
• Status codes matter. A 404 on /robots.txt means "crawl everything". A persistent 5xx means Google may stop crawling the site entirely — a failing robots.txt endpoint is an outage, not a no-op. Google caches the file for around 24 hours and honours a 500 KiB limit, truncating beyond it.

X-Robots-Tag: indexing rules for things without a <head>

PDFs, images, spreadsheets, video files, JSON endpoints and plain-text files cannot carry a meta tag. The X-Robots-Tag response header does the same job at the HTTP layer, and it works for HTML too.

nginx — noindex every PDF, allow link-following:

location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

Apache — one rule for several file types:

<FilesMatch "\.(pdf|docx?|xlsx?|zip)$">
    Header set X-Robots-Tag "noindex"
</FilesMatch>

Express / Node — per-route:

app.get("/exports/:id.csv", (req, res) => {
  res.set("X-Robots-Tag", "noindex, nofollow");
  res.type("text/csv").send(csv);
});

Cloudflare Workers or any edge function:

const res = await fetch(request);
const out = new Response(res.body, res);
out.headers.set("X-Robots-Tag", "noindex");
return out;

You can target a specific crawler by prefixing the user-agent token, and you can send multiple headers:

X-Robots-Tag: googlebot: noindex, nofollow
X-Robots-Tag: bingbot: noarchive
X-Robots-Tag: unavailable_after: 2027-01-01T00:00:00+00:00

Rules without a user-agent prefix apply to everyone. Note that unavailable_after uses a date format Google can parse — RFC 822, RFC 850 and ISO 8601 all work — and results in the URL being dropped after that time, which is handy for time-limited offers and event pages.

One warning: check that your CDN or proxy actually passes the header through. Some caching layers strip or normalise unknown response headers, and some strip them only on cache hits, which produces the maddening intermittent case where the header is present when you test with cache-busting and absent for the crawler.

The directive values worth knowing

Both the meta tag and the header accept the same comma-separated vocabulary:

<meta name="robots" content="index, follow, max-snippet:-1, max-image-preview:large" />

index and follow are defaults; writing them changes nothing but does no harm. Bot-specific meta tags use the crawler name in place of robots:

<meta name="googlebot" content="noindex" />
<meta name="googlebot-news" content="nosnippet" />

When a page carries both a generic robots tag and a bot-specific one, the bot-specific tag wins for that bot — and it wins entirely, so anything you omitted from it reverts to default. As with robots.txt groups, specificity replaces rather than merges.

Two more nuances. nofollow in the page-level robots tag is distinct from rel="nofollow" on an individual link, and since 2019 link-level rel values are treated as hints rather than strict directives. And combining noindex with a rel=canonical pointing elsewhere is a contradiction — you are saying "this cluster's representative is X" while suppressing something in the cluster; see canonical tag mistakes for why that produces unpredictable results.

Testing and verification

See what the crawler sees, headers included:

curl -sD - -o /dev/null -A "Googlebot" https://example.com/exports/report.pdf

Look for X-Robots-Tag in the output. Sending the crawler's user-agent matters because some sites vary behaviour by UA — if the header only appears for Googlebot, that is worth knowing; if it disappears for Googlebot, you have a cloaking-adjacent bug.

Check the meta tag in the served HTML:

curl -s https://example.com/internal-search \
  | grep -Eio '<meta[^>]+name=["'\'']?(robots|googlebot)[^>]*>'

If this returns nothing but the tag appears in your browser's DevTools, it is being injected by JavaScript. Google does execute JavaScript and will see a client-side noindex after rendering — but rendering is queued and delayed, and a page that is index in the initial HTML may be indexed before the render pass changes it. Server-render robots directives.

Confirm robots.txt parsing rather than eyeballing it:

curl -s https://example.com/robots.txt | head -40

Then use Search Console's URL Inspection on the specific URL: it reports crawl-allowed status, the indexing verdict, and the exact reason. That tool tests against the live file Google has cached, which is the authoritative answer — local parsers and third-party checkers disagree with Google's own parser more often than you would expect, particularly around Allow/Disallow precedence. Google open-sourced its robots.txt parser in 2019, so if you want a local check, that library is the one to match against.

Confirm removal actually happened. site:example.com/path in Google is a rough indicator, not a guarantee, and it lags. Search Console's Page Indexing report with the "Excluded by noindex" and "Indexed, though blocked by robots.txt" filters is the real signal. Expect days to weeks, not hours — unless you use the Removals tool, which hides a URL for about six months and buys you time to make the permanent fix stick.

Where this fits in an audit

Checking these three layers is mechanical: fetch robots.txt and evaluate the requested path against it, read the response headers for X-Robots-Tag, parse the head for robots meta, and flag the combinations that contradict each other — a Disallowed URL carrying a noindex, a noindex page listed in the sitemap, a canonical target that is itself noindexed. SEO Fix Agent's Site Audit does that pass on the page you give it, fetching the page along with its robots.txt and sitemap so those cross-checks are possible; it audits one page per run, so point it at one URL per template. What you get is the list of contradictions, and the AI Prompt Studio turns that list into a fix prompt for Claude Code or Cursor — useful here mainly because robots directives usually live in a layout or middleware file, one edit away from being correct everywhere.

The mental model is the part that lasts, though. Ask "am I trying to control access, or control what happens after access?" Access is robots.txt. Everything else is a meta tag or a header on a URL the crawler is allowed to fetch. Once you hold those apart, the failure modes stop being mysterious.