Writing the robots.txt and .htaccess Rules That Actually Control AI Bots

A wine estate that wanted to be findable, not scraped bare

A small winery's marketing manager had two conflicting goals: she wanted AI answer engines to be able to describe the tasting room's hours and wine list accurately when asked, but she didn't want a bulk AI training crawler vacuuming up every photo and blog post on the site to train an unrelated model with no attribution back. These are two different technical decisions, and conflating them is the most common mistake in this area — 'AI bot' is not one category of behavior.

Robots.txt basics for named AI user-agents

Robots.txt is a plain text file at the root of a domain that names user-agents and tells them which paths they may or may not fetch. It's advisory, not enforced — well-behaved bots respect it, and the major AI companies have published documentation for their own crawlers' user-agent strings specifically so site owners can target them. A block for a specific bot looks like this in structure: a User-agent line naming the exact bot string, followed by one or more Disallow lines.

• GPTBot — OpenAI's crawler used to gather training data
• ClaudeBot and anthropic-ai — Anthropic's crawlers, with different bots for training data collection versus real-time retrieval when a user asks Claude to browse a page
• Google-Extended — a control specifically for whether Google can use your content to train Gemini models, separate from the standard Googlebot used for search indexing
• PerplexityBot — used for Perplexity's live answer retrieval

Blocking a training-data crawler while leaving a retrieval-time crawler alone is exactly the winery's situation: disallow the bot associated with bulk model training, and leave the bot associated with real-time question answering untouched, since blocking the latter means the AI simply can't describe the business accurately when asked.

Where .htaccess comes in

Robots.txt only asks politely. For actual enforcement — blocking requests before they're served, rate-limiting a bot that's hitting the server too aggressively, or blocking a bot that ignores robots.txt entirely — the enforcement layer lives in server configuration, on Apache sites typically in .htaccess. A rewrite rule can match the User-Agent HTTP header against a bot's known string and return a 403 response instead of serving the page.

1. Identify the exact User-Agent string the crawler sends, from that provider's published documentation, not a guess
2. Write a RewriteCond that matches HTTP_USER_AGENT against that string, case-insensitively
3. Follow it with a RewriteRule that returns a 403 Forbidden for matching requests
4. Test the rule with a curl request that sets a matching User-Agent header before assuming it works — a UI-based check won't reveal server-level blocks
5. Re-check periodically, since AI companies do occasionally rotate or add new crawler identities

Robots.txt is a note left on the door. .htaccess is the lock. Most sites only need the note — but it's worth knowing where the lock is.

The llms.txt question

A newer, unofficial convention called llms.txt proposes a separate file summarizing a site's key pages specifically for large language models to consume efficiently, distinct from the search-oriented sitemap.xml. Adoption among AI companies is inconsistent and unconfirmed as a ranking or retrieval factor by any major provider, so it's reasonable to treat it as a low-cost experiment rather than a required piece of infrastructure — worth adding if there's spare development time, not worth prioritizing over getting robots.txt and core schema right first.

A sane default policy

For most small businesses across NetWebMedia's service niches — home services, healthcare, real estate, hospitality — the practical default is: allow retrieval-time bots so AI assistants can answer questions about the business accurately, disallow bulk training crawlers on original creative content like blog posts and photography if that matters to the business, and never rely on robots.txt alone if a bot's behavior actually needs to be stopped rather than just discouraged. NetWebMedia configures this per-property as part of standard technical setup; see https://netwebmedia.com/services.html for scope, or check current crawler directives on any live site with the audit tool at https://netwebmedia.com/free-audit.html.

← Back to all articles