robots.txt generator
Decide what search engines and AI bots can see on your site. Ready-made templates, separate settings for ChatGPT, Claude, Gemini and Perplexity, a live preview and warnings about costly mistakes.
Free for signed-in readers · your file is ready instantly
Generator
Your robots.txt file
Sign in to use the generator
Checking whether you are signed in…
# robots.txt — twojafirma.pl
# Generated on 2026-10-08 with the Mediarun.com generator
# Rules for all crawlers
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /search/
Disallow: /*?replytocom=
# AI bots — blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: CCBot
User-agent: Bytespider
Disallow: /
# Sitemap
Sitemap: https://twojafirma.pl/wp-sitemap.xml
How it works
Four steps to your file
Pick a template — WordPress, online store, Next.js — or start from an empty file.
Enter your domain, sitemap URL and the paths crawlers should not visit.
Choose which AI bots may read your site — including “search only” without model training.
Download the file and upload it to the root of your domain: yourdomain.com/robots.txt.
Questions and answers
Frequently asked questions
What is a robots.txt file?
It is a text file in the root of your domain that tells search engine crawlers and AI bots which URLs not to visit. Crawlers check it before visiting your site.
Does robots.txt protect a site from bots?
No. It is a request, not a safeguard — well-behaved crawlers follow it, but the file does not block access. Protect confidential content with a login, and set up server-level blocking with your hosting or CDN provider.
What does “search only” mean for AI bots?
We block bots that collect content to train models (e.g. GPTBot, ClaudeBot, Google-Extended) and keep AI search crawlers and user-requested fetching (e.g. OAI-SearchBot, ChatGPT-User, PerplexityBot). Your site can then appear in answers with a link to the source.
Will blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether your content is used for Gemini models — Google Search keeps indexing your site with Googlebot.
Where do I put the file and how do I check it works?
The file must be available at yourdomain.com/robots.txt — separately for every subdomain. After uploading, open that URL in your browser and check the robots.txt report in Google Search Console.
More tools
You may also need
llms.txt generator
Show AI models what matters most on your site. A file in the llmstxt.org format — from your sitemap or by hand, with a live preview.
HEXX GEO+SEO
See how search engines and AI models read your site — a page-by-page analysis with recommendations.