One wrong line in this little file can hide a whole website from Google. Tick a few boxes here and the file is written for you in the right format.
Create my file Free. No sign in. Nothing you type leaves your browser.Start with the main rule, then add anything you want to keep crawlers out of.
Pick a main rule for all crawlers, then list the folders you want kept out, such as cart or search pages.
One tick adds the standard rules for a WordPress site, keeping the admin area closed while leaving the part themes need open.
Another tick adds blocks for well known crawlers that collect text to train AI models.
Crawlers read robots.txt before anything else on your site. It is your first instruction to them.
Blocking endless search and filter pages leaves more attention for the pages you want found.
A sitemap line tells every search engine where your full page list lives.
The tool writes each rule in the correct form, with the slashes where they belong.
Choose to block everything and a warning appears, because that single line removes a site from search.
For nearly every public site, allow the whole site.
One folder or page per line. Paste full addresses if that is easier.
Paste the full address of your sitemap file.
Save the text as robots.txt and place it in the top folder of your site, so it opens at yoursite.com/robots.txt.
Robots.txt is simple, and that is exactly why small slips cause big trouble.
Good crawlers follow it. Bad ones ignore it, so it is not a security tool.
It must sit in the top folder and be named exactly robots.txt.
If no real file exists, WordPress serves a basic one. An SEO plugin lets you edit it.
A plain text file that tells search engine crawlers which parts of a site they may visit.
In the top folder of your website, so it is reachable at yourdomain.com/robots.txt.
No. It stops crawling, not indexing. To remove a page from results, use a noindex tag and let Google crawl it.
That is your choice. Blocking them does not affect normal Google search results.
Make one now and copy it in seconds.