Tools
All toolsRobots.txt Generator
SEO tool

A correct robots.txt without the guesswork

One wrong line in this little file can hide a whole website from Google. Tick a few boxes here and the file is written for you in the right format.

Create my file Free. No sign in. Nothing you type leaves your browser.

Robots.txt generator

Start with the main rule, then add anything you want to keep crawlers out of.

What it does

The whole file from a few choices

Allow or block

Pick a main rule for all crawlers, then list the folders you want kept out, such as cart or search pages.

WordPress preset

One tick adds the standard rules for a WordPress site, keeping the admin area closed while leaving the part themes need open.

AI crawler rules

Another tick adds blocks for well known crawlers that collect text to train AI models.

Benefits

Why this small file matters

Crawlers read robots.txt before anything else on your site. It is your first instruction to them.

1

Crawlers spend time on the right pages

Blocking endless search and filter pages leaves more attention for the pages you want found.

2

Your sitemap gets discovered

A sitemap line tells every search engine where your full page list lives.

3

Fewer mistakes

The tool writes each rule in the correct form, with the slashes where they belong.

4

You see the danger

Choose to block everything and a warning appears, because that single line removes a site from search.

Step by step

How to use it

1

Choose the main rule

For nearly every public site, allow the whole site.

2

List what to block

One folder or page per line. Paste full addresses if that is easier.

3

Add your sitemap

Paste the full address of your sitemap file.

4

Upload the file

Save the text as robots.txt and place it in the top folder of your site, so it opens at yoursite.com/robots.txt.

A few tips

Avoiding the classic mistakes

Robots.txt is simple, and that is exactly why small slips cause big trouble.

Good to know

How robots.txt works

It is a request

Good crawlers follow it. Bad ones ignore it, so it is not a security tool.

One file per site

It must sit in the top folder and be named exactly robots.txt.

WordPress makes one for you

If no real file exists, WordPress serves a basic one. An SEO plugin lets you edit it.

Help

Questions people ask

What is a robots.txt file?

A plain text file that tells search engine crawlers which parts of a site they may visit.

Where do I put robots.txt?

In the top folder of your website, so it is reachable at yourdomain.com/robots.txt.

Does robots.txt remove a page from Google?

No. It stops crawling, not indexing. To remove a page from results, use a noindex tag and let Google crawl it.

Should I block AI crawlers?

That is your choice. Blocking them does not affect normal Google search results.

Need a robots file?

Make one now and copy it in seconds.