Robots.txt Generator

Generate a robots.txt file for your website. Choose the rules, then copy or download the result.

Default rule

Sitemap URL (optional)

Blocked paths, one per line

Use paths beginning with /. Leave empty if you do not need exclusions.

Options for specific crawlers

A specific crawler rule takes precedence over the default. The blocked paths above are also included in groups that allow access.

Googlebot

Googlebot-Image

Bingbot

Crawl-delay

Google does not support Crawl-delay. This option is for crawlers that support it.

robots.txt file contents

Generation happens in your browser. The file on your website is not modified automatically.

How to use the result

Review the rules before publishing and save the file at the domain root, /robots.txt. A robots.txt file controls crawler access; it does not protect private information or guarantee removal of a URL from Google.

Google robots.txt documentation

Robots.txt Generator

Robots.txt Generator

How to use the result

Found the issue? Let’s fix it.