What Is a Robots.txt File and How to Write One Correctly for Your Website

Your website has a tiny text file that most people never look at, yet it can quietly decide whether Google sees your pages or ignores half your site. That file is robots.txt. It takes two minutes to read, five minutes to write, and about thirty seconds to break if you copy the wrong line from a forum post.

This guide explains what robots.txt actually does, where it must live, the exact syntax rules, and copy-ready templates for WordPress, Shopify and static sites. We also cover the classic blunders that accidentally block CSS, images or entire sections from search engines, and how to test your file before it costs you traffic.

What Is a Robots.txt File?

A robots.txt file is a plain text file placed at the root of a domain that tells automated crawlers which parts of your site they may or may not request. It is the implementation of the Robots Exclusion Protocol, a convention that has been in use since 1994 and was formalised as an internet standard (RFC 9309) in 2022.

Two points beginners almost always get wrong:

• It is a crawling instruction, not a security barrier. Well-behaved crawlers such as Googlebot and Bingbot obey it. Malicious scrapers can simply ignore it. Never use robots.txt to hide sensitive data.
• It does not reliably remove pages from Google. Blocking a URL stops it being crawled, but if other sites link to it, the URL can still appear in search results without a description. To keep a page out of the index, use a noindex meta tag or an HTTP header, and make sure the page is not blocked in robots.txt so Google can actually read that instruction.

What robots.txt is good for

• Preventing crawler overload on large or resource-heavy sites
• Keeping crawlers out of infinite spaces such as faceted filters, internal search results or calendar pages
• Stopping the crawl of duplicate or low-value URL patterns like ?sort=, ?sessionid= or print views
• Pointing crawlers to your XML sitemap
• Managing AI crawlers separately from search crawlers

What robots.txt is not good for

• Protecting private areas (use passwords or server-side authentication)
• De-indexing pages already in Google (use noindex or the Removals tool)
• Fixing duplicate content between near-identical pages (use canonical tags)

Where Does the Robots.txt File Live?

The location is not flexible. The file must sit at the root of the host, in lowercase, named exactly robots.txt.

URL	Valid?	Why
https://example.com/robots.txt	Yes	Correct root location
https://example.com/blog/robots.txt	No	Subdirectories are ignored
https://shop.example.com/robots.txt	Yes, but separate	Each subdomain needs its own file
https://example.com/Robots.TXT	No	The filename is case sensitive

To check your own file, simply type your domain followed by /robots.txt in a browser. If you see a 404, you do not have one. That is not automatically a problem: no robots.txt means everything is crawlable, which is a perfectly acceptable setup for many small sites.

Robots.txt Syntax: The Four Rules You Need

A robots.txt file is made of groups. Each group starts with one or more User-agent lines, followed by the rules that apply to those crawlers. Blank lines separate groups. Lines starting with # are comments.

1. User-agent

Names the crawler the group applies to. The asterisk * means “every crawler that does not have its own group”.

User-agent: *
User-agent: Googlebot
User-agent: Bingbot

Important: a crawler obeys only the most specific group that matches it. If you write a Googlebot group, Googlebot ignores the * group entirely, including any sitemap-adjacent rules you assumed were shared.

2. Disallow

Blocks a path. The value is matched from the start of the URL path.

Disallow: /admin/          # blocks the whole folder
Disallow: /private-page.html
Disallow:                  # empty value = block nothing
Disallow: /                # blocks the entire site

3. Allow

Creates an exception inside a blocked area. When two rules conflict, the most specific rule (the longest path match) wins, regardless of order.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

4. Sitemap

Declares the absolute URL of your XML sitemap. It is independent of any group, so place it on its own line, usually at the bottom. You can list several. Robots.txt and SEO: Everything You Need to Know tackles the same question from another angle.

Sitemap: https://example.com/sitemap_index.xml
Sitemap: https://example.com/image-sitemap.xml

Wildcards and pattern matching

Symbol	Meaning	Example
*	Any sequence of characters	Disallow: /*?sort=
$	End of URL	Disallow: /*.pdf$
#	Comment, ignored by crawlers	# staging rules below

Other notes worth remembering:

• Paths are case sensitive. Disallow: /Blog/ does not block /blog/.
• Directive names are not case sensitive: disallow and Disallow both work.
• Google ignores Crawl-delay. Bing and Yandex still read it.
• The file should be UTF-8 encoded and Google only processes the first 500 KiB.

Copy-Ready Robots.txt Examples

Minimal, safe default (works for almost any site)

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

This allows everything and tells crawlers where the sitemap is. If you are unsure what to do, this is the version that cannot hurt you.

WordPress

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Disallow: /*?replytocom=

Sitemap: https://example.com/sitemap_index.xml

Notes:

• Do not block /wp-content/, /wp-includes/ or /wp-content/uploads/. Old tutorials still recommend it, and it blocks your CSS, JavaScript and images from Google’s renderer.
• If you use Yoast, Rank Math or SEO Framework, edit the file from the plugin’s file editor rather than uploading manually, so the virtual file does not conflict.
• /?s= blocks internal search result pages, which have no business in the index.

Shopify

Shopify generates a default robots.txt automatically and it is already well configured. You can customise it by editing the robots.txt.liquid template (Online Store > Themes > Edit code > Templates > Add a new template > robots.txt). A typical customised output looks like this:

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /orders
Disallow: /search
Disallow: /account
Disallow: /*?sort_by*
Disallow: /*?filter*
Allow: /products

Sitemap: https://example.com/sitemap.xml

Resist the temptation to block /collections/. On most stores those are your highest-converting landing pages.

Static site (HTML, Hugo, Astro, Next.js export, Jekyll)

User-agent: *
Allow: /
Disallow: /drafts/
Disallow: /thank-you/
Disallow: /*.json$

Sitemap: https://example.com/sitemap.xml

Place the file in your build output root (/public, /static or /dist depending on the framework) so it is deployed to the domain root.

Staging or development site

User-agent: *
Disallow: /

Use this on staging only, and add HTTP authentication as well. Then double-check this file never ships to production. Accidentally deploying it live is the single most expensive robots.txt mistake in existence.

Blocking AI crawlers while keeping search crawlers

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Google-Extended controls use of your content for Gemini training without affecting your Google Search rankings. Blocking Googlebot itself would remove you from Search entirely, so keep the two separate. For the wider picture, see Free Robots.txt Generator.

The Classic Mistakes That Block Google

1. Blocking CSS, JavaScript and images

Google renders pages like a browser. If your stylesheets or scripts are disallowed, Google sees a broken layout and may misjudge mobile-friendliness or page content. Never write lines like:

Disallow: /wp-content/
Disallow: /assets/
Disallow: /*.css$
Disallow: /*.js$
Disallow: /images/

2. Using robots.txt to de-index a page

If a URL is blocked, Google cannot read the noindex tag on it. The page can stay in the index as a bare URL forever. The correct sequence is: allow crawling, add noindex, wait for it to drop out, then block it if you still want to.

3. Forgetting the leading slash

Disallow: admin/ matches nothing. It must be Disallow: /admin/.

4. Assuming Disallow blocks a subfolder only

Disallow: /print also blocks /printer-reviews and /printing-services. Add the trailing slash when you mean a folder: Disallow: /print/.

5. Writing separate groups and losing rules

# Wrong
User-agent: *
Disallow: /private/

User-agent: Googlebot
Disallow: /internal/

Googlebot here does not respect /private/, because it matched its own group. Repeat every rule you need inside the specific group.

6. Blocking URL parameters used for pagination

A blanket Disallow: /*? often kills paginated category pages, filtered product listings and tracked links. Be surgical: block specific parameters, not every query string.

7. Serving the wrong status code

If /robots.txt returns a 5xx error, Google may treat your entire site as disallowed for a period. A 404 is far safer than a broken server response.

How to Test Your Robots.txt in Google Search Console

1. Open Google Search Console and select the correct property (remember: https:// and www variants are different properties).
2. Go to Settings > Crawling > robots.txt. You will see the fetched version, the fetch date, the HTTP status and any parsing issues.
3. Click the report to view the exact content Google has cached, and use Request a recrawl after you make changes.
4. To test an individual URL, paste it into the URL Inspection tool at the top of Search Console. The Coverage section will say “Crawl allowed: Yes/No” and name the blocking rule if there is one.
5. Check the Pages report for the statuses Blocked by robots.txt and Indexed, though blocked by robots.txt. The second one almost always means you used robots.txt when you should have used noindex.
6. Run URL Inspection > Test live URL > View tested page > Screenshot to confirm the page renders correctly with all its CSS and images.

Google typically refreshes robots.txt about once a day, so changes are not always instant. Plan for up to 24 hours before a fix takes effect.

How to Fix a “Blocked by robots.txt” Error

1. Inspect the URL in Search Console and note the rule Google reports as the blocker.
2. Open your robots.txt and find that line. Check for case sensitivity, missing slashes and unintended wildcards.
3. Decide whether the block is intentional. Cart pages, admin areas and internal search should stay blocked.
4. If it is unintentional, remove the rule or add a specific Allow line above it.
5. Save, clear any caching layer or CDN cache, and reload yourdomain.com/robots.txt in a private window to confirm the new version is live.
6. Request indexing on the affected URL and resubmit your sitemap.

A Quick Robots.txt Checklist

• File is at https://yourdomain.com/robots.txt and returns HTTP 200
• Filename is lowercase and plain text, not HTML
• No Disallow: / anywhere on the production site
• CSS, JS, fonts and image folders are crawlable
• Sitemap line uses a full absolute URL with the correct protocol
• Every specific user-agent group repeats the shared rules it needs
• Tested in Search Console with URL Inspection
• Re-checked after every site migration, theme change or CDN switch

FAQ

What is robots.txt used for?

It tells search engine crawlers which URLs on your site they are allowed to request. It is mainly used to manage crawl load, keep bots out of low-value or infinite URL spaces, and point crawlers to your XML sitemap. It is not a way to secure content or remove pages from search results.

Is robots.txt legal?

Yes, publishing a robots.txt file is completely legal and standard practice. It is a voluntary protocol rather than a legal contract: compliant crawlers such as Googlebot and Bingbot honour it, but there is no automatic legal penalty for a bot that ignores it. Some jurisdictions and terms-of-service agreements do treat ignoring robots.txt as evidence of unauthorised access, but the file alone does not enforce anything.

Where do I find the robots.txt file?

Type your domain plus /robots.txt in a browser. To edit it: on WordPress use your SEO plugin’s file editor or upload the file to the site root via FTP; on Shopify create a robots.txt.liquid template; on a static site place robots.txt in your public or build output folder. Anyone digging further should read What is robots.txt? The Essentials.

Do I need a robots.txt file at all?

Not strictly. If you have nothing to block, a missing robots.txt means the whole site is crawlable, which is fine. That said, having one avoids 404 noise in your logs and gives you a place to declare your sitemap, so most sites benefit from a minimal file.

Does robots.txt affect rankings?

Not directly. There is no ranking bonus for having one. It affects rankings indirectly: block an important page or its resources and that page cannot rank properly. Used well, it helps search engines spend their crawl budget on the pages you actually care about.

Can I block a specific bot only?

Yes. Create a dedicated group with that bot’s user-agent string, for example User-agent: GPTBot followed by Disallow: /. Remember that the bot will then follow only that group, so repeat any other rules it should obey.

How long does it take for Google to see my changes?

Google generally refetches robots.txt around once every 24 hours. You can speed things up by requesting a recrawl in the robots.txt report inside Search Console settings.

Final Word

Robots.txt rewards restraint. The shortest file that does the job is nearly always the right one. Start with a minimal version, block only what genuinely needs blocking, keep every rendering resource crawlable, and validate the result in Search Console before you move on. If you would like our team at PixelBright to audit your crawl setup and technical SEO foundations, get in touch and we will review your file line by line.