Blocked by robots.txt: What It Means and How to Fix It

If you see “Blocked by robots.txt” in Google Search Console, it means Googlebot tried to crawl a URL but your website’s robots.txt file told it not to access that URL.

This is not always an error. Sometimes blocking certain URLs is intentional. The problem starts when an important page—such as a blog post, product page, category page, or landing page—is accidentally blocked.

Google explains that robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if Google discovers the URL through links or other sources.

What Is robots.txt?

The robots.txt file normally sits in the root directory of your website:

https://example.com/robots.txt

It gives instructions to search engine crawlers about which parts of the site they can or cannot crawl.

For example:

User-agent: *
Disallow: /admin/
Disallow: /private/

Here, crawlers are asked not to crawl URLs under /admin/ and /private/.

A simple mistake, however, can block much more than intended.

For example:

User-agent: *
Disallow: /

This tells crawlers not to crawl the entire website.

Why Does Google Show “Blocked by robots.txt”?

There are several common reasons:

1. The URL is intentionally blocked

You may have deliberately blocked login pages, admin areas, internal search pages, or other URLs that don’t need to be crawled.

In this situation, there may be nothing to fix.

2. An important directory was accidentally blocked

For example:

Disallow: /blog/

If your blog posts are inside /blog/, Googlebot won’t be able to crawl them.

3. WordPress or an SEO plugin changed the robots.txt rules

Changes made through WordPress, SEO plugins, hosting settings, or custom code can sometimes introduce unexpected rules.

Always check the actual robots.txt file rather than assuming what your CMS is generating.

4. A broad rule is blocking specific pages

For example:

Disallow: /category/

This could prevent Google from crawling category URLs that you actually want indexed.

How to Fix “Blocked by robots.txt”

Step 1: Find the affected URL

Open Google Search Console → Page Indexing and look for URLs reported as blocked by robots.txt.

You can also test an individual URL using the URL Inspection tool.

Google recommends checking whether Googlebot can access the page and reviewing your robots.txt rules when pages aren’t being crawled.

Step 2: Open your robots.txt file

Visit:

https://yourdomain.com/robots.txt

Look for Disallow rules that could be affecting the URL.

For example, if your page is:

and robots.txt contains:

User-agent: *
Disallow: /blog/

the page is being blocked.

Step 3: Remove or modify the incorrect rule

If the page should be accessible to Google, modify the rule.

For example, change:

User-agent: *
Disallow: /blog/

to something that doesn’t block your important blog content.

Be careful when editing robots.txt. A small change can affect hundreds or thousands of URLs.

Step 4: Check for noindex

This is an important distinction.

robots.txt controls crawling, while noindex controls indexing.

If your intention is to keep a page out of Google Search but still allow Googlebot to crawl it, use a noindex directive instead of blocking the page with robots.txt. Google specifically states that a noindex directive cannot be seen if the URL is blocked by robots.txt.

For an HTML page, this can look like:

<meta name="robots" content="noindex">

Step 5: Test the URL again

After fixing robots.txt, use URL Inspection in Search Console to check the URL again.

Make sure:

• Googlebot can access the URL
• The page returns a normal 200 response
• There isn’t an unwanted noindex
• The canonical URL is correct
• The page contains indexable content

Google’s technical requirements state that Googlebot must not be blocked, the page should work correctly, and it should contain indexable content.

Step 6: Request crawling

Once you’ve fixed the problem, use Request Indexing in URL Inspection if appropriate.

Keep in mind that requesting indexing doesn’t guarantee immediate indexing. Google still decides whether and when to crawl and index the page.

Don’t Make This Common SEO Mistake

One of the biggest mistakes is using robots.txt to remove a page from Google’s index.

For example:

User-agent: *
Disallow: /old-page/

This prevents Googlebot from crawling the page, but the URL itself may still appear in Search if Google discovers it elsewhere.

If you want a page removed from Google’s search results, allow Googlebot to crawl it and use noindex, or remove/password-protect the content when appropriate.

Quick Checklist

If an important page is showing “Blocked by robots.txt”, check:

• Is the URL supposed to be indexed?
• Is its directory blocked?
• Is there a broad Disallow rule?
• Is WordPress generating an unexpected robots.txt?
• Is an SEO plugin modifying the rules?
• Does the page have a noindex directive?
• Does the URL return HTTP 200?
• Is the canonical URL correct?
• Can Googlebot access the page in URL Inspection?

Final Thoughts

“Blocked by robots.txt” isn’t automatically a problem. For some URLs, blocking crawling is exactly what you want.

The key is to look at which URL is blocked and why.

If an important SEO page is blocked accidentally, fix the robots.txt rule first. Then check the page in Google Search Console and request a fresh crawl. For pages you don’t want in search results, don’t rely on robots.txt alone—use the appropriate indexing control such as noindex.