Blocked by robots.txt in Search Console: How to Fix It Step by Step
What 'Blocked by robots.txt' means in Search Console, when it is a real problem, and how to find and fix the rule that blocks your pages.
- "Blocked by robots.txt" means a rule in your robots.txt file tells Googlebot not to crawl that URL.
- It is fine for pages you do not want in Google, such as admin, cart or search pages. It is a problem only when important pages are blocked.
- To fix it, find the rule with the URL Inspection tool, edit robots.txt on your hosting, then ask Google to recheck.
- robots.txt stops crawling, not indexing. To keep a page out of Google, use a noindex tag instead and let Google crawl it.
You open Google Search Console and see some pages listed as "Blocked by robots.txt." Maybe they are your blog posts, maybe your whole site. Traffic is low, and you are not sure if this is the reason.
This guide explains what the message means, how to find the exact rule causing it, and how to fix your robots.txt file safely, step by step.
What does "Blocked by robots.txt" mean?
Every website can have a small text file at yourdomain.com/robots.txt. It gives instructions to search engine bots about which parts of the site they may crawl.
When Search Console shows "Blocked by robots.txt" in the Pages report, it means Googlebot wanted to visit a URL, read your robots.txt first, and found a rule telling it not to. So Google did not crawl that page.
Google's own robots.txt report help page makes an important point: robots.txt is used to stop crawling. If you want to keep a page out of search results, Google says to use noindex instead.
Is "Blocked by robots.txt" always a problem?
No. Many sites block some pages on purpose. Use this table to decide.
| Blocked URL type | Is it a problem? | What to do |
|---|---|---|
| Admin, login or dashboard pages | No | Leave as is |
| Cart, checkout and account pages of a store | No | Leave as is |
Internal search results (/?s= or /search) | Usually no | Leave as is |
| Blog posts or product pages | Yes | Fix the rule |
| Category pages you want to rank | Yes | Fix the rule |
| CSS, JavaScript or image folders | Often yes | Allow them so Google can see your page properly |
| Your whole site | Serious | Fix immediately |
"Blocked by robots.txt" vs "Indexed, though blocked by robots.txt"
These two look similar but mean different things.
- Blocked by robots.txt: Google did not crawl the page and it is not indexed.
- Indexed, though blocked by robots.txt: Google could not crawl the page, but found links to it and indexed the URL anyway. It may show in results with little or no description.
If you want the second type out of Google, do not just add more robots.txt rules. Remove the block, add a noindex tag to the page, and let Google crawl it so it can see the noindex. Our guide on the "Excluded by noindex tag" status explains how noindex works.
How to find which rule is blocking your page
Step 1: Open the Pages report
In Search Console, go to Indexing > Pages. Scroll to "Why pages aren't indexed" and click Blocked by robots.txt. You will see a list of example URLs.
Step 2: Check one URL with URL Inspection
Copy one important URL from the list and paste it into the search bar at the top of Search Console. In the result, open the page indexing details. Look for Crawl allowed? If it says no, robots.txt is blocking it.
Step 3: Open your robots.txt file
Visit https://yourdomain.com/robots.txt in your browser. Read each Disallow line and see which one matches the start of the blocked URL.
Step 4: Check the robots.txt report
Search Console also has a robots.txt report, found under Settings. It shows the robots.txt files Google found, when they were last checked, and any errors in them. Google notes that this report is available only for Domain properties, so you may not see it on a URL-prefix property.
Common robots.txt mistakes that block good pages
| Rule in robots.txt | What it blocks | Fix |
|---|---|---|
Disallow: / | Your entire site | Remove it, or change it to block only the folder you mean |
Disallow: /blog | Everything starting with /blog, including all posts | Delete it unless you truly want the blog hidden |
Disallow: /*? | Every URL with a question mark, including some product or post pages | Block only the exact parameter you need, such as /*?s= |
Disallow: /assets/ | CSS and JavaScript files | Allow these so Google can render pages |
Disallow: /Product/ | Only the capital-P path (rules are case-sensitive) | Match the real case used in your URLs |
Disallow: / while a site is "under construction." If you launched your site but forgot to turn this off, Google cannot crawl anything. Check this first.How to fix "Blocked by robots.txt" step by step
- Back up the current file. Copy your robots.txt text into a note before you change anything.
- Open the file on your hosting. In cPanel, go to File Manager, open
public_htmland editrobots.txt. On WordPress, your SEO plugin may manage it instead, so check its settings. - Remove or narrow the wrong rule. Delete the line that blocks your important pages, or make it more specific.
- Keep the file simple. Most small blogs need only a few lines.
- Save and check it live. Reload
yourdomain.com/robots.txtin a private browser window to make sure the change is showing. - Ask Google to recheck the file. In the robots.txt report, open the menu next to your file and choose Request a recrawl. Google recommends this after fixing an error or making an important change.
- Request indexing for key pages. Inspect two or three important URLs and click Request indexing. See our guide on how to request indexing.
- Validate the fix. Back in the Pages report, open the "Blocked by robots.txt" reason and click Validate fix if those URLs should now be crawled.
A safe, simple robots.txt for a small blog
Here is a basic example for a typical PHP or WordPress blog. Change the paths to match your own site.
User-agent: *
Disallow: /admin/
Disallow: /?s=
Sitemap: https://yourdomain.com/sitemap.xml
A few rules from Google's robots.txt guide to keep in mind:
- The file must sit at the root of your domain, such as
yourdomain.com/robots.txt. - It must be a plain UTF-8 text file.
- Rules and paths are case-sensitive.
- Paths should start with a forward slash.
- The sitemap line needs the full URL, including
https://.
Does robots.txt affect AdSense?
Yes, it can. AdSense uses its own crawler to understand your pages and show relevant ads. Google's ad placement policies mention that ads may be disabled on content that cannot be evaluated, such as content blocked by robots.txt.
So if you block your posts by mistake, it can hurt both search traffic and ads. If you use a ready-made website, check the robots.txt it ships with before you apply for AdSense. Our AdSense-ready website scripts are a good place to start, but it is always worth checking the file yourself after setup.
How long does the fix take?
Google recrawls robots.txt files often, and requesting a recrawl can speed this up. After that, Google still needs to crawl and process your pages again. Small sites may see changes within days, but it can take longer. Keep checking the Pages report rather than changing the file again and again.
Final word
"Blocked by robots.txt" is only a problem when it blocks pages you want people to find. Check the blocked URLs, find the matching rule, keep your file short and clear, then ask Google to recrawl. Remember that robots.txt controls crawling, and noindex controls indexing.
Next step: open your robots.txt file right now and read every Disallow line. If your pages are crawled but still not indexed, read our guide on "Crawled, currently not indexed."
Frequently asked questions
Does robots.txt remove a page from Google?
No. robots.txt stops Google from crawling a page, but the URL can still be indexed if other pages link to it. Use a noindex tag and allow crawling if you want a page out of search results.
Where is the robots.txt file on my website?
It must be at the root of your domain, such as yourdomain.com/robots.txt. On cPanel hosting it is usually inside the public_html folder.
Why can't I see the robots.txt report in Search Console?
Google says the robots.txt report is available only for Domain properties. If you use a URL-prefix property, add a Domain property to see it.
How long does it take after fixing robots.txt?
Google recrawls robots.txt often, and you can request a recrawl in the robots.txt report. Pages then need to be crawled again, which can take days or longer.