Indexed, Though Blocked by robots.txt: What It Means and Fixes
Search Console reports "Indexed, though blocked by robots.txt" or "Blocked by robots.txt" — your robots.txt file tells Google not to crawl certain pages, yet some of them still show up in search, or pages you wanted in Google are being kept out.
Common signs of this issue
- Search Console shows "Indexed, though blocked by robots.txt" as a warning, or lists pages under "Blocked by robots.txt" in the Pages report.
- Your page appears in Google with no description and the note "No information is available for this page."
- Admin, login, cart, or filter URLs you never meant to be found are showing in Google.
- Important pages — services, products, or the whole site — stopped appearing after a redesign, a new SEO plugin, or a move from a staging site.
- URL Inspection says "Blocked by robots.txt" when you test a page that loads fine in a browser.
- You added a noindex tag to pages weeks ago but they are still in Google.
Safe checks you can do yourself
None of these require sharing passwords with anyone.
- Open yourdomain.com/robots.txt in a browser. It is a short plain-text file. Look for lines starting with
Disallow:. A line readingDisallow: /underUser-agent: *blocks your entire site and is almost always a leftover from development. - In Search Console go to Indexing, then Pages, and click the Blocked by robots.txt reason, and the Indexed, though blocked by robots.txt warning if it appears. Look at the example URLs and sort them into pages you want in Google and pages you don't.
- If the listed URLs are things like
/wp-admin/,/cart/,/checkout/, internal search results, or URLs with?filters on the end, that is normally intentional and harmless. Nothing to fix. - If any real page you want found is in the list, paste it into the URL Inspection bar and click Test live URL. It will confirm whether robots.txt is the reason Google cannot fetch it.
- Check Settings, then robots.txt in Search Console. This report shows the robots.txt file Google last fetched, when it fetched it, and any errors — useful if you have edited the file and are unsure Google has seen the new version.
- On WordPress, check Settings, then Reading for "Discourage search engines from indexing this site" and look at your SEO plugin's robots.txt or crawl settings. Many plugins generate a virtual robots.txt, so the file you see may not exist on the server. See WordPress blocking Google.
- If the goal is to remove pages from Google, check whether they are blocked by robots.txt and carry a noindex tag. That combination does not work — Google cannot read the noindex on a page it is not allowed to crawl.
- After any change, use Validate fix on the issue in the Pages report and give it a few weeks. The report updates slowly.
What this usually means
robots.txt controls crawling, not indexing. A Disallow rule tells Google not to fetch a page. It does not stop Google from listing that page if other sites or your own menus link to it. When that happens, Google indexes the bare URL without reading it, which is why the result shows no description. That is exactly what Indexed, though blocked by robots.txt means: Google found the page through links, respected your instruction not to read it, and listed it anyway based on the links alone.
Blocked by robots.txt in the not-indexed list is simpler: the page is blocked and Google has left it out. Whether that is a problem depends entirely on which pages are listed. Blocking the admin area, cart, and search result pages is routine. Blocking your service pages, product pages, images or stylesheets, or the whole site by accident, is a serious problem, and it is common after a site goes live from a staging copy that was blocked on purpose during development.
The noindex versus disallow conflict trips up a lot of people. If you want a page out of Google, it needs a noindex tag and Google must be allowed to crawl it so it can see that tag. Blocking it in robots.txt at the same time hides the instruction, so the page can stay indexed indefinitely. The correct order is: allow crawling, add noindex, wait for it to drop out, and only then block it in robots.txt if you still want to. For pages you want in Google, the fix is the opposite — remove the Disallow line covering them. For more on getting pages out, see removing old pages from Google.
What not to do
- Don't use robots.txt to hide private or sensitive pages. The file is public, and blocked URLs can still appear in search. Use a password or login instead.
- Don't block a page in robots.txt and add noindex at the same time. Google can't see the noindex, so the page may never leave the index.
- Don't block your CSS, JavaScript, or image folders. Google needs them to see your pages the way visitors do.
- Don't panic about blocked admin, cart, or search URLs. Those are supposed to be blocked, and the report is just telling you so.
- Don't edit robots.txt directly on a live site without keeping a copy of the original. One wrong line can block the whole site.
When to get help
It is worth getting help if your important pages are in the Blocked list and you cannot find the rule responsible, if two or three plugins seem to be writing robots.txt rules, or if traffic dropped around the time the block appeared. A site-wide block after a launch or migration is urgent: every day it stays in place, more pages fall out of Google. Someone who works on this regularly will read the rules, test them against your real URLs, and check for noindex tags, redirects, and staging settings that often come along with the same mistake.
Glenn, who runs WebsiteSelfHelp, sorts out robots.txt and indexing problems for small business sites. Send a short description of what Search Console is showing and you will get a clear answer on whether it needs fixing and what it takes — you don't need to share any passwords to start.
Not sure what to do next?
Answer a few short questions and we'll point you to the safest next step — DIY, a freelancer, or a direct review. No passwords required.
Frequently asked questions
What does Indexed, though blocked by robots.txt mean?
Google found links to the page and indexed its address, but your robots.txt file tells Google not to crawl it, so Google never read the content. The page may appear in search with no description.
Should I fix Indexed, though blocked by robots.txt?
Only if the page matters. If you want it in Google, remove the rule blocking it. If you want it gone, allow crawling and add a noindex tag instead. If it is an unimportant URL nobody searches for, you can usually leave it.
What is the difference between noindex and disallow?
Disallow in robots.txt stops Google from crawling a page. Noindex is a tag on the page that tells Google not to show it in search. To keep a page out of results, use noindex and let Google crawl it so it can see the tag.
Why does my page say No information is available for this page?
That is what Google shows when a page is indexed but blocked from crawling by robots.txt. Google knows the page exists from links but was not allowed to read it, so it has no description to display.
Will Blocked by robots.txt hurt my SEO?
Not when the blocked pages are admin, cart, or search result pages. It hurts badly when real content is blocked, because those pages cannot rank. Check the example URLs to see which case you are in.
How long does it take Google to notice a robots.txt change?
Google generally rechecks robots.txt within about a day, but pages it had been kept away from then need to be recrawled, which can take days to weeks. The Search Console report catches up more slowly still.