Noindex and robots.txt mistakes hide a site from Google by telling it to skip pages you want found. A noindex instruction tells Google not to keep a page in its index. A robots.txt block tells Google not to crawl a path. Either one, applied to the wrong pages, can wipe out traffic. The usual cause is a setting left over from a staging site or a redesign.
Noindex and robots.txt do different jobs
The two are mixed up all the time, and the mix up causes real damage.
Robots.txt is a file at the root of your site. It tells crawlers which paths they may fetch. It controls crawling. It does not control whether a URL appears in the index.
Noindex is an instruction on the page itself, either in a robots meta tag in the page head or in an X-Robots-Tag HTTP header. It tells Google to leave that page out of the index. Our short answer on what a noindex tag does covers the basics.
Here is the trap. If you block a page in robots.txt, Google cannot fetch it, so it cannot see a noindex instruction on it. A page you wanted removed can stay in the results, and a page you wanted found can go missing. Pick the right tool for the job.
How to tell if you have this problem
Sudden, sitewide drops in indexed pages are the classic sign. Check these in order.
Quick check
Is something hiding your pages?
Do the checks on the live site, not on what you remember setting up. Staging settings have a habit of travelling to production.
If your homepage is excluded, treat it as an emergency. Fix first, investigate second.
Five checks, ten minutes
- Open the Page indexing report in Search Console and look for Excluded by noindex tag and Blocked by robots.txt.
- Run URL Inspection on your homepage and two key pages. Check Indexing allowed and Page fetch.
- Open yoursite.com/robots.txt in a browser and read every Disallow line.
- View the page source and search for a robots meta tag. Check what it says.
- Check the response headers for an X-Robots-Tag line, especially on PDFs and file downloads.
The most common mistakes
Common causes
How sites end up hidden
Staging settings went live
A site built behind a block or a noindex setting was launched without removing it.
A CMS discourage setting
A single checkbox in the CMS asks search engines not to index the site, and it was left on.
Disallow slash
A robots.txt rule that blocks the entire site, often a leftover from development.
Blocking CSS and scripts
Rules that stop Google fetching the files it needs to render the page properly.
Noindex on templates
A rule applied to a whole template or category, catching pages that should rank.
Header level noindex
An X-Robots-Tag set at the server, invisible when you look at page source.
Noindex and a block at the same time
Some sites add a noindex tag and also block the path in robots.txt. Google cannot crawl the page, so it never reads the tag. Remove the block, let Google crawl the page, and the noindex works as intended.
Plugin and theme updates
An SEO plugin update, a theme switch or a new caching layer can change robots settings without anyone noticing. Check after every major update.
Noindex used as a quality fix
Some owners noindex pages hoping to hide weak content. That can be a reasonable choice for pages with no search value, but applied broadly it can remove pages that bring in traffic. Know what each page earns before you hide it.
Step by step fix
1. Find every affected URL
Export the excluded URLs from the Page indexing report. Crawl your own site and record the robots meta tag, the X-Robots-Tag header and the robots.txt status for every URL. The difference between what you want indexed and what is indexed is your list.
2. Decide what should be indexed
Not every page belongs in Google. Login pages, internal search results and thank you pages usually should not be. Mark each URL as keep out or let in before you change anything.
3. Remove the wrong instructions
Take out the stray noindex setting, the CMS discourage option or the bad Disallow lines. Change one thing at a time on a large site so you know what moved the result. Keep a copy of the old robots.txt.
4. Test before you trust it
Use URL Inspection on a sample of fixed pages and run a live test. Confirm Google can fetch the page and that indexing is allowed.
5. Prompt a recrawl
Request indexing for your most important pages and submit an updated sitemap. Pages return as Google recrawls them, and that takes time.
6. Add a guard
Put robots and noindex checks into your launch checklist and your update routine. Most of these disasters are repeat offences.
If pages are crawled but still not indexed
Removing a block does not guarantee indexing. If Google can reach the page and still leaves it out, the problem has moved. Read our guide on crawled, currently not indexed for that case. Conflicting signals can also confuse things, so check for canonical tag mistakes at the same time.
When to call for help
Get help when a launch or redesign went wrong and traffic fell, when you cannot find where the instruction is coming from, or when thousands of URLs are affected. Header level and template level rules are easy to miss.
We find the cause before touching anything. Our technical SEO fixes work starts with evidence: which URLs are hidden, by which rule, and since when. Ask for a free SEO triage and we will check.
FAQ
Questions owners ask us
Does robots.txt stop a page from being indexed?
Not reliably. Robots.txt controls crawling, not indexing. A blocked page can still be indexed if other pages link to it, usually without a description. To keep a page out of the index, it must be crawlable and carry a noindex instruction.
How fast does Google drop a page after I add noindex?
It depends on when Google next recrawls the page. Once it sees the instruction, the page is removed from the index. Important pages get recrawled more often, so a mistake on a key page can show up quickly.
I removed the noindex. When will my pages return?
Only after Google recrawls them. Use URL Inspection to request indexing for key pages, keep your sitemap current, and expect the rest to return as Google works through the site. Nobody can promise a date.
Get started