indiaseofieldnotes

Robots.txt, Staging Leaks and Stray Noindex: The Monthly Ten-Minute Audit

Most of the ranking disasters I get called in for are not algorithmic. They are a line in a file. A developer shipped a staging robots.txt to production. A plugin update reinstated a noindex on a category. A canonical pointed at a URL that was retired two years ago. None of these are hard to find. They are just nobody's job, so nobody looks.

Here is the check I run monthly on every site I am responsible for. It takes about ten minutes and it has caught more lost traffic than any content project I have ever run.

1. Read robots.txt with your own eyes

Open yoursite.com/robots.txt in a browser. Not a tool, the actual file. You are looking for two things: a Disallow: / that should not be there, and a sitemap line that points somewhere real. A surprising number of Indian sites have a sitemap directive pointing at a path that 404s, usually because the blog moved to a subdirectory and nobody updated the line.

If you have a staging environment, check its robots.txt too. Staging should be blocked. Production should not be blocked by a file that was copied from staging.

2. Check indexed count against expected count

In Search Console, look at the Pages report. Compare the indexed number to the number of URLs in your sitemap. A gap is normal. A gap that grew by a few hundred since last month is a signal. Click into the exclusion reasons: "Excluded by noindex tag" and "Blocked by robots.txt" are the two that should be zero on pages you care about, and are the two that plugin updates quietly break.

3. Spot-check canonicals on your five money pages

View source on each page that actually earns revenue. Find the canonical tag. Confirm it points at that page's own live URL, with the same protocol, the same www-or-not, and the same trailing slash. Canonical drift after a URL migration is the quietest traffic killer there is, because nothing breaks visibly: the page loads fine, it just stops being the version Google ranks.

I have found this on sites that had been migrated cleanly and then had eleven pages canonicalising back to the pre-migration URLs. Nobody noticed for months.

4. Confirm your sitemap is honest

Your sitemap should list the URLs you want indexed and nothing else. Not redirects, not 404s, not noindexed pages, not parameter variants. Fetch it and check the count matches what you expect. If your CMS generates it automatically, verify it after any plugin or theme update.

5. Test one page as a crawler

Use the URL Inspection tool on a single important page and read the rendered HTML. You are checking that your main content is actually in there. On JavaScript-heavy sites, the body copy sometimes is not, and a page that looks perfect in a browser can be nearly empty to a crawler.

6. Look at the redirect chain on your most-linked old URL

If you have ever changed URL structure, pick the old URL with the most backlinks and follow it. One hop to the right destination is correct. Two or three hops loses value. A hop that lands on a 404 or the homepage instead of the equivalent page is throwing away the strongest links you own.

7. Check HTTPS and www consistency

All four variants of your homepage - http, https, with www, without - should end up at the same single URL with one redirect. Test them by hand. Hosting migrations break this regularly.

Why monthly, and why manual

Crawlers and audit tools are good, and you should run them. But they report against the last crawl, they bury the important finding in a list of two hundred warnings, and they cannot tell you that a noindex on your services category is catastrophic rather than cosmetic. Ten minutes by hand, once a month, on the pages you actually care about, catches the things that matter while they are still cheap to fix.

If a monthly check turns up something you cannot explain, that is the point to bring in an SEO expert in India rather than guessing at it - a misdiagnosed technical problem usually gets worse, not better, when content is thrown at it.

Related reading: Search Console for Indian Sites: The Four Reports That Actually Change Decisions goes deeper on reading the Pages report properly.