Skip to content

Search Console Says "Excluded by noindex" — Deliberate or Broken?

TL;DR — Google Search Console emails New reasons prevent pages from being indexed … Excluded by 'noindex' tag for both an accidental regression and a change you made on purpose. It cannot tell them apart, and it re-sends the alert as a "new reason" whenever the count returns from zero — so an intentional noindex shipped months ago can surface as a brand-new alarm. Before changing anything, verify on the box which URL types carry the tag. If real posts/pages are clean and only archives are tagged, there is nothing to fix. Critically: WP-core sitemaps do not auto-drop noindex'd URLs — noindexing archives and removing them from wp-sitemap.xml is a matched pair, and doing only half creates a genuine, much louder problem.

Symptom

An email from Search Console:

New reason preventing your pages from being indexed

Search Console has identified that some pages on your site are not
being indexed due to the following new reason:

Excluded by 'noindex' tag

The word "new" is the trap. It does not mean "this started today." GSC tracks each exclusion reason as a count; when a reason drops to zero and later comes back — even from a completely different and deliberate cause — it is reported as new.

This produces a specific, confusing sequence that is easy to misread months later:

  1. A real fault noindexes a handful of real pages (commonly leftover SEO-plugin postmeta after the plugin is removed). GSC emails. You fix it, count → 0.
  2. Separately, you deliberately noindex thin taxonomy archives to reclaim crawl budget. Count climbs from 0 into the thousands.
  3. GSC emails the same alert text again. It looks like the original fault regressed. It did not.

Why noindexing archives is usually correct

A WordPress site accumulates one archive URL per tag, per category, per author, and per date period. A site with ~1,500 posts can easily carry ~950 tags and ~470 categories — roughly 1,400 thin, near-duplicate pages whose only content is excerpt lists already published elsewhere.

Google spends crawl budget on them and (correctly) declines to index them, which shows up as large crawled - currently not indexed and discovered - currently not indexed counts. Sending noindex,follow on those archive types is the standard fix: follow still lets link equity flow through to the real posts.

So a large Excluded by 'noindex' tag count on a site that did this is the fix working, not the fix failing.

The gotcha that turns the fix into a real problem

WP-core sitemaps (wp-sitemap.xml, WordPress 5.5+) do not remove noindex'd URLs automatically. This is different from most SEO plugins, which link the two.

If you noindex tag/category/author archives but leave those providers in the sitemap, you are simultaneously telling Google "here are 1,400 URLs worth crawling" and "do not index any of them." That contradiction is its own Search Console warning ("noindex in sitemap"), and it is a genuine problem — at far greater scale than whatever you started with.

Ship both halves in the same change, in one mu-plugin:

<?php
/**
 * Plugin Name: Noindex Archives + Sitemap Hygiene
 * Description: noindex tag/category/author/date archives AND drop them from
 *              wp-sitemap so the two can never contradict each other.
 */

// 1. noindex,follow on the archive types
add_filter('wp_robots', function ($robots) {
    if (is_tag() || is_category() || is_author() || is_date()) {
        $robots['noindex'] = true;
        $robots['follow']  = true;
    }
    return $robots;
}, 20);

// 2. Drop those taxonomies from the sitemap
add_filter('wp_sitemaps_taxonomies', function ($taxonomies) {
    unset($taxonomies['post_tag'], $taxonomies['category']);
    return $taxonomies;
});

// 3. Drop the users/authors sitemap provider entirely
add_filter('wp_sitemaps_add_provider', function ($provider, $name) {
    return ('users' === $name) ? false : $provider;
}, 10, 2);

Use an mu-plugin (wp-content/mu-plugins/) rather than theme functions.php — it survives theme changes and can't be deactivated from the UI by accident. Note that mu-plugins don't appear in the plugins list, so document them somewhere findable; an undocumented mu-plugin altering site-wide indexing is exactly the thing that makes a future alert look inexplicable.

Diagnostic: is this deliberate or a real fault?

Run this before changing anything. It is entirely read-only. The --resolve flag hits the box directly, bypassing any CDN cache, and the random cache-buster defeats page caches like W3TC.

P="--allow-root --path=/var/www/html"
R="--resolve example.com:443:127.0.0.1"

POST=$(wp post list --post_type=post --post_status=publish --field=url --posts_per_page=1 $P | head -1)
PAGE=$(wp post list --post_type=page --post_status=publish --field=url --posts_per_page=1 $P | head -1)
CAT=$(wp term list category --field=slug --number=1 $P | head -1)
TAG=$(wp term list post_tag --field=slug --number=1 $P | head -1)

for u in "https://example.com/" "$POST" "$PAGE" \
         "https://example.com/category/$CAT/" \
         "https://example.com/tag/$TAG/" \
         "https://example.com/2026/"; do
  printf "%-60s -> " "$u"
  curl -sk $R "$u?cb=$RANDOM" | grep -o "<meta name=.robots.[^>]*>" | head -1
done

Watch the flag name. wp post list takes --posts_per_page, not --number. Passing --number is silently ignored and you get the full list — easy to miss when the output scrolls, and it makes the next step compare against the wrong URL.

Healthy, deliberate configuration:

URL type Expected
Homepage no noindex
A real post no noindex
A real page no noindex
/category/<x>/, /tag/<x>/ noindex, follow
/2026/ (date archive) noindex, follow

If a real post or page comes back noindex, that is a genuine fault — investigate rather than dismissing the alert.

Confirm the sitemap and the noindex don't overlap

This is the check worth doing even when everything above looks right, because it's the failure that scales:

# What the sitemap index still advertises
curl -sk $R "https://example.com/wp-sitemap.xml" | grep -o "<loc>[^<]*</loc>"

# Then fetch each remaining child sitemap's URLs and confirm they are indexable

Every URL type still listed in wp-sitemap.xml must be indexable. After the mu-plugin above, the index should list only posts, pages, and post_format — and post_format archives (/type/video/, /type/image/) are not covered by the wp_robots filter, so they stay indexable and remain legitimately sitemapped.

Rule out the other causes of a real noindex

# Leftover Yoast postmeta after the plugin was removed — expect 0
wp db query "SELECT COUNT(*) FROM wp_postmeta
             WHERE meta_key='_yoast_wpseo_meta-robots-noindex'
             AND meta_value='1'" $P

# Site-wide "discourage search engines" toggle — expect 1 (public)
wp option get blog_public $P

blog_public = 0 noindexes the entire site and is a frequent post-staging-migration mistake.

Should you click "Validate Fix"?

Only if the noindex was genuinely accidental. If the current state is intentional, there is nothing to validate — the URLs will keep returning noindex forever by design, and a validation attempt just fails against thousands of URLs. Leave the exclusion count high; it is descriptive, not a defect.

The count you actually care about is indexed real posts, not total excluded URLs.

Gotchas worth restating

  • "New reason" ≠ new problem. It means "returned from zero." Check the date of your own last indexing change before assuming a regression.
  • A large exclusion count is not a score. On a tag-heavy WordPress site, a big Excluded by 'noindex' tag number is the expected end state of a correct fix.
  • Never ship the noindex without the sitemap change. Half the fix is worse than none.
  • Flush caches before verifying. A page cache will happily serve pre-change HTML and convince you the fix didn't apply: wp cache flush, plus the page-cache directory for W3TC.
  • Removing a sitemap provider can leave a soft-404. The old wp-sitemap-users-1.xml may return HTTP 200 with an HTML body rather than a hard 404. Harmless when the URL is unlinked and absent from the index, but worth recording so it isn't re-investigated later.
  • Document mu-plugins on the host's own page. They are invisible in the WordPress admin UI, so the next person debugging an indexing alert has no way to discover them from the application side.