Skip to content

Castopod Has No XML Sitemap — Feed-Driven Generator + GSC Submission

Castopod (verified through 2.0.0-next.4) ships no XML sitemap — /sitemap.xml 404s, no route or config exists (spark routes | grep -i sitemap is empty), and robots.txt declares no Sitemap: line. Google discovers episodes only via internal links + RSS, so indexing coverage stays terrible (observed: 37 of 238 known URLs indexed, 15.5%) and Googlebot spends its crawl budget on /assets/* and the actor page instead of episodes.

Symptom

  • GSC shows large "Crawled – currently not indexed" and (after partial discovery) "Discovered – currently not indexed" buckets; episode pages barely indexed.
  • Verified-Googlebot log analysis (filter IPs 66.249.* — most "Googlebot" UA traffic is spoofed scanners) shows episode pages getting ~1 crawl/day while static assets take >50% of hits.

Fix — generate the sitemap from the public RSS feed

The feed is the authoritative episode list and requires no DB access. Cron-regenerate so new episodes auto-add.

/usr/local/bin/<site>-sitemap.sh (as deployed on tttpod for techtalkthursdays.net):

#!/usr/bin/env bash
# Generates /sitemap.xml (Castopod has no native sitemap).
# Source of truth = the public RSS feed. Re-run via cron so new episodes auto-add.
set -euo pipefail

SITE="https://techtalkthursdays.net"
HANDLE="@TechTalkThursdays"          # case-sensitive Castopod handle
OUT="/var/www/html/castopod/public/sitemap.xml"
RESOLVE="--resolve techtalkthursdays.net:443:127.0.0.1"
FEED="$SITE/$HANDLE/feed.xml"

TMP_URLS="$(mktemp)"; TMP_XML="$(mktemp)"
trap 'rm -f "$TMP_URLS" "$TMP_XML"' EXIT

# Static high-value pages + every episode page from the feed
# (strip query/fragment; the /episodes/[^/]+$ grep drops /comments, /embed, /oembed variants)
{
  echo "$SITE/$HANDLE"
  echo "$SITE/$HANDLE/episodes"
  echo "$SITE/credits"
  curl -fsS $RESOLVE "$FEED" \
    | grep -oE "$SITE/$HANDLE/episodes/[^<\" ]+" \
    | sed 's/[?#].*//' \
    | grep -E "/episodes/[^/]+$" \
    | sort -u
} | sort -u > "$TMP_URLS"

{
  echo '<?xml version="1.0" encoding="UTF-8"?>'
  echo '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">'
  while IFS= read -r u; do
    esc=${u//&/&amp;}
    printf '  <url><loc>%s</loc></url>\n' "$esc"
  done < "$TMP_URLS"
  echo '</urlset>'
} > "$TMP_XML"

install -o www-data -g www-data -m 0644 "$TMP_XML" "$OUT"
echo "Wrote $(grep -c '<url>' "$OUT") URLs to $OUT"

Notes: - Writing a static file into castopod/public/ works because Apache serves existing files before CodeIgniter's router sees the request — no Castopod patching, nothing lost on upgrade (unlike vendor/ or modules/ edits). - Credits is site-wide, not per-podcast: Castopod serves it at /credits. /@<handle>/credits is a 404 — an earlier version of this script listed it, and Search Console reported the dead page (fixed 2026-10-27). - --resolve pins the request to localhost so the cron doesn't depend on external DNS/proxy. - Cron: /etc/cron.d/<site>-sitemap → 30 4 * * * root /usr/local/bin/<site>-sitemap.sh >/var/log/<site>-sitemap.log 2>&1

Declare + submit — the step that actually matters

  1. Append to castopod/public/robots.txt: 
    Sitemap: https://<site>/sitemap.xml
  2. Submit it in GSC → Sitemaps. Do not rely on robots.txt discovery alone: on tttpod, robots.txt-only discovery produced exactly 2 Googlebot sitemap fetches in 5 weeks while episode-crawl share sat at 1.3% and indexed count fell. The generator ran perfectly for a month and accomplished nothing visible because the GSC submission step was skipped.

Verifying it's working

# sitemap live + fresh
curl -sI https://<site>/sitemap.xml | head -3
ls -l castopod/public/sitemap.xml   # mtime should be last cron run

# is real Googlebot fetching it? (spoofed-UA scanners are ~80% of "Googlebot" traffic)
zgrep -h "GET /sitemap.xml" /var/log/apache2/<vhost>-access.log* | awk '$1 ~ /^66\.249\./'

In GSC, expect "Discovered – currently not indexed" to grow first (sitemap URLs entering the queue), then convert to crawled/indexed over following weeks. Track indexed count and episode-crawl share — not the crawled-not-indexed pile, which is permanent Castopod system-page noise (embeds, oembed, fediverse endpoints, pagination).