Your docs are public on the web as soon as you publish. Should Google index every page, or should crawlers stay away while you draft internal documentation? docs.page handles both paths with generated discovery files and a single site-wide switch.
Published docs.page sites expose two HTTP endpoints crawlers expect:
robots.txtat/{owner}/{repo}/robots.txt: tells crawlers what they may fetch and where to find your sitemapsitemap.xmlat/{owner}/{repo}/sitemap.xml: lists every documentation page with update hints
Each HTML page also ships standard metadata: title, description, Open Graph tags, and a canonical URL when one applies. Together, these signals help search engines find and rank your content.
Set seo.noindex in docs.json when you want the opposite: block indexing across the entire site without taking the site offline.
When seo.noindex is false (the default), docs.page encourages crawlers to index your site:
robots.txtallows all paths and points to your sitemap URL.sitemap.xmlincludes every page file in your repository. The index page gets priority1.0; other pages get0.8. All entries usechangefreq: weekly.- Page
<head>metadata sets title and description from page frontmatter, falling back todocs.json. Open Graph and Twitter card tags use the same values plus an image from frontmatter,socialPreview, or a generated preview.
These endpoints are served from your production URL, custom domain, or vanity path, the same origin readers use. Local preview does not generate sitemap.xml or robots.txt; use a branch preview or production URL to verify them.
Add seo.noindex: true in docs.json when the entire site should stay out of search results, for example, internal runbooks on a public repository, pre-release documentation, or a staging site you do not want indexed.
{
"seo": {
"noindex": true
}
}This flag affects two layers:
| Layer | Behavior when seo.noindex is true |
|---|---|
robots.txt | Disallow: / for all user agents. The sitemap line is omitted. |
| Every page | <meta name="robots" content="noindex" /> in the document head |
Crawlers that respect robots.txt or the noindex meta tag should stop indexing new content. Pages may remain in search results until engines re-crawl and drop them. Allow time after you change the setting.
If you migrated from docs.page v1, the top-level noindex field maps to seo.noindex.
You can also set noindex: true in a page's YAML frontmatter. That adds the noindex meta tag for that page only. It does not change robots.txt or remove the page from sitemap.xml.
Use site-wide seo.noindex when nothing on the site should appear in search. Use per-page noindex when most pages should be indexed but a few (draft changelogs, legal stubs, or redirect helpers) should not.
sitemap.xml is built from the file tree docs.page reads from GitHub: one URL per documentation page. It does not filter by sidebar visibility or noindex frontmatter, so a page listed in the sitemap may still carry a noindex meta tag if you set one.
If your repository has an unusually large tree, the sitemap response may include an x-docs-page-tree-truncated header when the file list is truncated.