Skip crawling for new API documentation

This prepares for the launch of new API documentation, initially without indexing by search engines.
2021-01-19 23:27:05 +00:00 · 2021-01-19 23:27:05 +00:00 · ec8deb56e7
parent a5630c5dde
commit ec8deb56e7
1 changed files with 4 additions and 0 deletions
--- a/layouts/robots.txt
+++ b/layouts/robots.txt
@ -6,4 +6,8 @@ Disallow: /v1.1/
 Disallow: /404/
 Disallow: /404.html

+Allow: /docs/reference/kubernetes-api/api-index/
+Allow: /docs/reference/kubernetes-api/labels-annotations-taints/
+Disallow: /docs/reference/kubernetes-api/
+
 SITEMAP: {{ "sitemap.xml" | absLangURL }}