drupal/robots.txt

#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used:    http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html
#
# For syntax checking, see:
# http://www.frobee.com/robots-txt-check

User-agent: *
Crawl-delay: 10
# JS/CSS
Allow: /core/*.css$
Allow: /core/*.js$
# Directories
Disallow: /core/
Disallow: /profiles/
# Files
Disallow: /README.txt
Disallow: /web.config
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
# Paths (no clean URLs)
Disallow: /index.php/admin/
Disallow: /index.php/comment/reply/
Disallow: /index.php/filter/tips/
Disallow: /index.php/node/add/
Disallow: /index.php/search/
Disallow: /index.php/user/password/
Disallow: /index.php/user/register/
Disallow: /index.php/user/login/
Disallow: /index.php/user/logout/
#78652 by m3avrck, add CVS ID 2006-08-15 05:18:16 +00:00			`#`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# robots.txt`
			`#`
- Patch #79031 by jvandyck: corrections to robots.txt. 2006-08-16 22:03:56 +00:00			`# This file is to prevent the crawling and indexing of certain parts`
- Patch #77183 by m3vrck: cleaned up some of the CSS mess. 2006-08-14 20:35:11 +00:00			`# of your site by web crawlers and spiders run by sites like Yahoo!`
			`# and Google. By telling these "robots" where not to go on your site,`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# you save bandwidth and server resources.`
			`#`
- Patch #79031 by jvandyck: corrections to robots.txt. 2006-08-16 22:03:56 +00:00			`# This file will be ignored unless it is at the root of your host:`
			`# Used: http://example.com/robots.txt`
			`# Ignored: http://example.com/site/robots.txt`
			`#`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# For more information about the robots.txt standard, see:`
Issue #2290435 by PQ: Fix broken link in robots.txt file 2014-06-25 22:06:24 +00:00			`# http://www.robotstxt.org/robotstxt.html`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`#`
- Patch #79031 by jvandyck: corrections to robots.txt. 2006-08-16 22:03:56 +00:00			`# For syntax checking, see:`
Issue #2276469 by decibel.places: Fixed robots.txt syntax checking link broken. 2014-06-02 04:09:07 +00:00			`# http://www.frobee.com/robots-txt-check`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00
			`User-agent: *`
			`Crawl-delay: 10`
Issue #2364343 by ksenzee \| corbacho: Fixed robots.txt to allow Google access to CSS and JavaScript files. 2014-11-05 09:15:07 +00:00			`# JS/CSS`
			`Allow: /core/*.css$`
			`Allow: /core/*.js$`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# Directories`
Issue #22336 by quicksketch, scor, boombatower, and rfay. Move all core Drupal files under a core subdirectory. 2011-10-31 04:05:57 +00:00			`Disallow: /core/`
- Patch #299582 by hass: robots.txt maintenance. 2008-08-25 08:13:41 +00:00			`Disallow: /profiles/`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# Files`
Issue #22336 by quicksketch, scor, boombatower, and rfay. Move all core Drupal files under a core subdirectory. 2011-10-31 04:05:57 +00:00			`Disallow: /README.txt`
			`Disallow: /web.config`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# Paths (clean URLs)`
			`Disallow: /admin/`
			`Disallow: /comment/reply/`
Issue #1137848 by BrockBoland, tim.plunkett, jensimmons: Disallow /filter/tips page from search engines. 2011-06-01 08:27:15 +00:00			`Disallow: /filter/tips/`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`Disallow: /node/add/`
			`Disallow: /search/`
			`Disallow: /user/register/`
			`Disallow: /user/password/`
- Patch #75916 by pcwick: block user/login page. 2007-01-08 12:02:18 +00:00			`Disallow: /user/login/`
- Patch #337820 by Dave Reid, Damien Tournoud, et al: rename menu path 'logout' to 'user/logout'. For real now. 2008-11-29 09:33:51 +00:00			`Disallow: /user/logout/`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# Paths (no clean URLs)`
- Patch #1183208 by effulgentsia, Niklas Fiekas, katbailey, no_commit_credit, catch: remove variable_get()('clean_url()') and switch to index.php/path pattern for dirty URL support. 2012-04-29 15:16:27 +00:00			`Disallow: /index.php/admin/`
			`Disallow: /index.php/comment/reply/`
			`Disallow: /index.php/filter/tips/`
			`Disallow: /index.php/node/add/`
			`Disallow: /index.php/search/`
			`Disallow: /index.php/user/password/`
			`Disallow: /index.php/user/register/`
			`Disallow: /index.php/user/login/`
			`Disallow: /index.php/user/logout/`