drupal/robots.txt

#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used:    http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html

User-agent: *
# CSS, JS, Images
Allow: /core/*.css$
Allow: /core/*.css?
Allow: /core/*.js$
Allow: /core/*.js?
Allow: /core/*.gif
Allow: /core/*.jpg
Allow: /core/*.jpeg
Allow: /core/*.png
Allow: /core/*.svg
Allow: /profiles/*.css$
Allow: /profiles/*.css?
Allow: /profiles/*.js$
Allow: /profiles/*.js?
Allow: /profiles/*.gif
Allow: /profiles/*.jpg
Allow: /profiles/*.jpeg
Allow: /profiles/*.png
Allow: /profiles/*.svg
# Directories
Disallow: /core/
Disallow: /profiles/
# Files
Disallow: /README.txt
Disallow: /web.config
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
# Paths (no clean URLs)
Disallow: /index.php/admin/
Disallow: /index.php/comment/reply/
Disallow: /index.php/filter/tips
Disallow: /index.php/node/add/
Disallow: /index.php/search/
Disallow: /index.php/user/password/
Disallow: /index.php/user/register/
Disallow: /index.php/user/login/
Disallow: /index.php/user/logout/
#78652 by m3avrck, add CVS ID 2006-08-15 05:18:16 +00:00			`#`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# robots.txt`
			`#`
- Patch #79031 by jvandyck: corrections to robots.txt. 2006-08-16 22:03:56 +00:00			`# This file is to prevent the crawling and indexing of certain parts`
- Patch #77183 by m3vrck: cleaned up some of the CSS mess. 2006-08-14 20:35:11 +00:00			`# of your site by web crawlers and spiders run by sites like Yahoo!`
			`# and Google. By telling these "robots" where not to go on your site,`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# you save bandwidth and server resources.`
			`#`
- Patch #79031 by jvandyck: corrections to robots.txt. 2006-08-16 22:03:56 +00:00			`# This file will be ignored unless it is at the root of your host:`
			`# Used: http://example.com/robots.txt`
			`# Ignored: http://example.com/site/robots.txt`
			`#`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# For more information about the robots.txt standard, see:`
Issue #2290435 by PQ: Fix broken link in robots.txt file 2014-06-25 22:06:24 +00:00			`# http://www.robotstxt.org/robotstxt.html`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00
			`User-agent: *`
Issue #2364343 by damien_vancouver, criz, ksenzee, joegraduate, Neograph734, droplet, pounard, jp.stacey, ciss: Fix robots.txt to allow Google access to CSS, JavaScript and image files 2016-02-29 07:18:01 +00:00			`# CSS, JS, Images`
Issue #2364343 by ksenzee \| corbacho: Fixed robots.txt to allow Google access to CSS and JavaScript files. 2014-11-05 09:15:07 +00:00			`Allow: /core/*.css$`
Issue #2364343 by damien_vancouver, criz, ksenzee, joegraduate, Neograph734, droplet, pounard, jp.stacey, ciss: Fix robots.txt to allow Google access to CSS, JavaScript and image files 2016-02-29 07:18:01 +00:00			`Allow: /core/*.css?`
Issue #2364343 by ksenzee \| corbacho: Fixed robots.txt to allow Google access to CSS and JavaScript files. 2014-11-05 09:15:07 +00:00			`Allow: /core/*.js$`
Issue #2364343 by damien_vancouver, criz, ksenzee, joegraduate, Neograph734, droplet, pounard, jp.stacey, ciss: Fix robots.txt to allow Google access to CSS, JavaScript and image files 2016-02-29 07:18:01 +00:00			`Allow: /core/*.js?`
			`Allow: /core/*.gif`
			`Allow: /core/*.jpg`
			`Allow: /core/*.jpeg`
			`Allow: /core/*.png`
			`Allow: /core/*.svg`
			`Allow: /profiles/*.css$`
			`Allow: /profiles/*.css?`
			`Allow: /profiles/*.js$`
			`Allow: /profiles/*.js?`
			`Allow: /profiles/*.gif`
			`Allow: /profiles/*.jpg`
			`Allow: /profiles/*.jpeg`
			`Allow: /profiles/*.png`
			`Allow: /profiles/*.svg`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# Directories`
Issue #22336 by quicksketch, scor, boombatower, and rfay. Move all core Drupal files under a core subdirectory. 2011-10-31 04:05:57 +00:00			`Disallow: /core/`
- Patch #299582 by hass: robots.txt maintenance. 2008-08-25 08:13:41 +00:00			`Disallow: /profiles/`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# Files`
Issue #22336 by quicksketch, scor, boombatower, and rfay. Move all core Drupal files under a core subdirectory. 2011-10-31 04:05:57 +00:00			`Disallow: /README.txt`
			`Disallow: /web.config`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# Paths (clean URLs)`
			`Disallow: /admin/`
			`Disallow: /comment/reply/`
Issue #1137848 by Tor Arne Thune, mgifford, GaëlG, ericjenkins: /filter/tips page is listed by search engines 2018-03-15 15:02:58 +00:00			`Disallow: /filter/tips`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`Disallow: /node/add/`
			`Disallow: /search/`
			`Disallow: /user/register/`
			`Disallow: /user/password/`
- Patch #75916 by pcwick: block user/login page. 2007-01-08 12:02:18 +00:00			`Disallow: /user/login/`
- Patch #337820 by Dave Reid, Damien Tournoud, et al: rename menu path 'logout' to 'user/logout'. For real now. 2008-11-29 09:33:51 +00:00			`Disallow: /user/logout/`
- Patch #75916 by Robert: re-introduced default robot.txt. 2006-08-14 10:41:52 +00:00			`# Paths (no clean URLs)`
- Patch #1183208 by effulgentsia, Niklas Fiekas, katbailey, no_commit_credit, catch: remove variable_get()('clean_url()') and switch to index.php/path pattern for dirty URL support. 2012-04-29 15:16:27 +00:00			`Disallow: /index.php/admin/`
			`Disallow: /index.php/comment/reply/`
Issue #1137848 by Tor Arne Thune, mgifford, GaëlG, ericjenkins: /filter/tips page is listed by search engines 2018-03-15 15:02:58 +00:00			`Disallow: /index.php/filter/tips`
- Patch #1183208 by effulgentsia, Niklas Fiekas, katbailey, no_commit_credit, catch: remove variable_get()('clean_url()') and switch to index.php/path pattern for dirty URL support. 2012-04-29 15:16:27 +00:00			`Disallow: /index.php/node/add/`
			`Disallow: /index.php/search/`
			`Disallow: /index.php/user/password/`
			`Disallow: /index.php/user/register/`
			`Disallow: /index.php/user/login/`
			`Disallow: /index.php/user/logout/`