Magento robots.txt

Magento robots.txt

Magento comes without robots.txt functionality. It can be useful to add one yourself to tell the search engines where they are not allowed to index. It will hide your javascript files, hide SID parameters and prevent some duplicate content. It will help your SEO process and reduces resources on your server. In this blogpost I explain you how to set your own Magento robots.txt using an existing example and using an extension. Both solutions are easy to handle.

Manual installation of Magento robots.txt

The Magento robots.txt file we use a lot on websites is one that is around on the net since January 2010. I would love to credit the creator of the file, but the original forum post on Magento is not available anymore.
Implementation is easy. Just copy the content below, paste it in a new file called robots.txt, change the location of the sitemap.xml and upload the file into the root of your website. Be sure to upload it to the root even if your Magento installation is in a subdirectory. Search engines will only read robots.txt in the root of a website.

# $Id: robots.txt,v magento-specific 2010/28/01 18:24:19 goba Exp $ # Edited: 2012/06/11 by Byte Internet # # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these “robots” where not to go on your site, # you save bandwidth and server resources. # # This file will be ignored unless it is at the root of your host: # Used: http://example.com/robots.txt # Ignored: http://example.com/site/robots.txt # # For more information about the robots.txt standard, see: # http://www.robotstxt.org/wc/robots.html # # For syntax checking, see: # http://www.sxw.org.uk/computing/robots/check.html # Website Sitemap Sitemap: http://www.example.com/sitemap.xml # Crawlers Setup User-agent: * Crawl-delay: 10 # Allowable Index # Mind that Allow is not an official standard Allow: /*?p= Allow: /index.php/blog/ Allow: /catalog/seo_sitemap/category/ # Allow: /catalogsearch/result/ Allow: /media/catalog/ # Directories Disallow: /404/ Disallow: /app/ Disallow: /cgi-bin/ Disallow: /downloader/ Disallow: /errors/ Disallow: /includes/ Disallow: /js/ Disallow: /lib/ Disallow: /magento/ # Disallow: /media/ Disallow: /media/captcha/ # Disallow: /media/catalog/ Disallow: /media/css/ Disallow: /media/css_secure/ Disallow: /media/customer/ Disallow: /media/dhl/ Disallow: /media/downloadable/ Disallow: /media/import/ Disallow: /media/js/ Disallow: /media/pdf/ Disallow: /media/sales/ Disallow: /media/tmp/ Disallow: /media/wysiwyg/ Disallow: /media/xmlconnect/ Disallow: /pkginfo/ Disallow: /report/ Disallow: /scripts/ Disallow: /shell/ Disallow: /skin/ Disallow: /stats/ Disallow: /var/ # Paths (clean URLs) Disallow: /index.php/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ Disallow: /catalog/product/gallery/ Disallow: /catalogsearch/ Disallow: /checkout/ Disallow: /control/ Disallow: /contacts/ Disallow: /customer/ Disallow: /customize/ Disallow: /newsletter/ Disallow: /poll/ Disallow: /review/ Disallow: /sendfriend/ Disallow: /tag/ Disallow: /wishlist/ # Files Disallow: /cron.php Disallow: /cron.sh Disallow: /error_log Disallow: /install.php Disallow: /LICENSE.html Disallow: /LICENSE.txt Disallow: /LICENSE_AFL.txt Disallow: /STATUS.txt Disallow: /get.php # Magento 1.5+ # Paths (no clean URLs) Disallow: /*.js$ Disallow: /*.css$ Disallow: /*.php$ Disallow: /*?SID= Disallow: /rss* Disallow: /*PHPSESSID

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106

# $Id: robots.txt,v magento-specific 2010/28/01 18:24:19 goba Exp $ # Edited: 2012/06/11 by Byte Internet # # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these “robots” where not to go on your site, # you save bandwidth and server resources. # # This file will be ignored unless it is at the root of your host: # Used: http://example.com/robots.txt # Ignored: http://example.com/site/robots.txt # # For more information about the robots.txt standard, see: # http://www.robotstxt.org/wc/robots.html # # For syntax checking, see: # http://www.sxw.org.uk/computing/robots/check.html # Website Sitemap Sitemap: http://www.example.com/sitemap.xml # Crawlers Setup User-agent: * Crawl-delay: 10 # Allowable Index # Mind that Allow is not an official standard Allow: /*?p= Allow: /index.php/blog/ Allow: /catalog/seo_sitemap/category/ # Allow: /catalogsearch/result/ Allow: /media/catalog/ # Directories Disallow: /404/ Disallow: /app/ Disallow: /cgi-bin/ Disallow: /downloader/ Disallow: /errors/ Disallow: /includes/ Disallow: /js/ Disallow: /lib/ Disallow: /magento/ # Disallow: /media/ Disallow: /media/captcha/ # Disallow: /media/catalog/ Disallow: /media/css/ Disallow: /media/css_secure/ Disallow: /media/customer/ Disallow: /media/dhl/ Disallow: /media/downloadable/ Disallow: /media/import/ Disallow: /media/js/ Disallow: /media/pdf/ Disallow: /media/sales/ Disallow: /media/tmp/ Disallow: /media/wysiwyg/ Disallow: /media/xmlconnect/ Disallow: /pkginfo/ Disallow: /report/ Disallow: /scripts/ Disallow: /shell/ Disallow: /skin/ Disallow: /stats/ Disallow: /var/ # Paths (clean URLs) Disallow: /index.php/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ Disallow: /catalog/product/gallery/ Disallow: /catalogsearch/ Disallow: /checkout/ Disallow: /control/ Disallow: /contacts/ Disallow: /customer/ Disallow: /customize/ Disallow: /newsletter/ Disallow: /poll/ Disallow: /review/ Disallow: /sendfriend/ Disallow: /tag/ Disallow: /wishlist/ # Files Disallow: /cron.php Disallow: /cron.sh Disallow: /error_log Disallow: /install.php Disallow: /LICENSE.html Disallow: /LICENSE.txt Disallow: /LICENSE_AFL.txt Disallow: /STATUS.txt Disallow: /get.php # Magento 1.5+ # Paths (no clean URLs) Disallow: /*.js$ Disallow: /*.css$ Disallow: /*.php$ Disallow: /*?SID= Disallow: /rss* Disallow: /*PHPSESSID

We commented out the allow of catalogsearch/results. This is because we use Google CSE for our Magento shops. Read our previous blogpost on how to implement Google CSE on your Magento shop. Since we want to let the search engines index the images of our products we set /media/catalog to Allow and Disallow the rest of the directories in /media.

Use an extension for robots.txt instead

If you’re more comfortable working from Magento backend instead of changing files by hand there is an extension you can use to generate a Magento robots.txt file. Robots.txt management tool and can be downloaded via Magento connect. Out of the box this module can generate a robots.txt file for Magento. Via the settings you can alter some main options. After that go to CMS >> Robots.txt >> Manage and install some standard rules for Magento. You might want to change some standard settings. I would like to allow the search engines to index /media/catalog/. Out of the box this rule is not present therefor you have to add it as a new rule. When you’re done setting up the rules click the button to generate robots.txt

Reindex robots.txt

Sometimes it might take a while for search engines to read the changed Magento robots.txt. Using Google Webmasters Tools you can see when Google indexed your robots.txt for the last time. Google should download robots.txt every 24 hours or after 100 visits. If you want Google or other search engines to get the updated version sooner you can use Header Cache-Control in your .htaccess file. Copy the statement below into your .htaccess file.

<filesmatch "\.(txt)$"=""> Header set Cache-Control "max-age=60, public, must-revalidate"

1 2	<filesmatch "\.(txt)$"=""> Header set Cache-Control "max-age=60, public, must-revalidate"

With this statement you tell that all .txt files will expire after 60 seconds and require the user to download the file again. Depending on how often Google crawls your site it will notice an outdated robots.txt and downloads a fresh copy. Increase the max-age once you notice that Google uses the new robots.txt and set it back to 60 seconds the day before you’re going to change the file again. It will save resources.
Update 18-09-2012: naar aanleiding van blogpost Kennisartikel: serverload verlagen Byte hebben we de Magento robots.txt aangepast. URL-parameters welke we eerst blokkeerden worden nu weer toegelaten. Het is aan Google Webmasters tools om deze te blokkeren.
source of image: www.sxc.hu/photo/1171276