Should I Be Using A Robots.txt File

To decide weather or not you need to use a robots.txt you need to understand its purposes and uses before setting one up.

Robots and crawlers like the googlebot visit your site daily to check for new content amongst other things, one of thing you can use the robots.txt file for is to control what pages you don’t want the robots to crawl/index. The robots.txt file has been around for some time so it has become standard for bots to look for the file when they visit your site, you should always put the robots file in your root folder as the bots tend to add the word robots.txt to the end of your url to find it.

Using a robots.txt file can help prevent duplicate content issues and keep private information from appearing in the google results pages. Even if you don’t post duplicate content the chances are google will find another way of finding it for example lets say your homepage is at directorysubmissions.eu it will class directorysubmissions.eu/index.php and directorysubmissions.eu/home.php as duplicate content as they are different urls but display the same content.

There are several ways you can tell google to ignore certain pages and one of the best ways is by using a robots.txt file, you can specify what each search engine bot does when they visit your site or you can set a global command for all the search engines to follow.

The duplicate content issue can be a big problem for some sites so its best to try get on top of it before it becomes a problem. I have written out an example of what your robots.txt file should look like.

user-agent: Googlebot disallow: /mysite/test.php You can use the disallow command to prevent google from indexing directories or particular pages. By changing the user agent to another name you will be stating what that bot should do when it visits your website, you can do as many commands as you like for as many bots as you like so for examples;

user-agent: MSN disallow: /mysite/test.php

user-agent: Googlebot disallow: /mysite/test.php

user-agent: Yahoo

disallow: /mysite/test.php