What comes to your mind when I tell you to optmize your site? You will definitely say something like titles, friendly urls, quantity and quanlity of backlinks… etc. But do not ignore the trivial things when it comes to SEO since it still can impact the site to some extent and sometimes to a large extent.
95% of webmasters (or even more) know robots.txt tell search engines to exclude certain areas of your website such as /images and /temp. However, almost less than 35% of websites had a robots file and a lot of sites simply copying a robotx.txt file from another site. Spidering is a costly effort that search engines try to optimize. Therefore, it makes sense that websites that demonstrates optimal command will get rewarded.
At SES NewYork Summit, all search engines announced that they will identify site map via robots.txt file. So now, you got one more use with robots.txt by adding a line like this:
sitemap: <your sitemap URL>. A lot of people have been asking whether they can submit a sitemap to MSN. Well, here is your chance to tell MSN that you have a sitemap and hopefully that will get your site quickly indexed.
We know Google has alwasy been trying to return the best search results. Thus, Google wants to eliminate any pages that does not add any value. Google Webmasters Gudielines clearly tells that it wants webmasters to use robots.txt to to prevent crawling pages that does not add any value to users. So what are you waiting for, it is what Google want, if we want good ranking results on Google, then at least comply with their guidelines.
For more instructions, you can visit robotstxt.org

