Slashdot Disallows MSN and Yahoo but allows Google to crawl their site?
Did you know Slashdot does not allow MSN and Yahoo News Crawler or Slurp(Yahoo/Inktomi bot) into their site. Take a look at the robots.txt file of slashdot.org at http://slashdot.org/robots.txt
User-agent: Slurp
Crawl-delay: 100
Disallow:User-agent: Yahoo-NewsCrawler
Disallow:User-Agent: msnbot
Crawl-delay: 100
Disallow:
You will see three bots; Slurp, Yahoo News Crawler and msnbot being disallowed from crawling all parts of their website. Slurp and YahooNews crawler are bots for Yahoo, while msnbot crawls for msn.com and now live.com. Anyone knows why?
Slashdot, ‘news for nerds’ is one of THE places for geek/nerd information, news and related events. It was one of the biggest sites for user generated news and views site, untill Digg came along. Still a site with huge set of members, very loyal users. News for nerds !! I love the catch line.
Another geek/nerd hangout before the internet boom was deja.com which was later aquired by Google and renamed to Google Groups or Usenet as it is now called. Another place for nerds to hang out !
Update: Oops, I was wrong. Read the robots.txt file wrong !! My mistake. Ignore this post guys.
I would like to blame, posting at 2am for this mishap !!
About this entry
You’re currently reading “Slashdot Disallows MSN and Yahoo but allows Google to crawl their site?,” an entry on Sasidhar | Blog | Life
- Published:
- 10.14.06 / 2am
- Category:
- search engines /seo, technology

1 Comment
Jump to comment form | comments rss [?] | trackback uri [?]