Sasidhar | Blog

Just found some interesting stuff on Digg.com. I looked at the robots.txt file, and found that Digg disallows a bot by the name, “Referrer Karma”. Now, as a concerned webizen (web citizen, coined a new word right there !), I looked up on the internet what Referrer Karma is and found that DDoS attacks can be caused using Referrer Karma. Thought it was pretty cool, that Digg is taking precautions against it. For a site that is run totally on user generated content, this is something that they NEED to be doing better I guess.

Here is how the robots.txt file of Digg.com looks as:

User-agent: Referrer Karma/2.0 Disallow: /

Another interesting fact is, if you search for digg.com/robots.txt on Google, you get a link with Title as link20. Don’t know whats with that !! because robots.txt is a text file and can’t have a title corresponding to it.
At first, I thought this was the reason digg.com is not in Archive.org aka Wayback machine. But, I just found out that archive.org uses archive.org_bot as Referrer. So, there must be an other reason for that !

Something else that I found interesting was;

Digg was down in some regions in the world, if I’m not mistaken. No one in US or Europe noticed any downtime. Asia, though had a few minutes where Digg was down, and all the urls from the site were redirecting to http://down.digg.com. This page is still up and you can look at it here. I have a screenshot, just in case and I will put it up here.