How to use robots.txt to block the Internet Archiver

post Category: Code, Internet — Chris @ 3:36 am — post
postSeptember 18, 2006

A very simple robots.txt file placed in your docroot will stop your webpage being archived forever at the Internet Archive. You probably will wish to do this for privacy reasons. It’s interesting to note that myspace specifically block the way-back archiver bot.

Here’s what you need to include in your robots file:

User-agent: ia_archiver
Disallow: /

« Migrating ISPs
I have my MAC code..! »

1 person has left a comment

#1

Robots are cool -especially Megatron

ed wrote on September 27, 2006 - 9:37 am
You can leave a response, or trackback from your own site.

Write Your Comment

Comment Guidelines: Basic XHTML is allowed (a href, strong, em, code). All line breaks and paragraphs will be generated automatically.

You should have a name, right? 
Your email address, I promised I won't tell it to anyone. 
If you have a web site or blog, you can type the URL right here. 
This is where you type your comments. 
Remember my information for the next time I visit.
 

This is a captcha-picture. It is used to prevent mass-access by robots. (see: www.captcha.net)

You must read and type the 4 chars within 0..9 and A..F, and submit the form.

  

Oh no, I cannot read this. Please, generate a