A robots.txt Guide For Newspaper Publishers

January 21st, 2007 Ryan Jones

While more and more companies are entering into a paid search battle to get more traffic it seems that newspapers are trying to do just the opposite. This is especially the case in Belgium. First there was the court case that ruled against Google claiming that they couldn’t index a Belgian newspaper. Now, as TechDirt points out, it looks like they’re going after Yahoo.

In an effort to help out, I’m now going to tell all newspapers how to get their sites de-listed from ALL search engines without having to hire a lawyer.

Step 1: Create a new text file called robots.txt The easiest way to do this is to click on start in the lower left corner, click run. Now, type in “notepad robots.txt”. A Windows prompt will come up saying “cannot find file, would you like to create one?”. Select Yes.

Step 2: Enter the following text:

User-agent: *
Disallow: /


Don’t forget to save this file.

Step 3: Upload this file to your server. You may have to talk to somebody from your tech team for help with this. Another way is to go to start, run and this time type in ftp://Username:[email protected]. (of course, replace username, password and yoursite with the actual details. Then, you can just drag and drop the robots.txt file into the window. Congratulations! You’ve now blocked Google AND Yahoo from visiting your website, and it didn’t even cost you any retainer fees.

Of course you’ve also cut off millions of potential visitors, which begs the question “Why are you even putting your articles online if you don’t want anybody to see them?”

• Tweet
• Pin It

Entry Filed under: Main