robots.txt

Posted: Tue Apr 23, 2002 8:29 am (21 years, 12 months ago)

remember, there exist some agents/search engines who try to index only those pages that are disallowed in order to be able to get some possibly sensitive content.

Team Member Joined: 06 Jan 2002 Posts: 2564

Well there isn't really anything you can do about that.

Posted: Fri Apr 26, 2002 4:42 pm (21 years, 12 months ago)

I've got another question ...

I have a re-direction address on my website to make my URL smaller but when I tell a search engine to crawl my website, it's going to come up with the long URL. Is there anyway of telling the search engine to use my shorter URL ?

Thanks Smile

Team Member Joined: 06 Jan 2002 Posts: 2564

Nope, not really, since the URL to the pages will be the LONG one, not the SHORT one. It can't change the URL of the pages just because you don't like the short URL Smile

Posted: Fri May 03, 2002 6:23 pm (21 years, 11 months ago)

I may have missed something or may be it's just my ignorance on the subject.

Where would one write the code for the disallow statement.

Thanks

Team Member Joined: 06 Jan 2002 Posts: 2564

In a file called robots.txt, placed in the root of your site.

Posted: Sun May 12, 2002 5:58 pm (21 years, 11 months ago)

Thanks Daniel

Posted: Mon May 13, 2002 1:03 am (21 years, 11 months ago)

and its the first place a hacker will look when trying to find "secret areas" on your site Laughing

Team Member Joined: 06 Jan 2002 Posts: 2564

But you've got to be stupid not to password protect it Smile