4WebHelp
 FAQ  •  Search  •  User Groups  •  Forum Admins  •  Smilies List  •  Statistics  •  Rules   •  Login   •  Register
Toggle Navigation Menu

 robots.txt
Post New TopicReply to Topic
View Previous Topic Print this topic View Next Topic
Author Message
jayant
Team Member



Joined: 07 Jan 2002
Posts: 262
Location: New Delhi, India

PostPosted: Tue Apr 23, 2002 8:29 am (15 years, 6 months ago) Reply with QuoteBack to Top

remember, there exist some agents/search engines who try to index only those pages that are disallowed in order to be able to get some possibly sensitive content.

________________________________
Jayant Kumar
Member of the 4WebHelp Team
Nibble Guru - Computing Queries Demystified
GZip/ Page Compression Test
OfflineView User's ProfileFind all posts by jayantSend Personal MessageVisit Poster's WebsiteYahoo MessengerMSN Messenger
Daniel
Team Member



Joined: 06 Jan 2002
Posts: 2564

PostPosted: Wed Apr 24, 2002 7:53 pm (15 years, 6 months ago) Reply with QuoteBack to Top

Well there isn't really anything you can do about that.

________________________________
Image
OfflineView User's ProfileFind all posts by DanielSend Personal Message
Richard
WebHelper
WebHelper


Joined: 12 Jan 2002
Posts: 93
Location: Herts, UK

PostPosted: Fri Apr 26, 2002 4:42 pm (15 years, 6 months ago) Reply with QuoteBack to Top

I've got another question ...

I have a re-direction address on my website to make my URL smaller but when I tell a search engine to crawl my website, it's going to come up with the long URL. Is there anyway of telling the search engine to use my shorter URL ?

Thanks Smile
OfflineView User's ProfileFind all posts by RichardSend Personal Message
Daniel
Team Member



Joined: 06 Jan 2002
Posts: 2564

PostPosted: Fri Apr 26, 2002 5:05 pm (15 years, 6 months ago) Reply with QuoteBack to Top

Nope, not really, since the URL to the pages will be the LONG one, not the SHORT one. It can't change the URL of the pages just because you don't like the short URL Smile

________________________________
Image
OfflineView User's ProfileFind all posts by DanielSend Personal Message
JointComms
WebHelper
WebHelper


Joined: 03 May 2002
Posts: 79
Location: Windsor, CANADA

PostPosted: Fri May 03, 2002 6:23 pm (15 years, 5 months ago) Reply with QuoteBack to Top

I may have missed something or may be it's just my ignorance on the subject.

Where would one write the code for the disallow statement.

Thanks
OfflineView User's ProfileFind all posts by JointCommsSend Personal MessageSend email
Daniel
Team Member



Joined: 06 Jan 2002
Posts: 2564

PostPosted: Fri May 03, 2002 6:28 pm (15 years, 5 months ago) Reply with QuoteBack to Top

In a file called robots.txt, placed in the root of your site.

________________________________
Image
OfflineView User's ProfileFind all posts by DanielSend Personal Message
JointComms
WebHelper
WebHelper


Joined: 03 May 2002
Posts: 79
Location: Windsor, CANADA

PostPosted: Sun May 12, 2002 5:58 pm (15 years, 5 months ago) Reply with QuoteBack to Top

Thanks Daniel
OfflineView User's ProfileFind all posts by JointCommsSend Personal MessageSend email
Ben
Senior WebHelper
Senior WebHelper


Joined: 08 Jan 2002
Posts: 431
Location: Liverpool - UK

PostPosted: Mon May 13, 2002 1:03 am (15 years, 5 months ago) Reply with QuoteBack to Top

and its the first place a hacker will look when trying to find "secret areas" on your site Laughing

________________________________
Ben Scott

Red and White Kop
OfflineView User's ProfileFind all posts by BenSend Personal MessageSend emailVisit Poster's Website
Daniel
Team Member



Joined: 06 Jan 2002
Posts: 2564

PostPosted: Mon May 13, 2002 6:05 am (15 years, 5 months ago) Reply with QuoteBack to Top

But you've got to be stupid not to password protect it Smile

________________________________
Image
OfflineView User's ProfileFind all posts by DanielSend Personal Message
Display posts from previous:      
Post New TopicReply to Topic
View Previous Topic Print this topic View Next Topic


 Jump to:   


Go to page Previous  1, 2

You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot edit your posts in this forum.
You cannot delete your posts in this forum.
You cannot vote in polls in this forum.


Page generation time: 0.05323 seconds :: 17 queries executed :: All Times are GMT
Powered by phpBB 2.0 © 2001, 2002 phpBB Group :: Based on an FI Theme