Thread: spiders ?
View Single Post
  #2  
Old 2010-02-03, 12:11 PM
saltman's Avatar
saltman saltman is offline
Shareblue Platinum Member
TTD Staff
 
Join Date: Dec 2004
Re: spiders ?

a spider is a program that searches the web for information to propagate search sites such as google, bing, yahoo, etc.

Only certain pages here are viewable by spiders. ( at least the ones that obey robots.txt and login/pass rights ) Technobabble for instance.

http://en.wikipedia.org/wiki/Web_crawler
__________________
68 Stat. 775, 50 U.S.C. 841-844
Reply With Quote Reply with Nested Quotes