How do you control access of the web crawlers?

Author Topic: How do you control access of the web crawlers?  (Read 1407 times)

Offline pixhngehTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 24
  • Karma: 1
How do you control access of the web crawlers?
« on: 12-29-2011, 02:30:43 »
Hello,
There are numerous reasons as to why or when you should control the access of the web robots or web crawlers to your site.  As much as you want Googlebot to come to you site, you don’t want the spam bots to come and collect private information from your site.
How do you control access of the web crawlers?


Offline Hogward

  • Trade Count: (0)
  • Sr. Member
  • ****
  • Thank You 4
  • Posts: 420
  • Karma: 5
Re: How do you control access of the web crawlers?
« Reply #1 on: 12-29-2011, 06:29:55 »
By the effective use of robots.txt you can control the access of the web crawlers. whenever the bot crawls the site it checks with robot.txt file to crawl the pages. You can make the bot to allow/disallow the site with it.

Offline crowdfinch

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 40
  • Karma: 1
  • Gender: Female
    • CrowdFinch Technologies
Re: How do you control access of the web crawlers?
« Reply #2 on: 12-31-2011, 01:13:55 »
You can restrict web crawlers from robots.txt file.

Offline macjonshonm

  • Trade Count: (0)
  • Novice
  • *
  • Thank You 0
  • Posts: 0
  • Karma: 0
Re: How do you control access of the web crawlers?
« Reply #3 on: 01-12-2012, 02:58:07 »
As per my suggestion you can use robot.txt to not access any page of your site. Thank you.


newbielink:http://www.mbaupdates.com/index.aspx [nonactive] | newbielink:http://www.mbaupdates.com/index.aspx [nonactive]

Offline cpaoutsourcing

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 7
  • Karma: 1
Re: How do you control access of the web crawlers?
« Reply #4 on: 01-19-2012, 22:54:39 »
By the successful use of programs.txt you can management the accessibility of the web programs. whenever the bot crawls the website it assessments with automatic robot.txt computer file to examine the websites. You can create the bot to allow/disallow the website with it.
newbielink:http://www.cpa-outsourcing.com/ [nonactive]
newbielink:http://www.cpa-outsourcing.com/ [nonactive]


 

Related Topics

  Subject / Started by Replies Last post
5 Replies
1038 Views
Last post 11-05-2011, 00:01:18
by Vinil
1 Replies
2361 Views
Last post 07-28-2014, 02:03:44
by Siservices
3 Replies
1651 Views
Last post 02-28-2015, 02:42:43
by jannatul18
4 Replies
1249 Views
Last post 11-23-2015, 03:04:50
by seoroy
3 Replies
1044 Views
Last post 08-23-2016, 06:48:00
by TomClarke