Seo Forum

Search Engine Optimization => SEO Basics => Topic started by: pixhngeh on 12-29-2011, 02:30:43

Title: How do you control access of the web crawlers?
Post by: pixhngeh on 12-29-2011, 02:30:43
Hello,
There are numerous reasons as to why or when you should control the access of the web robots or web crawlers to your site.  As much as you want Googlebot to come to you site, you don’t want the spam bots to come and collect private information from your site.
How do you control access of the web crawlers?
Title: Re: How do you control access of the web crawlers?
Post by: Hogward on 12-29-2011, 06:29:55
By the effective use of robots.txt you can control the access of the web crawlers. whenever the bot crawls the site it checks with robot.txt file to crawl the pages. You can make the bot to allow/disallow the site with it.
Title: Re: How do you control access of the web crawlers?
Post by: crowdfinch on 12-31-2011, 01:13:55
You can restrict web crawlers from robots.txt file.
Title: Re: How do you control access of the web crawlers?
Post by: macjonshonm on 01-12-2012, 02:58:07
As per my suggestion you can use robot.txt to not access any page of your site. Thank you.


Latest MBA News (http://www.mbaupdates.com/index.aspx) | CAT 2012 Exams (http://www.mbaupdates.com/index.aspx)
Title: Re: How do you control access of the web crawlers?
Post by: cpaoutsourcing on 01-19-2012, 22:54:39
By the successful use of programs.txt you can management the accessibility of the web programs. whenever the bot crawls the website it assessments with automatic robot.txt computer file to examine the websites. You can create the bot to allow/disallow the website with it.
Title: Re: How do you control access of the web crawlers?
Post by: Rattan11 on 03-03-2019, 03:44:50
There are numerous reasons as to why or when you should control the access of the web robots or web crawlers to your site.  As much as you want Googlebot to come to you site, you don’t want the spam bots to come and collect private information from your site. Not to mention that when a robot crawls your site it uses the website’s bandwidth too!

Why use ‘robots.txt’ file?
Gooble bot may be crawling your site to provide better search results but at the same time other spam bots may be collecting personal information such as email addresses for spamming purpose. If you want to control the access of the web crawlers on your site, you can do so by using the “robots.txt” file.

How do I create ‘robots.txt’ file?
‘robots.txt’ is a plain text file. Use any text editor to create the ‘robots.txt’ file.

Examples
The following will stop all robots from crawling your site (‘*’ means all and ‘/’ is the root directory.)

User-agent: *
Disallow: /

The following will stop all robots from crawling the ‘/private’ directory.

User-agent: *
Disallow: /private
Title: Re: How do you control access of the web crawlers?
Post by: lishmaliny on 10-18-2019, 04:47:50
As much as you want Googlebot to come to you site, you don’t want the spam bots to come and collect private information from your site. Not to mention that when a robot crawls your site it uses the website’s bandwidth too! In this post I have explained how you can control the access of the web robots to your site through the usage of a simple ‘robots.txt’ file.