What is robots.txt? Does it have any value for SEO?

Author Topic: What is robots.txt? Does it have any value for SEO?  (Read 10368 times)

Offline hieronymusf01

  • Trade Count: (0)
  • Jr. Member
  • **
  • Thank You 10
  • Posts: 50
  • Karma: 0
    • Cheap seo services India
Re: What is robots.txt? Does it have any value for SEO?
« Reply #30 on: 04-14-2016, 06:06:44 »
How can robots.txt be used for SEO?

    You can use robots.txt to block Google or Bing from crawling folders or pages you don't want them to crawl.
    Be absolutely sure that you do not block any URL that you want Google to index and display in search results.
    In general, it's better to use a robots meta tag with parameters "noindex, follow" to keep search engines from indexing a page on your site. This way, you won't lose any PageRank or link juice pointed at the page.


Offline nguyenmailan

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 8
  • Posts: 41
  • Karma: 0
    • imeo
Re: What is robots.txt? Does it have any value for SEO?
« Reply #31 on: 04-24-2016, 20:52:14 »
There are two important considerations when using robots.txt:

- Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
- The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

 

Related Topics

  Subject / Started by Replies Last post
4 Replies
2717 Views
Last post 09-20-2018, 22:24:07
by MVMInfotech18
11 Replies
4264 Views
Last post 10-05-2016, 06:39:48
by mukeshkr
5 Replies
2227 Views
Last post 04-12-2016, 23:39:58
by jaysh4922
4 Replies
1901 Views
Last post 12-28-2016, 03:47:42
by wellliving
4 Replies
2850 Views
Last post 12-27-2016, 03:47:18
by damponting44