robots.txt and Wordpress

Author Topic: robots.txt and Wordpress  (Read 2065 times)

Offline itraiderTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 45
  • Karma: 1
robots.txt and Wordpress
« on: 05-23-2010, 16:23:27 »
What's your typical robots.txt file look like? I think it's good to get this done right at the start to make sure only relevant pages get indexed.
The virtual robots.txt that gets generated does not really exclude the admin related pages.


Offline aprilhomer

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 2
  • Posts: 115
  • Karma: 2
  • Gender: Female
    • http://www.cafe24.com
Re: robots.txt and Wordpress
« Reply #1 on: 02-16-2012, 22:40:19 »
you can see robots.txt on many sites. like http://www.google.com/robots.txt most of the websites have robots.txt by the way, If you want to create a new website you can signup Cafe24. Its free!

 

Related Topics

  Subject / Started by Replies Last post
0 Replies
1540 Views
Last post 09-10-2011, 10:16:39
by Maddy
15 Replies
4796 Views
Last post 03-03-2014, 05:55:34
by kumar29
11 Replies
4042 Views
Last post 10-05-2016, 06:39:48
by mukeshkr
5 Replies
2116 Views
Last post 04-12-2016, 23:39:58
by jaysh4922
11 Replies
3883 Views
Last post 06-01-2016, 05:36:15
by urbansoft