robots.txt and Wordpress

Author Topic: robots.txt and Wordpress  (Read 2127 times)

Offline itraiderTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 45
  • Karma: 1
robots.txt and Wordpress
« on: 05-23-2010, 16:23:27 »
What's your typical robots.txt file look like? I think it's good to get this done right at the start to make sure only relevant pages get indexed.
The virtual robots.txt that gets generated does not really exclude the admin related pages.


Offline aprilhomer

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 2
  • Posts: 115
  • Karma: 2
  • Gender: Female
    • http://www.cafe24.com
Re: robots.txt and Wordpress
« Reply #1 on: 02-16-2012, 22:40:19 »
you can see robots.txt on many sites. like http://www.google.com/robots.txt most of the websites have robots.txt by the way, If you want to create a new website you can signup Cafe24. Its free!

 

Related Topics

  Subject / Started by Replies Last post
0 Replies
1593 Views
Last post 09-10-2011, 10:16:39
by Maddy
15 Replies
4873 Views
Last post 03-03-2014, 05:55:34
by kumar29
11 Replies
4162 Views
Last post 10-05-2016, 06:39:48
by mukeshkr
5 Replies
2180 Views
Last post 04-12-2016, 23:39:58
by jaysh4922
10 Replies
3992 Views
Last post 05-27-2016, 03:16:06
by Kate Evans