robots.txt and Wordpress

Author Topic: robots.txt and Wordpress  (Read 2401 times)

Offline itraiderTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 45
  • Karma: 1
robots.txt and Wordpress
« on: 05-23-2010, 16:23:27 »
What's your typical robots.txt file look like? I think it's good to get this done right at the start to make sure only relevant pages get indexed.
The virtual robots.txt that gets generated does not really exclude the admin related pages.


Offline aprilhomer

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 2
  • Posts: 115
  • Karma: 2
  • Gender: Female
    • http://www.cafe24.com
Re: robots.txt and Wordpress
« Reply #1 on: 02-16-2012, 22:40:19 »
you can see robots.txt on many sites. like http://www.google.com/robots.txt most of the websites have robots.txt by the way, If you want to create a new website you can signup Cafe24. Its free!

 

Related Topics

  Subject / Started by Replies Last post
0 Replies
2224 Views
Last post 09-10-2011, 10:16:39
by Maddy
15 Replies
5196 Views
Last post 03-03-2014, 05:55:34
by kumar29
11 Replies
4593 Views
Last post 10-05-2016, 06:39:48
by mukeshkr
5 Replies
2425 Views
Last post 04-12-2016, 23:39:58
by jaysh4922
10 Replies
4289 Views
Last post 05-27-2016, 03:16:06
by Kate Evans