robots.txt and Wordpress

Author Topic: robots.txt and Wordpress  (Read 1924 times)

Offline itraiderTopic starter

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 45
  • Karma: 1
robots.txt and Wordpress
« on: 05-23-2010, 16:23:27 »
What's your typical robots.txt file look like? I think it's good to get this done right at the start to make sure only relevant pages get indexed.
The virtual robots.txt that gets generated does not really exclude the admin related pages.


Offline aprilhomer

  • Trade Count: (0)
  • Full Member
  • ***
  • Thank You 2
  • Posts: 115
  • Karma: 2
  • Gender: Female
    • http://www.cafe24.com
Re: robots.txt and Wordpress
« Reply #1 on: 02-16-2012, 22:40:19 »
you can see robots.txt on many sites. like http://www.google.com/robots.txt most of the websites have robots.txt by the way, If you want to create a new website you can signup Cafe24. Its free!

 

Related Topics

  Subject / Started by Replies Last post
0 Replies
1393 Views
Last post 09-10-2011, 10:16:39
by Maddy
15 Replies
4324 Views
Last post 03-03-2014, 05:55:34
by kumar29
11 Replies
3642 Views
Last post 10-05-2016, 06:39:48
by mukeshkr
5 Replies
1827 Views
Last post 04-12-2016, 23:39:58
by jaysh4922
11 Replies
3247 Views
Last post 06-01-2016, 05:36:15
by urbansoft