Robots.txt language understanding

Author Topic: Robots.txt language understanding  (Read 2736 times)

Offline RossKariTopic starter

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 2
  • Karma: 0
Robots.txt language understanding
« on: 01-08-2022, 14:42:27 »
Hi All,

I am trying to read the code on robots.txt but can't understand everything properly.

Could you help me to find the source from where I can learn language of robots.txt.

As you can see from below, someone blocked a lot of pages, but I can't understand the intention of those blocked urls.

Thanks,

The code itself looks like this:



User-agent: *
Allow: /wp-content/*.js
Allow: /wp-content/*.css
Disallow: /cgi-bin
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /wp-trackback
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /trackback
Disallow: /tag
Disallow: /category
Disallow: /archive
Disallow: */trackback/
Disallow: */*/trackback
Disallow: */comments/
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?
Disallow: /*?*
Disallow: /*.php


User-agent: Yandex
Allow: /wp-content/*.js
Allow: /wp-content/*.css
Disallow: /cgi-bin
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /wp-trackback
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /trackback
Disallow: /tag
Disallow: /category
Disallow: /archive
Disallow: */trackback/
Disallow: */*/trackback
Disallow: */comments/
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?
Disallow: /*?*
Disallow: /*.php


User-agent: Googlebot
Allow: *.css
Allow: *.js
Allow: /wp-content/uploads/
Allow: /wp-content/uploads/*.png
Allow: /wp-content/uploads/*.jpg
Allow: /wp-content/uploads/*.jpeg
Allow: /wp-content/uploads/*.gif
Allow: /wp-content/cache/minify/*.js*
Allow: /wp-content/cache/minify/*.css*
Allow: /wp-content/themes/*.png*
Allow: /wp-content/themes/*.jpg*
Allow: /wp-content/themes/*.jpeg*
Allow: /wp-content/themes/*.gif*
Allow: /wp-content/themes/*.webp*
Allow: /wp-content/themes/astra-child/images/restyle/*.png
Allow: /wp-content/themes/astra-child/images/restyle/*.jpg
Allow: /wp-content/themes/astra-child/images/restyle/*.jpeg
Allow: /wp-content/themes/astra-child/images/restyle/*.gif
Allow: /wp-content/themes/astra-child/images/restyle/*.webp
Allow: /wp-content/themes/astra/assets/fonts/*.ttf*
Allow: /wp-content/themes/astra/assets/fonts/*.woff*
Allow: /wp-content/*.svg*
Allow: /wp-content/*.js*
Allow: /wp-content/*.css*
Allow: /wp-includes/*.js
Allow: /wp-includes/*.ttf
Allow: /wp-includes/*.woff
Allow: /wp-includes/*.js*
Allow: /wp-includes/*.ttf*
Allow: /wp-includes/*.woff*
Disallow: /cgi-bin
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /wp-trackback
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /trackback
Disallow: /tag
Disallow: /category
Disallow: /archive
Disallow: */trackback/
Disallow: */*/trackback
Disallow: */comments/
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?
Disallow: /*?*
Disallow: /*.php


Offline Naad Nodal

  • Trade Count: (0)
  • Semi-Newbie
  • *
  • Thank You 0
  • Posts: 30
  • Karma: 0
  • Gender: Male
    • Best Music School Near You | Naad Acadamy
Re: Robots.txt language understanding
« Reply #1 on: 01-09-2022, 08:34:25 »
Hello,

thank you for sharing this wonderful information.
best music school in surat, best music classes in surat, best guitar classes in surat, best flute classes in surat, best keyboard classes near me, best music classes near me, best music school near me


Offline RossKariTopic starter

  • Trade Count: (0)
  • Newbie
  • *
  • Thank You 0
  • Posts: 2
  • Karma: 0
Re: Robots.txt language understanding
« Reply #2 on: 01-09-2022, 09:43:35 »
For example,

Disallow: /*?   Disallow: /*?
Disallow: /*?*   Disallow: /*?*

why they disallowed *? and *?* on for all agents and for googlebot?

and also what are the reasons for disallowing the rest such as disallow: /tag

Thanks,

 

Related Topics

  Subject / Started by Replies Last post
0 Replies
1497 Views
Last post 09-10-2011, 10:29:05
by Maddy
4 Replies
2429 Views
Last post 07-06-2012, 05:02:21
by jamesdavid
0 Replies
4116 Views
Last post 07-02-2016, 05:54:13
by chinmay.sahoo
4 Replies
2334 Views
Last post 12-28-2016, 03:47:42
by wellliving
4 Replies
4173 Views
Last post 12-27-2016, 03:47:18
by damponting44