Search Engine Optimization > Advanced Search Engine Optimization

Robots.txt language understanding

(1/1)

RossKari:
Hi All,

I am trying to read the code on robots.txt but can't understand everything properly.

Could you help me to find the source from where I can learn language of robots.txt.

As you can see from below, someone blocked a lot of pages, but I can't understand the intention of those blocked urls.

Thanks,

The code itself looks like this:



User-agent: *
Allow: /wp-content/*.js
Allow: /wp-content/*.css
Disallow: /cgi-bin
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /wp-trackback
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /trackback
Disallow: /tag
Disallow: /category
Disallow: /archive
Disallow: */trackback/
Disallow: */*/trackback
Disallow: */comments/
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?
Disallow: /*?*
Disallow: /*.php


User-agent: Yandex
Allow: /wp-content/*.js
Allow: /wp-content/*.css
Disallow: /cgi-bin
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /wp-trackback
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /trackback
Disallow: /tag
Disallow: /category
Disallow: /archive
Disallow: */trackback/
Disallow: */*/trackback
Disallow: */comments/
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?
Disallow: /*?*
Disallow: /*.php


User-agent: Googlebot
Allow: *.css
Allow: *.js
Allow: /wp-content/uploads/
Allow: /wp-content/uploads/*.png
Allow: /wp-content/uploads/*.jpg
Allow: /wp-content/uploads/*.jpeg
Allow: /wp-content/uploads/*.gif
Allow: /wp-content/cache/minify/*.js*
Allow: /wp-content/cache/minify/*.css*
Allow: /wp-content/themes/*.png*
Allow: /wp-content/themes/*.jpg*
Allow: /wp-content/themes/*.jpeg*
Allow: /wp-content/themes/*.gif*
Allow: /wp-content/themes/*.webp*
Allow: /wp-content/themes/astra-child/images/restyle/*.png
Allow: /wp-content/themes/astra-child/images/restyle/*.jpg
Allow: /wp-content/themes/astra-child/images/restyle/*.jpeg
Allow: /wp-content/themes/astra-child/images/restyle/*.gif
Allow: /wp-content/themes/astra-child/images/restyle/*.webp
Allow: /wp-content/themes/astra/assets/fonts/*.ttf*
Allow: /wp-content/themes/astra/assets/fonts/*.woff*
Allow: /wp-content/*.svg*
Allow: /wp-content/*.js*
Allow: /wp-content/*.css*
Allow: /wp-includes/*.js
Allow: /wp-includes/*.ttf
Allow: /wp-includes/*.woff
Allow: /wp-includes/*.js*
Allow: /wp-includes/*.ttf*
Allow: /wp-includes/*.woff*
Disallow: /cgi-bin
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /wp-trackback
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /trackback
Disallow: /tag
Disallow: /category
Disallow: /archive
Disallow: */trackback/
Disallow: */*/trackback
Disallow: */comments/
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?
Disallow: /*?*
Disallow: /*.php

Naad Nodal:
Hello,

thank you for sharing this wonderful information.

RossKari:
For example,

Disallow: /*?   Disallow: /*?
Disallow: /*?*   Disallow: /*?*

why they disallowed *? and *?* on for all agents and for googlebot?

and also what are the reasons for disallowing the rest such as disallow: /tag

Thanks,

Navigation

[0] Message Index