Hello friends,
Which is better Robots.txt or Meta Robots Tag?
Hello,
The Meta tag option tells bots not to index individual files, whereas Robots.txt can be used to restrict access to entire directories and also show them the sitemap location.
I suggest you to write a robots.txt file.
You can also combine tags & robots.txt for your needs.
More informations here: https://www.weareinfront.com/learn-article/robots_txt/
If you are trying to improve SEO it's ok to use both. You may want to and noindex with meta for the RSS feed, all back-end pages, comments and other pages without important content.
Robots.txt works fine, but keep in mind settings within robots.txt are just guidelines for search engines. They can choose to ignore them, so don't use this as a way to secure/hide web pages.
robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.
whereas meta tag ,you have to provide allow!/disallow to read the content of your web page !!bu default allow exists
The purpose of a robots.txt file, also known as the robots exclusion protocol, is to give webmasters control over what pages robots (commonly called spiders) can crawl and index on their site. A typical robots.txt file, placed on your site's server, should include your sitemap's URL and any other parameters you wish to put in place.