Would you like to optimize your WordPress "robots.txt" file?
You may not know how and why this file is so important for your SEO ?
We have what you need.
In this tutorial, I'll show you how to optimize the file " robots.txt For better SEO and I will help you better understand the importance of these files.
Recently one of our subscribers wanted to know how important the file really was " robots.txt For his blog.
First, you need to know that this is a communication gateway between you and the search engines, through which you give them details of how your blog should be indexed.
Do you really need this file?
The absence of this file will not prevent search engines from indexing your blog. However, it is strongly recommended to create one.
If you want to submit a file « XML sitemap »To search engines, this is where the engines will search for the XML file, unless it is manually specified on the Google Webmaster tools (Google Webmasters Tools)
We strongly recommend that you immediately create a “robots.txt” file if you do not have one.
But before, if you have never installed WordPress discover How to Install a WordPress Blog in 7 Steps et How to search, install and activate a WordPress theme on your blog
Then back to why we are here.
Where is the "robots.txt" file, how to create one?
The file " Robots.txt Is usually available at the root of your installation. You will need to use an FTP client to access it or use your cPanel to manage your files.
Go further by discovering How to manage WordPress files and folders
It is an ordinary file like all the others and you just have to open it with a text editor any like notepad.
If you can not find the file " robots.txt At the root of your installation, then you can always create one. All you need to do is create a text file named " robots.txt »On your computer and send it to the root of your hosting using an FTP client as specified above.
How to use the file "robots.txt"?
The syntax to use for the file robots.txt Is very simple. The first line usually specifies the name of an Agent. Agent is the name of a robot you are communicating with. For example " Googlebot " or " Bingbot For the respective Google and Bing robots. You can use "*" to give instructions to all robots at the same time.
The following line begins with " Allow " or " Disallow "(Authorize or prohibit) indexing a path. This way, robots will know what to index and what not to index.
Here is an example of a " robots.txt »
In this WordPress “robots.txt” draft file, we gave the robots instructions to index the entire media file directory.
The last two lines prohibit indexing the plugins directory and the “readme.html” file.
How to optimize your "robots.txt" file for better SEO
Google recommends against using the "robots.txt" file to hide irrelevant content. If you were going to use it to prevent Google from indexing your categories, archives and so on, then you should know that this is not a good practice.
Remember, the purpose of the file is to give instructions to robots so that they can properly index your blog. It does not prevent robots from analyzing your blog.
There is a WordPress Plugin which allows you to add tags " arches " as " nofollow " and " noindex »On the pages of your archives, this is the plugin WordPress SEO. We're not saying that you absolutely should deindex your archives, but if you want to do so, this is the most appropriate way.
You don't need to add the login page, or the administration directory or even the registration page in your "robots.txt" file because these pages already have the tags " noindex ».
It is recommended to disable playback of the readme.html file from your file robots.txt ". This file can be used by anyone to retrieve the version of WordPress that you are using to check for possible vulnerabilities.
But also, if another person runs malicious code to search for blogs using a specific version of WordPress, then disabling this file can protect you from multiple attacks.
Also discover our 10 WordPress plugins to strengthen the security of your blog
You can disable the plugins directory. This will add an extra barrier of protection to your blog.
How to add the XML map site to the "robots.txt" file
If you are using a plugin to generate your XML file, then your plugin will automatically try to add a sitemap to the " robots.txt ».
Discover Using Sitemaps WordPress
However, if it doesn't, you can do so by adding the following lines:
What does a complete "robots.txt" file look like?
Honestly, most blogs use a simple "robots.txt" file. Their content varies according to the needs of each blog.
This file tells robots to index all content and provide the path to the XML files.
Discover also some premium WordPress plugins
You can use other WordPress plugins to give a modern appearance and to optimize the handling of your blog or website.
We offer you here some premium WordPress plugins that will help you do that.
1. CommentPress
CommentPress is a powerful WordPress compact plugin that lets you insert, edit and delete comments quickly and easily. CommentPress uses Ajax, jQuery and PHP to give visitors the ability to insert comments without having to reload the page.
This plugin has many customization options, finally giving it the look and feel you want. It has a captcha to block spam and brings to your comment section more security.
Its main features are: the easy insertion of comments, the possibility for users to reply to specific comments, the pagination of comments, complete management of the administration of the comments section, buttons for easy insertion of images, videos and links, a fully responsive layout, a Captcha to secure the form and prevent spam, and much more.
Download | Demo | Web hosting
2. jqEasytooltip
JqEasytooltip is a publisher for WordPress easy to use and customizable tooltips. VSe WordPress Plugin Comes with lots of tooltip templates, css3 transition effects, and 300+ amazing font icons included for buyers. All of its configurations can be managed through the editor integration tinymce. 
And its main features are: aQuick and easy tooltip editor (Integration tinymce), compatible with other plugins, a layout efully responsive, the support of jquery plugin Jqeasytooltip jquery, 35 theme models to choose from and customize, pMore than 300 ready-to-use icons, 10 animations in pure CSS3 to appear and disappear, etc ...
Download | Demo | Web hosting
3. Social Auto Poster
Social Auto Poster is a WordPress Plugin and a perfect choice for automatically publishing your content to social media platforms such as: Facebook, LinkedIn, Twitter, Tumbler, BufferApp, Instagram and Pinterest. 
You will be able to set up your social networking accounts and publish new content, or republish an old one to keep it alive and reach a wider audience.
This WordPress plugin provides an amazing feature that allows you to schedule the publication of your content anytime. It also works with personal profiles, professional pages, groups, etc.
Download | Demo | Web hosting
Recommended Resources
Find out about other recommended resources to help you build and manage your website.
- How to add interactive maps to your WordPress website
- How to create Excel and Word documents on WordPress
- How to compress and resize images on WordPress
- How to find the login URL of your WordPress blog
Conclusion
Here ! That's all for this tutorial on file optimization " robots.txt ». Feel free to share the tip with your friends on your favorite social networks.
However, you will also be able to consult our resources, if you need more elements to carry out your projects of creation of Internet sites, by consulting our guide on the WordPress blog creation.
If you have suggestions or remarks, leave them in our section Comments.
...
