Would you like to optimize your WordPress "robots.txt" file?

You may not know how and why this file is so important for your SEO?

We have what you need.

In this tutorial, I'll show you how to optimize the file " robots.txt For better SEO and I will help you better understand the importance of these files.

Recently one of our subscribers wanted to know how important the file really was " robots.txt For his blog.

First, you need to know that this is a communication gateway between you and the search engines, through which you give them details of how your blog should be indexed.

Do you really need this file?

The absence of this file will not prevent search engines from indexing your blog. However, it is strongly recommended that you create one.

If you want to submit a file « XML sitemap »To search engines, this is where the engines will search for the XML file, unless it is manually specified on the Google Webmaster tools (Google Webmasters Tools)

We strongly recommend that you immediately create a “robots.txt” file if you do not have one. 

But before, if you have never installed WordPress discover How to Install a WordPress Blog in 7 Steps et How to search, install and activate a WordPress theme on your blog 

Then back to why we are here.

Where is the "robots.txt" file, how to create one?

The file " Robots.txt Is usually available at the root of your installation. You will need to use an FTP client to access it or use your cPanel to manage your files.

Go further by discovering How to manage WordPress files and folders

It is an ordinary file like all the others and you just have to open it with a text editor any like notepad.

If you can not find the file " robots.txt At the root of your installation, then you can always create one. All you need to do is create a text file named " robots.txt »On your computer and send it to the root of your hosting using an FTP client as specified above. 

How to use the file "robots.txt"?

The syntax to use for the file robots.txt Is very simple. The first line usually specifies the name of an Agent. Agent is the name of a robot you are communicating with. For example " Googlebot " or " Bingbot For the respective Google and Bing robots. You can use "*" to give instructions to all robots at the same time.

The following line begins with " Allow " or " Disallow "(Authorize or prohibit) indexing a path. This way, robots will know what to index and what not to index.

Here is an example of a " robots.txt »


In this WordPress “robots.txt” draft file, we gave the robots instructions to index the entire media file directory.

The last two lines prohibit indexing the plugins directory and the “readme.html” file.

How to optimize your "robots.txt" file for better SEO

Google recommends against using the "robots.txt" file to hide irrelevant content. If you were going to use it to prevent Google from indexing your categories, archives and so on, then you should know that this is not a good practice.

Remember, the purpose of the file is to give instructions to the robots so that they can properly index your blog. It does not prevent bots from crawling your blog.

There is a WordPress plugin that allows you to add " goal " as " nofollow "And" noindex »On the pages of your archives, this is the plugin WordPress SEO. We're not saying that you absolutely should deindex your archives, but if you want to do so, this is the most appropriate way.


You don't need to add the login page, or the administration directory or even the registration page in your "robots.txt" file because these pages already have the tags " noindex ».

It is recommended to disable playback of the readme.html file from your file robots.txt ". This file can be used by anyone to retrieve the version of WordPress that you are using to check for possible vulnerabilities.

But also, if another person runs malicious code to search for blogs using a specific version of WordPress, then disabling this file can protect you from multiple attacks.

Also discover our 10 WordPress plugins to strengthen the security of your blog

You can disable the plugins directory. This will add an extra barrier of protection to your blog.

How to add the XML map site to the "robots.txt" file

If you are using a plugin to generate your XML file, then your plugin will automatically try to add a sitemap to the " robots.txt ».

Discover Using Sitemaps WordPress

However, if it doesn't, you can do so by adding the following lines:


What does a complete "robots.txt" file look like?

Honestly, most blogs use a simple "robots.txt" file. Their content varies according to the needs of each blog.


This file tells robots to index all content and provide the path to the XML files. 

Discover also some premium WordPress plugins  

You can use other WordPress plugins to give a modern look and optimize the grip of your blog or website.

We offer you here some premium WordPress plugins that will help you do that.

1. CommentPress

CommentPress is a powerful WordPress compact plugin that lets you insert, edit and delete comments quickly and easily. CommentPress uses Ajax, jQuery and PHP to give visitors the ability to insert comments without having to reload the page.CommentPress Ajax Comments Insert Edit and Delete Comments

This plugin has many customization options, finally giving it the look and feel you want. It has a captcha to block spam and brings to your comment section more security.

Its main features are: the easy insertion of comments, the possibility for users to reply to specific comments, the pagination of comments, complete management of the administration of the comments section, buttons for easy insertion of images, videos and links, a fully responsive layout, a Captcha to secure the form and prevent spam, and much more.

Download | Demo | Web hosting

2. jqEasytooltip

JqEasytooltip is a publisher for WordPress easy to use and customizable tooltips. VSThe WordPress plugin comes with plenty of tooltip templates, css3 transition effects, and over 300 amazing font icons included for buyers. All its configurations can be managed via the integration of the editor tinymce.  jqEasytooltip WordPress Tooltip Plugin

And its main features are: aQuick and easy tooltip editor (Integration tinymce), compatible with other plugins, a layout efully responsive, the support of jquery plugin Jqeasytooltip jquery, 35 theme models to choose from and customize, pread from 300 ready-to-use icons, 10 animations in pure CSS3 to appear and disappear, etc ...

Download | Demo | Web hosting

3. Social Auto Poster

Social Auto Poster is a WordPress plugin and a perfect choice for automatically publishing your content on social networking platforms such as: Facebook, LinkedIn, Twitter, Tumbler, BufferApp, Instagram and Pinterest. Social Auto Post WordPress Plugin

You will be able to set up your social networking accounts and publish new content, or republish an old one to keep it alive and reach a wider audience.

 This WordPress plugin provides an amazing feature that allows you to schedule the publication of your content anytime. It also works with personal profiles, professional pages, groups, etc.

DownloadDemo |Web hosting

Recommended Resources

Find out about other recommended resources to help you build and manage your website.


Here ! That's all for this tutorial on file optimization " robots.txt ». Feel free to share the tip with your friends on your favorite social networks

However, you will also be able to consult our resources, if you need more elements to carry out your projects of creation of Internet sites, by consulting our guide on the WordPress blog creation.

If you have suggestions or remarks, leave them in our section Comments.