How to edit robots.txt in WordPress

Robots are very important nowadays to handle bots from crawling your website or pages, Hence it is important to add robots.txt file in your website’s root directory.

One of the main reason of using robots.txt is hiding your private pages from search results like for .e.g. your-website.com/wp-admin

Generally, If you don’t use robots.txt file, bots (spiders) will crawl your entire webpages, some are not necessary or it can also be duplicate contents.

I would better suggest you to use this file to guide bots to crawl your websites/blogs properly.

There is no need to create robots.txt nowadays because while installing WordPress, It automatically creates a virtual robots file with default coding. (Example given below)

How should I verify If I have default/virtual robots.txt file ?

Just you need to verify is by typing this url in your browser:

http://your-website.com/robots.txt

User-agent:  *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/

Still there are many threads on WordPress support forums related to editing this virtual robots.txt, In my opinion editing this virtual robots file is difficult.

This is what I have faced with my blog (BlogDum); I tried every possible ways to solve but failed to edit this virtual file, Hence after doing a good research.

I found out two ways to edit robots.txt file one with plugin and other without plugin, both worked for me like a charm.

Method 1 – Without plugin.

1. Create a text file
2. Add your desired code in this file or you can check mine robots.txt
3. Save file as “robots.txt” – Don’t misspell
4. Upload this file in your root directory (Same place where you add Google webmasters verification code).
5. Now check www.yourdomainname.com/robots.txt
6. If you can see the file and codes then you have successfully uploaded it.