Create a custom or tailor-made Robots.txt for your Blog 4

Learn to create the perfect search engine friendly Robots.txt that’s just right for you.

Creating a Robots.txt for your website or Blog is probably one of the smartest things that you can do for optimizing your websites SEO next to creating and submitting your XML sitemap to Webmaster tools, which is one of those other things that you should not put off doing.

For today readers I would like to help you in creating your very own robots.txt that you can set and forget,however it will continue to serve you and your blog for years to come.

What is a Robots.txt?

A robots.txt is a simple text document that tells the search engines bots weather or not they can craw your website and index the contents. This can also be used to exclude particular search engines or blocking certain portions of your website from being indexed if you so choose.In extreme cases you can block these bots all together, but that would mean no one would find your website in search.

Issues often Encountered with Robots.txt

The problem is that most bloggers do not know what a robots.txt is a and they would like a hassle free method to creating one, we have you covered so read on.

Most only find out about their robots.txt when they are having website errors or they are receiving errors in Adsense control panel or webmasters tools.

What does my robots.txt look like?

You may view your current or default robots.txt or that of any website by placing this after the domain in the address field:

/robots.txt

So for example that would be in the address bar:

YourWebSite.com/robots.txt

You may look at ours as well, go ahead and give it a try by simply adding /robots.txt to our domain name or clicking here.

Create Your Own Robots.txt

Each line of a robots.txt basically directs bots telling them what they should do and what they should not do or index, to put it simply the house rules.You can also lay out the welcome mat and tell them where your Site Map is, if you have yet to create a XML SiteMap make sure to do so here as it a very good idea for indexing of your content and people finding you in search engines.Its also a good idea to get rid of your default Robots.text and loading a custom one as the examples below:

Example 1: For those seeking a simple no hassle robots text that require no editing and that allows all search engines to crawl your content for indexing, simply copy the code below and move to the instructions towards the end of the article  to easily apply your robots.txt to your blog.(Also note that your content will only be indexed if it original and provides value to readers)

 

# robots.txt created at https://www.blogtechtips.com
User-agent: *
Disallow:
Disallow: /cgi-bin/

Example 2: A robots.txt that allows all search engines and includes a site Map to tell search engines where to find your websites XML Sitemap, be sure to place the address of your site map on the line that says Sitemap: below in the sample.

# robots.txt created at https://www.blogtechtips.com
User-agent: *
Disallow:
Disallow: /cgi-bin/
Sitemap:

You can learn how to create and get your XML site map address in our tutorial here.

Eliminate Adsense Crawler errors that’s preventing targeted ads from appearing on your blog.

Example 3: This robots.txt contains the string of code that allows the Google Adsense crawlers to display the appropriate information on your website that targeted to your content by allowing them full access through robots.txt identified by the addition of the Mediapartners-Google addition at the very top of the robots.txt, also remember to edit and include your site map url in the example below on the line that says Sitemap:

User-agent: Mediapartners-Google
Disallow:

# robots.txt created at https://www.blogtechtips.com
User-agent: *
Disallow:
Disallow: /cgi-bin/
Sitemap:

Generate Your Own Robots.txt with custom Exclusions or settings

If none of the above is doing it for you or you would like to generate your own, and block specific search bots or block specific directories from being indexed then you may want to click below and use this highly recommended generator to easily create a Robots.txt tailor fitted for you. Simply fill out the info,make your selections and click create to make your own.

Click here: Robots.txt Generator

How to implement your Robots.txt hassle free?

You could take the long and complicated route of actually logging into the backend of your website through CPanel where you will have access to your website control panel or some other complicated method that involves editing your robots.txt file and placing it in the root directory of your website, or you could use our so simple a baby could do it method:

1. Install Yoast All in one SEO plugin (or go to plugins and search for it) and install and activate it. If you have the plugin already you are ahead of the game as its the best for SEO optimization.

2.  After activation look for a SEO option in the left panel and select edit files.

3. You should see a robots.txt option at the top click the button to edit or modify the settings.

4. Copy one of our Examples or generate your own and paste it in the robots.txt field, save changes to the robots.txt and you are finished.

5. Your new robots.txt is now in effect.

Testing Your Robots.txt

To test the robots.txt simply go to your webmaster tools and in the left panel select Crawl and fetch as Google and attempt to fetch the default URL or a specific page using the settings on that page, if you get a success that means crawlers are not blocked and are accessing your pages fine. You are now finished.

How easy did you find this exercise or did you fail miserably in implementing and following the instructions? let us know what you experience has been below as a part of the comments, looking forward to hearing from you.