Instructing GoogleBot using robots.txt

According to Google .., their crawler for indexing purpose (GoogleBot) do respect the robots.txt content..

If we happened to specify something in robots.txt .. such as of

Disallow: /restrict_folder/

Its crawler then will respect this directive.. and will not crawl whatever inside /restrict_folder/ ..
somehow some other crawler might not respect this directive though..
so Google recommend us to protect our .. not so public page with a password or some sort of authentication ..

Ok.. but if you don’t have robots.txt defined.. or robots.txt is just allowing .. no restricting to folder..
Only then GoogleBot will crawl the page and read its meta directive..

and depending on the instruction at meta for robots.. it might index, archive .. or not archiving based on META tag directive..
If everything is okay.. It will then archive, index and all sort of thing that can be done for searching purpose.

then come the canonical directive in META tag for robot..

what does this one define is…
if the page happened to have two different link pointing to it but displaying same content..
using this directive.. we can define which one to be indexed..

example :

1. https://namran.com/2009/05/19/instructing-googlebot-using-robotstxt
2. https://namran.com/2009/05/19/instructing-googlebot-using-robotstxt#comment

both link are pointing to the same page.. and we prefer its to index the first one only..
instead of both two page..

we can then write the canonical as

<link rel='canonical' href='https://namran.com/instructing-googlebot-using-robotstxt'>

more detail can be found here.

.. and to examine you robots setting..

1. login to Google webmaster tool
2. click to Tools at the left menu.
3. the can see “Analyze robots.txt”

my link would be something like .. https://www.google.com/webmasters/tools/robots?siteUrl=http%3A%2F%2Fnamran.com%2F&hl=en
this one can test if the robots.txt is properly written.. and either it is blocking crawler to access certain page or not..
just fill in the desired URL into the box provided.. you will be able to see its analyze..

something like this..

robot-analyzer

p/s : still can’t understand why my recent post can’t be archived/indexed ..though.. since 10th May 2009… can’t recall why.. *sigh*

Related Post

7 Responses

  1. Harvey says:

    It is the best time to make some plans for the future and it’s time to be happy. I’ve read this post and if I could I want to suggest you some interesting things or advice. Maybe you can write next articles referring to this article. I desire to read even more things about it!

    Reply
  2. Rachel says:

    Thanks for sharing your info. I really appreciate your efforts and I am waiting for your further write ups thank you once again.

    Reply
  3. Brittney says:

    I do not even know the way I ended up here, however I thought this put up used to be great. I don’t recognize who you might be but certainly you’re going to a well-known blogger in case you are not already. Cheers!

    Reply
  4. website says:

    What’s up, after reading this amazing article i am too delighted to share my knowledge here with colleagues.

    Reply
  5. http://www.ogloszenia-matrymonialne.org.pl says:

    Do you mind if I quote a couple of your articles as long as I provide
    credit and sources back to your site? My blog site is in the very same niche as yours
    and my users would genuinely benefit from some
    of the information you present here. Please let me know if this okay with you.
    Many thanks!

    Reply
  6. calvin klein underwear uk says:

    Incredible! This blog looks just like my old one! It’s on a completely different subject but it has pretty much the same page layout and design. Great choice of colors!

    Reply
  7. calvin klein underwear uk says:

    I really like what you guys are usually up too. Such clever work and coverage!
    Keep up the amazing works guys I’ve included you guys to my own blogroll.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *