 |
| DistrictND |
You may have heard of seo and you've always wanted every page in your blog whether useful or less important to be indexed by search engines, well custom robots.txt will do that for you.
Robots.txt
Robots.txt is a text file containing some lines of codes. Well it's a custom file which means you can customize it to index every page in your blog whether useful or less important.
Remember robots.txt are normally scanned by by search engines before crawling any web page.
How to add Custom Robots.Txt to Blogger.
1. Go to your blogger blog dashboard.
2. Go to Settings >> Search Preferences ››
Crawlers and indexing ››
3.pick Custom robots.txt ›› Edit ›› Yes
Now paste your robots.txt file in that box provided below.
This is what a custom robots.txt file look like.
User-agent: Mediapartners-Google
Disallow:
User-agent: * Disallow: /search Allow: / Sitemap: http://example.blogspot.com feeds/
posts/default?orderby=UPDATED
How to Check & Get Your Robots.txt File
You can check & get this file on your blog by
adding /robots.txt at the end of your blog URL
in the web browser. For example:
http://www.exampleblog.blogspot.com/robots.txt
Once you visit the robots.txt file URL, you will
see the entire code which you are using in your
custom robots.txt file.
Let me explain how the codes work.
This code is divided into three sections.
We will discuss about the 3 parts an it's
usefulness.
User-agent: Mediapartners-Google
This code is for Google Adsense robots which
help them serve better ads on your blog.
Either you are using Google Adsense on your
blog or not simply leave it as it is if you're not
using it there is also a custom ads.txt I'll explain that soon.
User-agent: *
This is for all robots marked with asterisk (*).
In default settings our blog's labels links are
restricted to indexed by search crawlers that
means the web crawlers will not index your
labels page links because of below code and
it's really good in making your blog look
reasonable
Disallow: /search
That means the links having
keyword search just after the domain name will
be ignored. And if we remove Disallow: /search from the above code then crawlers will access our entire blog to index and crawl all of its content and web pages.
Here Allow: / refers to the Homepage that
means web crawlers can crawl and index our
blog's homepage.
There are other site maps which are useful and
can be submitted Google.com /webmaster
Hence, there are better chances that web
crawlers crawl all of our blog posts without
ignoring a single one.
Note:
The Sitemap added in custom robot.
The Sitemap added in custom robot.
Text will only tell the web crawlers about the
recent 25 posts.
If you want to increase the
number of link in your sitemap then replace or edit default sitemap with below one. It will work for first 500 recent posts.
Sitemap1: http://example.blogspot.com/
atom.xml?redirect=false&start-index=1&max-
results=500
If you have more than 500 published posts in
your blog then you can should add this site
map too
Sitemap2: http://example.blogspot.com/
atom.xml?redirect=false&start-
index=500&max-results=1000
Now your blog will now be crawled by search engines, note any wrong setting to this file can result to your blog being ignored by search engines.
0 Comments