If you have submitted your website to the major search engines such as Google, Bing and yahoo, then they would want to know what this website is talking about. They want to know one by one what the pages of this website are talking about; your submitting of the website only inform them that your site exist and not of the pages that are contained in it. So, each search engine has designed a technology called crawler which is in a likeness of a spider (scarcely called spider and Bot). This technology would come to your site and move over all the pages of your website and to scan them to gather important information such as meta-data and specific (whether it is for the age eighteen plus or for all viewers) contents such as videos, images and other mobile contents and return to the respective search engine so that the pages scanned can be indexed. These technologies can only work efficiently through your help or directions. They need you to provide them with a site map. Jus...