Although the robots.txt file was invented to tell serps what internet pages never to crawl, the robots.txt file may also be used to level serps into the XML sitemap. This can be supported by Google, Bing, Yahoo and Ask. Referencing the XML sitemap within the robots.txt file is one of https://rankingseotoolz.com/robots-txt-generator