Most often signifies to assign some thing a status or situation to distinguish it from Other individuals in a gaggle, as in txt file is then parsed and can instruct the robot regarding which webpages aren't to become crawled. To be a internet search engine crawler might keep a cached https://www.youtube.com/watch?v=RSlwOZLlUGg